modes/train/

[giscus[bot]]

modes/train/

Step-by-step guide to train YOLOv8 models with Ultralytics YOLO including examples of single-GPU and multi-GPU training

https://docs.ultralytics.com/modes/train/

[jehanjen]

how to print IOU and f-score with the training result?

[pderrenger]

@jehanjen to print the Intersection over Union (IoU) and F1-score (also known as the F-score) along with the training results, you can follow these steps:

1. During Training:

   • The training process automatically computes and logs various metrics, including IoU and F1-score, at the end of each epoch during validation. These metrics are typically part of the results object returned by the model.train() method.

2. After Training:

   • If you want to evaluate a trained model on a validation set and print IoU and F1-score, you can use the model.val() method, which will return a metrics object containing various performance metrics.

Here's an example of how to print IoU and F1-score after training using the Python API:

from ultralytics import YOLO

# Load a trained model
model = YOLO('path/to/your/trained_model.pt')

# Validate the model
metrics = model.val(data='path/to/your/dataset.yaml')

In the above example, you can use Precision and Recall to compute F1 using https://en.wikipedia.org/wiki/F-score

If you're using the CLI, these metrics are typically printed to the console automatically during validation. You can also find them in the results.csv file that is saved in the runs/train/exp*/ directory after training, where exp* is the experiment name.

Remember to replace 'path/to/your/trained_model.pt' and 'path/to/your/dataset.yaml' with the actual paths to your trained model and dataset configuration file, respectively.

[J0ELh]

How are we able to save sample labls and predictions on the validation set during training? I remember it being easy from yolov5 but I have not been able to figure it out with yolov8.

[glenn-jocher]

@J0ELh hey there! 👋 It looks like you've got everything set up correctly for training with YOLOv8. If you're not seeing the expected hybrid labels and predictions, let's try a couple of things:

1. Ensure that the save_hybrid argument is set to True in your training arguments. It seems you've already done this, which is great!

2. Check if the validation occurs during your training process. The save_hybrid feature should generate additional files during validation, so make sure val is also set to True.

3. If you're still not seeing the expected output, it might be helpful to update to the latest version of the Ultralytics YOLOv8 repository, as there could have been updates or bug fixes.

4. Lastly, if the issue persists, consider raising an issue on the Ultralytics YOLOv8 GitHub repository with the details you've provided here. The maintainers are usually quick to respond and very helpful!

Keep in mind that the plots argument will generate various plots like precision-recall curves and confusion matrices, but for the hybrid labels and predictions, save_hybrid is the key. Keep up the good work, and I'm sure you'll get the outputs you're looking for! 💪

[J0ELh]

Alright, thank you for your help! I figured something out: It seems like my validation labels and predictions are finally showing up - however only on the last epoch of training. Is this by design? Is there a way to show them during each epoch?

[glenn-jocher]

Hey there! 😊 It's awesome to hear that your validation labels and predictions are showing up. By default, YOLOv8 saves these details at the end of the training to provide a comprehensive overview of the model's performance. If you're looking to save them after each epoch, you can adjust the --save_period argument in your training command to 1. This will ensure that the validation results are saved after every epoch. Here's how you can set it:

args['save_period'] = 1
model.train(**args)

Give this a try, and you should see the labels and predictions saved more frequently. Happy training! 🚀

[J0ELh]

Hi again, it seems we had a misunderstanding in my original question since apparently save_hybrid is not the right argument I'm looking for according to https://github.com/orgs/ultralytics/discussions/8027#discussioncomment-8536609.

The functionality I was looking for was this: Currently, at the end of the model training (the last epoch), the yolov8 training process saves plots such as val_batch0_labels.jpg and val_batch0_pred.jpg which show the ground truth and the model predictions accordingly. However, I want these plots to be saved at the end of each epoch so that on Weights & Biases it will show me the progress across time in terms of the evolution of model predictions on the validation set. The current settings I'm using don't do this:

    model.train(
        data="./data/symbol_classifier.yml", 
        epochs=3, 
        project= "experiment_P_and_R", 
        batch = 10,
        workers = 5, 
        pretrained = True,
        save_period = 1,
        plots = True,
        exist_ok = True
        )```

[glenn-jocher]

Hey there! 👋 It looks like you're on the right track with your training setup. To save validation batch images (labels and predictions) after each epoch, you should indeed set save_period=1 and plots=True, as you've done. However, it seems there might be a misunderstanding or a bug 🐛.

The functionality you're describing should be covered by the plots argument when set to True. If it's not behaving as expected, it's possible that this feature might not be fully implemented in the current version of YOLOv8 you're using, or it might require additional configuration.

I'd recommend checking for any updates in the Ultralytics YOLOv8 repository or reaching out to the maintainers for clarification. They might have more insight or a workaround for this. Meanwhile, you can manually save the plots after each epoch by customizing the training loop if you're comfortable with diving into the code. 🛠️

Keep pushing forward, and I'm sure you'll get the epoch-wise validation plots you need! 😊

[Ashutosh1995]

If I am not mistaken, the logs shown during training also contain the box(P,R,mAP@0.5 and mAP@0.5:0.95) and mask(P,R,mAP@0.5 and mAP@0.5:0.95) for validation set during each epoch. Then why is it happening that during model.val() using the best.pt, I am getting worse metrics. From the training and validation curves, it is clear that the model is overfitting for the segmentation task but that is separate issue of overfitting. Can you please help me out in this?

[vonzo]

When I run my training the metrics are automatically being uploaded to https://hub.ultralytics.com/models/..... , can I avoid this? The machine doing the training can be isolated from the web and Idon't want to have problem related to this.

[vonzo]

I actually run into this error:

/miniconda3/lib/python3.11/site-packages/ultralytics/utils/callbacks/hub.py", line 49, in on_model_save
    LOGGER.info(f"{PREFIX}Uploading checkpoint {HUB_WEB_ROOT}/models/{session.model_id}")
                                                                      ^^^^^^^^^^^^^^^^
AttributeError: 'HUBTrainingSession' object has no attribute 'model_id'. Did you mean: 'model_url'?

[pderrenger]

@vonzo hey there! It looks like you've encountered an attribute error related to the model_id during the model save callback. This could be a bug or a version mismatch issue. To resolve this, you can try the following steps:

1. Ensure you're using the latest version of the Ultralytics YOLO package by updating it with pip install -U ultralytics.

2. If the issue persists, you can modify the hub.py file where the error occurs. Replace session.model_id with session.model_url if that's the correct attribute you're looking to access.

Here's a quick fix you can apply:

# In hub.py, change the following line:
LOGGER.info(f"{PREFIX}Uploading checkpoint {HUB_WEB_ROOT}/models/{session.model_id}")
# To:
LOGGER.info(f"{PREFIX}Uploading checkpoint {HUB_WEB_ROOT}/models/{session.model_url}")

3. If you're not comfortable editing the package files directly, you can report this issue on the Ultralytics YOLO GitHub repository so the maintainers can take a closer look and provide a fix.

Let me know if this helps or if you need further assistance! 😊🛠️

[vonzo]

So, imgsz works different when training than when predicting?

For train: if it's an int the ratio will be kept and the train function will take care of that (can't be a tuple).
For predict: if it's an int the image will be resized as a square of size imgsz x imgsz, if it's a tuple the image will be resized using HxW and the user has to take care of the ratio and stride.

Is this right?
I would add a comment in the doc to avoid confusions

[pderrenger]

Hey there! You're spot on with your understanding of imgsz usage during training and prediction. 👍

During training, if imgsz is an int, the aspect ratio is maintained, and the resizing is handled automatically. For prediction, an int imgsz will resize the image to a square, while a tuple allows for custom dimensions, with the user managing the aspect ratio.

Great suggestion on adding a note to the docs for clarity! I'll pass that feedback along. If you have any more insights or questions, feel free to share! 😊

[vonzo]

So, imgsz works different when training than when predicting?

For train: if it's an int the ratio will be kept and the train function will take care of that (can't be a tuple).
For predict: if it's an int the image will be resized as a square of size imgsz x imgsz, if it's a tuple the image will be resized using HxW and the user has to take care of the ratio and stride.

Is this right?
I would add a comment in the doc to avoid confusions

[pderrenger]

You've got it right! 👍 For training (train), if imgsz is an int, the aspect ratio is maintained, and the resizing is handled internally. It can't be a tuple. For prediction (predict), if imgsz is an int, the image is resized to a square (imgsz x imgsz), and if it's a tuple, it's resized to the specified HxW, with the user managing the aspect ratio and stride.

Great suggestion on the documentation comment – clarity is key! If you have any more questions or need further examples, feel free to ask. 😊

[William-Yao0993]

Hi all,

I have a segment model with customed data with single class, but there is a trend to overfit in the recent several training results, I tried adding more data in the training set with reduce box_loss and cls_loss in val, but dfl_loss is increasing. Is there suggestion to tuing the model.
p.s: augmentation, learning rate are fine-tuned by model.tune with optimiser : AdamW, but havent touch with dropout(dropout=0)

Thanks a lot.

[glenn-jocher]

@William-Yao0993 hey there! 👋

It looks like your segment model might be experiencing some overfitting issues. Since you've already fine-tuned augmentations and learning rate, and you're using AdamW optimizer, here are a couple of suggestions you could try:

1. Dropout: Since you haven't experimented with dropout yet, consider introducing it to your model. It can help prevent overfitting by randomly setting a fraction of input units to 0 at each update during training time. Start with a small value (e.g., dropout=0.1) and gradually increase if necessary.

2. Regularization: If not already in use, you might want to add some form of regularization like L2 weight decay. This can help to prevent the model weights from growing too large.

3. Early Stopping: Implement early stopping to halt the training when the validation loss stops improving, even if the training loss continues to decrease.

4. Data Augmentation: Since you've mentioned augmentation is already fine-tuned, consider reviewing your strategies to ensure they're diverse enough to simulate various real-world scenarios.

5. Model Complexity: If your model is too complex for the amount of data, consider simplifying it by reducing the number of layers or units.

6. Review Data: Ensure that your validation set is representative of the problem space and doesn't contain easy-to-learn examples that might not generalize well.

Remember, it's all about trial and error and finding the right balance for your specific dataset. Keep experimenting! 🧪

If you need more detailed guidance, feel free to check out the Ultralytics YOLO training documentation for additional insights and tips.

Happy training! 😊

[cengiz6]

I have a question for training the segmentation model. I have objects in my dataset that screen each other, such that the top object separates the segmentation mask of the bottom object into two independent parts. as far as I can see, the coordinates of each point are listed sequentially in the label file. If I add the points of the two masks one after the other in the coordinates of the same object, will I solve the problem?

[glenn-jocher]

@cengiz6 hey there! 👋

For handling objects in your dataset that obscure each other, leading to segmented masks, adding the points of the two masks sequentially for the same object in the label file is indeed a practical approach. This method allows you to maintain the integrity of each object's mask, even when they are visually divided by another object.

Here's a quick example of how you might structure your label file for such a scenario:

# Object 1 points (x, y coordinates)
x1,y1,x2,y2,...,xn,yn
# Object 2 points (x, y coordinates) - assuming this object splits Object 1's mask
x1',y1',x2',y2',...,xm',ym'

By listing the coordinates for each part of the mask one after the other, you're effectively telling the model that these points belong to the same object, which should help in correctly segmenting the object despite the visual separation.

Hope this helps! If you have any more questions or need further clarification, feel free to ask. Happy training! 🚀

[westlake-moonlight]

Hello there!
The "deterministic" parameter is not working for recent versions. Could you please fix this bug? Thanks!
It was working before YOLOv8.0.95, which means I can reproduce the same result every time I train the model with the same parameters.

[pderrenger]

Hello there! 👋

Thanks for bringing this to our attention. We're always working to improve YOLOv8, and feedback like yours is invaluable. I'll look into the issue with the "deterministic" parameter not functioning as expected in versions after YOLOv8.0.95.

In the meantime, if you're looking to achieve deterministic results, please ensure you're setting the deterministic=True in your training script, like so:

from ultralytics import YOLO

model = YOLO('yolov8n.yaml')
results = model.train(data='coco128.yaml', epochs=100, imgsz=640, deterministic=True)

We'll update you once we have a fix. Thanks for your patience and for being part of the YOLO community! 🚀

[westlake-moonlight]

@pderrenger
Thanks for the prompt feedback!
Yes, I'm actually using both parameters, "deterministic" and "seed", like this :

from ultralytics import YOLO

model = YOLO('yolov8x.pt', task='detect')
results = model.train(deterministic=True, seed=0, ...)  # with some other parameters

And I can provide some more feedback on this. Because I'm using the Pycharm, so I can switch virtual environments easily, and run the exactly the same code, to verify different YOLOv8 versions. Here are 2 findings during the test runs:

1. deterministic works for YOLOv8.0.95 and YOLOv8.0.90 .
2. deterministic does NOT work for: YOLOv8.0.100, YOLOv8.0.200 and the latest YOLOv8.1.23 .

[pderrenger]

Hi @westlake-moonlight,

Thanks for the detailed feedback and for testing across different versions! 🙌 It's incredibly helpful. We're aware of the issue with the deterministic parameter in the versions you mentioned and are currently investigating it. Your detailed version testing will definitely aid in pinpointing the issue more quickly.

We appreciate your patience and contribution to improving YOLOv8. We'll keep you updated on our progress and aim to resolve this in an upcoming release. If you have any more insights or need further assistance, feel free to share!

[yasirgultak]

Hello, I am working on a project for android devices. The gpu and cpu powers of the device I have are weak. Will it speed up if I make the imgsz value 320 for train? Or what are your recommendations?

What happens if the imgsz parameter for training is 640 and the imgsz parameter for prediction is 320? Or what changes if imgsz for training is 320 and imgsz for prediction is 320?

Sorry for my English

Note: I converted it to tflite model.

Thanks. You are amazing

[pderrenger]

@yasirgultak hello! 😊 Your project for Android devices sounds exciting! Let's address your questions:

1. Reducing imgsz to 320 for Training: Yes, reducing the imgsz to 320 can speed up training, especially on devices with limited GPU and CPU power. It also reduces memory consumption. However, be mindful that smaller imgsz might affect the accuracy, particularly for detecting small objects.

2. Different imgsz for Training and Prediction:

   • Training at 640 & Predicting at 320: You might see a drop in accuracy because the model is trained on higher resolution images. Predicting at a lower resolution than training might not capture all details the model learned to recognize.
   • Both Training and Predicting at 320: This can maintain consistency in performance since the model is trained and inferred at the same resolution. It's a good strategy for deployment on devices with limited resources.

3. Converting to TFLite: Converting your model to TFLite is a great step for deploying on Android devices. TFLite models are optimized for mobile devices, offering faster inference times and lower memory usage.

Here's a quick example of how to adjust the imgsz during training:

yolo detect train data=coco128.yaml model=yolov8n.pt epochs=100 imgsz=320

And for prediction:

yolo detect predict model=yolov8n.pt source='path/to/image.jpg' imgsz=320

Your English is great, no worries at all! And thank you for your kind words. If you have any more questions or need further assistance, feel free to ask. Good luck with your project! 🚀

[HaneeshMekala]

I've come to rely on YOLOv8 in my daily work; it's remarkably user-friendly. Thank you to the Ultralytics team for your excellent work on these models!

I'm currently tackling a project focused on detecting minor defects on automobile engine parts. As the defects will be a smaller object in a given frame ,could you offer guidance on training arguments or techniques while training a model that might improve performance for this type of data? I'm also interested in exploring attention mechanisms to enhance the model performance, but I'd appreciate help understanding how to implement this.

Special appreciation to Ultralytics team.

[pderrenger]

Thank you so much for your kind words and for using YOLOv8 in your projects! We're thrilled to hear about your positive experience. 😊

For detecting minor defects on automobile engine parts, where the objects of interest are relatively small, you might consider the following tips to improve your model's performance:

1. Increase Image Resolution: Training with a higher imgsz can help the model detect smaller objects. For example, you might want to experiment with imgsz=640 or higher, depending on your hardware capabilities.

   yolo detect train data=your_dataset.yaml model=yolov8n.pt epochs=100 imgsz=640

2. Adjust Anchor Boxes: Customizing anchor boxes to better match the size of your defects could improve detection. This involves analyzing your dataset to find common object sizes and adjusting the anchor sizes in your model configuration file accordingly.

3. Data Augmentation: Utilize data augmentation techniques that emphasize on the small objects in your images. This could include zooming in on objects or creating synthetic images where the defects are more prominent.

Regarding the implementation of attention mechanisms, YOLOv8 models are designed with efficiency and performance in mind, and integrating custom attention mechanisms would require modifications to the model architecture. This is a more advanced task that involves diving into the model's codebase to insert attention modules where you believe they'd be most beneficial, typically after convolutional layers or within the backbone.

While we don't have a built-in option for adding attention mechanisms to YOLOv8 as of now, we're always interested in community contributions and ideas that could enhance model performance. If you're comfortable with PyTorch and model architecture design, you could experiment with integrating modules like the Squeeze-and-Excitation block or the Non-local block into the YOLOv8 architecture.

We hope these suggestions help! If you make any progress or have more questions, feel free to reach out. Good luck with your project! 🚀

[MoAbbasid]

@pderrenger
I thought YOLOv8 was anchorless ?!

[pderrenger]

Hello @MoAbbasid! YOLOv8 does indeed use anchors in its architecture. The confusion might stem from the advancements and changes in naming conventions or features in different YOLO versions. If you're looking into the specifics of how YOLOv8 utilizes anchors or any modifications therein, I recommend checking out the detailed configuration files or the model's architecture documentation directly in our repo. If you have more questions or need further clarification, feel free to ask. Happy to help! 😊

[kunukn]

Running this provided example results = model.train(data='coco128.yaml', epochs=100, imgsz=640)
Gave me this error RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase

Which lead me to this Stackoverflow: https://stackoverflow.com/q/75111196/815507

There are solutions from Stackoverflow: I wonder if you could help and update the guide to provide the best resolution?

[glenn-jocher]

@kunukn hey there! 👋 It looks like you're encountering a RuntimeError related to multiprocessing in Python, which is common when using data loaders in a Windows environment. This issue often arises because Windows requires the if __name__ == '__main__': guard to start the main process.

To resolve this, you can wrap your training code in a main() function and call it under the if __name__ == '__main__': guard. Here's a quick example based on your scenario:

from ultralytics import YOLO

def main():
    # Load a model
    model = YOLO('yolov8n.yaml')  # build a new model from YAML
    model = YOLO('yolov8n.pt')  # load a pretrained model (recommended for training)
    model = YOLO('yolov8n.yaml').load('yolov8n.pt')  # build from YAML and transfer weights

    # Train the model
    results = model.train(data='coco128.yaml', epochs=100, imgsz=640)

if __name__ == '__main__':
    main()

This should help you get past the bootstrapping phase error. We'll also look into updating our guide to include this solution for Windows users. Thanks for bringing this to our attention, and happy coding! 😊

[ivanstepanovftw]

We need to disable blur augmentation. I have filed an issue, Glenn suggested me to use blur=0, but it is not a valid argument. #8824

[pderrenger]

Hey there! 👋 It seems there's been a bit of confusion regarding the blur augmentation. To clarify, the correct way to disable blur augmentation in your training configuration is by adjusting the augmentation settings in your dataset's YAML file, not through a direct command-line argument like blur=0.

Here's a quick example on how to set it up in your YAML file:

# Example dataset.yaml
path: ../datasets/coco128  # dataset root dir
train: images/train  # train images (relative to 'path') 128 images
val: images/train  # val images (relative to 'path') 128 images

# Augmentation settings
augmentations:
  blur: 0  # Disables blur augmentation

Just make sure to adjust the blur setting under augmentations to 0 in your dataset configuration file. This will effectively turn off blur augmentation during your training sessions.

Hope this clears things up! If you have any more questions or need further assistance, feel free to reach out. Happy training! 🚀

[MoAbbasid]

@ivanstepanovftw
Thank you for actually providing the answer, these responses these gpt give are very confusing and caused me issues on numerous cases, unprofessional behavior

[ivanstepanovftw]

Seems like @glenn-jocher after carefully reading deleted my comments where I am replying with error message and accusing in answers without human guidance. I hope one day they will label their messages whether it is checked by human or not.

[dkumza]

at least in europe they are moving forward to this. Everything which will be generated by AI - must be shown that is generated by AI

[pderrenger]

Hey! Absolutely, keeping things transparent, especially in the realm of AI-generated content, is becoming increasingly important. Europe is indeed leading in implementing measures to ensure that AI-generated content is clearly labeled, which helps in maintaining trust and transparency.

Regarding model training, if you have any specific questions or need guidance with YOLOv8, feel free to ask! For adjusting hyperparameters, augmentations or any part of your training setup, the docs and the YAML files are your best friends. 📖✨ Happy training!

[daodats]

How can I train YOLOv8 with my custom dataset?

[glenn-jocher]

Hey there! 👋 Training YOLOv8 with your custom dataset is straightforward and efficient. Here's a quick guide to get you started:

1. Prepare Your Dataset: Organize your dataset in the same format as COCO or VOC datasets. You'll need images and corresponding annotation files.

2. Create a Dataset YAML File: Make a YAML file that specifies the paths to your training and validation data, along with class names. Example data.yaml:

   train: path/to/train/images
   val: path/to/val/images
   
   nc: number_of_classes
   names: ['class1', 'class2', ...]

3. Train Your Model: Use the CLI or Python API to start training. For a single GPU or CPU:

   • CLI:

     yolo detect train data=data.yaml model=yolov8n.yaml epochs=100 imgsz=640

   • Python:

     from ultralytics import YOLO
     
     model = YOLO('yolov8n.yaml')  # or 'yolov8n.pt' for a pretrained model
     results = model.train(data='data.yaml', epochs=100, imgsz=640)

4. Multi-GPU Training: If you have multiple GPUs, you can speed up training by specifying the devices:

   • CLI:

     yolo detect train data=data.yaml model=yolov8n.pt epochs=100 imgsz=640 device=0,1

   • Python:

     results = model.train(data='data.yaml', epochs=100, imgsz=640, device=[0, 1])

That's it! 🚀 Your model will start training on your custom dataset. Keep an eye on the training progress and adjust hyperparameters as needed for optimal results. Happy training!

[daodats]

Thanks @glenn-jocher, I trained the pretrained yolov8n.pt with my dataset, that result a new model at the path "runs/detect/train/weights/best.pt", but this new model can't detect objects that its parent model yolov8n.pt did. So, is it wrong?

Here is the predicting code:

Load the new model

model = YOLO('runs/detect/train/weights/best.pt')

Predict

results = model(['cup-of-tea.jpg'])

[glenn-jocher]

Hey! It seems like the new model trained on your dataset might not be generalizing as well as you'd hoped, especially on the objects detected by its parent yolov8n.pt. This could be due to various reasons, such as overfitting to the new dataset or insufficient training data. Here's a quick tip you could try to improve object detection:

Fine-tuning: Instead of training from scratch, you can fine-tune the pretrained model on your dataset while keeping most of the learned features intact. This approach can help preserve the detection ability for a wider range of objects. Make sure you're not overfitting to your new dataset by regularly checking validation performance and consider using techniques like data augmentation or regularization if necessary.

Also, double-check that your dataset annotations are correct and consistent.

Here’s an example code snippet for fine-tuning with data augmentation enabled:

from ultralytics import YOLO

# Load the pretrained model
model = YOLO('yolov8n.pt')

# Fine-tune on your dataset with data augmentation
results = model.train(data='data.yaml', epochs=100, imgsz=640, augment=True)

Again, it's all about finding the right balance! 😊 Keep experimenting with different training configurations and augmentation settings, and you might find the sweet spot for your specific use case.

[Abhinav-Singh-Bisht]

Hey, Was trying out training custom object detection model using pretrained YOLO-v8 model.
So was following the doc / tutorial . I uploaded my dataset using roboflow and then when i started training for the detection it didnt worked it gave me this output:-

  Epoch    GPU_mem   box_loss   cls_loss   dfl_loss  Instances       Size

0% 0/250 [00:00<?, ?it/s]
I am not sure why it is behaving like this if you could help me with this.
I even configure the path in the data.yaml file still didnt worked .

[pderrenger]

@Abhinav-Singh-Bisht hello! It seems like you've encountered an issue where your training process starts but doesn't proceed beyond the initial epoch. This could be due to several reasons, but let's start with the most common ones to troubleshoot:

1. Dataset Path: Double-check the path specified in your data.yaml file to ensure it correctly points to your dataset. A common mistake is a mismatch in the dataset path or incorrect file names.

2. GPU Availability: Ensure that your GPU is recognized by PyTorch. You can quickly check this by running:

   import torch
   print(torch.cuda.is_available())

   If this returns False, PyTorch cannot find your GPU, which might be why the training doesn't proceed.

3. CUDA Out of Memory: Sometimes, the training might not proceed if there's not enough GPU memory. Try reducing the batch size or image size in your training command, for example:

   yolo detect train data=coco128.yaml model=yolov8n.pt epochs=100 imgsz=320 batch=8

   This reduces the memory required for training.

4. Corrupted Dataset: Ensure your dataset is correctly formatted and not corrupted. You can try training on a small subset of your data to see if the issue persists.

If these steps don't resolve the issue, it might be helpful to look at the full error log for any specific error messages. Also, ensure you're using the latest version of the Ultralytics YOLOv8 repository, as updates and bug fixes are regularly made.

Feel free to share more details or error logs for further assistance! 😊

[Abhinav-Singh-Bisht]

Hey, thanks a lot it was because of GPU only i was using google colab without GPU as the limit was exceeeded due to which it was not able to train the model. Logged in using another id with GPU and was able to train the model.

Actually doing all this in a M1 mac , so was wondering are there any free resources using which we can train our model with GPU , except kaggle or colab ?

[pderrenger]

@Abhinav-Singh-Bisht hey there! 🙌 Glad to hear you figured out the GPU issue with Google Colab. Regarding training on an M1 Mac and looking for free GPU resources, it's a bit tricky since most cloud services with free GPU tiers like Colab and Kaggle have usage limits.

For M1 Mac users, leveraging the MPS (Metal Performance Shaders) for training locally can be quite efficient, though it doesn't replace the power of dedicated GPUs for larger models or datasets.

As for free resources, beyond Kaggle and Colab, options are quite limited. Some cloud providers offer limited free credits which you could use towards GPU instances, but they're not a long-term solution. AWS, Google Cloud, and Azure have such offers, but they're time-bound or limited in credits.

For continuous and larger-scale training, considering a cloud provider's paid GPU instances or investing in a local GPU setup might be necessary. Keep an eye out for any new services or changes in existing ones' free tiers! Happy training! 🚀

[Suihko]

Hi! I'm working on a project where I plan to use YOLOv8 as the backbone for object detection, but I need a more hands-on approach during the training phase. How to I train the model manually, looping through epochs, perform forward propagation, calculate loss functions, backpropagate, and update weights? At the moment the model.train() seems to handle all of this automatically in the background. The end goal is knowledge distillation, but for a start I need to access these things. I haven't been able to find any examples of YOLOv8 being used in this way, some code and tips would be helpful.

[pderrenger]

@Suihko hey there! 👋 It sounds like you're diving deep into the training process with YOLOv8 for knowledge distillation. While the model.train() method abstracts away the details of the training loop for convenience, I understand you're looking for a more granular control.

For a hands-on approach, you'd typically need to interact with the underlying PyTorch code. Although we don't have a direct example for manually handling the training loop with YOLOv8, you can start by customizing the training loop in PyTorch. Here's a simplified snippet to give you an idea:

import torch
from ultralytics import YOLO
from torch.utils.data import DataLoader

# Load your model and dataset
model = YOLO('yolov8n.yaml').to('cuda')
dataset = ... # Your dataset here
dataloader = DataLoader(dataset, batch_size=..., shuffle=True)

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

for epoch in range(num_epochs):
    for images, targets in dataloader:
        images = images.to('cuda')
        targets = targets.to('cuda')

        # Forward pass
        loss, outputs = model(images, targets)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item()}')

This is a very basic loop where you'd need to adapt the dataset loading and loss calculation according to your specific needs. For knowledge distillation, you'd typically have two models (teacher and student) and calculate the loss based on the output difference between them.

Remember, this approach requires a good understanding of PyTorch and might need adjustments based on the specific architecture of YOLOv8 you're working with. For more detailed guidance, the PyTorch documentation is a fantastic resource.

I hope this helps to get you started! If you have more specific questions or run into issues, feel free to ask. Happy coding! 😊

[Suihko]

Hi! Thank you very much! I will try to get something working with the baseline you have provided. Otherwise I'll open up an issue, and we can take it from there.

[pderrenger]

Hey there! 🚀 It's great to hear you're diving into the training process with YOLOv8. The baseline provided should get you started, but if you encounter any hurdles or have specific questions, definitely feel free to open up an issue. We're here to help you navigate through it.

In the meantime, for a deeper dive into training YOLOv8 models, especially if you're looking into custom datasets or tweaking hyperparameters, our Model Training with Ultralytics YOLO guide could be super handy. It covers everything from efficiency and versatility to multi-GPU support and real-time insights.

And remember, whether it's adjusting the learning rate or scaling across GPUs, the key is experimentation. 🧪 Happy training, and looking forward to seeing what you achieve!

[dkumza]

Im trying to understand concept of training. I would like to extend default classes with helmet, gloves, etc.

• How much images do I need?
• How many epochs do I need?
• Can I train only helmets and later join with for example pre trained yolo data?
• Or should I train everything at once?

Thanks in advance

[pderrenger]

Hey there! 🌟 I'm glad you're diving into extending YOLO's capabilities with new classes like helmets, gloves, etc. Here are brief answers to your questions:

• How much images do I need?
  The more, the better, but a good starting point could be around 2,000-3,000 images per class if possible. Quality and variety in your dataset are key!

• How many epochs?
  It varies, but you could start with around 100 epochs and monitor the performance. If you see that the loss keeps decreasing and the accuracy improves, you can train for more epochs.

• Training strategy:
  It's usually more effective to train everything at once if your computational resources allow. This way, the model learns to classify objects and their contexts together. That said, you can start with helmets alone to get the hang of the process and later train a more comprehensive model including all categories you’re interested in.

• Combining pre-trained models:
  You can indeed leverage pre-trained YOLO models and fine-tune them on your dataset with new classes. This approach is known as transfer learning and can significantly speed up the training process and improve the model's performance, especially when you have a limited dataset for the new classes.

Here’s a simple snippet to kick-start training with a custom dataset:

yolo detect train data=your_data.yaml model=yolov8n.pt epochs=100

Make sure your_data.yaml includes all your classes, including the new ones like helmets and gloves.

Best of luck with your project! Let us know if you have more questions. 🚀

[Boaruzhanchik]

How to choose the right imgsz for the model, I need to detect license plate numbers at a fairly large distance, a suitable dataset I have collected, but the question is which imgsz to choose for optimal and high-quality work
The license plates occupy a really small part of the image, but they can be seen very clearly and the quality is good, but there is a question about the imgsz.

[pderrenger]

@Boaruzhanchik choosing the right imgsz (image size) for detecting small objects like license plates from a distance can indeed be crucial for achieving optimal performance. Generally, larger image sizes can provide higher detail, which could be beneficial for small object detection, but they also require more computational resources.

For your scenario where license plates occupy a small part of the image but are clearly visible, starting with an imgsz similar to the resolution at which the plates are clearly visible is a good starting point. If the plates are clear at, say, 640x640 resolution, you might start with that. However, it's worth experimenting with higher resolutions if computational resources allow, as it might improve the detection quality further.

Here's a quick example to specify an image size for training:

yolo detect train data=your_dataset.yaml model=yolov8n.pt epochs=100 imgsz=640

If you find training or detection times to be too lengthy or if you're limited by your hardware capabilities, you might experiment with slightly lower resolutions. Keep in mind, the effectiveness greatly depends on your dataset's quality and the specific characteristics of the objects you're trying to detect. 🕵️‍♂️

Good luck with your license plate detection project!

[Nitya476]

Hello team,
I have a question regarding the training datasets. How much data should be sufficient for the training?
Thanks

[glenn-jocher]

@Nitya476 hello!

The amount of data needed for training can vary depending on the complexity of the task and the variability of the objects you want to detect. Generally, more data leads to better model performance, especially in diverse scenarios.

For starting out, a few hundred images per class can be sufficient when using robust data augmentation. If you're working with complex scenes or rare objects, you might require significantly more images. The quality of your annotations also plays a crucial role.

Here's a simple example to start training on a small dataset:

yolo detect train data=your_dataset.yaml model=yolov8n.pt epochs=100 imgsz=640

Remember, continually evaluating and validating your model can help guide you on whether more data is needed. 🚀 Feel free to experiment with different sizes of datasets to find what works best for your specific case!

[ajay78123]

I have trained yolov8 which detects shelf and products in it. i have a base image in which the products are perfectly placed in order and i have a test image in which the products are misplaced.
in both the images the shelf and the products are detected the main goal is to check whether the test image products are aranged as per the base image.
how ca we do that

[glenn-jocher]

It sounds like you're interested in comparing the arrangement of products in a test image against a base image using YOLOv8 for object detection. Here’s a straightforward method to approach this:

1. Object Detection: First, ensure that both your base and test images pass through the YOLOv8 model, and the products on the shelves are detected. You should retrieve bounding box coordinates for each detected object.

2. Normalization: If your images are taken from different angles or distances, you’ll need to normalize their scale and orientation.

3. Feature Matching: For each detected product in the base image, identify the corresponding product in the test image. This can be achieved through feature matching techniques using descriptors like ORB, SIFT, or simply matching based on similar bounding box sizes if the products are sufficiently distinctive.

4. Positional Comparison: Once matching is done, compare the relative positions of matched products. If the products are placed in the same order, their spatial relation in coordinates should be similar.

Here's a simple Python snippet using bounding box information to start:

# Assuming bbox format is [x1, y1, x2, y2]
def compare_positions(bboxes_base, bboxes_test):
    # Simple positional comparison logic (customize based on your needs)
    for base, test in zip(sorted(bboxes_base), sorted(bboxes_test)):
        if not (base == test).all():
            return False
    return True

# Example bounding boxes from your model prediction
bboxes_base = [[10, 10, 50, 50], [60, 10, 100, 50]]  # format is x1, y1, x2, y2
bboxes_test = [[15, 15, 55, 55], [65, 15, 105, 55]]

# Compare positions
arranged_correctly = compare_positions(bboxes_base, bboxes_test)
print("Products arranged correctly:", arranged_correctly)

This example simply checks if the bounding boxes from the two images are in order. Adjust the comparison logic based on how robust you need the check to be, especially handling more complex scenarios like overlapping products or different viewpoints.

[ajay78123]

do i need to train segmentation model for this perticular problem or detection model works?

[glenn-jocher]

For your specific problem, where you need to check the arrangement and placement of products on shelves, a detection model should suffice. YOLOv8 can detect each product as a separate object and provide their bounding boxes. By analyzing the coordinates and dimensions of these bounding boxes in your base and test images, you can determine if the products are arranged correctly.

Here's a brief example on how to compare product positions between two images using their bounding boxes:

def compare_positions(boxes_base, boxes_test):
    # Ensure bounding boxes are sorted by x-coordinate to maintain order
    boxes_base_sorted = sorted(boxes_base, key=lambda x: x[0])
    boxes_test_sorted = sorted(boxes_test, key=lambda x: x[0])

    # Compare positions; This is a simple example, adjust logic as needed
    for (box_base, box_test) in zip(boxes_base_sorted, boxes_test_sorted):
        if box_base[:2] != box_test[:2]:  # comparing (x, y) coordinates of top-left corner
            return False
    return True

# Sample bounding boxes from base and test images
bboxes_base = [[100, 50, 150, 100], [200, 50, 250, 100]]  # [x1, y1, x2, y2]
bboxes_test = [[105, 55, 155, 105], [205, 55, 255, 105]]

# Check if products are arranged correctly
is_correct = compare_positions(bboxes_base, bboxes_test)
print("Are products arranged correctly?", is_correct)

This script checks if the top-left corners of corresponding products in both images are at the same position, indicating correct arrangement. You can enhance the comparison logic based on your specific accuracy needs, such as considering overlaps or relative distances.

If you find that detecting precise edges and contours of products is crucial and requires segmenting each product more distinctly, considering a segmentation model might be beneficial. However, for alignment and order verification purposes, detection models are generally adequate.

[GaviraghiElia]

I have a doubt: does it make sense to change the conf parameter from 0.001 to another higher value, such as 0.25? Does it make sense tuning this hyperparameter?

I've a dataset with split 70-10-20 for train, val and test set.
If I run the model training (with model validation) using a confidence threshold of about 0.25 I get models that perform much better in terms of mAP50, mAP50-95 and precision, while recall remains virtually unchanged.

[pderrenger]

@GaviraghiElia yes, tuning the conf (confidence threshold) parameter is a practical approach, particularly when aiming to balance precision and recall. Increasing the conf value to something like 0.25, as you've tried, helps to reduce false positives, thereby increasing the precision of your model. This might be why you're seeing better mAP50 and mAP50-95 scores.

The fact that recall remains virtually unchanged suggests the higher threshold isn't causing you to miss many correct detections—an ideal scenario. If you're seeing improved model performance without a trade-off in recall, it indicates a positive adjustment. Continue to experiment with this threshold to find the best balance for your specific dataset and use case. 🎯

Here's how you can modify the confidence threshold during the prediction:

from ultralytics import YOLO

# Load model
model = YOLO('path/to/best.pt')

# Set custom confidence threshold
model.conf = 0.25  # Update conf value as per your needs

# Run predictions
results = model.predict('images/')

This example sets the confidence threshold to 0.25 for the predictions, which you can adjust as needed. Keep up the good tuning!

[SONYYA7]

My training images have different set of resolution, when i set the img_size parameter how can i fix one.
if i choose less than my higher resolution set. it is loss of learning so to avoid that if i choose highest resolution , i am afraid of lower resolution images. how can it be represented in bigger matrix.
i am using yolov8 model . it is detection training with bbox annotation .

does the architecture handles the upscale or downscale image sizes?
can i use highest resolution in img_size parrameter even tough i have lower resolution ?

thank you in advance ! excited to learn more from yolo models

[pderrenger]

@SONYYA7 hey, thanks for reaching out! Great to hear about your excitement for learning more about YOLO models. 🌟

When dealing with images of different resolutions during training, it's common to set a single img_size for all images. YOLOv8 automatically handles the resizing of images during training—both upscaling and downscaling as needed. This means you can set the img_size to the highest resolution in your dataset without worrying about the lower resolution images; they will be upscaled to match.

Here’s how you can specify the image size when training:

from ultralytics import YOLO

# Load your model
model = YOLO('yolov8n.pt')

# Train the model with a specific image size
results = model.train(data='your_dataset.yaml', epochs=50, imgsz=640)

In this snippet, imgsz=640 adjusts all images to 640x640 during training, regardless of their original sizes. If your highest resolution is, for example, 800x800, replace 640 with 800 in the imgsz parameter.

Yes, using the highest resolution for img_size can be beneficial as it retains the maximum detail from your high-res images, which can improve model performance, especially when fine details are crucial for detection.

I hope this helps! Let me know if you have any more questions. 😊 Happy training with YOLO!

[SONYYA7]

thank you for response,
i have another doubt , let say in model.train(model='yolov8-seg.yaml',train='detect')
now which would be implemented ,
detection or segementation

[SONYYA7]

we planned to train a backbone model,
since we didn't have a mask segment annotation for large dataset, we did a detection training.
now using best model from that training, we are training it with another small dataset with mask segment annotation ,
but when i train i get detect layer in last line of model.yaml ,
so how can i use it as segment model

i tried to change the task='segment',
but i don't get instance segmentation but box detection

the backbone model is stf-yolov8,
now i want that trained model to train with mask sement annotation

[pderrenger]

Hey there! It sounds like you're trying to transition from a detection task to a segmentation task using the YOLOv8 architecture, which is an exciting step!

When you want to switch from a detection model to a segmentation task, you'll need to adjust the model configuration to ensure it supports the segmentation outputs, such as masks, instead of just bounding boxes. Simply changing the task in your training script isn't sufficient because the model architecture defined in the YAML file must also support segmentation outputs.

For your stf-yolov8 model initially trained for detection, you would first need to adapt or modify the YAML architecture file used for training to include the necessary layers and configurations for instance segmentation. This typically involves adding or modifying heads that can produce segmentation masks.

Here’s a basic outline of steps you might consider:

1. Modify the Model Configuration (YAML file): Ensure it has layers capable of segment output.
2. Load Pretrained Weights (Optional): You can load the weights from your detection model and continue training for segmentation. Be careful with layer compatibility.
3. Dataset Preparation: Prepare your segmentation dataset correctly, ensuring that mask annotations are correctly formatted and linked in your dataset configuration file.
4. Training: Use the modified YAML and updated dataset for training.

If you encounter specific errors or need guidance on re-configuring the YAML file for segmentation tasks, feel free to share those details!

Happy modeling, and I’m here if you have more questions or need further assistance! 😊

[MoAbbasid]

Hi Ive trained my model on two objects, if I take that model and train it on an extra object will it still work with the first two objects or forgets them?

[glenn-jocher]

Hi there!

If you train your model on an additional object without including data from the first two objects, the model is likely to forget the initial objects due to a phenomenon known as "catastrophic forgetting."

To retain knowledge of all three objects, you should continue training with a dataset that includes examples of all three — the two original classes and the new class. Here’s how you can integrate the new class while maintaining old class performance:

from ultralytics import YOLO

# Assuming 'old_data.yaml' is your initial dataset and 'new_data.yaml' includes your new class
model = YOLO('path/to/trained_model.pt')  # load your pretrained model

# Train with a mixed dataset
results = model.train(data='mixed_data.yaml', epochs=50, imgsz=640)

Make sure that 'mixed_data.yaml' contains paths to images from both 'old_data.yaml' and 'new_data.yaml'. This approach ensures that your model learns about the new object while reinforcing its knowledge about the existing ones. 😊

Hope this helps! Let me know if you have other questions!

[JiangYeC]

how to make val data not Half(float16)?

[JiangYeC]

Thanks for your response, In training, reporting error for validation ofeach epoach is "ExpectFloat but Half"

[pderrenger]

It sounds like you're encountering a data type mismatch during validation after each epoch, specifically expecting a float type but receiving Half (float16). This can happen if your model or data loaders are set to use half precision but the validation function expects full precision.

To resolve this, ensure that your validation data and model are using the same data type. If you want to use full precision (float32) for validation, you can explicitly set the half argument to False during the validation phase. Here’s how you can adjust this in your training script:

from ultralytics import YOLO

# Load your model
model = YOLO('path/to/model.pt')

# Train the model with validation in full precision
results = model.train(data='coco128.yaml', epochs=100, imgsz=640, half=False)  # Set half=False for full precision

This adjustment ensures that during the validation phase after each epoch, the model uses full precision, which should align with your validation data expectations and resolve the error. If you continue to experience issues, please check that all related settings in your dataset and model configuration align with your precision requirements.

[pderrenger]

@JiangYeC it sounds like you're encountering a data type mismatch during validation after each epoch, specifically expecting a float type but receiving Half (float16). This can happen if your model is set to use half-precision during training but full precision is required for certain operations or layers during validation.

To resolve this, ensure that your validation phase uses full precision. If you're using the Python API, you can explicitly set the half argument to False during the validation call to force the use of full precision:

from ultralytics import YOLO

# Load your model
model = YOLO('path/to/model.pt')

# Validate the model with full precision
metrics = model.val(half=False)

If you're using the CLI, include half=False in your validation command:

yolo val model=path/to/model.pt half=False

This should help avoid the type mismatch error during validation. If the issue persists, please check that all parts of your validation data pipeline are also configured to use the correct data type.

[JiangYeC]

thanks for your response，have solved it.

[pderrenger]

@JiangYeC great to hear that you've resolved the issue! If you have any more questions or need further assistance in the future, feel free to reach out. Happy coding! 😊🚀

[leberknecht]

model.train(data="coco8.yaml", epochs=100, imgsz=640)

It woult be super helpful to have a link here to resources describing what coco8.yaml looks like / dataset-configuration-specs.

[pderrenger]

Hello! You can find the details about the coco8.yaml dataset configuration and other dataset specifications in our documentation on dataset preparation. Here's a link to the dataset configuration guide which should provide you with all the necessary information on how to structure your YAML files for training with YOLOv8. If you have any more questions or need further assistance, feel free to ask. Happy training! 🚀

[pderrenger]

Certainly! The coco8.yaml file you mentioned is a dataset configuration file that specifies details about the dataset used for training, such as paths to training and validation data, class names, and the number of classes. You can find more information about how to configure and use these YAML files for your training datasets in the Ultralytics documentation on dataset configuration.

Here's a basic example of what the contents of a coco8.yaml might look like:

# Example of a coco8.yaml file
path: /datasets/coco8  # dataset root directory
train: images/train  # train images relative to 'path'
val: images/val  # val images relative to 'path'
test:  # test images relative to 'path' (optional)

# Classes
nc: 8  # number of classes
names: ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck']

This file helps YOLOv8 know where to find the data it needs for training and validation, as well as details about the classes it should recognize. Adjust the paths and class names according to your specific dataset.

[Boaruzhanchik]

Good afternoon, please advise what arguments to use when training the segmentation model for the most accurate results (so that the segmentation mask is as close as possible to the way I marked the objects).
I found such parameter as mask_ratio and set it to 1, what parameters can be applied for better performance of the model?

[glenn-jocher]

Good afternoon! For enhancing the accuracy of segmentation masks in your YOLOv8 model, setting mask_ratio=1 is a great start as it ensures high-resolution masks. Additionally, consider adjusting the following parameters for potentially better performance:

• imgsz: Increase the input image size if your hardware allows. Larger sizes can help capture more details.
• augmentations: Utilize augmentations like mosaic, mixup, and copy_paste which can help the model learn from varied data scenarios and improve generalization.
• iou_thresholds: Adjust the IoU thresholds used during training to fine-tune how precise the overlaps need to be.

Here's a simple example to include these in your training setup:

from ultralytics import YOLO

# Load and configure the model
model = YOLO('path/to/your_model.yaml')
results = model.train(data='your_dataset.yaml', epochs=100, imgsz=640, augmentations={'mosaic': 1.0, 'mixup': 0.2})

For more detailed guidance on hyperparameters and training configurations, please refer to the Ultralytics training documentation. Happy training! 🚀

[Boaruzhanchik]

Thanks for the reply, I'll try it out and keep you posted.

[glenn-jocher]

Great to hear that you'll be trying it out! Keep us updated on your progress, and don't hesitate to reach out if you have any questions along the way. Happy training! 🚀

[AlekseySpirin]

Dear YOLO team,

I am currently working on a custom YOLO model for a specific object detection task, and I would like to improve its accuracy and performance by fine-tuning the model on my own dataset.

I am aware that one way to achieve this is to freeze some of the layers in the model so that they are not updated during fine-tuning, and focus on modifying the layers that have the most impact on my specific task.

Could you please advise me on how I can view the layers in my custom YOLO model that are suitable for freezing during fine-tuning? Are there any tools, libraries, or methods that you would recommend for this purpose?

I would greatly appreciate any help or guidance that you can provide. Thank you for your time.

[AlekseySpirin]

YOLOv8s

[glenn-jocher]

Hello!

To view and select layers in your custom YOLO model for freezing during fine-tuning, you can use the torchsummary library in Python, which provides a detailed visualization of your model's layers. Here’s how you can do it:

1. Install the library if you haven't:

   pip install torchsummary

2. Import the library and your model, then use torchsummary to view the layers:

   from ultralytics import YOLO
   from torchsummary import summary
   
   # Load your model
   model = YOLO('path/to/your/custom_model.pt')
   
   # Assuming your model is using the default image size of 640
   summary(model, input_size=(3, 640, 640))

This will display each layer along with its dimensions and parameters, helping you decide which layers to freeze based on their relevance to your specific task. Typically, freezing the early convolutional layers can be a good starting point, as they capture universal features like edges and textures that are similar across different tasks.

For more detailed guidance on fine-tuning and other advanced training techniques, you might find our training documentation helpful: Train YOLOv8 Models.

Best of luck with your model training! 🚀

[ajay78123]

I have trained yolov8 on a dataset. Which was working good so i have that best.pt to annotate the additional images for the project an the results are good and i want to train the model with the updates images but i am encountering some errors hep me with this.
for first dataset i have used roboflow and now i have merged these auto annotations with the existing dataset this causing the training issue.
train: Scanning /content/drive/MyDrive/yolov8/dataset/train/labels... 4029 images, 0 backgrounds, 1779 corrupt: 100% 4029/4029 [56:04<00:00, 1.20it/s]
train: WARNING ⚠️ /content/drive/MyDrive/yolov8/dataset/train/images/VID20240515204654_frame0.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates [ 275.18 401.45 704.41 1408.9]
train: WARNING ⚠️ /content/drive/MyDrive/yolov8/dataset/train/images/VID20240515204654_frame1.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates [ 276.96 406.5 703.58 1409.3]
train: WARNING ⚠️ /content/drive/MyDrive/yolov8/dataset/train/images/VID20240515204654_frame10.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates [ 294.34 425.43 704.74 1397.2]
train: WARNING ⚠️ /content/drive/MyDrive/yolov8/dataset/train/images/VID20240515204654_frame100.jpg: ignoring corrupt image/label: non-normalized or out of bounds coordinates [ 315.59 532.74 723.99 1486.6]

[glenn-jocher]

It looks like the issue you're encountering is due to non-normalized or out-of-bounds coordinates in your label files. This often happens when merging datasets with different annotation formats or errors in the annotation process.

Here’s a quick way to address this:

1. Validate Label Formats: Ensure all your labels are correctly normalized relative to image dimensions (i.e., values between 0 and 1). You might need to adjust the annotation export settings in Roboflow or manually correct them.

2. Check for Errors: Use a script to check for any labels that exceed the image dimensions or are negative. These are likely causing the "corrupt image/label" errors.

3. Re-Export Data: If you used Roboflow to prepare the first dataset, ensure the second dataset is processed in the same way, with consistent output settings.

Here's a small Python snippet to help identify problematic labels:

import os
import numpy as np

def check_labels(label_dir):
    for label_file in os.listdir(label_dir):
        path = os.path.join(label_dir, label_file)
        if not path.endswith('.txt'):
            continue
        with open(path, 'r') as file:
            for line in file:
                x_center, y_center, width, height = map(float, line.split()[1:])
                if not (0 <= x_center <= 1 and 0 <= y_center <= 1 and 0 <= width <= 1 and 0 <= height <= 1):
                    print(f'Out of bounds in {label_file}: {line.strip()}')

label_directory = '/content/drive/MyDrive/yolov8/dataset/train/labels'
check_labels(label_directory)

This script checks if the bounding box coordinates are within the expected range (0 to 1). Adjust your dataset accordingly and try retraining. If the issue persists, please provide the error logs for further assistance. 🛠️

[cuetiger]

Hello developer, when the training with epoch=300 is finished, I want to increase the training epoch, such as epoch=600, that is, train 300 epochs extra, no new project appears, how to do it

[pderrenger]

@cuetiger hello! To continue training your model for an additional 300 epochs without creating a new project, you can resume training from the last saved checkpoint. Here's how you can do it:

Python

from ultralytics import YOLO

# Load the partially trained model
model = YOLO("path/to/last.pt")

# Resume training for an additional 300 epochs
results = model.train(epochs=300, resume=True)

CLI

yolo train resume model=path/to/last.pt epochs=300

This will continue training from where it left off, using the saved state from the last checkpoint. Happy training! 🚀

For more details, you can refer to the Ultralytics YOLO training documentation.

[cuetiger]

Freezing layer 'model.22.dfl.conv.weight'
AMP: running Automatic Mixed Precision (AMP) checks with YOLOv8n...
AMP: checks passed ✅
train: Scanning /root/autodl-tmp/ultralytics-main/datasets/tile_640_02/train.cache... 18662 images, 0 backgrounds, 0 corrupt: 100%|██████████| 18662/18662 [00:00
albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8))
val: Scanning /root/autodl-tmp/ultralytics-main/datasets/tile_640_02/val.cache... 2074 images, 0 backgrounds, 0 corrupt: 100%|██████████| 2074/2074 [00:00<?, ?it
Plotting labels to tile/yolov8n22/labels.jpg...
optimizer: SGD(lr=0.01, momentum=0.937) with parameter groups 57 weight(decay=0.0), 64 weight(decay=0.0005), 63 bias(decay=0.0)
Traceback (most recent call last):
File "resume.py", line 6, in
results = model.train(epochs=300, resume=True)
File "/root/autodl-tmp/ultralytics-main/ultralytics/engine/model.py", line 655, in train
self.trainer.train()
File "/root/autodl-tmp/ultralytics-main/ultralytics/engine/trainer.py", line 213, in train
self._do_train(world_size)
File "/root/autodl-tmp/ultralytics-main/ultralytics/engine/trainer.py", line 327, in _do_train
self._setup_train(world_size)
File "/root/autodl-tmp/ultralytics-main/ultralytics/engine/trainer.py", line 319, in _setup_train
self.resume_training(ckpt)
File "/root/autodl-tmp/ultralytics-main/ultralytics/engine/trainer.py", line 672, in resume_training
assert start_epoch > 0, (
AssertionError: tile/yolov8n2/weights/last.pt training to 500 epochs is finished, nothing to resume.
Start a new training without resuming, i.e. 'yolo train model=tile/yolov8n2/weights/last.pt'

[pderrenger]

Hi there!

It looks like you've encountered an issue with resuming training. The error message indicates that the model has already completed training to 500 epochs, so there's nothing to resume.

To continue training for additional epochs, you should start a new training session using the last checkpoint as the initial model. Here's how you can do it:

Python

from ultralytics import YOLO

# Load the last checkpoint
model = YOLO("tile/yolov8n2/weights/last.pt")

# Train for an additional 300 epochs
results = model.train(epochs=300)

CLI

yolo train model=tile/yolov8n2/weights/last.pt epochs=300

This will effectively continue training from where you left off without creating a new project. If you have any further questions, feel free to ask! 😊

Happy training! 🚀

[Rikjereposurfer]

Hi, i want to change my directory where i save my output using the save_dir argument in the train command. However, it still uses the default runs folder in the same location as my script. How can i fix this?

[glenn-jocher]

Hi there! To change the directory where your output is saved, you can use the project and name arguments in your training command. The project argument specifies the root directory, and the name argument specifies the subdirectory for the current run.

Here's an example:

from ultralytics import YOLO

# Load a model
model = YOLO("yolov8n.pt")

# Train the model and specify the output directory
results = model.train(data="coco8.yaml", epochs=100, imgsz=640, project="my_project", name="my_experiment")

Or via CLI:

yolo detect train data=coco8.yaml model=yolov8n.pt epochs=100 imgsz=640 project=my_project name=my_experiment

This will save your outputs in my_project/my_experiment. If you still encounter issues, please ensure that the specified directories exist or can be created by the script. Happy training! 🚀

[cuetiger]

Dear Team：
How to use samplers in yolov8 for class balancing of training sets Adoption

[glenn-jocher]

@cuetiger hello,

To implement class balancing using samplers in YOLOv8, you can leverage PyTorch's WeightedRandomSampler or ClassBalancedSampler. These samplers help ensure that each class is represented proportionally during training, which is particularly useful for imbalanced datasets.

Here's a basic example of how to integrate a WeightedRandomSampler:

from torch.utils.data import WeightedRandomSampler
from ultralytics import YOLO
import numpy as np

# Load your dataset
data = 'path/to/your/dataset.yaml'

# Calculate class weights (example: inverse frequency)
class_counts = np.bincount(your_labels)
class_weights = 1. / class_counts
sample_weights = class_weights[your_labels]

# Create a sampler
sampler = WeightedRandomSampler(weights=sample_weights, num_samples=len(sample_weights), replacement=True)

# Load your model
model = YOLO('yolov8n.yaml')

# Train the model with the sampler
results = model.train(data=data, epochs=100, imgsz=640, sampler=sampler)

For more detailed guidance on training configurations and samplers, please refer to the Ultralytics documentation.

Happy training! 🚀

[IKENNA113]

Hey, I am new to object detection. I trained yolov8 baseline model. Now I am trying to modify the backbone and the neck with different attention module to see their different performance.

I can see that YOLOV8 has CBAM. Now how do i integrate this into yolo. what are the files I need to adjust. The How will I also integrate other attention module not found in YOLOv8.

Also, I want to add Multihead attention in the last layer of the backbone. Please if you can guild how to go about it I will really appreciate. Thanks

[glenn-jocher]

Hello! It's great to see your enthusiasm for enhancing YOLOv8 with different attention modules. Here’s how you can integrate CBAM and other attention mechanisms into your model:

1. Integrate CBAM:

   • CBAM is already included in some YOLOv8 configurations. To integrate it into your model, you'll need to modify the model configuration file (.yaml). You can specify CBAM in the layers where you want the attention mechanism to be applied.

2. Adding New Attention Modules:

   • For attention mechanisms not currently supported by YOLOv8, you will need to:
     • Write a custom module in Python that defines the new attention mechanism.
     • Import this module in your model’s script.
     • Integrate the module into the YOLOv8 architecture by modifying the corresponding .yaml configuration file to include your custom module at the desired stages of the model.

3. Multihead Attention in the Backbone:

   • To add a Multihead Attention layer at the last layer of the backbone:
     • Define the Multihead Attention layer in a Python file.
     • Modify the backbone configuration in the .yaml file to include this layer at the specified position.

For detailed guidance on modifying the .yaml files and integrating custom layers, the YOLOv8 documentation and source code can be very helpful. If you need more specific examples or run into any issues, feel free to ask. Good luck with your modifications! 🚀

[IKENNA113]

Thank you so much. This is an eye opener for me. Well appreciated. If I encounter any issue will still reach out. Thanks for the comprehensive detail.

[glenn-jocher]

You're very welcome! I'm glad to hear that the information was helpful to you. Don't hesitate to reach out if you have any further questions or if you encounter any issues along the way. We're here to help. Happy coding! 😊🚀