modes/val/

[giscus[bot]]

modes/val/

Guide for Validating YOLOv8 Models. Learn how to evaluate the performance of your YOLO models using validation settings and metrics with Python and CLI examples.

https://docs.ultralytics.com/modes/val/

[shubhamjanu]

This line of code "metrics.box.map50" gives error saying that "'NoneType' object has no attribute 'box' ".

My code is described below:

Load a YOLOv8 model

model = YOLO('yolov8n.pt')

Train the model

results_train = model.train(data='japan5.yaml', epochs=1, imgsz=600)
model = YOLO('runs/detect/train11/weights/best.pt') # load a custom model

Validate the model

metrics = model.val() # no arguments needed, dataset and settings remembered
map50=metrics.box.map50
print(map50)

Please suggest.

[glenn-jocher]

@shubhamjanu hello! 😊 Let's dive right into your queries:

1. For training with high-resolution images while maintaining the aspect ratio, YOLOv8's training mechanism automatically handles resizing and aspect ratios elegantly. However, dealing with very high-resolution images does require a balance. For your case, using imgsz=640 (or slightly higher if your GPU supports it) is a sensible choice. It scales down images, maintaining their aspect ratio. YOLOv8 interprets imgsz as [height, width]. The worse results with [1920,1024] could be due to several factors, including overfitting or not fine-tuning other hyperparameters accordingly.

2. Yes, resizing with the imgsz parameter handles annotations accordingly. YOLOv8 takes care of scaling the annotations, so you don't have to manually adjust your annotation files.

3. Whether downsampling or upsampling is better can depend on various factors, including the nature of your dataset and what you're trying to detect. Generally, slight upsampling (e.g., 600x600 to 608x608) can be beneficial as it introduces minimal distortion while adhering to the model's architectural constraints. However, extensive experimentation with your specific dataset is the best way to determine the ideal approach.

4. The IoU threshold differences during training (0.7) and validation (0.6) are predefined settings based on empirical performance metrics. If your goal is to calculate mAP at IoU=0.5, setting iou=0.5 during both training and validation is a reasonable approach. This aligns the model's detection criteria closely with your evaluation metric and can provide more relevant performance insights.

Remember, experimenting with these settings in the context of your specific dataset and objectives is key to achieving the best results. Happy experimenting! 🚀

[DoFulangChen]

hi, when i use the pycharm, the code is (not cmd line)
model.train(data=r"datasets\data.yaml", epochs=5, batch=1, device='cpu', task='segment', imgsz=[1920, 1200])

one warning comes out
WARNING ⚠️ updating to 'imgsz=1920'. 'train' and 'val' imgsz must be an integer, while 'predict' and 'export' imgsz may be a [h, w] list or an integer, i.e. 'yolo export imgsz=640,480' or 'yolo export imgsz=640'

how can i train my own dataset ? img_width = 1920, img_height = 1200, many tks

[glenn-jocher]

Hey there! 👋 It looks like you've encountered a warning due to a slight misunderstanding in specifying the image size for training in PyCharm.

For training, YOLOv8 expects the imgsz argument to be a single integer, representing both the height and width since it scales the images keeping their aspect ratio intact. This means when you provide a single integer (like imgsz=640), it maintains the aspect ratio for you automatically. If you're seeing worse results at a specific size, consider different dimensions close to your target but as a single value, or experiment with the rect parameter as you mentioned.

The warning you encountered suggests a syntax more aligned with export or predict tasks, which can accept [height, width] lists. For training your dataset that has images of resolution 1920x1200, start with something manageable for your GPU like imgsz=640 and see if your hardware can handle higher resolutions gradually.

To answer your question, for training, you should adjust the imgsz to a single value that ensures your hardware supports it, without explicit width and height distinction. The annotation adjustments are taken care of by YOLOv8 based on the resizing.

Regarding whether to downsample or upsample, both have benefits and drawbacks, and the choice heavily depends on your specific problem and dataset. Downsampling retains more original content, but might miss fine details, whereas upsampling might introduce artifacts. Given the two options, I'd slightly lean towards downsampling for less distortion, but encourage testing both to see what works better for your case.

The IoU parameter differences during training and validation reflect typical use cases, but for mAP calculation at IoU=0.5, you should indeed adjust the IoU threshold to 0.5 during both training and validation for consistency with your evaluation criteria.

Hope this helps clarify things! Let me know if there's anything else I can do for you. Happy training! 😊

[DoFulangChen]

i see, thanks for your answer, regards

[glenn-jocher]

Hey there! 😊 Glad you found the answer helpful. For training your dataset with varying resolutions while preserving aspect ratios and avoiding out-of-memory issues, you might consider trying adaptive resizing with smaller imgsz values that fit your GPU capabilities, like imgsz=640. YOLOv8 indeed auto-scales annotations, so you're all set on that front!

Regarding image resolutions, the decision between downsampling and upsampling (e.g., 576 vs. 608) generally depends on your specific dataset and task. Experimenting is key, but starting with downsampling might be slightly more efficient to retain original content without artificially introducing pixels.

For calculating mAP at IoU=0.5, setting iou=0.5 for both training and validation phases would align your training closer with your evaluation metric.

Remember, YOLOv8 will take care of aspect ratios and annotations for you during training. Just specify your desired imgsz (as a single integer for training) and experiment to find what works best for your setup. Happy training! 🚀

[devandress]

i was trying to train my model but, i dont find de predict file after training either best.pt

[pderrenger]

Hey there! 👋 It sounds like you're having trouble locating the predict file and the best.pt model after training. No worries, let's sort this out!

After training with YOLOv8, the best.pt model (which is the best performing model on your validation set during training) should be saved in your runs/train/exp*/weights/ directory. The directory might have a different name if you've set a custom --project or --name during training.

As for the predict functionality, it's not a separate file but a mode you can use with the YOLO CLI or Python API. To make predictions using your trained model, you can use the following command if you're using the CLI:

yolo detect predict model=runs/train/exp*/weights/best.pt source='path/to/your/image/or/video'

Or, if you're using Python:

from ultralytics import YOLO

# Load your trained model
model = YOLO('runs/train/exp*/weights/best.pt')

# Make predictions on an image or video
results = model(['path/to/your/image/or/video'])

# To view results
results.show()

Make sure to replace 'path/to/your/image/or/video' with the actual path to your image or video file. Also, adjust the path to best.pt based on where your trained model is saved.

If you're still having issues or if there's anything else I can help you with, feel free to ask. Happy coding! 😊

[eddmar1993]

After the training of YOLOv8, I got my metrics results in csv file. However, I did validation. But I cannot get my results in csv format. Only images (png & jpeg). How to get validation results in csv?

[glenn-jocher]

Hey there! 😊 To get your validation results in CSV format, you can utilize the save_json argument during validation, which will save the results in a JSON file. From there, you can easily convert the JSON file to CSV using Python. Here's a quick example:

from ultralytics import YOLO
import json
import pandas as pd

# Validate the model and save results to JSON
model = YOLO('path/to/your/model.pt')
model.val(save_json=True)

# Assuming your JSON results are saved as 'results.json'
with open('results.json') as f:
    data = json.load(f)

# Convert JSON to DataFrame then save as CSV
df = pd.DataFrame(data)
df.to_csv('validation_results.csv', index=False)

This will give you a CSV file with your validation results. Hope this helps! Let me know if you have any more questions.

[riubotin]

Hello! Glenn.. 😊 I'm Jeff. I found this post about getting validation results in CSV format very helpful! I'm currently working on integrating YOLO with a database, specifically MySQL, to store detection results and calculations. I noticed that you suggested saving the validation results as JSON before converting them to CSV. However, I'm wondering if there's a more direct way to save the detection results directly to a MySQL database without the intermediate step of JSON conversion.

My goal is to continuously run YOLO on a Jetson device with a remote camera, while sending the detection results to a database. I've created a separate GUI for monitoring purposes. Could you provide some guidance on how to achieve this? I haven't come across many resources that explain how YOLO interacts with a database server.

Any insights or tips on how to seamlessly integrate YOLO with a MySQL database would be greatly appreciated. Thank you in advance for your help!

Additionally, I'm specifically interested in extracting data such as the type of vehicles, ID, and speed. As for the video, I'd like it to be streamed in real-time to my GUI.

[RudeNinja]

Hey, I am trying to get metrics such as recall and also trying to save some images in validation. I am using coco.yaml. Any idea how I might do it?

[glenn-jocher]

Hey there! 👋

For calculating recall and saving validation images using coco.yaml, you can simply run the validation process with the save_json and plots arguments set to True. This will save the results in JSON format (which includes recall among other metrics) and also save plots of predictions vs. ground truth for visual evaluation.

Here's how you can do it:

yolo detect val model=yolov8n.pt data=coco.yaml save_json=True plots=True

This command will validate your model and save the detailed metrics and images in the runs directory. You can find recall in the saved JSON file and the validation images in the plots directory.

Hope this helps! Let me know if you have any other questions. 😊

[vanquang203546]

I want to get parameters of a class in metrics and save it to csv. Thank you very much.

[pderrenger]

Hi there! 👋 To save class-specific metrics like mAP, precision, and recall to a CSV file after validation, follow these steps:

Python API Example

After running validation, you can access class-specific metrics from the metrics object and save them using pandas:

import pandas as pd
from ultralytics import YOLO

# Validate your model
model = YOLO('yolov8n.pt')  # or your custom model
metrics = model.val()

# Assuming you want mAP for each class
map50_95_per_class = metrics.box.maps.tolist()

# Create a DataFrame
df = pd.DataFrame({
    'class': model.names,
    'mAP50-95': map50_95_per_class
})

# Save to CSV
df.to_csv('class_metrics.csv', index=False)

CLI Approach

If you're looking to perform post-processing after using the CLI for validation, you'd need to parse the output logs or JSON file (--save_json for generating a JSON file) and then manually write to a CSV using a similar approach or any preferred scripting language.

Remember, if you choose to save results directly from validation with --save_json, the resulting file can be easily parsed and converted into a CSV with your preferred data handling tools like pandas or even Excel.

Let me know if this helps or if you have any more questions! 😊

[HafizRaheem]

hey

i want to do the evaluate my model i already get the detection images using ignition gazebo and ros2 but now i want to evaluate my model like accuracy or MAP or recall how to do it i did not have any dataset or any annotations i am using pretrained yolov8 model and using coco8.yaml

[HafizRaheem]

for option 2 Run Predictions on Real-Time Images. what it mean for this sentence (could be a ROS2 subscriber saving images to disk temporarily) before start the detection i have to save images after the ros2 but before the detection start using yolo and then pass those images to the yolov8 for detection

and for predict does it give me the precision or recall
that is the main thing i have to do it other than that i have images with scores
file:///home/hafiz/ros2_ws/processed_images/processed_10.jpg

[HafizRaheem]

or first i have to train my model and then do everything download the dataset from the kaggle and trian it and do everything

[glenn-jocher]

Hey there! 👋

Absolutely, if you've already got a pretrained model and have newly annotated images in COCO format, you’re on a fantastic track. Since the images are already detected using Ignition Gazebo and ROS2, and now that you've annotated some of them for evaluation, you’re ready to move on to the validation part using Ultralytics YOLOv8.

For validation, you don't need to re-train your model unless you're aiming to fine-tune it on very specific data from your environment. Given you're using a pretrained model, here’s how you can proceed:

1. Prepare your dataset YAML file, ensuring it points to your annotated images and COCO-style annotations. Your YAML might look something like this (adjust paths as necessary):

val: path/to/your/images
annotations: path/to/your/annotations.json

2. Run validation to evaluate your model against your manually annotated dataset. This will give you precision, recall, and mAP metrics, providing insight into how well your model is performing.

yolo detect val model=path/to/your/pretrained/model.pt data=path/to/your/dataset.yaml

This step uses the annotations you've created to compare against the model's predictions, allowing you to assess the model's accuracy.

Since you're handling real-time images and have already annotated them, this validation process will enable you to evaluate the YOLO model's effectiveness in your specific application scenario.

For real-time detection, there's typically no need for a separate dataset aside from monitoring the model's general performance or unless you're using it for continuous learning. So, you're correct; for real-time detection, you proceed with the model prediction directly on incoming images from Ignition Gazebo and ROS2.

Please let me know if you need further guidance or have more questions. Best of luck with your project and your upcoming deadline! 😊

[HafizRaheem]

Thank You but confused how to do it this part i share my code do i have to do this in sperate or do it in same file that i share i have some images with bounding box and scores. now i don't need to rerun the detection or my node so i have annotaton json file and images that i use and draw bounding box using cvat and save as coco format just used those for precison and recall.
sorry i know i am asking stupid question but i have no choice

[glenn-jocher]

Hey there! 😄 It looks like you're on the right track with creating annotations for your images in COCO format using CVAT. Now, since you have both images and annotations, you don't need to rerun detections for validation purposes. You can use your annotations and images directly to calculate precision, recall, and mAP metrics.

Here’s the simplified process:

1. Ensure your annotated dataset (images + COCO format annotations) is organized correctly. Your .yaml file for the dataset should correctly point to the location of your images and annotations.

2. Validate your model using the dataset to get precision, recall, and mAP. Here's how you can do it:

   from ultralytics import YOLO
   
   # Load your model
   model = YOLO('path/to/your/pretrained/model.pt')
   
   # Validate the model
   metrics = model.val(data='path/to/your/dataset.yaml')
   print(metrics.box.map, metrics.box.map50, metrics.box.map75)

Or via CLI,

yolo detect val model=path/to/your/pretrained/model.pt data=path/to/your/dataset.yaml

3. No need for separate detection runs for validation. The val process will internally use your model to predict on the images specified in your dataset YAML and compare those predictions against your annotations to compute the evaluation metrics.

4. Prediction and Validation are separate steps. Predictions are for visual inspection or application integration (using detected bounding boxes and scores). Validation (with .val()) computes how well your model is performing statistically (precision, recall, mAP).

For real-time detections coming from Ignition Gazebo and ROS2 to YOLOv8, you're essentially running predictions. If you wish to evaluate these real-time detections, you would need a way to capture the inferred annotations and have a ground truth to compare against, which might require building a custom process.

No questions are stupid; we all start somewhere! 👍 If you're under a tight deadline and need more detailed help or file reviews, consider reaching out directly on GitHub issues or discussions specific to the Ultralytics YOLO repository. The community and developers are pretty responsive there. Best of luck with your project and deadline!

[UsmanRashid12]

Hello i have two questions.
First one: i have trained model on custom dataset for object detection. After testing the model is giving false positives i want to calculate the false positives? How to calculate the false positives that how many times it is giving me false positive.
Second one: I want to what save_hybrid does? and how to implement it what are the benefits of it.

[UsmanRashid12]

Hello thankyou for your reply. To the first answer i have this already but wanted to see the FP directly. Secondly what if i want to run evaluations on my whole model and then make a confusion matrix on it. Like run the model on all my dataset including all the images of train test valid and then create a confusion matrix of it. The issue i face is when running inference on each jpeg file it saves the image just by save=True but the evaluations such as mask, bbox, keypoints, names and orig img. when i try to save them it gives me error even i cant convert them to JSON because of object error in list. so how do you recommend i do this.

Issue with my model is it is detecting false positives with a high percentage so i want to see what the problem really is between those two wrongly detected classes.

[UsmanRashid12]

Also there is not much information available about the save_hybrid=True argument ive searched it online that how does it shows us the predictions with the model against actual labels. Because i get results in batches of images , confusion matrix, and others like P-curve, R_curve. after val

[pderrenger]

Hello! Thanks for reaching out. 🙌

For the save_hybrid functionality, by setting save_hybrid=True during the validation process, the model will save a hybrid version of labels. This means it will combine your model predictions with the original labels in your validation dataset. This can immensely help in analyzing the model's performance by visually comparing its predictions against the ground truth directly on the images. You can see these comparisons in the resulting batch of images, confusion matrices, P and R curves, etc., provided after running yolo detect val with save_hybrid=True. It's a powerful tool for diagnosing model behavior, especially in identifying false positives or understanding class-wise performance.

To address the issue of saving evaluations like masks, bboxes, keypoints, names, and original images, it sounds like you're looking for a structured output. Typically, these details are part of the Python object returned by validation or prediction methods and might not be directly serializable to JSON due to their complex structures (e.g., tensors or custom objects).

For a comprehensive evaluation, including generating a confusion matrix that visually represents model performance across all your dataset images (train, test, validate), you'd likely benefit from aggregating predictions across your dataset in a structured form like a CSV or a database, then using a separate script or tool to calculate and visualize the confusion matrix.

If you're encountering serialization issues, consider converting these objects to more JSON-friendly structures (like converting tensors to lists) before attempting to convert to JSON. Here’s a quick snippet for converting a prediction result to a list (Pythonic way), which might give you an idea:

# Assuming 'results' is the output from model validation or prediction
predictions = [result.xyxy[0].tolist() for result in results]

For identifying why your model is giving false positives between two classes, diving into the save_hybrid results can provide visual clues. Furthermore, systematically reviewing the confusion matrix and the specific images associated with those false positives can reveal patterns or specific features the model might be misinterpreting.

I hope this helps! If you have any more questions or need further assistance, feel free to ask. Happy coding! 😊

[UsmanRashid12]

Hello thankyou so much for this detailed answer i used save_hybrid feature. i have one more question

If i want to see the wrongly predicted image in the model how can i do that. For example my model is wrongly predicting an image giving me a false positive can i see through validation or any argument that the model predicted which image wrong?

[pderrenger]

@UsmanRashid12 hey there! 😀 Really glad the save_hybrid feature was helpful for you! To identify and inspect wrongly predicted images or false positives more directly, you might consider running validation with detailed logging enabled or utilizing the save_hybrid=True feature to visually inspect outputs.

For validation, running the command below generates a detailed report, including a confusion matrix that can highlight where your model tends to predict wrong classes:

yolo detect val model=path/to/your/model.pt data=your_dataset.yaml

For save_hybrid=True, unfortunately, the YOLOv8 documentation is sparse on this. Generally, it combines model predictions with ground-truth annotations on saved output images, allowing you to visually compare predictions against truth directly for each image.

To further drill down on specific false positives or to build a custom confusion matrix from your entire dataset, consider collating predictions programmatically:

from ultralytics import YOLO

model = YOLO('path/to/your/model.pt')

# For each image in your dataset...
for path in dataset_paths:
    results = model.predict(path)
    # Process each result

You can structure predictions alongside labels for all images to analyze/model performance comprehensively, export them to JSON or another format, and then use that data to specifically identify and examine false positives more closely.

Hope this helps! Let me know if there's anything else you're curious about. 🌟

[UsmanRashid12]

Hello i have two questions kindly help me out.

when i have custom trained model. if i load the model and then to detect use" !yolo detect val model=/weights/best.pt data=/data.yaml save=True save_json=True conf=0.85 iou=0.5 split=val " can i give it a new yaml file other than the one which model was trained. For example i used an old dataset to train the model. Now i have new data on which i want to validation of the model i upload data in valid folder and give its yaml file to the model to load those images and validate against the model. (THOSE ARE NEW IMAGES ADDED AFTER TRAINING IN VALID) will it give me accurate results on the new dataset against the model trained? I added 10k images to my Valid folder and want to detection on those new images and see the validation of the model.

Second question is that does conf threshold matters while doing validation? and can you please explain how is the confusion matrix is build after validation because its prediction against classes so it goes and compares with the ground truth? Please explain thankyou

[pderrenger]

@UsmanRashid12 hello! I'm happy to help with your questions. 😊

1. Validating on a New Dataset: Yes, you can specify a new YAML file that points to a different dataset for validation, such as your new data.yaml with the updated valid folder images. YOLOv8 models adapt to the dataset specified during the validation phase. So, if your model was trained on an older dataset and you now have a new set of images, simply adjust your data.yaml to point to these new images for validation. This approach will give you insights into how your model performs on the new dataset. However, keep in mind, if the new dataset is significantly different or more challenging, the accuracy might vary.

Example command adjustment:

!yolo detect val model=/weights/best.pt data=/new_data.yaml save=True save_json=True conf=0.85 iou=0.5 split=val

2. Confidence Threshold and Confusion Matrix: Yes, the confidence (conf) threshold matters during validation because it filters out detections below the specified confidence level, affecting the final metrics like precision, recall, and mAP. Lowering the threshold might increase the number of detected objects but can also introduce more false positives. Raising it can reduce false positives but might also miss some true positives.

The confusion matrix is constructed by comparing the predicted class labels against the true class labels for all the validations or test examples. It helps in understanding the performance of the model for each class, indicating how often it confuses two classes. For building it after validation, each prediction is matched with ground truth based on IoU criteria. True Positives (TP), False Positives (FP), and False Negatives (FN) are then counted for each class which goes into making the confusion matrix.

Hope this clarifies your queries! Let me know if there's anything else you want to know.

[TersteegC]

Hi, I want to use the validate function to finetune my trained model and see how it performs against real life pictures. There is however a difference in parameters available between "predict" mode and val "mode". For instance, I would like to use specific settings for Augment and retina_mask as part of the validation. Is this possible? And out of curiosity, why are the default settings for Val different from Predict?

[glenn-jocher]

Hey there! ✨ Thanks for reaching out with your query.

To tailor your validation settings for Augment and retina_mask, you can directly modify these parameters in your dataset's YAML file or provide them as additional arguments in your custom script. These options, however, are not directly available through the CLI for the val mode. Changes to parameters like these are usually done within the training script or the dataset configuration to affect the validation phase indirectly.

Regarding defaults differences between val and predict modes – these modes serve different purposes and thus have distinct default settings optimized for their specific use-cases. Val mode focuses on model evaluation against a known dataset with ground truth for performance metrics computation, hence might prefer higher accuracy over inference speed. In contrast, predict mode is tailored towards real-time inference, possibly preferring speed and real-world applicability.

Here's a quick Python snippet to kickstart customization for validation (ensure your dataset YAML reflects your augmentation desires):

from ultralytics import YOLO

# Load your custom model
model = YOLO('path/to/your/model.pt')

# Validate the model with custom settings
validation_results = model.val(data='your_dataset.yaml', imgsz=640, batch=16, conf=0.25, iou=0.6)

Remember, Augment and custom settings may require adjustments within your training/validation Python script or dataset YAML rather than through CLI commands. Hope this helps clear things up! 🚀

[SimCromarty]

Hi, I am completing a vision project for an automated industrial packing cell. Currently I have trained 3 custom models for the different tasks. I want to test the performance of each with different scenarios like bright, dim, natural, artificial lighting and varied backgrounds or similar objects. What would be the best method to do so? Would it be best to create these standard test datasets, link with yaml then use the val function? Or use predict and count correct / incorrect detections and work out metrics from there? Once I have done this I also want to investigate performance with different model architecture sizes and training times so has to be easily repeatable. How is best to implement basically, thanks in advance!

[pderrenger]

@SimCromarty hello! 🌟 For evaluating your models under different lighting conditions and backgrounds, using the val function with standard test datasets linked via YAML is indeed a good approach. This allows you to leverage YOLOv8's built-in metrics like mAP50, mAP75, and others for a comprehensive evaluation. Here's a quick example on how you might set up your val step:

yolo val model=path/to/your/model.pt data=your_test_dataset.yaml

For repeated experiments across various model architectures and training durations, setting up separate YAML files for each scenario can streamline the process. This ensures that you can easily switch between datasets and model configurations without manually counting detections.

Additionally, remember that YOLOv8 models remember their training configurations, making it straightforward to repeat validation steps with different settings. Modifying the imgsz, batch, and other parameters in your dataset's YAML file allows for flexible adaptation to varied scenarios.

Hope this helps! If you have any more questions or need further assistance, feel free to ask. Good luck with your project! 🚀

[slhermite]

Hi, an iou (1) threshold is used during detection to prevent duplicate bounding boxes.
But another threshold ior (2) is used during validation to determine when a predicted bounding box corresponds to the ground truth, this allows us to "decide" whether a prediction is true or false positive.
If I've understood correctly, in metrics = model.val(ior=0.6) you fill in the threshold (1) and metrics contains precision and recall. My question is as follows: With what threshold ior (2) are precision and recall calculated? Can you choose this threshold (2) in model.val ?

[glenn-jocher]

Hey there! I see where the confusion is coming from, and I appreciate your patience and the detailed examination. 🙏

In Ultralytics YOLOv8, the iou parameter provided in model.val() function indeed controls the IoU threshold for NMS (iou1) during the validation phase, as you've correctly pointed out from the val.py script. This acts to filter detections before calculating validation metrics such as mAP.

The mAP metrics like mAP50 are indeed computed with a fixed IoU threshold of 50% for matches between the predicted bounding boxes and the ground truth (iou2), which is a standard practice for these evaluations. This threshold is internally set and not exposed for adjustment through the model.val() method.

So, to clarify, while the iou parameter in model.val() adjusts the NMS threshold (iou1), the IoU threshold for calculating precision and recall metrics (iou2) is traditionally set at 0.5 (or 50%) for mAP50 calculations and isn't directly adjustable through the model.val() arguments.

I hope this clears up the point of confusion! Thanks for diving deep into this, and if you have more questions or need further clarifications, feel free to ask. Happy to help!

[slhermite]

Hi, thank you for your reply.

On the other hand, you repeated my assertion without unfortunately answering the only question I asked.

Nevertheless, I appreciate the effort you've put in.

[glenn-jocher]

Hi there! 👋 No worries, I'm here to help clarify things.

The iou parameter you pass to model.val(iou=0.6) actually sets the IoU threshold for Non-Maximum Suppression (NMS) during the validation phase (what you referred to as iou1). This impacts which detections are kept based on their confidence and overlap. It's not directly the IoU threshold used for calculating mAP50 or other precision/recall metrics.

For calculating mAP50 or similar validation metrics, the threshold (iou2 for matching detections to ground truths) is typically fixed at 0.5 (50%) standard across most object detection benchmarks. This isn't adjustable in the model.val() function and is the standard practice for consistency in evaluation metrics.

In summary, through model.val(iou=0.6), you're adjusting how detections are filtered before they're evaluated for metrics like mAP50, which itself uses a standard IoU threshold (0.5) to determine true positives vs. false positives.

Hope that clears it up! Let me know if there's anything else you're curious about. 😊

[siGrau]

Hi there,

I have a similar follow-up question regarding the iou thresholds: As you already said, the iou2 threshold for validation metrics like mAP50 or precision and recall is fixed at 0.5. Why is the iou threshold for the confusion matrix set to 0.45?

[glenn-jocher]

Hi there!

Great question! The IoU threshold for the confusion matrix is set slightly lower at 0.45 to provide a more lenient criterion for considering detections as matches to the ground truth. This can help in analyzing scenarios where the bounding boxes are not perfectly aligned but are close enough to be considered correct detections, thus giving a broader perspective on the model's performance.

This setting helps in understanding the model's behavior under slightly less strict conditions, which can be particularly useful in practical applications where perfect alignment of predicted and actual bounding boxes is rare.

If you have more questions or need further clarification, feel free to ask. Happy to help! 😊

[HumairaBano]

hi,
i am validating yolov8 on pidray dataset using the following code but got this error. I am unable to resolve it as I could not find any difference in class labels. there are 12 classes.
from ultralytics import YOLO
path = "/content/drive/MyDrive/datasets/pidray/yolov8-pidray/weights/yolov8_pidray.pt"
model = YOLO(path)
metrics=model.val(data="/content/drive/MyDrive/datasets/pidray/yolov8-pidray/data/pidray.yaml")

the error is
Ultralytics YOLOv8.1.47 🚀 Python-3.10.12 torch-2.2.1+cu121 CPU (Intel Xeon 2.20GHz)
YOLOv8s summary (fused): 168 layers, 11130228 parameters, 0 gradients, 28.5 GFLOPs
val: Scanning /content/drive/MyDrive/datasets/pidray/labels/hidden.cache... 1241 images, 3764 backgrounds, 0 corrupt: 100%|██████████| 5005/5005 [00:00<?, ?it/s]
Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 313/313 [1:09:00<00:00, 13.23s/it]

KeyError Traceback (most recent call last)
in <cell line: 1>()
----> 1 history=model.val(data="/content/drive/MyDrive/datasets/pidray/yolov8-pidray/data/pidray.yaml")

7 frames
/usr/local/lib/python3.10/dist-packages/ultralytics/utils/metrics.py in plot_mc_curve(px, py, save_dir, names, xlabel, ylabel, on_plot)
476 if 0 < len(names) < 21: # display per-class legend if < 21 classes
477 for i, y in enumerate(py):
--> 478 ax.plot(px, y, linewidth=1, label=f"{names[i]}") # plot(confidence, metric)
479 else:
480 ax.plot(px, py.T, linewidth=1, color="grey") # plot(confidence, metric)

KeyError: 11

[glenn-jocher]

Hey there! 🚀 It looks like you're encountering a KeyError during validation, which often indicates a mismatch between the class labels defined in your dataset and those the model is expecting. Since you mentioned having 12 classes, ensure your pidray.yaml file accurately reflects all your class labels and that they are zero-indexed (i.e., the first class is 0, the second is 1, etc.).

Here's a quick check you can do:

• Open your pidray.yaml and ensure the nc: (number of classes) is set to 12 and the names: list includes all your classes indexed correctly.

If your pidray.yaml looks correct, the issue might be with the model expecting a different number of classes. This can happen if the model was trained on a dataset with a different class structure. Make sure you're using the correct pre-trained model or that your custom model was trained with the same class setup.

As a simple code snippet for validating, your setup seems fine. Just ensure the paths and class definitions align perfectly.

Hope this helps! If you're still stuck, could you share the class definitions from your pidray.yaml? That might provide more clues. 🕵️‍♂️

[HumairaBano]

model is trained on same pidray dataset. It is github project https://github.com/haksorus/yolov8-pidray.
yes sure the .yaml file is here

path: /content/drive/MyDrive/datasets/pidray
train: images/train
val: images/hidden
test: images/hidden

names:
0: baton
1: pliers
2: hammer
3: powerbank
4: scissors
5: wrench
6: gun
7: bullet
8: sprayer
9: handcuffs
10: knife
11: lighter

[glenn-jocher]

Hey there! 👋 It looks like your pidray.yaml is set up correctly with 12 classes, which matches the class indices you've provided. The KeyError: 11 suggests an issue accessing the last class, "lighter". This could be due to how the classes are interpreted during the validation process.

Since the model and the dataset are correctly aligned, ensure that:

• Your dataset annotations are consistent with the class labels in pidray.yaml.
• There's no missing/corrupt label file in the dataset that might cause this error during validation.

Remember, YOLOv8 seamlessly integrates the configuration from your training for validation, so mismatches are usually about dataset labeling or class indices. If the issue persists, revisiting the dataset to confirm all annotations are correct and complete for each class might help.

For clarity, validation is as simple as:

from ultralytics import YOLO

# Assuming your model and dataset are correctly set
model = YOLO('/content/drive/MyDrive/datasets/pidray/yolov8-pidray/weights/yolov8_pidray.pt')
metrics = model.val(data="/content/drive/MyDrive/datasets/pidray/yolov8-pidray/data/pidray.yaml")

# This should run without the KeyError if classes match dataset annotations

I hope this points you in the right direction! Let us know if there's anything else we can help with. 🚀

[YunYeTang]

hi,
when I use yolov8-obb to validate DOTAv1, I found that confusion matrix is empty. Does it mean that confusion matrix is not supported in OBB validation?

[glenn-jocher]

Hi there! 👋

At the moment, confusion matrices might not be directly supported for OBB validation in the way they are for standard bounding box detections. This could explain why you're encountering an empty confusion matrix during your DOTAv1 validation using yolov8-obb.

However, evaluating the model's performance can still be done using other metrics provided during validation, such as mAP. If you need more detailed insights or have a specific evaluation need, you might consider manually computing confusion matrices based on the raw prediction and ground truth data obtained during validation.

Here's a quick snippet on how you could potentially start to manually calculate a confusion matrix (for a hypothetical scenario):

from sklearn.metrics import confusion_matrix
y_true = [...]  # True labels
y_pred = [...]  # Predicted labels by your model
conf_mat = confusion_matrix(y_true, y_pred)
print(conf_mat)

Remember, this requires you to post-process the model outputs and ground truth labels to fit into a format suitable for confusion matrix computation.

We're always looking to improve, so feedback like yours is invaluable. Feel free to share any more thoughts or questions you might have! 🚀

[YunYeTang]

thank u very much

[glenn-jocher]

Hey there! 🚀

Absolutely, thank you for reaching out!

Regarding your observation about the confusion matrix being empty during DOTAv1 validation with yolov8-obb, it's essential to note that certain specialized metrics or visualizations might need specific support or additional setup procedures when dealing with oriented bounding boxes (OBB). At the moment, direct confusion matrix visualization support may not be explicitly designed for OBB validation in the documentation.

But, worry not! For a comprehensive evaluation, you can still rely on other metrics that are provided during the validation process, such as mAP. If a confusion matrix is critical for your analysis, consider manually extracting prediction and ground truth data to compute it as needed.

If there's anything more specific you'd like assistance with or any further questions, feel free to ask. Cheers! 🌟

[atharva-hude-asu]

I use my own post-processing logic on the txts that i get from running inference the validation set. So now I have the post processed txts for the validation set and the original validation txts that i used for training. How to compute the val performance results using the txt files ?

[pderrenger]

@atharva-hude-asu hey there! 👋 It sounds like you've already taken some great steps with your post-processing and have your processed txt files ready to go. To calculate the validation performance using these txt files, you typically need to compare your post-processed predictions against the ground truth annotations in your validation set.

One straightforward approach is to use the evaluation script provided in many object detection repositories, which might require slight modifications to match your data format. However, since you're interested in Ultralytics YOLO, I'd recommend feeding your post-processed results directly back into the YOLO val command and letting it handle the performance calculations.

Here's a basic idea of how to proceed:

1. Ensure that your post-processed txt files follow the same format as the YOLO prediction files, where each line corresponds to a detection formatted as [class_id] [confidence] [x_center] [y_center] [width] [height], with values normalized to the image width and height.

2. Use these txt files in a manner consistent with YOLO expectations for validation datasets. You might need a custom script to align your post-processed results with the expected input for validation.

Unfortunately, if the val mode directly doesn't support external result inputs, you might consider calculating key metrics like mAP, precision, and recall using external scripts or tools designed for evaluating object detection models (many are available online).

Lastly, if you have access to a custom setup that calculates these metrics, simply adjust it to work with your format. This discussion might provide some insights or direction on handling custom evaluation scripts.

If there's anything more specific you'd need help with, please feel free to ask! Happy coding! 🚀

[Huo-Yukang]

[pderrenger]

@Huo-Yukang hello! 😊 To obtain APS (AP small), APM (AP medium), and APL (AP large) metrics through your trained model weight file, you'll simply need to run a validation using the model. Here's how you can do it:

If you're using Python:

from ultralytics import YOLO

# Load your trained model
model = YOLO('path/to/your/trained_weights.pt')

# Validate the model
metrics = model.val()  # Ensure you're validating on a dataset that has been seen during training for accurate metrics.

# Access APS, APM, and APL
print(f"APS: {metrics.box.aps}, APM: {metrics.box.apm}, APL: {metrics.box.apl}")

Or, if you prefer using the CLI:

yolo detect val model=path/to/your/trained_weights.pt

(Note: The CLI will display a comprehensive metrics summary including APS, APM, and APL in the output)

This process evaluates your trained model's performance across different object sizes — small, medium, and large — giving you insightful metrics on how well your model detects objects of varying scales. 📊

If you have any more questions, feel free to ask. Happy validating!

[pderrenger]

Hey there! 😊 It seems like you encountered an AttributeError because aps, apm, and apl metrics might not be directly accessible attributes of the Metrics object after validation. Instead, you can extract similar size-specific AP metrics using the COCO API if you're working with COCO datasets or any custom evaluation script tailored for your specific dataset format.

For a closer look at how metrics like mAP are calculated and possibly dive into size-specific AP calculations, using COCO tools directly might offer more insights.

For COCO datasets, after saving the results in the COCO format (save_json=True during validation), you could use something like this with pycocotools:

from pycocotools.coco import COCO
from pycocotools.cocoeval import COCOeval

# Assuming val.json is your validation set annotations in COCO format
# and results.json is what you saved using save_json=True during validation
cocoGt=COCO('val.json')  
cocoDt=cocoGt.loadRes('results.json')

cocoEval = COCOeval(cocoGt,cocoDt,'bbox')
cocoEval.evaluate()
cocoEval.accumulate()
cocoEval.summarize()

# Access size-specific AP metrics
print(f"APS: {cocoEval.stats[4]}, APM: {cocoEval.stats[5]}, APL: {cocoEval.stats[6]}")

Please note, this example uses COCO evaluation tools directly for size-specific metrics, which might differ slightly from how Ultralytics processes these metrics internally. For the differences you're observing in mAP calculations, ensure the evaluation settings (like IoU thresholds and object sizes) match between your Ultralytics validation process and the pycocotool process. Adjustments to parameters may bring the evaluations more in line with each other. 👍

[dntcoder]

When I use the command line： yolo task=detect mode=val model=best.pt data=data.yaml device=0 to verify, the results I get are very different from when I use default.yaml to verify. Why is this?

[pderrenger]

@dntcoder hello! When you see different results between using data.yaml and default.yaml for model validation, it's likely due to variations in the configuration settings defined in these files. Each .yaml file contains specific parameters that can impact model performance, such as dataset paths, class names, number of classes, input image size (imgsz), batch size, etc.

To ensure consistency in your validation results, carefully check and compare the settings within data.yaml and default.yaml. Ensure that important parameters like imgsz, batch, conf, and others match between the two. Any discrepancies in these settings could lead to the observed differences in validation outcomes.

Here's a quick example of how to validate using Python with specific arguments overriding the .yaml configurations:

from ultralytics import YOLO

# Load model and override settings for validation
model = YOLO('path/to/best.pt')
metrics = model.val(data='data.yaml', imgsz=640, batch=16, conf=0.25, iou=0.6, device='0')

# View validation metrics
print(metrics.box.map)

Please let us know if adjusting these settings helps or if you need further assistance! Happy validating! 🚀

[GuangPanSir]

When we use model.val（） for validation，It is found that the last output confusion_matrix_normalized.png confusion matrix has no number, that is, the relevant accuracy is printed. What is going on?

[pderrenger]

Hey there! 👋 It seems like the issue you're encountering with the missing numbers in the confusion_matrix_normalized.png might be related to the visualization settings or a minor glitch. Typically, the confusion matrix should display the accuracy rates for each class. Let's try a couple of things to troubleshoot:

1. Ensure that your dataset has a sufficient number of examples for each class. A very small dataset might result in visualization issues.
2. You might want to check if the version of Ultralytics YOLO you're using is up to date. If not, updating to the latest version could help resolve the issue.

Regarding code, the process you're following with model.val() looks correct. If continuing to face this issue, consider manually adjusting the plot settings or reporting this as a potential bug on the GitHub repo for further assistance.

If there's anything specific you're looking for in the numbers (like precision, recall, etc.), the metrics returned by model.val() allow access to detailed class-wise and overall statistics, which might provide the insights you need without relying solely on the confusion matrix visualization.

Hope this helps, and let us know if you have any more questions or if there's anything else we can do for you! 🚀

[elifmeliskulbay]

Hi!

I am working with a dataset that includes only the 'car' class, where their corresponding label is '2' in my dataset. When I attempt to generate confusion matrix and other plots, I encounter a problem: the base model "yolov8n.pt" includes all the other classes while calculating the metrics. However, I want to use model.val( ) specifically for objects detected as a 'car'.

Yaml file looks like this:

path: # dataset root directory

train:  # training images folder
val:    # validation images folder
test: # test images folder

nc: 1
names: ['car']  

[pderrenger]

Hi there!

It looks like you want your model validation to focus specifically on the 'car' class in your dataset. You've already configured your yaml file correctly with nc: 1 and names: ['car']. For the validation to consider only the 'car' class, ensure that your dataset labels are correctly mapped to this configuration.

When using the model.val() function, it will automatically use the classes defined in your dataset config, so it should only validate the 'car' class if that's the only class provided. Here's a quick example to ensure your setup is correct:

from ultralytics import YOLO

# Load your model
model = YOLO('path/to/your/yolov8n.pt')

# Perform validation
metrics = model.val(data='your_yaml_file.yaml')
print(metrics.box.map)  # mAP50-95 for 'car'

This should give you the metrics for the 'car' detections only. If you encounter any specific errors during this process, feel free to share the error messages, and I'll be glad to help further! 🚗

[elifmeliskulbay]

Thank you for your reply.

When I check the output in the 'run' folder, the metrics have been evaluated only for 'car' class, except for confusion matrix. The confusion matrix includes all classes. My images have, for example, person, truck, cars etc. However, my labels only include car class.

Other plots in the 'run' folder only include car, however the confusion matrix has been evaluated for all classes the base pre-trained model found in the images. Since I only have car annotations in the ground truth, I have many false positives for other classes. I am using the pre-trained Yolo model and cannot make the validation only for car class and have the confusion matrix accordingly.

Please find my config file and python script for validation below:

from ultralytics import YOLO
def val_yolo():

    model = YOLO('C:/Users/mk/PycharmProjects/pythonProject2/yolov8n.pt')
    val_results = model.val(data='C:/Users/mk/PycharmProjects/pythonProject2/dataset_config_base.yaml', batch=16)
    print(val_results.box.map)  # mAP50-95 for 'car'

if __name__ == '__main__':
    val_yolo()

and

path: C:/Users/mk/PycharmProjects/pythonProject2/Experiments/val2  # dataset root directory

train: C:/Users/mk/PycharmProjects/pythonProject2/Experiments/val2/images  # training images folder
val: C:/Users/mk/PycharmProjects/pythonProject2/Experiments/val2/images      # validation images folder
test: C:/Users/mk/PycharmProjects/pythonProject2/YOLO/dataset_train_Yolo/test/images    # test images folder

nc: 1  
names: ['car'] 

[pderrenger]

Hello!

For the concern you've described about the confusion matrix including all classes, you might need to adjust the post-processing or prediction filtering specifically for your use case where only the 'car' class is relevant. In the standard YOLOv8n model validation, it indeed considers all classes detected by the model.

A possible workaround is to modify the output processing part of your validation script to solely focus on the 'car' class. Since your model is detecting other classes which are being labeled as false positives, you could modify the evaluation scripts to ignore these classes when generating the confusion matrix.

Here's a pseudo approach to how you might modify your script:

1. After running the predictions, filter out the detections to include only those for 'car' class.
2. Use these filtered predictions to compute your confusion matrix.

Unfortunately, directly altering the internal computations for model validation to exclude specific detections from the confusion matrix isn't straightforward without adjusting the source code or your post-processing steps.

If you need further guidance on how to specifically adjust the confusion matrix calculation, feel free to reach out! 😊

[jiayuanz3]

Hi there!

I'd like to get all visualisation results from all validation images. Then I try to run the following, but it only gives the first three batches, what should I do?

model = YOLO('../best.pt')
validation_results = model.val(data='../data.yaml', imgsz=640, batch=16, plots=True)

Thanks in advance!

[glenn-jocher]

Hello!

It seems like your model is set to generate plots only for a limited number of batches due to the plots=True setting. By default, plotting is typically configured to limit the output to prevent excessive file output, particularly for larger datasets.

If you want to visualize results for all validation images, you might consider either:

1. Manually tweaking the internal settings related to plot generation in the YOLOv8 source code, specifically where the plotting function is called during validation. This might involve removing or adjusting any conditions that limit the number of batches plotted.

2. Running a loop that validates smaller subsets of your data at a time and captures the output plots for each subset. This way, you ensure all data is covered without overwhelming your system at once.

Here’s a quick example to modify your existing code for the second suggestion:

from ultralytics import YOLO

model = YOLO('../best.pt')
total_images = 1000  # total number of images in your validation set
batch_size = 16

for i in range(0, total_images, batch_size):
    validation_results = model.val(data='../data.yaml', imgsz=640, batch=batch_size, plots=True)

This example assumes you know the total number of images in your validation dataset. Adjust total_images and batch_size as needed for your specific dataset.

Hope this helps! 😊

[glenn-jocher]

Hello!

It seems like your model is set to generate plots only for a limited number of batches by default. If you want to get visualization results for all validation images across all batches, you might need to modify the number of batches processed for visualization or ensure that all validation batches are being processed and plotted.

Here's a simple modification you might try to ensure plots are created for every batch:

from ultralytics import YOLO

# Load your model
model = YOLO('../best.pt')

# Validate the model with settings to generate plots for every batch
results = model.val(data='../data.yaml', imgsz=640, batch=16, plots=True)

Make sure that the plots=True flag is correctly being handled in your validation loop, and your dataset doesn't have restrictions on the number of batches it can process visually due to memory constraints or other settings.

If problems persist, it may be helpful to check the logs for any errors that might indicate why plots are not being generated beyond the initial batches. You could also manually iterate through your data in smaller chunks to control the flow and ensure all are visualized.

Hope this helps! Let me know if there's anything else you need! 😊

[jiayuanz3]

Thanks for your quick response! I tried this one, but it creates a series of val folders. And inside each folder, it predicts three batches, which are same across all val folders.

[glenn-jocher]

Hello!

It sounds like you're encountering an issue with plotting during validation where the same batches are being plotted across multiple validation folders. This might be related to how the dataset is being loaded or iterated during the validation phase.

For a quick fix in this scenario, you can ensure that your data loader is shuffling the batches adequately, or manually implement a loop to handle data splitting and visualization more directly. Here’s a basic modification to your code that might help:

from ultralytics import YOLO

model = YOLO('../best.pt')

# Manually iterate over dataset
for batch in dataloader:  # Ensure dataloader is defined to shuffle data if not done
    validation_results = model(batch, imgsz=640, batch_size=16, plots=True)
    # handle/save results here

Make sure your dataloader is set to shuffle the evaluation data properly. If you continue to see duplications or have further issues, checking the dataset and the data loader configuration for potential repetitive data handling might be needed.

Hope this helps resolve the issue! 😊

[echo385]

hi， is it possible to integrate sahi during the verification process? To draw the corresponding evaluation curve on the high-resolution image

[glenn-jocher]

Hello! Integrating Sahi for drawing evaluation curves on high-resolution images during the validation process isn't directly supported by Ultralytics YOLO's Val mode. However, after you've validated your model and obtained the metrics, you can use those metrics to manually plot curves using libraries like Matplotlib in Python.

Here's a basic example of how you might plot the precision-recall curve using Matplotlib after validation:

from ultralytics import YOLO
import matplotlib.pyplot as plt

# Load your model and validate
model = YOLO('path/to/your/model.pt')
metrics = model.val()

# Precisions and recalls from validation metrics
precisions = [p for p in metrics.box.precisions]
recalls = [r for r in metrics.box.recalls]

# Plotting the precision-recall curve
plt.figure(figsize=(10, 6))
plt.plot(recalls, precisions, marker='o')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision-Recall Curve')
plt.grid()
plt.show()

This code assumes you have access to precision and recall data from your validation metrics, which may need to be adjusted based on the actual output format of your validation results. Let me know if you need more details on this! 😊

[1JayPeng]

Why does this happen when I run val?Is my model based on yolov9e.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\multiprocessing\spawn.py", line 116, in spawn_main     
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\multiprocessing\spawn.py", line 125, in _main
    prepare(preparation_data)
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\multiprocessing\spawn.py", line 236, in prepare        
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\runpy.py", line 288, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "c:\Users\Administrator\Desktop\yolov92\val.py", line 8, in <module>
    metrics = model.val(
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\site-packages\ultralytics\engine\model.py", line 528, in val
    validator(model=self.model)
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\site-packages\ultralytics\engine\validator.py", line 154, in __call__
    self.dataloader = self.dataloader or self.get_dataloader(self.data.get(self.args.split), self.args.batch)   
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\site-packages\ultralytics\models\yolo\detect\val.py", line 226, in get_dataloader
    return build_dataloader(dataset, batch_size, self.args.workers, shuffle=False, rank=-1)  # return dataloader
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\site-packages\ultralytics\data\build.py", line 135, in build_dataloader
    return InfiniteDataLoader(
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\site-packages\ultralytics\data\build.py", line 39, in __init__
    self.iterator = super().__iter__()
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\site-packages\torch\utils\data\dataloader.py", line 439, in __iter__
    return self._get_iterator()
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\site-packages\torch\utils\data\dataloader.py", line 387, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\site-packages\torch\utils\data\dataloader.py", line 1040, in __init__
    w.start()
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\multiprocessing\process.py", line 121, in start        
    self._popen = self._Popen(self)
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\multiprocessing\context.py", line 224, in _Popen       
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\multiprocessing\context.py", line 327, in _Popen       
    return Popen(process_obj)
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\Administrator\miniconda3\envs\yolo8\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

from ultralytics import YOLO

# Load a model
# model = YOLO('yolov8n.pt')  # load an official model
model = YOLO(r"my\12\weights\best.pt")  # load a custom model

# Validate the model
metrics = model.val(
    data=r"my.yaml",
    device=[0, 1, 2, 3, 4, 5],
    split="test",
    batch=16 * 6,
)

cuda 12.4 windows11 torch-2.3.0

# Name                    Version                   Build  Channel
aiosignal                 1.3.1                    pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
blas                      1.0                         mkl
brotli-python             1.0.9            py39hd77b12b_7
ca-certificates           2024.3.11            haa95532_0
certifi                   2024.2.2         py39haa95532_0
charset-normalizer        2.0.4              pyhd3eb1b0_0
click                     8.1.7                    pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
comet-ml                  3.40.0                   pypi_0    pypi
configobj                 5.0.8                    pypi_0    pypi
contourpy                 1.2.1                    pypi_0    pypi
cuda-cccl                 12.4.127                      0    nvidia
cuda-cudart               12.1.105                      0    nvidia
cuda-cudart-dev           12.1.105                      0    nvidia
cuda-cupti                12.1.105                      0    nvidia
cuda-libraries            12.1.0                        0    nvidia
cuda-libraries-dev        12.1.0                        0    nvidia
cuda-nvrtc                12.1.105                      0    nvidia
cuda-nvrtc-dev            12.1.105                      0    nvidia
cuda-nvtx                 12.1.105                      0    nvidia
cuda-opencl               12.4.127                      0    nvidia
cuda-opencl-dev           12.4.127                      0    nvidia
cuda-profiler-api         12.4.127                      0    nvidia
cuda-runtime              12.1.0                        0    nvidia
cycler                    0.12.1                   pypi_0    pypi
dill                      0.3.8                    pypi_0    pypi
dulwich                   0.22.1                   pypi_0    pypi
everett                   3.1.0                    pypi_0    pypi
filelock                  3.13.1           py39haa95532_0
fonttools                 4.51.0                   pypi_0    pypi
freetype                  2.12.1               ha860e81_0
frozenlist                1.4.1                    pypi_0    pypi
fsspec                    2024.3.1                 pypi_0    pypi
gmpy2                     2.1.2            py39h7f96b67_0
idna                      3.4              py39haa95532_0
importlib-resources       6.4.0                    pypi_0    pypi
intel-openmp              2021.4.0                 pypi_0    pypi
jinja2                    3.1.3            py39haa95532_0
jpeg                      9e                   h2bbff1b_1
jsonschema                4.21.1                   pypi_0    pypi
jsonschema-specifications 2023.12.1                pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
lerc                      3.0                  hd77b12b_0
libcublas                 12.1.0.26                     0    nvidia
libcublas-dev             12.1.0.26                     0    nvidia
libcufft                  11.0.2.4                      0    nvidia
libcufft-dev              11.0.2.4                      0    nvidia
libcurand                 10.3.5.147                    0    nvidia
libcurand-dev             10.3.5.147                    0    nvidia
libcusolver               11.4.4.55                     0    nvidia
libcusolver-dev           11.4.4.55                     0    nvidia
libcusparse               12.0.2.55                     0    nvidia
libcusparse-dev           12.0.2.55                     0    nvidia
libdeflate                1.17                 h2bbff1b_1
libjpeg-turbo             2.0.0                h196d8e1_0
libnpp                    12.0.2.50                     0    nvidia
libnpp-dev                12.0.2.50                     0    nvidia
libnvjitlink              12.1.105                      0    nvidia
libnvjitlink-dev          12.1.105                      0    nvidia
libnvjpeg                 12.1.1.14                     0    nvidia
libnvjpeg-dev             12.1.1.14                     0    nvidia
libpng                    1.6.39               h8cc25b3_0
libtiff                   4.5.1                hd77b12b_0
libuv                     1.44.2               h2bbff1b_0
libwebp-base              1.3.2                h2bbff1b_0
lz4-c                     1.9.4                h2bbff1b_0
markdown-it-py            3.0.0                    pypi_0    pypi
markupsafe                2.1.3            py39h2bbff1b_0
matplotlib                3.8.4                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mkl                       2021.4.0                 pypi_0    pypi
mkl-service               2.4.0            py39h2bbff1b_0
mkl_fft                   1.3.1            py39h277e83a_0
mkl_random                1.2.2            py39hf11a4ad_0
mpc                       1.1.0                h7edee0f_1
mpfr                      4.0.2                h62dcd97_1
mpir                      3.0.0                hec2e145_1
mpmath                    1.3.0            py39haa95532_0
msgpack                   1.0.8                    pypi_0    pypi
networkx                  3.1              py39haa95532_0
numpy                     1.24.3           py39hf95b240_0
numpy-base                1.24.3           py39h005ec55_0
opencv-python             4.9.0.80                 pypi_0    pypi
openjpeg                  2.4.0                h4fc8c34_0
openssl                   3.0.13               h2bbff1b_0
packaging                 24.0                     pypi_0    pypi
pandas                    2.2.2                    pypi_0    pypi
pillow                    10.2.0           py39h2bbff1b_0
pip                       23.3.1           py39haa95532_0
protobuf                  5.26.1                   pypi_0    pypi
psutil                    5.9.8                    pypi_0    pypi
py-cpuinfo                9.0.0                    pypi_0    pypi
pyarrow                   16.0.0                   pypi_0    pypi
pygments                  2.17.2                   pypi_0    pypi
pyjion                    0.15.0                   pypi_0    pypi
pyparsing                 3.1.2                    pypi_0    pypi
pysocks                   1.7.1            py39haa95532_0
python                    3.9.19               h1aa4202_0
python-box                6.1.0                    pypi_0    pypi
python-dateutil           2.9.0.post0              pypi_0    pypi
pytorch                   2.3.0           py3.9_cuda12.1_cudnn8_0    pytorch
pytorch-cuda              12.1                 hde6ce7c_5    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pytz                      2024.1                   pypi_0    pypi
pyyaml                    6.0.1            py39h2bbff1b_0
ray                       2.20.0                   pypi_0    pypi
referencing               0.35.0                   pypi_0    pypi
requests                  2.31.0           py39haa95532_1
requests-toolbelt         1.0.0                    pypi_0    pypi
rich                      13.7.1                   pypi_0    pypi
rpds-py                   0.18.0                   pypi_0    pypi
scipy                     1.13.0                   pypi_0    pypi
seaborn                   0.13.2                   pypi_0    pypi
semantic-version          2.10.0                   pypi_0    pypi
sentry-sdk                1.45.0                   pypi_0    pypi
setuptools                68.2.2           py39haa95532_0
simplejson                3.19.2                   pypi_0    pypi
six                       1.16.0             pyhd3eb1b0_1
sqlite                    3.41.2               h2bbff1b_0
sympy                     1.12             py39haa95532_0
taichi                    1.7.1                    pypi_0    pypi
tbb                       2021.12.0                pypi_0    pypi
tensorboardx              2.6.2.2                  pypi_0    pypi
thop                      0.1.1-2209072238          pypi_0    pypi
torchaudio                2.3.0                    pypi_0    pypi
torchvision               0.18.0                   pypi_0    pypi
tqdm                      4.66.2                   pypi_0    pypi
typing_extensions         4.9.0            py39haa95532_1
tzdata                    2024.1                   pypi_0    pypi
ultralytics               8.2.14                   pypi_0    pypi
urllib3                   2.1.0            py39haa95532_1
vc                        14.2                 h21ff451_1
vs2015_runtime            14.27.29016          h5e58377_2
websocket-client          1.3.3                    pypi_0    pypi
wheel                     0.41.2           py39haa95532_0
win_inet_pton             1.1.0            py39haa95532_0
wrapt                     1.16.0                   pypi_0    pypi
wurlitzer                 3.0.3                    pypi_0    pypi
xz                        5.4.6                h8cc25b3_0
yaml                      0.2.5                he774522_0
zipp                      3.18.1                   pypi_0    pypi
zlib                      1.2.13               h8cc25b3_0
zstd                      1.5.5                hd43e919_0

[pderrenger]

Hi there! It looks like you are encountering a multiprocessing issue common in Windows when using Python's multiprocessing with YOLO. This typically happens if the script is not protected with if __name__ == '__main__': to ensure it runs in the main environment.

Here’s a quick fix:

1. Wrap your entry-point code with the following check:

if __name__ == '__main__':
    from ultralytics import YOLO

    # Load a model
    model = YOLO(r"my\12\weights\best.pt")  # load a custom model

    # Validate the model
    metrics = model.val(
        data=r"my.yaml",
        device=[0, 1, 2, 3, 4, 5],
        split="test",
        batch=16 * 6,
    )

This prevents the fork of a new process before the current process completes its setup, which is what causes the RuntimeError.

Make sure you add this snippet to your script, replacing existing top-level code. Give it a try, and let me know if it resolves your issue! 🚀

[GuenKainto]

Hi,
I have a question. Why we need to split data to validate , why we can't use all of data in val dataset

[pderrenger]

Hello! Great question. Splitting the data into training and validation sets is essential for several reasons: 🌟

1. Assess Generalization: Using a separate validation set helps evaluate how well your model generalizes to new, unseen data, which is crucial for real-world applications.

2. Avoid Overfitting: Validation helps check if the model is just memorizing the training data (overfitting) or if it's learning to generalize from patterns in the data.

3. Model Tuning: It allows you to fine-tune model parameters and select the best model version based on its performance on the validation set without using the test data.

Using all data for validation wouldn't provide these insights since there would be no data left to train the model effectively. Plus, understanding model performance on unseen data is critical before deploying it.

Here's an example snippet on how to validate your model with YOLOv8:

from ultralytics import YOLO

# Load your model
model = YOLO('path/to/your/model.pt')

# Validate the model
metrics = model.val()  # no arguments needed if using settings stored in the model
print(metrics.box.map)  # Print mAP score

Let me know if you need more details! 😊

[GuenKainto]

Sorry I seem to be mistaken, I mean why when valid model we can use all of the images in the val directory folder (in data.yaml) , but we split the number images in val folder to valid the model ?
my dataset have 3 directory for train, val and test. So with split i not need 3 folder any more ? Just one and we auto split it to 3 path?

[pderrenger]

Hello! It's good to clarify these concepts. 🌟 Even though you have data directories for training, validation, and testing, splitting the images within each directory allows for a more robust evaluation. Here’s why:

• Diversity in Data: By splitting the images in the validation folder, you ensure a diverse range of examples. This helps in evaluating how well your model performs across different types of data within the same category.

• Consistency in Training and Evaluation: Keeping separate directories for training, validation, and test retains an organizational structure that prevents data leakage and ensures that the model is tested on completely unseen data.

If you just had one folder and split data into three paths automatically for training, validation, and testing, it could simplify the process but might risk mixing data, leading to potential issues in model evaluation.

Here’s a simple example to use validation:

from ultralytics import YOLO

# Load your model
model = YOLO('path/to/your/model.pt')

# Validate the model using the validation dataset specified in your model's settings
metrics = model.val()

# Output some key validation metrics
print(f"mAP50-95: {metrics.box.map}, mAP50: {metrics.box.map50}")

This setup helps ensure your validation process is straightforward while maintaining high standards for model accuracy and reliability.

[GuenKainto]

Thankyou for this information

[pderrenger]

Hello! You're welcome! If you have any more questions or need further examples on how to implement validation or any other feature in YOLOv8, feel free to ask. Happy modeling! 😊🚀

[1411734304]

How can I output the value of F1 just like P and R, or simply print the value of F1

[pderrenger]

To output the F1 score along with Precision (P) and Recall (R) during model validation, you can calculate the F1 score using the precision and recall values. The F1 score is the harmonic mean of precision and recall and is calculated as follows:

F1 = 2 * (precision * recall) / (precision + recall)

If you're using the Ultralytics YOLO library, you'll need to manually calculate the F1 score after obtaining the precision and recall from your validation results. Here's an example of how you can do this:

from ultralytics import YOLO

# Load the model
model = YOLO('yolov8n.pt')

# Validate the model
metrics = model.val()

# Calculate F1 score
precision = metrics.box.map50  # Assuming map50 as precision
recall = metrics.box.map75  # Assuming map75 as recall
F1 = 2 * (precision * recall) / (precision + recall)

print(f'Precision: {precision}, Recall: {recall}, F1 Score: {F1}')

This code snippet will print the precision, recall, and F1 score based on your model's validation on the dataset. Adjust the precision and recall terms as necessary based on your specific metrics.

[Cheterizzin]

why in the confusion matrix there is data that is considered background in the actual condition, not in the predicted condition. Even though in actual conditions the data has been given its respective label.

[pderrenger]

@Cheterizzin it sounds like you're encountering an issue where some data that has been assigned specific labels is being categorized as background in the actual predictions. This can sometimes occur due to several reasons, such as model misclassification, inadequate training on the specific classes, or an imbalance in the dataset where background classes may be overrepresented.

To troubleshoot this, you might want to check the following:

• Ensure that the dataset is correctly labeled and balanced in terms of class representation.
• Consider fine-tuning the model further, adjusting the confidence threshold (conf) and Intersection Over Union (IoU) threshold (iou) settings during validation to better differentiate between background and foreground.
• Review the training process to confirm the model was exposed to sufficient examples of each class.

Here is a quick example on how to adjust the conf and iou settings during validation:

yolo detect val model=path/to/your/model.pt data=coco128.yaml conf=0.25 iou=0.5

Adjusting these parameters can help minimize instances where objects are falsely classified as background. If the problem persists, you might consider re-evaluating the dataset or augmenting it with more varied examples.

[BastonGazan]

Hi! I'm evaluating a model with a custom dataset. I'm adjusting all the hyper-parameters until i get the best posible recall (because I'm more interested in detect objects more than the precision).
The method that i came with for comparing the different models is to obtain the Recall-Confidence AUC because I assumed that it represented the Average Recall. Is my intuition correct or there's something that i'm missing?
You think that evaluating the model under that pespective is usefull or might be a better method?

[BastonGazan]

I just saw that metrics.box.map75 is used to return a number for "recall".
Why is considered as such when that means "mean average presition with a IoU threshold of 0.75"?

[pderrenger]

Hello! Great question! The metrics.box.map75 indeed refers to the mean average precision at an IoU threshold of 0.75, not recall. It measures how well the model predicts the correct class and location of objects with a higher precision requirement due to the stricter IoU threshold.

For focusing on recall, which emphasizes the model's ability to detect all relevant objects, you might consider looking at metrics that specifically measure recall or the True Positive Rate at various confidence thresholds. This could give you a better understanding of how well your model detects all objects, which seems to be your primary concern.

Evaluating models based on the Recall-Confidence AUC can be useful, especially if your application requires maximizing object detection without missing any. This method gives you insight into how recall changes with different confidence thresholds, helping to balance detection sensitivity with precision.

If you need further details on how to implement or interpret specific metrics, feel free to ask! 🚀

[naumovan]

Hello!
i'd like to test my custom detection model predictions with Ultralytics mAP calculation and compare results between Yolov8 ad my model. I already have images,predicted and ground truth labels, so i dont need to use model class.
Could you help me how can use mAP calculation without initializing model? Maybe there is way to initialize model but dont make prediction with that?
Thank you!

[glenn-jocher]

Hello!

To calculate mAP for your custom detection model without running predictions, you can use the validation functionality directly with your precomputed predictions and ground truth labels. Here’s a simplified approach:

1. Prepare your data: Ensure your predictions and labels are formatted correctly, typically in a JSON or similar format that includes class IDs, confidence scores, and bounding box coordinates.

2. Use the Val mode: Although you mentioned not wanting to initialize the model for predictions, you'll still need to initialize a YOLO object to access the validation functionality. However, you can bypass the prediction step by directly feeding your precomputed results.

Here's a basic example in Python:

from ultralytics import YOLO

# Initialize the model (required to access the validation tools)
model = YOLO('path/to/your/model.pt', pretrained=False)

# Assuming 'results' is a list of dictionaries formatted like your prediction output
# You can directly calculate mAP from these results
metrics = model.val(precomputed=results)
print(metrics.box.map)  # Print mAP scores

In this snippet, replace 'path/to/your/model.pt' with the path to your model and results with your precomputed predictions.

This approach allows you to utilize Ultralytics' robust mAP calculation without performing additional predictions. For more detailed guidance, please refer to the validation documentation.

Hope this helps! 😊