yolov5/tutorials/running_on_jetson_nano/

[giscus[bot]]

yolov5/tutorials/running_on_jetson_nano/

Detailed guide on deploying trained models on NVIDIA Jetson using TensorRT and DeepStream SDK. Optimize the inference performance on Jetson with Ultralytics.

https://docs.ultralytics.com/yolov5/tutorials/running_on_jetson_nano/

[sanvir19]

can i use jetson nano dev kit for .engine model

[lakshanthad]

Hi @sanvir19 ,

Currently Jetson Nano is kind of a legacy device where NVIDIA has stopped support for latest Python and Ubuntu versions. And at the time of writing this guide, everything was working, but now we have updated ultralytics packages along with the latest Python updates.

It is also not recommended to manually update Python versions on the Jetson because it might break other packages such as TensorRT in the process.

My recommended is to keep the default Python3.6.9 version as it is on the Jetson along with JetPack 4.6.4 and then do the following. Try to go with a fresh JetPack install and start over to rule out any other issues:

1. Flash JetPack with SDK components
2. sudo apt update
3. sudo apt install nvidia-jetpack -y (If SDK components are not installed)
4. sudo apt install python3-pip -y
5. pip3 install -U pip
6. pip install ultralytics
7. pip uninstall torch torchvision
8. sudo apt-get install -y libopenblas-base libopenmpi-dev
   wget https://nvidia.box.com/shared/static/fjtbno0vpo676a25cgvuqc1wty0fkkg6.whl -O torch-1.10.0-cp36-cp36m-linux_aarch64.whl
   pip3 install torch-1.10.0-cp36-cp36m-linux_aarch64.whl
9. sudo apt install -y libjpeg-dev zlib1g-dev
   git clone --branch v0.11.1 https://github.com/pytorch/vision torchvision
   cd torchvision
   python3 setup.py install --user
10. yolo checks

After that, please share the output.

[sanvir19]

This problem is related to TensorRT. If the issue with TensorRT is resolved, the YOLOv8 TensorRT model should run successfully.

nano@ubuntu:~/People_counter_v2/People_counter_v2$ python3
Python 3.8.0 (default, Dec 9 2021, 17:53:27)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import tensorrt
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'tensorrt'

nano@ubuntu:~/People_counter_v2/People_counter_v2$ python3.6
Python 3.6.9 (default, Mar 10 2023, 16:46:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import tensorrt
print(tensorrt.version)
8.2.1.9

[sanvir19]

I wanted to express my gratitude for your guidance. I followed the steps as you advised, but encountered an issue while attempting to install the Torch package. Specifically, when executing the command "pip3 install torch-1.10.0-cp36-cp36m-linux_aarch64.whl", I received the following error message:

Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
ERROR: torch-1.10.0-cp36-cp36m-linux_aarch64.whl is not a supported wheel on this platform.

Thank you once again for your support.

[sanvir19]

Thank you for your response.

You mentioned the importance of selecting the correct version of PyTorch. However, it's crucial to note that the version you suggested is compatible with Python 3.6, not Python 3.8. Unfortunately, there isn't a PyTorch version available for Python 3.8, as my JetPack version is 4.6.4. You can verify this information by checking the link provided; it confirms the absence of a PyTorch version for Python 3.8.

For my Jetson Nano Dev Kit with JetPack 4.6.4, PyTorch is compatible with Python 3.6 but not Python 3.8. Additionally, Ultralytics does not support Python 3.6; it only functions on Python 3.8. When I attempt to run the YOLOv8 .pt model on Python 3.8, it executes, albeit with reduced frames per second.

However, when I try to convert the YOLOv8 model to TensorRT, I encounter the error "No module named tensorrt." Ultralytics claims on their website, and as suggested by others, that TensorRT models should run on the Jetson Nano Dev Kit.

I kindly request a thorough understanding of my problem and a solution grounded in logic and accuracy, enabling me to effectively address the issue.

Thank you for your attention to this matter.

[pderrenger]

Hi there!👋

Absolutely, it sounds like you're on the right track but hit a snag with PyTorch and TensorRT compatibility on your Jetson Nano. Not to worry, these hiccups are pretty common.

Indeed, the Jetson environment does present some unique challenges given its architecture and supported software versions. As of JetPack 4.6.4, it's true that Python 3.6 is the go-to because of its compatibility with the CUDA and TensorRT versions pre-installed on the Nano. That said, for Python 3.8 support and PyTorch, you'd typically need a more recent JetPack version like 5.x, which supports Python 3.8 out of the box.

For the tensorrt module error, it's crucial to ensure that the TensorRT Python API is installed and matches your TensorRT version. Since you've got it working with Python 3.6 but not 3.8, it looks like the TensorRT bindings might not be available or correctly set up for Python 3.8 on your device.

Here's a quick fix you could try for installing PyTorch for Python 3.6, ensuring your environment is pointing correctly to Python 3.6 libraries:

# Assuming you're in the Python 3.6 environment
cd ~
wget https://nvidia.box.com/shared/static/fjtbno0vpo676a25cgvuqc1wty0fkkg6.whl -O torch-1.10.0-cp36-cp36m-linux_aarch64.whl
pip3 install torch-1.10.0-cp36-cp36m-linux_aarch64.whl

For Python 3.8 and TensorRT, unfortunately, if the official TensorRT Python API isn't provided for this specific combination on JetPack 4.6.4, you might be out of direct options. A possible workaround involves using Docker with a container that's set up for a newer environment, but that comes with its own set of challenges and overhead on the Jetson devices.

Finally, for converting YOLOv8 models to TensorRT to run on the Jetson, sticking to an environment confirmed to be fully supported (like Python 3.6 for now) might be the best immediate path, upgrading your JetPack version if feasible for your project.

Keep us posted on how things go! Always here to help if you hit any more bumps. 🚀

[S20-Raj]

Heyy.
I have a caliberation.txt file ready as mentioned above in the docs. But still I am getting this error:
File does not exist: /home/nvidia/DeepStream-Yolo/calib.table
Can someone help me to solve this? I

[S20-Raj]

@glenn-jocher

[glenn-jocher]

Hey there! It seems like there's a bit of a mix-up with the file path for your calib.table. Here's a quick checklist to help you troubleshoot this issue:

1. Double-check the file path: Make sure the calib.table file is indeed located at /home/nvidia/DeepStream-Yolo/. It's easy to overlook a typo or a missing directory.

2. Verify file permissions: Sometimes, the issue might be with the file permissions. You can use ls -l /home/nvidia/DeepStream-Yolo/calib.table to check the permissions and use chmod to modify them if necessary.

3. Update the config file: If your calib.table is in a different location, make sure to update the path in your DeepStream configuration file accordingly.

4. Environment Variables: If you're setting paths through environment variables, ensure they're correctly set in your session. You can check with echo $YOUR_VARIABLE_NAME.

If you've gone through these steps and the issue persists, could you provide more details about your setup? Specifically, the section of your config file where you're specifying the calib.table path would be helpful. This way, we can offer more targeted advice. 😊

[aidan-kk]

I'm following this tutorial with a jetson nano 2gb but I get an error as soon as i try to install requirements.txt... I'm pretty new to jetson/linux but I've followed every step exactly as its listed, on both Jetpack 4.6.1 and 4.6.
The error I get states there is no matching distribution found for gitpython>=3.1.30, and the installation stops there.
After editing requirements to pull 3.1.20 i get installation errors for all other requirements in the file. Does anyone know what might be the issue? Thanks!

[glenn-jocher]

@aidan-kk hey there! 👋 It seems like you're encountering an issue with installing dependencies on your Jetson Nano 2GB. The error with gitpython>=3.1.30 might be due to the pip version or Python compatibility. Here's a couple of things you could try:

1. Ensure your pip is up to date:

   pip3 install --upgrade pip

2. Try installing gitpython directly to see if it gives a more detailed error:

   pip3 install gitpython==3.1.30

3. If the above steps don't work, it might be helpful to manually install the dependencies listed in requirements.txt one by one to identify which ones are causing issues.

For the Jetson platform, some packages might need to be installed from source or might have specific versions that are compatible. If you continue to face issues, consider checking the NVIDIA forums for Jetson-specific advice on package installations.

Hope this helps! Let us know how it goes. 😊

[aidan-kk]

This didn't work, but I ended up figuring it out, and this was the solution for anyone else running into the same problem:

Run the following before you run pip3 install -r requirements.txt (it should still work if you've already done this)

(before this step ensure you are in yolo directory with: cd yolov5)

pip3 install --upgrade pip setuptools

This next part is crucial, and is the reason I couldn't install any of the dependencies. Python 3.7 is required for YOLOv5 and JetPack 4.6 comes with Python 3.6.9. However, the jetson apparently is unstable with versions beyond...? So what I found was that you should use a venv but i'm not a pro so proceed at your own risk.

sudo apt-get install python3.8-venv 
python3.8 -m venv yolov5-venv 

Activate virtual environment running python 3.8 for yolov5

source yolov5-venv/bin/activate

These 3 were needed, otherwise the device complained when installing requirements

pip3 install pybind11 
pip3 install cython
pip3 install numpy

Now you can try running

pip3 install -r requirements.txt

and the install should work :)

I quickly ran into this next issue though, and I believe this is the fix:

For Step 2 of DeepStream Config for YOLOV5: wts was replaced by export...? Can't find any info on this though.

cp DeepStream-Yolo/utils/export_yoloV5.py yolov5

Now my issue is that I get a fatal error at step 6 of the DeepStream Configuration for YOLOv5 step.
I followed the SD card flash setup with Jetpack 4.6 and now realize that JetPack doesn't come with DeepStream SDK (the Nvidia tutorial doesn't mention any clear way to do this via SD card flash), and can't find a straightforward way to do that without running ubuntu on my desktop and using the Nvidia SDK manager, which i'm just assuming is straightforward lol.
Anyway, I am determined to get yolov5 running on my Jetson 2gb. Does anyone know a workaround?

[glenn-jocher]

@aidan-kk hey there! It looks like you've been making great progress with YOLOv5 on your Jetson Nano 2GB. The gitpython and other dependency issues can indeed be a bit tricky. You've done an excellent job navigating through them and finding a solution that works. 🎉

Clearly, ensuring you're working in a Python environment that meets YOLOv5's requirements is key, as you've discovered with Python 3.8. The use of virtual environments (venv) is a smart move for keeping dependencies managed and avoiding any system-wide conflicts.

Regarding your experience with the DeepStream configuration, it's correct that certain steps, like replacing wts with an export script, have evolved to streamline the process with newer versions and tools. Your modification to include export_yoloV5.py aligns well with maintaining compatibility and following the latest practices in model deployment.

For installing DeepStream SDK without directly using NVIDIA SDK Manager, one workaround might be looking into containerized versions of DeepStream that can run on Jetson devices. NVIDIA NGC (NVIDIA GPU Cloud) often provides Docker containers pre-built with necessary software, including DeepStream, that you can deploy directly to your Jetson Nano. This method can sometimes bypass the intricacies of manual installation and get you up and running more quickly.

Thanks for sharing your journey and the solutions you've found. It's invaluable for the community! If you hit any more snags or have new insights as you proceed, feel free to reach out. Good luck with your YOLOv5 deployment on the Jetson Nano 2GB! 🚀

[sushantnair]

Hello @lakshanthad.
Thank you for this article.
I have a question in mind. I have a Jetson Nano Developer Kit. I am working on an Autonomous Wheelchair project which involves the use of Depth Estimation models from Hugging Face in order to convert input video stream to Depth Estimation output video stream to be used for navigation porposes. For that we need GPU accessibility. So, by following your tutorial would I be able to achieve what I need? Of course I need to install additional python libraries like Pillow, OpenCV and Transformers. But would that work?
Thank you.

[shepark]

[glenn-jocher]

@shepark hi there! 😊

Apologies for any confusion surrounding the calibration table steps. You're correct; the deepstream-app uses the calibration table but does not generate it. The missing step between 7 and 8 involves creating this calibration table, which is needed before running inference with INT8 precision.

Here's a quick explanation for clarification:

1. Generate Calibration Table: You'd typically generate a calibration table using a set of calibration images. This process involves running part of your dataset through the INT8 calibration process to determine the optimal scales for quantization. This step is crucial for moving to INT8 precision while maintaining accuracy.

2. Use Calibration Table: With the calibration table generated, deepstream-app can then use this table to perform INT8 inference.

If you're working with DeepStream and need to generate the calibration table, NVIDIA provides tools and documentation on how to perform this step, typically involving the TensorRT framework for optimizing the models.

Here's a generic command you might find in the TensorRT documentation to generate a calibration table:

trtexec --onnx=model.onnx --saveEngine=model_int8.engine --int8 --calib=<path_to_calibration_images>

Make sure to replace placeholders with your actual model and paths. This command is just an example and might need adjustments based on your setup.

If there's anything more specific you need help with or further clarification needed, feel free to ask! 🚀

[zosskr]

@glenn-jocher About your answer in the previous thread,
The help message from trtexec in nvcr.io/nvidia/tensorrt:23.12-py3 docker is this,
--calib=<file> Read INT8 calibration cache file
It seems like for reading calibration table, not for images to generate calibration table.
Maybe, operation depends on the target devices, like Jetson specific?

[glenn-jocher]

Hello! Regarding your observation on the trtexec --calib=<file> flag, you're correct that this flag is used to read an INT8 calibration cache file rather than directly processing images to generate a calibration table. The calibration process typically involves running inference on a subset of your dataset to determine optimal scale factors for quantization, before deploying on target devices like the Jetson.

For Jetson-specific deployments, particularly when using TensorRT for optimization, the initial calibration to generate this cache file is indeed an essential step for INT8 precision optimization. The specific approach to generating this calibration file can vary based on the toolchain and environment you're using. In scenarios where direct image processing for calibration is needed, alternative scripts or NVIDIA's recommended guidelines for your specific Jetson setup and TensorRT version should be considered.

In our Jetson Deployment Guide for YOLOv8, while we focus on optimizing and deploying models, considerations for calibration and the necessary steps for INT8 model optimization are highlighted as part of achieving the best inference performance. This includes ensuring your model and deployment strategy align with the JetPack SDK's capabilities and any device-specific requirements.

If you're working towards INT8 optimization, ensuring that the calibration step is accurately performed according to your device's specifics is crucial. Sometimes, this might require a separate calibration step before deployment. If there's any particular area of the process you're looking at or further questions on optimizing your models on Jetson devices, feel free to share! 😊

[zosskr]

Hi Glenn, Somehow, I can't see your reply in the github, so I write this. Thank you for the details. :)

…

On Thu, Mar 28, 2024 at 6:09 PM Glenn Jocher ***@***.***> wrote: Hey there! Thanks for bringing this up! 🚀 Yes, you're correct. The --calib=<file> option in trtexec is indeed for specifying an existing INT8 calibration cache file, not for providing images to generate one. The process of creating a calibration table does indeed vary based on the target device, and Jetson devices have their own specific workflow for this. For Jetson, when working with TensorRT for optimizing models for inference, the calibration process typically involves running a set of calibration data (images) through the network and collecting statistics that are then used to create the calibration table. This step is crucial for achieving optimal performance when moving to INT8 precision. The documentation you pointed out generally covers deploying YOLO models on Jetson devices, including using TensorRT and the necessary setup, but it seems we might need to clarify the calibration part a bit more for Jetson specifics. The provided guide focuses on deployment steps after you already have your model ready or converted, including TensorRT optimization and inference steps. For anyone looking to generate an INT8 calibration cache specifically for Jetson devices using TensorRT, I'd recommend checking NVIDIA's official TensorRT documentation and guides on INT8 calibration, as this process can be quite nuanced and device-specific. NVIDIA provides tools and examples that show how to perform calibration using a dataset representative of your application's use case. If you're diving into TensorRT optimization on Jetson, here’s a simple snippet on how you might proceed after setting up your environment and having your model: # pseudo code for INT8 calibration process outlineimport tensorrt as trt builder = trt.Builder(TRT_LOGGER)network = builder.create_network()parser = trt.OnnxParser(network, TRT_LOGGER) # Parse your modelwith open("your_model.onnx", 'rb') as model: parser.parse(model.read()) builder.int8_mode = Truebuilder.int8_calibrator = YourCalibratorClass(...) # Implement this based on TensorRT docs # Proceed with the rest of your optimization steps... And yes, you should adjust to your specific use case, whether it's for general deployment or targeting devices like Jetson. Hope this helps clarify a bit! If you have further questions, feel free to ask. 📚 — Reply to this email directly, view it on GitHub <#8386 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AX6OYJVJAUC6UHJXCMKDJLTY2SWKDAVCNFSM6AAAAABDU3EABOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSNBXGAZTA> . You are receiving this because you commented.Message ID: ***@***.*** com>

[glenn-jocher]

Hey! I'm glad you found the details useful. 😊

It sounds like the calibration process for Jetson devices using TensorRT is a key focus. If you're diving into INT8 calibration for optimizing your model's inference performance on Jetson, always consider NVIDIA's guidelines to ensure compatibility and optimal results.

For anyone else following along or running into similar questions, don't hesitate to reach out if you need further clarification or assistance. Our aim is to make deploying and optimizing models as smooth as possible for you. Happy coding! 🚀

[BJagruti]

Hello I am using Nvidia Xavier NX to deploy custom trained yolo model using deep streams.
I am stucked at this steps
Step 4. Generate the cfg and wts files

python3 gen_wts_yoloV5.py -w yolov5s.pt

Not able to find the gen_wts_yoloV5.py in https://github.com/marcoslucianops/DeepStream-Yolo

Found this file: export_yoloV5.py
Then I run the following commands
python3 export_yoloV5.py -w hatav2-80epc-best.pt.

Getting errors:
nvidia@tegra-ubuntu:~/Desktop/yolov5$ python3 export_yoloV5.py -w hatav2-80epc-best.pt
/usr/lib/python3/dist-packages/scipy/init.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.4
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"

Starting: hatav2-80epc-best.pt
Opening YOLOv5 model

YOLOv5 🚀 v6.1-306-gfbe67e46 Python-3.10.12 torch-2.1.0 CPU

Fusing layers...
Model summary: 224 layers, 7067395 parameters, 0 gradients, 16.3 GFLOPs
Traceback (most recent call last):
File "/home/nvidia/Desktop/yolov5/export_yoloV5.py", line 120, in
sys.exit(main(args))
File "/home/nvidia/Desktop/yolov5/export_yoloV5.py", line 54, in main
if len(model.names.keys()) > 0:
AttributeError: 'list' object has no attribute 'keys'

Please help me with this issue

[pderrenger]

Hello!

It seems like you're trying to use a script (gen_wts_yoloV5.py) that doesn't exist in the repository you mentioned. Instead, export_yoloV5.py is the correct script to use for exporting weights from YOLO models when working with DeepStream on Nvidia platforms.

The error you're seeing (AttributeError: 'list' object has no attribute 'keys') typically suggests that there might be a version mismatch or an expected object format isn't being met. Here are a couple of things you could try:

1. NumPy Version: Ensure your NumPy version falls within the required range. Your error message suggested a compatibility issue. You might consider downgrading NumPy to a compatible version, for instance:

   pip install numpy==1.25.0

2. Model Loading: Make sure that the model loading code in export_yoloV5.py accurately reflects the structure of your trained model. It seems like the script expects a dictionary where it's actually receiving a list.

If these suggestions don't resolve the issue, could you please provide more details on the structure of your model's output or the specific line of code causing the error? This might give more insight into what needs to be adjusted.

Good luck, and I hope this helps you move forward with your deployment! 🚀

[BJagruti]

Again the same issue i am getting error change the numpy version to 1.25.0 till not able to do Please help me with this issues

/usr/lib/python3/dist-packages/scipy/init.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.25.0
warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"

Starting: hatav2-80epc-best.pt
Opening YOLOv5 model

YOLOv5 🚀 v6.1-306-gfbe67e46 Python-3.10.12 torch-2.1.0 CPU

Fusing layers...
Model summary: 224 layers, 7067395 parameters, 0 gradients, 16.3 GFLOPs
Traceback (most recent call last):
File "/home/nvidia/Desktop/yolov5/export_yoloV5.py", line 120, in
sys.exit(main(args))
File "/home/nvidia/Desktop/yolov5/export_yoloV5.py", line 54, in main
if len(model.names.keys()) > 0:
AttributeError: 'list' object has no attribute 'keys'

Thanks & Regards

[pderrenger]

Hey there! 🚀 It seems like you're encountering a compatibility issue with NumPy and SciPy versions. Based on the error message you shared, you're using NumPy 1.25.0 which is incompatible with your version of SciPy. Here’s how we can try to solve this:

1. First, uninstall the current NumPy version:

   pip uninstall numpy

2. Reinstall a compatible version of NumPy. Since your SciPy version requires a NumPy version less than 1.25.0, you can try installing NumPy 1.24.x:

   pip install numpy==1.24.2

For the AttributeError pointing out that 'list' object has no attribute 'keys', you might need to debug or adjust the handling of model.names. If model.names is expected to be a dictionary but came out as a list, you should check the code segment that manipulates or assigns values to model.names.

Hope this helps! Let me know how it goes. 😊

[anthai0908]

Hi there, i’m using jetpack 4.5.1 and with python 3.6.9 on jetson nano. I’m wondering whether i can implement yolov5 with this config. Thanks

[glenn-jocher]

Hi there! 👋

For your Jetson Nano with JetPack 4.5.1 and Python 3.6.9, you can indeed implement YOLOv5, but you'll need to manually install compatible versions of PyTorch and Torchvision since the pip versions won't work directly on the ARM architecture of the Jetson.

Here's a quick guide:

1. Uninstall existing PyTorch and Torchvision (if installed):

   pip3 uninstall torch torchvision

2. Install the compatible PyTorch:

   wget https://nvidia.box.com/shared/static/fjtbno0vpo676a25cgvuqc1wty0fkkg6.whl -O torch-1.10.0-cp36-cp36m-linux_aarch64.whl
   pip3 install torch-1.10.0-cp36-cp36m-linux_aarch64.whl

3. Compile and install Torchvision from source:

   sudo apt install -y libjpeg-dev zlib1g-dev
   git clone --branch v0.11.1 https://github.com/pytorch/vision torchvision
   cd torchvision
   sudo python3 setup.py install

After these steps, you should be set to run YOLOv5 on your Jetson Nano! 🚀

For more detailed guidance, you can refer to the Ultralytics documentation on deploying models on NVIDIA Jetson: Deploy on NVIDIA Jetson using TensorRT and DeepStream SDK.

Happy coding!

[anthai0908]

Hi Glenn, the problem is there is no relevant ultralytics version for python 3.6.9. What should I do to install it?

[glenn-jocher]

Hi there! 👋

For Python 3.6.9 on Jetson Nano, you can still implement YOLOv5 by manually installing compatible versions of PyTorch and Torchvision. Here’s a quick guide:

1. Uninstall existing PyTorch and Torchvision:

   pip3 uninstall torch torchvision

2. Install PyTorch 1.10.0:

   wget https://nvidia.box.com/shared/static/fjtbno0vpo676a25cgvuqc1wty0fkkg6.whl -O torch-1.10.0-cp36-cp36m-linux_aarch64.whl
   pip3 install torch-1.10.0-cp36-cp36m-linux_aarch64.whl

3. Install Torchvision 0.11.1:

   sudo apt install -y libjpeg-dev zlib1g-dev
   git clone --branch v0.11.1 https://github.com/pytorch/vision torchvision
   cd torchvision
   sudo python3 setup.py install

This setup should get you up and running with YOLOv5 on your Jetson Nano. For more detailed instructions, you can refer to the Ultralytics documentation on deploying models on NVIDIA Jetson.

Happy coding! 🚀