Enable GUI Visualization on Docker with X11 Forwarding

[Jpfonseca]

This PR solves the issues identified in #882 and #4885.
In specific @gadhane and me have investigated the issues related with visualizing the GUI when running the docker image with X11 Forwarding in Ubuntu.

The issue and details are thoroughly discussed here.

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Enhancements in Docker compatibility and improved display checks in environment.

📊 Key Changes

• Added libsm6 installation to the Dockerfile, essential for creating QT-based windows for object visualization.
• Modified environment check logic in checks.py for better compatibility and error handling related to displaying images.

🎯 Purpose & Impact

• 🚀 Enhanced Visualization Support: Installing libsm6 allows Docker environments to more easily visualize detection results, aiding in debugging and development processes.
• 🛠 Improved User Experience: Modified display checks now more accurately identify when and why image displays might fail, leading to clearer error messages and less confusion for users setting up or debugging their environments.
• ✅ Broader Compatibility: These changes improve the software’s compatibility with various platforms, making it more accessible and easier to use across different setups.

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Enhancements to Docker setup and GUI support for Ultralytics.

📊 Key Changes

• Dockerfile Update: Added libsm6 for creating QT-based windows, enabling visualization in Docker containers. 🐳
• Docker Quickstart Guide Enhancements: Expanded with instructions for running Ultralytics using CPU or GPU, and incorporating GUI applications in Docker containers. 📖
• Experimental GUI Support: Provided instructions for displaying Ultralytics detection results through GUI within a Docker container, marking a significant step towards more interactive usage scenarios. ⚠️🖥️
• Check Update in Utils: Simplified conditions for the environment's capability to display images, specifically refining the checks for DISPLAY environment variable presence. 🛠️

🎯 Purpose & Impact

• Enhanced Visualization: The inclusion of libsm6 and detailed guide for GUI support allows users to visualize object detection results directly, significantly improving the user experience for those working in Docker environments. 🎉
• Greater Flexibility: By addressing CPU and GPU usage more explicitly, and offering guidance on utilizing GUI, users gain more flexibility in how they deploy and interact with Ultralytics software, catering to a wider range of use cases. 🌈
• Improved Accessibility: Simplifying environment check conditions makes it easier for developers to troubleshoot and ensure their setup is capable of visual output, streamlining the setup process. 🚀

This PR facilitates a more interactive and user-friendly approach to using Ultralytics in varied environments, making it accessible to both developers and users interested in advanced object detection without sacrificing the convenience of Docker.

[github-actions]

All Contributors have signed the CLA. ✅
_{Posted by the CLA Assistant Lite bot.}

[github-actions]

👋 Hello @Jpfonseca, thank you for submitting an Ultralytics YOLOv8 🚀 PR! To allow your work to be integrated as seamlessly as possible, we advise you to:

• ✅ Verify your PR is up-to-date with ultralytics/ultralytics main branch. If your PR is behind you can update your code by clicking the 'Update branch' button or by running git pull and git merge main locally.
• ✅ Verify all YOLOv8 Continuous Integration (CI) checks are passing.
• ✅ Update YOLOv8 Docs for any new or updated features.
• ✅ Reduce changes to the absolute minimum required for your bug fix or feature addition. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." — Bruce Lee

See our Contributing Guide for details and let us know if you have any questions!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.56%. Comparing base (bd50162) to head (ae788dd).

❗ Current head ae788dd differs from pull request most recent head 36a7afc

Please upload reports for the commit 36a7afc to get more accurate results.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #11729      +/-   ##
==========================================
- Coverage   70.63%   70.56%   -0.07%     
==========================================
  Files         122      122              
  Lines       15635    15636       +1     
==========================================
- Hits        11043    11033      -10     
- Misses       4592     4603      +11     

Flag	Coverage Δ
Benchmarks	35.35% <0.00%> (-0.19%)	⬇️
GPU	37.27% <0.00%> (-0.01%)	⬇️
Tests	66.74% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Jpfonseca]

I have read the CLA Document and I sign the CLA

[ambitious-octopus]

Seems like a good solution. However, I would also add to the documentation the specific arguments for launching the container in order to connect to the X server.

[glenn-jocher]

@ambitious-octopus hey that's a great idea!

@Jpfonseca can you add an example to the Docker Quickstart Guide here with the code to visualize from a Docker image?
https://docs.ultralytics.com/guides/docker-quickstart/
https://github.com/ultralytics/ultralytics/blob/main/docs/en/guides/docker-quickstart.md

[glenn-jocher]

@Jpfonseca PR looks but could you please update the docs with example code for remote visualization here:
https://github.com/ultralytics/ultralytics/blob/main/docs/en/guides/docker-quickstart.md

[Jpfonseca]

@glenn-jocher just added the examples for both X11 and Wayland. Tested both solutions in Xorg (gnome and xfce) and Wayland (sway and gnome) .

[ambitious-octopus]

Thank you for your contribution. I appreciate your input, as it adds significant value to the docs. I'll integrate this section more seamlessly, ensuring clarity and accuracy. Additionally, I'll correct any typos. Thanks for including instructions on using Docker without a GPU. It would also be useful to make a small integration here:

ultralytics/docs/en/guides/docker-quickstart.md

Lines 17 to 32 in fcfc44e

	## What You Will Learn
	- Setting up Docker with NVIDIA support
	- Installing Ultralytics Docker images
	- Running Ultralytics in a Docker container
	- Mounting local directories into the container
	---
	## Prerequisites
	- Make sure Docker is installed on your system. If not, you can download and install it from [Docker's website](https://www.docker.com/products/docker-desktop).
	- Ensure that your system has an NVIDIA GPU and NVIDIA drivers are installed.
	---

[Jpfonseca]

@ambitious-octopus we addressed the comments you made.

[ambitious-octopus]

What do you think of this solution? I tried to follow what you wrote but made it more concise and used the ultralytics documentation standard. You can test it with mkdocs serve. What do you think @Burhan-Q?

Also, I haven't tested with wayland. Have you already tested @Jpfonseca? With what machine?

[Burhan-Q]

@Jpfonseca and @ambitious-octopus a few notes in general on these additions/changes. Expand the section at the bottom of this comment to see a preview of the changes.

• Let's make the Visualization it's own section using an H2 header

• Double check alignment with proposed changes from @ambitious-octopus as they are in the GitHub preview, the alignment is off.

• Consider the section title Using a Display Server with Docker to Show Ultralytics Detection Results instead of Visualize Ultralytics' Object detection in your GNU-Linux Display Server as Ultralytics can be used for more than object detection and is more generally descriptive (although still more verbose than I like).

  • Also generally, use Ultralytics instead of Ultralytics' (no trailing apostrophe) as it's not required

• Make certain to always capitalize Ultralytics and Docker whenever used in text. If referring to the Ultralytics package, you can use ultralytics

• Make sure to always include new lines before and after code blocks.

    In both cases, don't forget to revoke access from the docker group when you're done.
  
    ```bash
    xhost -local:docker
    ```
  
    Next line starts here...
  

• ???+ danger "Caution: Highly Experimental" is a good change, however I'd recommend !!! danger "Highly Experimental - User Assumes All Risk" but at the very least, don't think it should be collapsible (change from ???+ to !!!); expand section at end of comment to see preview.

• It would be good to see more specifics about what the user should be looking to learn/understand from More information on the subject [here](http://users.stat.umn.edu/~geyer/secure.html) as it's not clear what a user should look to understand specifically (all commands from the link, 1 command, or something else?).

  • I also think it might be good to use a source that we can rely on. If looking to point users to these specific commands, perhaps using ssh, scp, xauth, and xhost

• Be sure to include an example of "what you should see and what it means" after instructing to use

  env | grep -E -i 'x11|xorg|wayland'

since it's not obvious what a user should expect to see based on what's available (or when nothing is available).

• I think the tabbed example can be changed (expand section at end of comment to see preview)

- !!! example ""
+ !!! example

• I like the tabbed examples, but maybe break up the lines with \ so there's no need to scroll to view the entire command (expand section at end of comment to see preview)

  xhost +local:docker && docker run -e DISPLAY=$DISPLAY -v \
  $XDG_RUNTIME_DIR/$WAYLAND_DISPLAY:/tmp/$WAYLAND_DISPLAY \
  --net=host -it --ipc=host $t

Preview single line with horizontal scrolling

• I think this block can be moved to it's own section (above) the tabbed examples so that it's visible for both tabs and doesn't need to be duplicated (expand section at end of comment to see preview):

  !!! example ""
  
      ??? info "Use GPUs"
      
          If you're using [GPUs](#using-gpus), you can add the `--gpus all` flag to the command.
      
          === "X11"
      
              If you're using X11, you can run the following command to allow the Docker container to access the X11 socket:
          
              ...

• Make this non-collapsible as I think it's important for users to always see (expand section at end of comment to see preview)

  - ???+ warning "Revoke access"
  + !!! warning "Revoke access"

• I'll admit I'm biased here (I wrote the guide), but I think it would be useful to mention the view-results-in-terminal.md guide as it's somewhat related. I think this could be good to include in the introduction to this section

Preview of suggestions

Table of Contents

Section

[glenn-jocher]

@Burhan-Q is this PR good to merge now?

[Burhan-Q]

@glenn-jocher I went ahead and made a few changes, fixes. I believe it's good to go.

• capitalize all text references to Docker
• reformat citations and links
• fix warning "Revoke access" (not rendering), now not collapsible, and new line before code block
• line spacing clean up
• Reorder sections, add sub-headers, move reference to view images in terminal guide, and add more docs reference links
• add external reference links and statement about display server setup

Page Preview

[Burhan-Q]

Also, I can't speak to the functionality of this guide and have assumed it to be correct.

[ambitious-octopus]

Tested with x11, everything works fine. I'm testing with wayland, I'll confirm asap!

[Jpfonseca]

@ambitious-octopus Thanks, if needed, I can record it now .

[ambitious-octopus]

It might be useful, thanks @Jpfonseca !

[Jpfonseca]

Here is the video as requested by @ambitious-octopus.
I started the session with xhost +local:docker enabled, but afterwards I revert the command. and use the command again to validate the access to the $DISPLAY

docker_ultralytics_wayland.mp4

[ambitious-octopus]

Hey @Jpfonseca thanks for the video, everything seems to work correctly!

[Jpfonseca]

@glenn-jocher I believe it is ready to be merged now.

[glenn-jocher]

@Jpfonseca @ambitious-octopus @Burhan-Q awesome, thank you all, PR merged!

[Jpfonseca]

@glenn-jocher @ambitious-octopus @Burhan-Q if needed, i can also share how this can be used in a K8s Deployment in a different PR

[Burhan-Q]

@Jpfonseca you never have to ask permission to open a PR 👍 however I will only be able to provide support from the docs formatting perspective, as I have only gotten as far as learning k8s == kubernetes 😆