CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected

[kunmonster]

Hi , when i run call_variant , it arises this warn which means can't use the gpu,but i can make sure that the tensorflow can use the gpu.There are the screen shots of the warn and the existence of the gpu.

• the gpu existence
  

tensorflow.test.is_gpu_available()
WARNING:tensorflow:From <stdin>:1: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2024-05-12 21:36:00.744470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /device:GPU:0 with 15089 MB memory:  -> device: 0, name: Vega 20, pci bus id: 0000:26:00.0
True

• the warn
  

  warnings.warn(
2024-05-12 21:43:29.067332: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:267] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected

• Operating system: Linux
• DeepVariant version: 1.6.1-gpu
• Installation method (Docker, built from source, etc.): Docker
• Type of data: (sequencing instrument, reference genome, anything special that is unlike the case studies?)

[akolesnikov]

Hi @kunmonster ,
Could you please provide a command line that you use to run DeepVariant?

[kunmonster]

Hi @kunmonster , Could you please provide a command line that you use to run DeepVariant?

Sorry,the command line is in the top of the second picture. Actually,i run the docker container in interactive mode,then run the call_variants line within the container

[akolesnikov]

Since sometimes warning messages from Tensorflow may be misleading. Could you please try running call_variants and at the same time monitor the GPU load to make sure the GPU is not used?
You can use watch -n0.5 nvidia-smi to check the GPU load in real time.

[kunmonster]

Happy to see the reply,actually, i run this on hpc and the usage of the gpu is very low , but the cpu and memory usage is extremely high , then i run this command line on my laptop with gtx 1050ti and compare the time of the prediction one batch ,the time in the hpc is longer than my laptop , but the truth is the performance of the hpc gpu is better than gtx1050ti. So, the gpu don't work. I will post what you want later.Thx!

[kunmonster]

There are the pictures for the usage of gpu

[akolesnikov]

Did you run tensorflow.test.is_gpu_available() from the DeepVariant docker?

Could you try the suggestion from this thread

[kunmonster]

Did you run tensorflow.test.is_gpu_available() from the DeepVariant docker?

Yes i did , I have posted the result which shows that in python shell the gpu can be identified with tensorflow in the first comment.

Could you try the suggestion from this thread

Actuallly i have tried to set CUDA_VISIBLE_DEVICES=0 in System ENV ,it did't work .So I tried to find the place where sets the value of the env in your code , and want to set the CUDA_VISIBLE_DEVICES=0 , but i did't find. So ,i turn to ask for your help.

I think the reason why the error occurs may be in your code the value of the CUDA_VISIBLE_DEVICES does't match with my device.