Reliance on CUDA for FFMPeg Hardware encoding causes codec warnings with AMD GPUs

[jeremymeyers]

Describe the bug
Stash appears to call on CUDA for hardware-level decoding. CUDA is a library specific to NVidia GPUs, called in the FFMPeg init

stash/pkg/ffmpeg/codec_hardware.go

Line 120 in 911da87

videoFilter = videoFilter.Append("hwupload_cuda")

and seemingly connected to support for some VP9 and h264 codecs.

I haven't been able to test whether it breaks vp9 playback as i don't seem to have vp9 files in my library but thought it was worth mentioning.

There are apparently multiple options when it comes to FFMpeg hardware encoding, as detailed here https://trac.ffmpeg.org/wiki/HWAccelIntro and NVENC(of which CUDA is a part) is very limited in its compatibility relative to other options, only being compatible with Linux and Windows with NVidia GPUs.

To Reproduce
Steps to reproduce the behavior:

1. Have computer with AMD GPU
2. launch stash.exe

The following errors appear:

DEBU[2024-04-16 15:09:02] [InitHWSupport] Codec h264_nvenc not supported. Error output:
[AVHWDeviceContext @ 000002511ddc7a00] Cannot load nvcuda.dll
[AVHWDeviceContext @ 000002511ddc7a00] Could not dynamically load CUDA
[AVFilterGraph @ 000002511ddf0040] Error initializing filter 'hwupload_cuda'
Error reinitializing filters!
Failed to inject frame into filter network: Unknown error occurred
rror while processing the decoded data for stream #0:0
DEBU[2024-04-16 15:09:02] [InitHWSupport] Codec h264_qsv not supported. Error output:
[AVHWDeviceContext @ 00000229ce23e5c0] Error initializing an MFX session: -3.
Device creation failed: -1313558101.
Failed to set value 'qsv=hw' for option 'init_hw_device': Unknown error occurred
rror parsing global options: Unknown error occurred
DEBU[2024-04-16 15:09:02] [InitHWSupport] Codec h264_vaapi not supported. Error output:
Unrecognized option 'vaapi_device'.
rror splitting the argument list: Option not found
DEBU[2024-04-16 15:09:02] [InitHWSupport] Codec h264_v4l2m2m not supported. Error output:
nknown encoder 'h264_v4l2m2m'
DEBU[2024-04-16 15:09:02] [InitHWSupport] Codec vp9_qsv not supported. Error output:
[AVHWDeviceContext @ 000001caecade600] Error initializing an MFX session: -3.
Device creation failed: -1313558101.
Failed to set value 'qsv=hw' for option 'init_hw_device': Unknown error occurred
rror parsing global options: Unknown error occurred
DEBU[2024-04-16 15:09:02] [InitHWSupport] Codec vp9_vaapi not supported. Error output:
Unrecognized option 'vaapi_device'.
rror splitting the argument list: Option not found
INFO[2024-04-16 15:09:02] [InitHWSupport] Supported HW codecs:

Expected behavior
Global compatibility

Screenshots
If applicable, add screenshots to help explain your problem please ensure that your screenshots are SFW or at least appropriately censored.

Stash Version: (from Settings -> About):

Desktop (please complete the following information):

• OS: Win11
• Browser n/a
• Version [n/a]

Additional context

[feederbox826]

I don't believe AMF is explicitly supported, but the presets for hwaccel are generally troublesome, there was someone in discord reporting decoder surfaces missing because NVDEC doesn't support .wmv

I think it would be better to move configuration to the user side or pick from a dropdown instead of auto-configurations, which would allow for AMF support

[jeremymeyers]

I would say okay if there are robust presets. Selecting hardware codec options is a really niche subject that most users are not going to be familiar with and if there a chance they could break playback by selecting the wrong thing then it should not be that open