h100: Worse output & 20x slower inference?

[addytheyoung]

We're testing finetuning on an h100 and 4090, here are the results:

4090: https://voca.ro/11mtxzLHzzih
h100: https://voca.ro/15QldVjuG7nu

Almost identical finetune, but h100 is output is SIGNIFICANTLY worse. It isn't a config issue, and we've replicated it twice with LJSpeech as well.

4090 is also faster during training and considerably faster during inference, almost 20x faster than h100:

h100:

And during training, one epoch took the 4090 about 3 minutes, while the h100 took 4.12 minutes.

Does anyone know what could be going on here? Never seen an issue like this on an h100 before with a diffusion like model. Thanks

[AMEERAZAM08]

I'm using on T4 2 GPU kaggle its working good check you requirement.txt version

[yl4579]

Can you be sure they’re on the same machine with the same library and everything, except the GPU difference?

[devidw]

@addytheyoung and me are working together on this

We are running this on 2 different machines, one is a lambdalabs h100 and one is a runpod rtx 4090

The env is the same with both running in a python3.10 virtualenv with the latest repo and dependencies

In the screenshots shared earlier there was an issue with non http 200 responses leading to invalid timing

However, rerunning the following simple web api on both machines, and the 4090 seems to be 2x as fast as it should be 3x slower than the h100

4090 ~12s @ 100 concurrent requests

h100 ~24s @ 100 concurrent requests

[yl4579]

@devidw Have you tried to match the CUDA driver version as well? Have you tried the docker version?

[devidw]

@devidw Have you tried to match the CUDA driver version as well? Have you tried the docker version?

Yeah, so both GPUs are using cuda 12.x

By docker version, you mean running the inference server within the same cuda/pytorch base docker image on both GPUs? (Currently, the runpod 4090 is using runpod/pytorch and the h100 on lambda doesn't require one, tho we could test that)

@yl4579

[yl4579]

@devidw This is so weird, I've tested the code on A40, A100 and L40, all good with CUDA 11.8. I'm not sure if it is the CUDA driver version that causes this. I don't have H100 so I can't test it. The worst case scenario is you compare the output for each module with the same input and see which module causes the issue.

[addytheyoung]

@yl4579 I believe we've done that already, oh well. If anyone else has or can test on an H100, we'd love to hear what's going on. This is a big deal for scaling inference, if the model just doesn't work with an H100 at all that's not good, but hopefully it's on our end.

[yl4579]

Did you find any output difference or timed each line and find the module that causes the bottleneck?

[addytheyoung]

@yl4579 Yes the output is considerably worse, I shared some examples in the main post. We have not tested if the module itself causes the bottleneck I don't think? cc @devidw

[yl4579]

@addytheyoung I think it’s better to check the output from each module and setting a static random seed (don’t use diffusion model) to check which module produces different results, so we can debug from there.

[korakoe]

Odd, I've been training a model on a h100 for a day or so, and haven't noticed a detriment in speed or quality, is your repo up to date?

Edit:
I retract this statement, it appears to emerge on larger datasets

[sch0ngut]

Very interesting, I just found this issue by pure chance, and indeed i had the same problem a few weeks ago. I ran 6 fine-tunings in parallel for several days on a single H100, and once the trainings finished I noticed that all models were absolutely useless.
I reran the same trainings on 8xA100 and the results were good! The only differences (besides the number & type of GPUs used) was the per GPU batch size, but I didn't think that this would have caused the issue. I was (and still am) extremely puzzled...