Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After the last update black images after few steps on Mac mini M2 pro with Sonoma 14.5 #3521

Open
Superacf opened this issue May 19, 2024 · 37 comments

Comments

@Superacf
Copy link

How wrote in the subject, almost every workflows not working, I see the preview, but after few steps the image goes black.
After many test I've discovered that "Karras" scheduler was causing this problem.

@tsevis
Copy link

tsevis commented May 19, 2024

I have the exact same issue here.
Small number of steps is ok but anything above 10 steps isn't working.
Started since Friday night.
Any help is very appreciated.

@comfyanonymous
Copy link
Owner

Can you tell me which checkpoint and workflow you are using?

@Superacf
Copy link
Author

Happen with all checkpoint

@comfyanonymous
Copy link
Owner

comfyanonymous commented May 19, 2024

If you run it with --force-upcast-attention does that fix the issue?

@Superacf
Copy link
Author

Here in Italy is 01:00 night, so I've made only one test and it works with that fix, but Comfyui was very very slow to start, but it is a begin to discover a better solution.

@lovelyesz
Copy link

lovelyesz commented May 20, 2024

I have same issue on mac m3 pro sonoma. but use --force-upcast-attention can fix the issue

@tsevis
Copy link

tsevis commented May 20, 2024

Hi everyone,
Using --force-upcast-attention resolved the problem for me.
I hope it won't change. :-)
Thank you!

@comfyanonymous
Copy link
Owner

If you have the issue can you post your pytorch version here, you can print it with: python -c 'import torch;print(torch.version.__version__)'

@tsevis
Copy link

tsevis commented May 20, 2024

2.4.0.dev20240518

@comfyanonymous
Copy link
Owner

If you install stable 2.3 is the issue present?

@tsevis
Copy link

tsevis commented May 20, 2024

I am not sure it's appropriate to do that, as Apple recommends the nightly version. I will probably try but so far, using --force-upcast-attention has resolved the issue.
I will follow your advise if I will face the problem again.
Thanks for your support.

@Superacf
Copy link
Author

If you have the issue can you post your pytorch version here, you can print it with: python -c 'import torch;print(torch.version.__version__)'

Mine is :2.4.0.dev20240413
The nightly version when I installed ComfyUI

@MattaTatTat76
Copy link

Noob here. Where do I enter --force-upcast-attention ? I'm on a Macbook. Do I enter it in terminal by itself or in addition to python3 main.py ?

@lquesada
Copy link

Possibly related: #3523

Will try the suggested fix tomorrow

@tsevis
Copy link

tsevis commented May 21, 2024

Noob here. Where do I enter --force-upcast-attention ? I'm on a Macbook. Do I enter it in terminal by itself or in addition to python3 main.py ?

you are adding --force-upcast-attention after python3 main.py

@BannyLon
Copy link

为什么我的是:main.py: error: unrecognized arguments: --force-upcast-attention

@nataliyakarat
Copy link

2.4.0.dev20240521 on M3

dpmpp samplers (with and without karras) caused a black image. The --force-upcast-attention option solved it

@nataliyakarat
Copy link

为什么我的是:main.py: error: unrecognized arguments: --force-upcast-attention

update comfyui?

@BannyLon
Copy link

为什么我的是:main.py: error: unrecognized arguments: --force-upcast-attention

update comfyui?

没有update comfyui,因为更新到最新版后采样器几乎全部失效,而且用ComfyUI Manager 这个插件,只要选择将Preview method:更改为Auto,结果运行生图就会出现以下错误,python意外退出。Python(5424,0x16b8ef000) malloc: Incorrect checksum for freed object 0x33c1a4c00: probably modified after being freed.
Corrupt value: 0x385229353f2d3444
Python(5424,0x16b8ef000) malloc: *** set a breakpoint in malloc_error_break to debug
zsh: abort python3 main.py
habhy@192 ComfyUI % /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

@udiai
Copy link

udiai commented May 23, 2024

Same issue, almost all workflows produce black frames. (OS 14.5 on M3 Max) Starting comfyUI with python3 main.py --cpu seems to solve it, but it's very very slow

@Creative-comfyUI
Copy link

Creative-comfyUI commented May 25, 2024

The problem is not only on Sonoma, it is also on Ventura. It starts in May with some ksampler before being extended to all.

Same issue, almost all workflows produce black frames. (OS 14.5 on M3 Max) Starting comfyUI with python3 main.py --cpu seems to solve it, but it's very very slow

Exactly Karras now seems to be very slow now

If you run it with --force-upcast-attention does that fix the issue?

Read what I wrote in the issue #3567
I explain the bug in detail, it will help you to understand what is happening. It seems that the origin of the problem is not the sampler.

@comfyanonymous
Copy link
Owner

If you update ComfyUI the issue should be solved.

@Creative-comfyUI
Copy link

If you update ComfyUI the issue should be solved.

Black Image again. --force-upcast-attention solve the problem yesterday but now the problem is here again after the update

@Creative-comfyUI
Copy link

Creative-comfyUI commented May 25, 2024

If you update ComfyUI the issue should be solved.

The problem arrive if the batch size is > 1

with --force-upcast-attention everything is ok

@firestar222
Copy link

I am on a Mac M2 Max with 64 gig ram. Same issue.

--force-upcast-attention doesn't do anything to help.

Comfy has always been very reliable and stable on my machine, now I can't even generate using SD1.5 models. It's either black or gobbledegook depending on the sampler I choose.

@udiai
Copy link

udiai commented May 25, 2024

Thank you [comfyanonymous], I'm on MacOS 14.5 with M3Max. --cpu did not generate black frames but was painfully slow. Non workable. I tried to load an archived folder of ComfyUI, before my IPAdapter update, but it didn't work. I tried removing the entire ComfyUI directory and reinstalling fresh ComfyUI, also didn't work. With the help of ChatGPT4o, I uninstalled home-brew and removed all versions of python I could find. Re-installed Home-brew, Python 3.11.9 (also verifying that this is what's active and it's not calling an earlier version when I start Comfy) and latest Pytorch Nighly. Took me 3 days to troubleshoot, but now everything is working as it did before with my full ComfyUI directory.

@firestar222
Copy link

Update- I had luck by updating torch to 2.3, now everything seems to be working again?

@tsevis
Copy link

tsevis commented May 27, 2024

If you update ComfyUI the issue should be solved.

Everything seems to work smoothly again. A bit slower though...
Thank you!

@Creative-comfyUI
Copy link

If you update ComfyUI the issue should be solved.

It is repair thanks. No need --force-upcast-attention

@Creative-comfyUI
Copy link

If you update ComfyUI the issue should be solved.

It is repair thanks. No need --force-upcast-attention

The problem starts again. I don't explain why.

@thanhan-a17
Copy link

After update, any execution took almost x2 time than before. When steps >= 8, black image appear again.
--force-upcast-attention still fix the issue with the cost of speed...

Torch ver: 2.4.0.dev20240526
Comfyui ver: 221116a493
Macbook M3 pro, 18Gb

@wanderingstan
Copy link

FWIW, same issues for me.

--force-upcast-attention doesn't seem to help, but --cpu does work but at super slow speed.

Model: Macbook Pro, 14-inch, 2023
Chip: Apple M2 Pro
Memory: 32 GB
OS: macOS 14.4.1 (23E224) Sonoma
Torch: 2.3.0.dev20231215

$ python -c 'import torch;print(torch.version.__version__)'
2.3.0.dev20231215

@treeshark12
Copy link

Unexpected value in cast, It seems to be linked to denoise and seed number. Persisted after update but deleting samplers from earlier workflows and replacing them seemed to fix it for now. No SD1.5 models have been working and very few SDXL ones. I updated everything today PyTorch etc.

@treeshark12
Copy link

Further to my last comment for me it only happens on smallish (768 or 1024) square images, Ultimate upscale fails every time with black tiles everywhere.

@Creative-comfyUI
Copy link

Unexpected value in cast, It seems to be linked to denoise and seed number. Persisted after update but deleting samplers from earlier workflows and replacing them seemed to fix it for now. No SD1.5 models have been working and very few SDXL ones. I updated everything today PyTorch etc.

The first problem you mention seems to occur before the black screen for me too. It looks like Ksampler is sometimes corrupted we have to replace it.
The black screen problem doesn't appear when the batch size is one for me. Have a look at what I wrote here: #3567
Your problem seems to confirm that there is something wrong with Ksampler. The problem is with all Ksamplers and in my opinion the problem is not really the black image but the way the information is treated by the Ksampler. If we read all the comments and I look at all the tests I have done, the black image is not always due to the same operation but always in relation to the Ksampler. @comfyanonymous

@treeshark12
Copy link

It happens partway through the sampling, at about step 6 or whatever it goes black. It doesn't always report the cast error which is very strange. By changing denoise and steps I can get it to work again and once going it won't happen again on that seed.

@abumolly
Copy link

abumolly commented Jun 3, 2024

my error is ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite/videohelpersuite/nodes.py:95: RuntimeWarning: invalid value encountered in cast
return tensor_to_int(tensor, 8).astype(np.uint8)
anyone met the same situation? I used mac M3 and seem only animatediff working flow will cause the error. that's really suffer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests