Prediction mode missing?

[quixot1c]

Hi, thanks for your great work!

In the GIF in the readme an animation of Ironman can be seen. Under it the text "Prediction" is shown. However, when looking through the code I cannot find a way to enable this prediction mode. Is this code not available or am I missing something?

[ExponentialML]

This may be what you're looking for. Change args.mask_type at this line to 'all'.
If you don't want to hardcode it, you can just add a custom argument like args.mask_before and use that in your yaml. The function shows the available choices to use.

SEINE/sample_scripts/with_mask_sample.py

Line 228 in 3795b24

mask = mask_generation_before(args.mask_type, video_input.shape, video_input.dtype, device) # b,f,c,h,w

[quixot1c]

@ExponentialML Thanks for your response. Using 'all' made the video look only very vaguely like the input image. A bit like img2img on high denoising strength. Is that to be expected?

[ExponentialML]

@quixot1c No problem. Yes, that's to be expected as it's essentially generating frames "on the go".

If you want to generate it in an autoregressive manner, it's not yet implemented script wise. A makeshift, albeit slower way would be to generate a video based on one init frame, then use the last frame from the new generated video as the new first frame, rinse repeat, then join all the videos together.

I have a custom script that works like this locally, but it's not quite ready to release at the moment as it's still in testing.