[Feature Request]: make resume from timestring also work for Video Input, Hybrid Mode and ControlNet input

[kabachuha]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Grab the animation frame marked with the timestring; grab the input video frame, if it doesn't exist unpack the video and grab the frame corresponding to the timestring; if it's the Hybrid mode, grab the previous animation and video frames as well; then continue rendering the animation. The same goes for Video Mask and ControlNet input.

Proposed workflow

1. Go to Run tab
2. Enter the timestring
3. Continue rendering the animation

Additional information

No response

Are you going to help adding it? If not, tell why.

As a part of Deforum I'll either to it myself or help the rest of the team by working on other issues if someone will take it for me

[cragginstylie]

Hi! The "Resume from Timestring" used to work fine. At least for Hybrid Mode, which also requires Video Input. Video Mask has never worked correctly AFAIK. ControlNet is new enough that it not working doesn't surprise me. But, ControlNet capability is the only one that should require a feature request, IMHO. The others exist as features already, but they no longer work or never completely did.

I've relied heavily on this capability, and with it broken, was forced to figure out a work-around or a hack. Here's what I've had to do so far:

1. Initialize all the appropriate settings and start a render
2. Interrupt the render job
3. With a text editor, open the settings file in the current output folder
4. Find the line with the option hybrid_use_first_frame_as_init_image and set it to false - this option has disappeared from the UI, so this is the only way to fix the issue, and probably a huge reason the feature is broken. At least for Hybrid Mode anyway.
5. Save the changes to the settings file.
6. Copy the settings file as path and paste string into the General Settings File text box and hit Load Settings button
7. The remaining steps below can now be done within the UI
8. Tick the box to Resume from timestring, and enter the appropriate timestring value
9. Untick the option under Hybrid Mode to Generate Inputframes (once this has been done, leaving it checked causes it to re-extract frames all over again - a waste of time and processing.
10. Create a "temp" folder within the current output folder
11. Move the last (2) rendered frames into this temp folder
12. Make a note of the new last frame number in the sequence of completed frames thus far
13. Navigate into the inputframes folder and search for the frame that you noted in above step
14. Copy frame file as path, and paste string into Init Image text box
15. Hit the Generate button to resume animation
16. Once it has rendered a new frame, go back into that temp created in step 10
17. Move the (2) files back into the parent folder
18. The frames should now be restored into a proper sequence with the animation correctly creating new frames.

Always check all the settings before hitting the Generate button. I've had other settings get mysteriously botched up, like the total frame count not what I had originally set.

I typed this up to hopefully help you or any other devs understand what may be various issues or lack of code needed for this feature to work correctly. Unfortunately, I haven't yet done anything with ControlNet to add any comments or reproducible steps in this regard.

I also can't adequately comment on the Cadence issue because so far in my Hybrid Mode videos I'm always using Cadence set to (1). However, if the video isn't a Hybrid, and rather completely txt2img, then Cadence set higher than (1) seems to work just fine.

Hope this helps, and happy to test any fixes as they're introduced.

[reallybigname]

I use resume all the time, with video. But, it only works with cadence 1 "properly". However, I've been seeing a new issue where I have to delete the last image that it rendered sometimes before it will resume properly.

That being said, I resume from cadence videos too.. it's just not quite right, but rarely very noticeable.

I suggest that it should just delete whatever cadence frames are leftover at the end, mathematically, and then just resume as normal. That seems like the most elegant solution to me.

[cragginstylie]

But that isn't the only issue atm. When it resumes the animation, it will always use the very first frame of the init video when it renders the next frame in the resumed sequence. Currently, the only way to avoid that is to use my workaround, where you find the extracted source frame in the inputframes folder and put that path/filename into the init image field. End user shouldn't have to do this.

This could be caused by the missing UI option for hybrid_use_first_frame_as_init_image, which was there at one time, but has disappeared. Hence the 1st part of my workaround about editing the settings file to set this parameter to false.

But, you are correct that "just delete whatever cadence frames are leftover at the end, mathematically, and then just resume as normal" is required nonetheless.

Ultimately, a Resume feature should enable/disable/correct settings (such as hybrid_use_first_frame_as_init_image for example) or ignore them in the code if they conflict with how a resume function should work.

[GevatterWuff]

I agree with the cadence/resume from timestring problem. But I'm not sure whether some other bugs with controlNet inits exist. While doing some youtube clips I've used resume and import excessively and often got no or a completely different init than expected. Sometimes it looked like the one frame I've used for txt2img or img2img shortly before. Other times it used an init without activating "enable" in the controlNet tab.

[reallybigname]

But that isn't the only issue atm. When it resumes the animation, it will always use the very first frame of the init video when it renders the next frame in the resumed sequence. Currently, the only way to avoid that is to use my workaround, where you find the extracted source frame in the inputframes folder and put that path/filename into the init image field. End user shouldn't have to do this.

This could be caused by the missing UI option for hybrid_use_first_frame_as_init_image, which was there at one time, but has disappeared. Hence the 1st part of my workaround about editing the settings file to set this parameter to false.

But, you are correct that "just delete whatever cadence frames are leftover at the end, mathematically, and then just resume as normal" is required nonetheless.

Ultimately, a Resume feature should enable/disable/correct settings (such as hybrid_use_first_frame_as_init_image for example) or ignore them in the code if they conflict with how a resume function should work.

I resume videos all the time and it doesn't put the first frame in there, and it's always on by default.. because it only affects the first frame, and only if you're using hybrid compositing.

And, just to be clear, I can resume normal animation, controlnet animation, hybrid animation, and any combination of any of those.

[reallybigname]

I agree with the cadence/resume from timestring problem. But I'm not sure whether some other bugs with controlNet inits exist. While doing some youtube clips I've used resume and import excessively and often got no or a completely different init than expected. Sometimes it looked like the one frame I've used for txt2img or img2img shortly before. Other times it used an init without activating "enable" in the controlNet tab.

I have just completely re-worked the resume mechanism and torture tested it. As long as you have at least the number of frames that your cadence is set to, it will resume. Like, if you are on cadence 5, you need to have rendered at least the first 5 frames. And, it will do it even if you delete a few random frames going backwards, as long as the minimum frame count is there.

[cragginstylie]

I resume videos all the time and it doesn't put the first frame in there, and it's always on by default.. because it only affects the first frame, and only if you're using hybrid compositing.

This is not how it was behaving at one time. It would always insert the first frame no matter where it was at in the sequence. But, I just did a test and it seems to be working acceptably. The resumed sequence still doesn't fit exactly with the previous sequence of frames in terms of the various strength/noise/etc settings that would have been calculated had the animation not been interrupted. But, it's really only noticeable if you are watching for it when playing back the stitched animation. This might be fixed given your comment of "just completely re-worked" came in only 5 hours ago. I'll retest after I update later this evening.