Skip to content

Experimental render core with keyframe redistribution logic for Parseq. #978

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 156 commits into
base: automatic1111-webui
Choose a base branch
from

Conversation

Tok
Copy link

@Tok Tok commented Aug 16, 2024

Contains a new rendering package with a refactored version of render.py module that may be used as an alternative to the existing render core in some conditions. It was mostly created with Parseq in mind and provides a new function to align keyframes in a way that allows for fast generation at high cadence.

The conditions of whether the new core is used are defined in render.py at the beginning of the 'render_animation' method (this would be a good place to start reviewing the code).
If the new core is used, everything is delegated to /rendering/experimental_core.py.
One of the conditions is that a new 'keyframe redistribution mode' is activated in a new dropdown on the Parseq UI tab.
If the new distribution mode is set to "off" (default setting), then the experimental render core is not called and everything else in render.py proceeds as usual.

The aim of creating the new core was mostly to reduce complexity by separating it into smaller parts and to bring it to a state where I would be able to replace the two nested main loops (frames and tween frames) and their indexes by another main loop that is doing the same thing, but by processing a collection of frame-objects that can have a variable number of (inbe)tween-frames that aren't actually diffused.
This makes it possible to realign the frames-to-actually-diffuse with the keyframes provided by the Parseq manifest. The new code related to keyframe redistribution can be found in /rendering/data/frame/key_frame_distribution.py

For a more detailed description and some progsplaining about refactoring and stuff, you may also check this slightly outdated README that I left on an older branch: https://github.com/Tok/sd-webui-deforum/tree/feature/integration

Other things

  • Short explanation and selection of the new redist mode on the Parseq UI tab.
  • There is a new property with the keyframe redistribution mode for deforum-settings.txt in the parseq namespace.
  • There is new orange progressbar that replaces current tween-frame related logging in some conditions, because the current default is over-verbose at high cadence.
  • The restart logic was basically reimplemented directly in the core (works on my system ;)
  • Also added and used /rendering/util/log_utils.py with some new methods that use log-levels like warning or debug
  • I was trying hard to not change existing code as much as possible. Also see long comment in /rendering/util/call/init.py
  • /rendering/img_2_img_tubes.py isolates some image to image transformations (numpy and PIL) into lambda chains which should make it more open for experimentation and future changes.
  • Even tough the code is well linted and should mostly look clean etc., truth is that I'm not really well versed in using Python, so there may be some noob mistakes in there. If you're more fam with py, please especially review the pythonisms (new init.py files and all the importing stuff etc).
  • Because of the complexity, the testing I could make was limited. Control net and color coherence was barely tested, but should work fine.
  • "optical flow" behaves weird, so there is a warning when it is in use in combination with the new core. The same is true for "hybrid video" and related functions that are mostly untested and may currently not run stable (or not run at all).
  • There are some active asserts related to the new keyframe redistribution modes in the new KeyFrame class.

If you have any questions, please ask away. I may also be around on discord sometimes.

I agree that my work will be granted copyright to Deforum LLC and be licensed under the terms of the GNU Affero General Public License version 3.

Tok and others added 30 commits June 1, 2024 11:42
…into separate methods and regrouped data accordingly into new tuples. Replaced some comments with samely named method calls. Prefixed some booleans with "is_" for better readability. Reorganized imports. Marked some initialization code that seems dead and added a few TODOs for further consideration and optimization.
… vars to avoid reassignments and consolidate calls.
… package to render_data because, really it's all just data.
… isolate and group side effects. More stuff moved to static context.
…d renamed the render package to avoid conflicts and confusion with the render module.
…on a button-click in PyCharm (Interpreter path pointing to standard Windows powershell dir by default, all other paths provided relative).
… prepare for reduction of complexities in method calls. render_animation method torn appart to ensure tightly scoped args and also unfroze the AnimationMode class again to allow for the late initialization of prev_flow for now.
…s that broke in the last commit. All original arguments have been taken out of scope for now. More TODOs added.
Zir Teq and others added 30 commits July 7, 2024 04:19
…s in some cases with PARSEQ_ONLY setup. Parse table printout reactivated.
…xisting Parseq tab. Moved new property to parseq namespace. Reverted render core back to original and added a switch for using the new core.
…es. Conversion between PIL and numpy images extracted.
…tical flows, but handling it as an experimental feature by printing a warning as it doesn't quite behave as one shoul expect and may also be quetionable in general when used in a variable cadence context.
…a new bool to turn it on or off and disabled it by default.
…p to be used together with hybrid video and updated the UI text.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant