-
-
Notifications
You must be signed in to change notification settings - Fork 410
[WIP] Second inpainting pass to extrapolate padding pixels smarter #74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
4963ea7
to
386e16d
Compare
8cfe24f
to
15a13f6
Compare
fixup
15a13f6
to
911b2e5
Compare
d426e63
to
6a06a47
Compare
2a804aa
to
0243e1f
Compare
For some reason it stops diffusing after some frames now 🤔. Argh... |
I'm adding separate inpainting settings for the changes pass, so you can set different fill mode |
Im trying to reproduce this, out of the box settings haven't dropped a frame yet. |
it usually happens on the second video |
I'm running it with the default settings for now |
Now i see it. It basically just zooms in and adds noise if Im looking at it correctly. |
This might be nothing but this allowed me to see something I haven't seen before. After some time of this non diffusion, the noise starts to create kinda a distinct line horizontally and vertically which slices the image in quadrants. Ill make a mental note to understand why a uniform pattern is emerging and if its caused by noise or some other image process. |
yep. It stops diffusing for some unknown reason. Whether it's because something resets after the iteration or whatever... |
There exists the same problem in a branch that I am working on with masking. After reading your notes about changes in inpainting, its starting to come together. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Propose a change in ln:194 as commented. Its not a fix.
New image areas are not diffused correctly still.
With the standard OOTB settings, this generates. However this is broken where the sample is no longer able to fill the output, meaning when animating translations that produce out of camera frames.
|
||
processed = processing.process_images(p) | ||
|
||
init_image = processed.images[0].convert('RGB') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am working locally of this branch. When removing .conver('RGB') so that LN: 194 is
init_image = processed.images[0]
The problem with the standard settings where it stops diffusing. (THIS IS NOT A FIX its an observation).
There still exists problems now when we rotate and novel space is introduced. This seems to address only the image not diffusing in the out of the box settings.
p.image_mask = None | ||
p.inpainting_fill = 0 | ||
mask_image = None | ||
processed = None |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is where it is passed to the masking portion
I believe that if processed is None then it executes line LN:243
if processed is None:
then we are now changing things with different variables.
I am assuming that this is the second pass you referred to.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LN:254 should reflect the zeros fill
p.inpainting_fill = args.zeros_fill_mode
not args.fill as I originally added to the init tab.
Am I wrong about that?
If I am correct, then we need to move the ui args and unify them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We are using a init sample as the mask, the way that we are processing that is masking diffusion. I realize that the idea is to make the inpainting work when novel image area is present.
There are also coloring issues. However the diffusion will continue if we do not essentially mask all of it.
Remember that the noise that we see added, it is only added because add_noise has no idea of masks. Yet.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Last comments and review in discovery process before working on final solution.
@@ -160,10 +176,13 @@ def render_animation(args, anim_args, animation_prompts, root): | |||
|
|||
# apply transforms to previous frame | |||
if prev_sample is not None: | |||
prev_img_cv2 = sample_to_cv2(prev_sample) | |||
# anti-blur | |||
prev_img_cv2 = unsharp_mask(prev_img_cv2, (kernel, kernel), sigma, amount, threshold) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this only applied in 3d?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no, should be both in 2d and 3d
if mask_image.getpixel((x,y)) < 1: | ||
mask_image.putpixel((x,y), 255 ) | ||
else: | ||
mask_image.putpixel((x,y), 0 ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I dont completely understand why, but this is sort of completely masking diffusion. I thought that white was complete masked and the code made sense in those terms before. This is indeed one location where diffusion stops.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This mask is supposed to be only for the zeroes inpainting pass. Then the mask is reset, so the regular diffusion should go fine after that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so when we get a mask from a noised sample then we cannot expect to get the actual mask we wanted if we wanted things with zeros.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that is part of the problem.
need to clear p.mask after processing to make it diffuse after this step is processed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I pulled that nested loop apart enough over last couple of days to realize it almost never had 0's and I nailed it down to noise added. Perhaps sending an argument of the clean sample would make it work. However I think its required to generate an explicit mask when transforming the image and then supplying that mask in args before generate is called on. Then we could use that mask.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
test.mp4
I got it working. This is what happens translating x by 10 that is the only motion setting.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One side effect is anything black is noised with "Latent Noise" if you use that setting.
That why I think a transformation mask makes sense. Looks cool.
At first I screamed that its making Klingon words.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fantastic!
Bringing this back up. I have thought about how to generate the mask for this inpainting pass. The masks are generated directly from the transform_image_3d(...) function
then I modified the return of the function to also return the mask. So this mask is generated along with the frame transformation. |
For the next progress, this should reverse the mask so that the newly created areas are not over converged. Ill try to work that out. |
My conclusion is that the frame warping in 3d causes artefacts that do not match the intended results. Much of the warping causes distortions that do not match the settings in this version. A new way is required. That is my conclusion. |
This looks promising for a fix for stripes. Lets keep the momentum on this. |
@Funofabot That looks really promising! I'll make some tests with the stuff as soon as I'll have time for this. Did you make any additional changes except for the mask part?
|
QoL-only features from #74: blank frames reroll, anti-blur
from #74 sets border = 'smart' by default
Closing in favor for #176 |
Mainly to fix 'border stripes'
Also a couple quality of life features such as opencv anti-blur filter and blank images reroll