Skip to content

[WIP] Second inpainting pass to extrapolate padding pixels smarter #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 22 commits into from

Conversation

kabachuha
Copy link
Member

Mainly to fix 'border stripes'

Also a couple quality of life features such as opencv anti-blur filter and blank images reroll

@kabachuha kabachuha added enhancement New feature or request UI labels Nov 6, 2022
fixup
@kabachuha
Copy link
Member Author

For some reason it stops diffusing after some frames now 🤔. Argh...

@kabachuha
Copy link
Member Author

kabachuha commented Nov 8, 2022

Quick question kabachuha: Are you changing the args.fill to args.zeros_fill_mode

I'm adding separate inpainting settings for the changes pass, so you can set different fill mode

@Funofabot
Copy link
Contributor

Funofabot commented Nov 8, 2022

Im trying to reproduce this, out of the box settings haven't dropped a frame yet.
Roughly how many frames until it stops diffusion?
I've went through some 2D and 3D keyframe settings, appears normal.
Any chance you can shoot me deforum settings?

@kabachuha
Copy link
Member Author

it usually happens on the second video

@kabachuha
Copy link
Member Author

I'm running it with the default settings for now

@Funofabot
Copy link
Contributor

Now i see it. It basically just zooms in and adds noise if Im looking at it correctly.

@Funofabot
Copy link
Contributor

This might be nothing but this allowed me to see something I haven't seen before. After some time of this non diffusion, the noise starts to create kinda a distinct line horizontally and vertically which slices the image in quadrants. Ill make a mental note to understand why a uniform pattern is emerging and if its caused by noise or some other image process.

@kabachuha
Copy link
Member Author

Now i see it. It basically just zooms in and adds noise if Im looking at it correctly.

yep. It stops diffusing for some unknown reason. Whether it's because something resets after the iteration or whatever...

@Funofabot
Copy link
Contributor

image
Happens when this statement is printed.

@Funofabot
Copy link
Contributor

Funofabot commented Nov 8, 2022

I see comments about the new inpainting. Now I am aware of why you want help with this because it is indeed very confusing. The mask area changes with the animation and it is not consistent. When zooming out, the first frame reduces size and by the third frame the noise is actually complete vertical of the centered image and not symmetric in one axis.
image

@Funofabot
Copy link
Contributor

There exists the same problem in a branch that I am working on with masking. After reading your notes about changes in inpainting, its starting to come together.
Im active looking into this.

Copy link
Contributor

@Funofabot Funofabot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Propose a change in ln:194 as commented. Its not a fix.
New image areas are not diffused correctly still.
With the standard OOTB settings, this generates. However this is broken where the sample is no longer able to fill the output, meaning when animating translations that produce out of camera frames.


processed = processing.process_images(p)

init_image = processed.images[0].convert('RGB')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am working locally of this branch. When removing .conver('RGB') so that LN: 194 is
init_image = processed.images[0]
The problem with the standard settings where it stops diffusing. (THIS IS NOT A FIX its an observation).
There still exists problems now when we rotate and novel space is introduced. This seems to address only the image not diffusing in the out of the box settings.

p.image_mask = None
p.inpainting_fill = 0
mask_image = None
processed = None
Copy link
Contributor

@Funofabot Funofabot Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is where it is passed to the masking portion
I believe that if processed is None then it executes line LN:243
if processed is None:

then we are now changing things with different variables.
I am assuming that this is the second pass you referred to.

Copy link
Contributor

@Funofabot Funofabot Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LN:254 should reflect the zeros fill
p.inpainting_fill = args.zeros_fill_mode
not args.fill as I originally added to the init tab.

Am I wrong about that?
If I am correct, then we need to move the ui args and unify them.

Copy link
Contributor

@Funofabot Funofabot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using a init sample as the mask, the way that we are processing that is masking diffusion. I realize that the idea is to make the inpainting work when novel image area is present.
There are also coloring issues. However the diffusion will continue if we do not essentially mask all of it.
Remember that the noise that we see added, it is only added because add_noise has no idea of masks. Yet.

Copy link
Contributor

@Funofabot Funofabot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last comments and review in discovery process before working on final solution.

@@ -160,10 +176,13 @@ def render_animation(args, anim_args, animation_prompts, root):

# apply transforms to previous frame
if prev_sample is not None:
prev_img_cv2 = sample_to_cv2(prev_sample)
# anti-blur
prev_img_cv2 = unsharp_mask(prev_img_cv2, (kernel, kernel), sigma, amount, threshold)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this only applied in 3d?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, should be both in 2d and 3d

if mask_image.getpixel((x,y)) < 1:
mask_image.putpixel((x,y), 255 )
else:
mask_image.putpixel((x,y), 0 )
Copy link
Contributor

@Funofabot Funofabot Nov 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont completely understand why, but this is sort of completely masking diffusion. I thought that white was complete masked and the code made sense in those terms before. This is indeed one location where diffusion stops.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mask is supposed to be only for the zeroes inpainting pass. Then the mask is reset, so the regular diffusion should go fine after that

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so when we get a mask from a noised sample then we cannot expect to get the actual mask we wanted if we wanted things with zeros.

Copy link
Contributor

@Funofabot Funofabot Nov 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that is part of the problem.
need to clear p.mask after processing to make it diffuse after this step is processed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pulled that nested loop apart enough over last couple of days to realize it almost never had 0's and I nailed it down to noise added. Perhaps sending an argument of the clean sample would make it work. However I think its required to generate an explicit mask when transforming the image and then supplying that mask in args before generate is called on. Then we could use that mask.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test.mp4

I got it working. This is what happens translating x by 10 that is the only motion setting.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One side effect is anything black is noised with "Latent Noise" if you use that setting.
That why I think a transformation mask makes sense. Looks cool.
At first I screamed that its making Klingon words.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic!

@Funofabot
Copy link
Contributor

Funofabot commented Dec 13, 2022

Bringing this back up. I have thought about how to generate the mask for this inpainting pass. The masks are generated directly from the transform_image_3d(...) function
with this code

warp_mask = torch.ones_like(image_tensor) * 255
warp_mask_image = torch.nn.functional.grid_sample(
    warp_mask.add(1/512 - 0.0001).unsqueeze(0), 
    offset_coords_2d, 
    mode="nearest", 
    padding_mode="zeros", 
    align_corners=False
)

result_mask = rearrange(
    warp_mask_image.squeeze().clamp(0,255), 
    'c h w -> h w c'
).cpu().numpy().astype(prev_img_cv2.dtype)

then I modified the return of the function to also return the mask.

So this mask is generated along with the frame transformation.

@Funofabot
Copy link
Contributor

Funofabot commented Dec 13, 2022

curious things happen however. For example when looking at the generated masks, in this example rotation_3d_z is the only option used in the animation, and is set to 1.

frame 35
mask20221212165458_00035
frame 36
mask20221212165458_00036

@Funofabot
Copy link
Contributor

20221212165458

produces images without border stripes. Still not a great example.

@Funofabot
Copy link
Contributor

x translation only

20221212173821

@Funofabot
Copy link
Contributor

For the next progress, this should reverse the mask so that the newly created areas are not over converged. Ill try to work that out.

@Funofabot
Copy link
Contributor

My conclusion is that the frame warping in 3d causes artefacts that do not match the intended results. Much of the warping causes distortions that do not match the settings in this version. A new way is required. That is my conclusion.

@Funofabot
Copy link
Contributor

This looks promising for a fix for stripes. Lets keep the momentum on this.

@kabachuha
Copy link
Member Author

@Funofabot That looks really promising! I'll make some tests with the stuff as soon as I'll have time for this. Did you make any additional changes except for the mask part?

Bringing this back up. I have thought about how to generate the mask for this inpainting pass. The masks are generated directly from the transform_image_3d(...) function with this code

warp_mask = torch.ones_like(image_tensor) * 255
warp_mask_image = torch.nn.functional.grid_sample(
    warp_mask.add(1/512 - 0.0001).unsqueeze(0), 
    offset_coords_2d, 
    mode="nearest", 
    padding_mode="zeros", 
    align_corners=False
)

result_mask = rearrange(
    warp_mask_image.squeeze().clamp(0,255), 
    'c h w -> h w c'
).cpu().numpy().astype(prev_img_cv2.dtype)

then I modified the return of the function to also return the mask.

So this mask is generated along with the frame transformation.

kabachuha added a commit that referenced this pull request Dec 19, 2022
using unsharp mask, from #74
kabachuha added a commit that referenced this pull request Dec 22, 2022
QoL-only features from #74: blank frames reroll, anti-blur
kabachuha added a commit that referenced this pull request Dec 27, 2022
from #74

sets border = 'smart' by default
@kabachuha
Copy link
Member Author

Closing in favor for #176

@kabachuha kabachuha closed this Dec 27, 2022
stride-srl pushed a commit to stride-srl/sd-webui-deforum that referenced this pull request Jun 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request UI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants