[WIP] Second inpainting pass to extrapolate padding pixels smarter #74

kabachuha · 2022-11-05T23:19:26Z

Mainly to fix 'border stripes'

Also a couple quality of life features such as opencv anti-blur filter and blank images reroll

fixup

fixup fixup

…for-automatic1111-webui into fix-stripes

kabachuha · 2022-11-08T19:24:54Z

For some reason it stops diffusing after some frames now 🤔. Argh...

kabachuha · 2022-11-08T21:03:14Z

Quick question kabachuha: Are you changing the args.fill to args.zeros_fill_mode

I'm adding separate inpainting settings for the changes pass, so you can set different fill mode

Funofabot · 2022-11-08T21:06:36Z

Im trying to reproduce this, out of the box settings haven't dropped a frame yet.
Roughly how many frames until it stops diffusion?
I've went through some 2D and 3D keyframe settings, appears normal.
Any chance you can shoot me deforum settings?

kabachuha · 2022-11-08T21:12:27Z

it usually happens on the second video

kabachuha · 2022-11-08T21:13:06Z

I'm running it with the default settings for now

Funofabot · 2022-11-08T21:33:30Z

Now i see it. It basically just zooms in and adds noise if Im looking at it correctly.

Funofabot · 2022-11-08T21:36:15Z

This might be nothing but this allowed me to see something I haven't seen before. After some time of this non diffusion, the noise starts to create kinda a distinct line horizontally and vertically which slices the image in quadrants. Ill make a mental note to understand why a uniform pattern is emerging and if its caused by noise or some other image process.

kabachuha · 2022-11-08T21:40:08Z

Now i see it. It basically just zooms in and adds noise if Im looking at it correctly.

yep. It stops diffusing for some unknown reason. Whether it's because something resets after the iteration or whatever...

Funofabot · 2022-11-08T23:18:31Z

Happens when this statement is printed.

Funofabot · 2022-11-08T23:52:13Z

I see comments about the new inpainting. Now I am aware of why you want help with this because it is indeed very confusing. The mask area changes with the animation and it is not consistent. When zooming out, the first frame reduces size and by the third frame the noise is actually complete vertical of the centered image and not symmetric in one axis.

Funofabot · 2022-11-09T02:07:28Z

There exists the same problem in a branch that I am working on with masking. After reading your notes about changes in inpainting, its starting to come together.
Im active looking into this.

Funofabot

Propose a change in ln:194 as commented. Its not a fix.
New image areas are not diffused correctly still.
With the standard OOTB settings, this generates. However this is broken where the sample is no longer able to fill the output, meaning when animating translations that produce out of camera frames.

Funofabot · 2022-11-09T02:27:09Z

scripts/deforum/generate.py

+
+            processed = processing.process_images(p)
+
+            init_image = processed.images[0].convert('RGB') 


I am working locally of this branch. When removing .conver('RGB') so that LN: 194 is
init_image = processed.images[0]
The problem with the standard settings where it stops diffusing. (THIS IS NOT A FIX its an observation).
There still exists problems now when we rotate and novel space is introduced. This seems to address only the image not diffusing in the out of the box settings.

Funofabot · 2022-11-09T03:03:55Z

scripts/deforum/generate.py

+            p.image_mask = None
+            p.inpainting_fill = 0
+            mask_image = None
+            processed = None


This is where it is passed to the masking portion
I believe that if processed is None then it executes line LN:243
if processed is None:

then we are now changing things with different variables.
I am assuming that this is the second pass you referred to.

LN:254 should reflect the zeros fill
p.inpainting_fill = args.zeros_fill_mode
not args.fill as I originally added to the init tab.

Am I wrong about that?
If I am correct, then we need to move the ui args and unify them.

Funofabot

We are using a init sample as the mask, the way that we are processing that is masking diffusion. I realize that the idea is to make the inpainting work when novel image area is present.
There are also coloring issues. However the diffusion will continue if we do not essentially mask all of it.
Remember that the noise that we see added, it is only added because add_noise has no idea of masks. Yet.

Funofabot

Last comments and review in discovery process before working on final solution.

Funofabot · 2022-11-10T02:47:34Z

scripts/deforum/render.py

@@ -160,10 +176,13 @@ def render_animation(args, anim_args, animation_prompts, root):

        # apply transforms to previous frame
        if prev_sample is not None:
+            prev_img_cv2 = sample_to_cv2(prev_sample)
+            # anti-blur
+            prev_img_cv2 = unsharp_mask(prev_img_cv2, (kernel, kernel), sigma, amount, threshold)


Is this only applied in 3d?

no, should be both in 2d and 3d

Funofabot · 2022-11-10T02:53:20Z

scripts/deforum/generate.py

+                if mask_image.getpixel((x,y)) < 1:
+                    mask_image.putpixel((x,y), 255 )
+                else:
+                    mask_image.putpixel((x,y), 0 )


I dont completely understand why, but this is sort of completely masking diffusion. I thought that white was complete masked and the code made sense in those terms before. This is indeed one location where diffusion stops.

This mask is supposed to be only for the zeroes inpainting pass. Then the mask is reset, so the regular diffusion should go fine after that

so when we get a mask from a noised sample then we cannot expect to get the actual mask we wanted if we wanted things with zeros.

that is part of the problem.
need to clear p.mask after processing to make it diffuse after this step is processed.

I pulled that nested loop apart enough over last couple of days to realize it almost never had 0's and I nailed it down to noise added. Perhaps sending an argument of the clean sample would make it work. However I think its required to generate an explicit mask when transforming the image and then supplying that mask in args before generate is called on. Then we could use that mask.

test.mp4

I got it working. This is what happens translating x by 10 that is the only motion setting.

One side effect is anything black is noised with "Latent Noise" if you use that setting.
That why I think a transformation mask makes sense. Looks cool.
At first I screamed that its making Klingon words.

Funofabot · 2022-12-13T00:09:56Z

Bringing this back up. I have thought about how to generate the mask for this inpainting pass. The masks are generated directly from the transform_image_3d(...) function
with this code

warp_mask = torch.ones_like(image_tensor) * 255
warp_mask_image = torch.nn.functional.grid_sample(
    warp_mask.add(1/512 - 0.0001).unsqueeze(0), 
    offset_coords_2d, 
    mode="nearest", 
    padding_mode="zeros", 
    align_corners=False
)

result_mask = rearrange(
    warp_mask_image.squeeze().clamp(0,255), 
    'c h w -> h w c'
).cpu().numpy().astype(prev_img_cv2.dtype)

then I modified the return of the function to also return the mask.

So this mask is generated along with the frame transformation.

Funofabot · 2022-12-13T00:12:39Z

curious things happen however. For example when looking at the generated masks, in this example rotation_3d_z is the only option used in the animation, and is set to 1.

frame 35

frame 36

Funofabot · 2022-12-13T00:16:48Z

produces images without border stripes. Still not a great example.

Funofabot · 2022-12-13T04:39:54Z

x translation only

Funofabot · 2022-12-14T03:31:37Z

For the next progress, this should reverse the mask so that the newly created areas are not over converged. Ill try to work that out.

Funofabot · 2022-12-15T05:16:39Z

My conclusion is that the frame warping in 3d causes artefacts that do not match the intended results. Much of the warping causes distortions that do not match the settings in this version. A new way is required. That is my conclusion.

Funofabot · 2022-12-19T04:49:35Z

This looks promising for a fix for stripes. Lets keep the momentum on this.

kabachuha · 2022-12-19T10:39:38Z

@Funofabot That looks really promising! I'll make some tests with the stuff as soon as I'll have time for this. Did you make any additional changes except for the mask part?

Bringing this back up. I have thought about how to generate the mask for this inpainting pass. The masks are generated directly from the transform_image_3d(...) function with this code
warp_mask = torch.ones_like(image_tensor) * 255
warp_mask_image = torch.nn.functional.grid_sample(
    warp_mask.add(1/512 - 0.0001).unsqueeze(0), 
    offset_coords_2d, 
    mode="nearest", 
    padding_mode="zeros", 
    align_corners=False
)

result_mask = rearrange(
    warp_mask_image.squeeze().clamp(0,255), 
    'c h w -> h w c'
).cpu().numpy().astype(prev_img_cv2.dtype)
then I modified the return of the function to also return the mask.

So this mask is generated along with the frame transformation.

using unsharp mask, from #74

QoL-only features from #74: blank frames reroll, anti-blur

from #74 sets border = 'smart' by default

kabachuha · 2022-12-27T20:24:33Z

Closing in favor for #176

kabachuha added enhancement New feature or request UI labels Nov 6, 2022

kabachuha added 14 commits November 7, 2022 20:04

wip border stripes fix

93e835d

fixes

f6b2cf3

fix border mode

5fe543c

name fixup

48e0b44

troubleshooting

76e6341

remove debug for zeros mask

03ea5bf

anti-blur

7bc1bc1

name fixup

0089d19

kernel size to int

631e6e3

reset mask after zeros inpainting

9c89ff9

zeroes mask smallness check

f402cc4

name fixup

656bf95

add blank frames reroll

293d1cb

correct tqdm when inpainting pass is skipped

386e16d

kabachuha force-pushed the fix-stripes branch from 4963ea7 to 386e16d Compare November 7, 2022 17:04

fix image not diffusion after zeroes inpaint

3719388

kabachuha force-pushed the fix-stripes branch from 8cfe24f to 15a13f6 Compare November 7, 2022 18:06

new settings load fix

911b2e5

fixup

kabachuha force-pushed the fix-stripes branch from 15a13f6 to 911b2e5 Compare November 7, 2022 18:13

fixup settings loading

6a06a47

fixup

kabachuha force-pushed the fix-stripes branch from d426e63 to 6a06a47 Compare November 7, 2022 19:32

fixup

0243e1f

fixup

kabachuha force-pushed the fix-stripes branch from 2a804aa to 0243e1f Compare November 7, 2022 19:46

kabachuha added 4 commits November 7, 2022 22:50

fixup

c7fd009

fixup fixup

Merge branch 'fix-stripes' of https://github.com/deforum-art/deforum-…

993cdd8

…for-automatic1111-webui into fix-stripes

fixup model reset for zeros inpaint

d664bc8

blank frames reroll model fix

8280b82

Funofabot reviewed Nov 9, 2022

View reviewed changes

Funofabot requested changes Nov 10, 2022

View reviewed changes

Funofabot reviewed Nov 10, 2022

View reviewed changes

kabachuha mentioned this pull request Nov 29, 2022

Use Inpainting for Consistency on Single Prompt Animations #120

Open

kabachuha added a commit that referenced this pull request Dec 19, 2022

anti-blur

78ed389

using unsharp mask, from #74

kabachuha mentioned this pull request Dec 19, 2022

QoL-only features from #74: blank frames reroll, anti-blur #159

Merged

kabachuha added a commit that referenced this pull request Dec 22, 2022

Merge pull request #159 from deforum-art/QOL-only-from-pr74

5aa9f38

QoL-only features from #74: blank frames reroll, anti-blur

kabachuha added a commit that referenced this pull request Dec 27, 2022

wip border stripes fix

267c0e6

from #74 sets border = 'smart' by default

kabachuha mentioned this pull request Dec 27, 2022

Smart border inpainting mode #176

Closed

kabachuha closed this Dec 27, 2022

stride-srl pushed a commit to stride-srl/sd-webui-deforum that referenced this pull request Jun 26, 2023

Parallel Downloads (deforum-art#74)

17b01a7


		processed = processing.process_images(p)

		init_image = processed.images[0].convert('RGB')

Uh oh!

[WIP] Second inpainting pass to extrapolate padding pixels smarter #74

[WIP] Second inpainting pass to extrapolate padding pixels smarter #74

Uh oh!

Conversation

kabachuha commented Nov 5, 2022

Uh oh!

kabachuha commented Nov 8, 2022

Uh oh!

kabachuha commented Nov 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Funofabot commented Nov 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kabachuha commented Nov 8, 2022

Uh oh!

kabachuha commented Nov 8, 2022

Uh oh!

Funofabot commented Nov 8, 2022

Uh oh!

Funofabot commented Nov 8, 2022

Uh oh!

kabachuha commented Nov 8, 2022

Uh oh!

Funofabot commented Nov 8, 2022

Uh oh!

Funofabot commented Nov 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Funofabot commented Nov 9, 2022

Uh oh!

Funofabot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Funofabot Nov 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Funofabot Nov 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Funofabot left a comment

Choose a reason for hiding this comment

Uh oh!

Funofabot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Funofabot Nov 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Funofabot Nov 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Funofabot commented Dec 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Funofabot commented Dec 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

kabachuha commented Nov 8, 2022 •

edited

Loading

Funofabot commented Nov 8, 2022 •

edited

Loading

Funofabot commented Nov 8, 2022 •

edited

Loading

Funofabot left a comment •

edited

Loading

Funofabot Nov 9, 2022 •

edited

Loading

Funofabot Nov 9, 2022 •

edited

Loading

Funofabot Nov 10, 2022 •

edited

Loading

Funofabot Nov 11, 2022 •

edited

Loading

Funofabot commented Dec 13, 2022 •

edited

Loading

Funofabot commented Dec 13, 2022 •

edited

Loading