Image tab and T2I tab batch generations behave differently when enabling refiner pass?

[atomicstrawberry]

New user, migrating over from A1111 in the last couple of days so this might be a known issue but this took me forever to figure out and it is, in my opinion, pretty non-intuitive if it is intentional. Also not sure if, assuming this is not intentional, it is a ModernUI bug or an SD.Next bug. I was going to file this as an issue but decided to post a discussion first.

This is on the latest commit, Windows 11, extremely standard install that I haven't really done much tweaking with, no extensions, pretty much only thing I've touched settings-wise (at least until trying to troubleshoot this) was to direct SD.Next to pick up my models etc from my old A1111 install so I can still use both side by side if needed.

The UI starts on the "Images" workflow tab for SD models, so that was where I was also starting.

When you add a simple text prompt, turn on eg 3 batches, batch size 1. Set seed to random (-1) and generate, you will get three different images based on your prompt and seed, exactly like you would expect. No controlnets, no image sources or sketches or anything, no additional networks, just raw simple text prompt to image.

Enable the refiner pass, upscale to 2x with any upscaler and force hires. Set Refiner start to 0, refiner steps to 0 (though this does not seem to make a difference for the issue). You will get three images, but they will be almost identical, which is really not what you expect especially since this isn't how it works with the refiner off. Inspecting them, each will have gotten a new seed applied. If you are using wildcards in your prompt, the prompt in the metadata will reflect that, but the images themselves are essentially identical, but not quite.

Saving out the "before hires fix" image shows something more curious - the before image for the second and third generation in the batch is already upscaled, and while the first pre-hires image saved out is tagged with pipeline "StableDiffusionXLPipeline" all subsequent images will have changed to StableDiffusionXLImg2ImgPipeline instead. I didn't even notice this initially because I assumed that the refiner & hires pass would be img2img normally since that's what's happening under the covers. Plus looking at my logs I still see it doing a 30-step generation and then a 10-step hires pass on each one.

Basically, I assume it is feeding each image of the batch back into itself via img2img, rather than doing what you would expect it to do based on the behavior without refiner pass on, which is just doing a standard text to image generation with a fresh / fixed seed a number of times in a row, applying hires each time.

If you actually go down to the T2I tab, copy prompt and all settings over, enable a batch of images, enable refiner pass, everything will work exactly the way you would expect it to, which is largely the way it worked in A1111 and family as well, you'll get three different images that have been upscaled and hiresed, same as you would with hires.fix, albeit sadly without the ability to do the hires pass with a different model.

I'm not sure if this is actually an issue or not, but in my opinion if it is a deliberate choice the behavior is unexpected enough that it should probably be documented somewhere. Happy to file an issue for this if it sounds like an actual bug, would need guidance on whether it would be a SD.Next bug or a ModernUI bug as well.

[vladmandic]

thanks for the detailed write-up it helps. i'll investigate and update here.

[vladmandic]

prompt:
batch-count=2
refine:
enable refine pass: yes
resize mode: fixed
resize method: resize lanczos
resize scale: 1.2
force hires: yes
refiner start: 0
refiner steps: 0

i cannot reproduce the problem, what am i missing?

and while the first pre-hires image saved out is tagged with pipeline "StableDiffusionXLPipeline" all subsequent images will have changed to StableDiffusionXLImg2ImgPipeline instead.

this is fine, expected. first hires caused a switch to StableDiffusionXLImg2ImgPipeline. for following images, sdnext simply avoided switching back to StableDiffusionXLPipeline pipeline before each image since there was no operation to do before it would have to switch back to StableDiffusionXLImg2ImgPipeline anyhow.

Plus looking at my logs I still see it doing a 30-step generation and then a 10-step hires pass on each one.

refine steps=0 doesn't mean 0-steps, it means use same steps as base generate, but adjusted for denoise strength.

[atomicstrawberry]

The pipeline part may be a red herring, I am not sure. I tried it again after restarting everything, issue still reproduces but everything out of the unified image creation tab is img2img apparently so that might be normal after all and I may have seen a symptom of fiddling trying to solve the issue.

Starting it in Unified interface is where it behaves differently for me, and only in there. It's entirely possible also that maybe this is some kind of weird deep settings issue on my end.

I was originally doing this with refine steps = 20, the default, and seeing same issue so 0 vs 20 should not be relevant (also assuming refine start = 0 means no refiner anyway? I definitely don't see it being used).

Here's some examples I just generated

No refiner pass:

Refiner pass enabled:

In this case it clearly added detail the second time around but you can see that it's recycled the image back through. My expectation would be that if you asked for 2 batches, it would do:

generate latent noise based off seed -> initial gen steps -> refiner steps -> upscale -> hires steps -> vae

For each generation, but instead it seems like it might be doing

generate latent noise based off seed -> initial gen steps -> refiner steps -> upscale -> hires steps -> vae -> add noise -> gen steps -> refiner steps -> upscale -> hires steps -> vae (save)

i.e. instead of generating a fresh initial latent noise, it is taking the result of the previous gen of the batch as input each time. It's hard to be sure but that looks like what it's doing. I didn't have it set to output the result before refiner & hires when doing this, sorry - I can run again if that would help narrow down what might be happening.

Here's the log file, I only did two batches, one batch of 2 with refiner pass enabled, and one batch of 2 without - results are what I attached up thread.
sdnext.log

[atomicstrawberry]

Actually just noticed in the log that there's local commits:

D:\sd.next>git diff diff --git a/extensions-builtin/sdnext-modernui b/extensions-builtin/sdnext-modernui index 60b3ea8b1..73a6cbd24 160000 --- a/extensions-builtin/sdnext-modernui +++ b/extensions-builtin/sdnext-modernui @@ -1 +1 @@ -Subproject commit 60b3ea8b13fc7a3aa101add11c8e18b2326399d9 +Subproject commit 73a6cbd24d584a9a18790a85ee669c89ceb45ec6

I wonder if somehow I managed to get an updated UI and have pulled in a ModernUI bug?

[vladmandic]

Actually just noticed in the log that there's local commits:

i doubt this has anything to do with modernui, this feels like a core thing. but please confim after updating modernui.

I was originally doing this with refine steps = 20, the default, and seeing same issue so 0 vs 20 should not be relevant (also assuming refine start = 0 means no refiner anyway? I definitely don't see it being used).

not relevant for this, but refiner start 0 or 1 means no refiner, so original model is going to be used.
in which case refiner steps sets steps for hires pass if its larger than 0 and if its 0 it will use base steps modified by denoising strength.

generate latent noise based off seed -> initial gen steps -> refiner steps -> upscale -> hires steps -> vae

this would happen if you're using batch-size.
if you're using batch-count, then entire sequence would be repeated for each batch entry.

anyhow, i cant reproduce using the steps i wrote down - did i miss anything?
you posted the issue is still here, but did not confirm the steps i wrote.
and please write down your EXACT steps - from clean server start to end.

[atomicstrawberry]

Clean server start -> add prompt in default window, set batch count = 2, set sampler to DPM++ 2M SDE, hit generate.

Result is top two images.

Go to refiner tab, enable it, set resize scale to 1.5x, enabled force hires, set hires steps to 10, hit generate (forgot to set a resize method, I guess None still worked), hit generate

Result is bottom two.

Log attached was from startup to the end of this sequence.

this would happen if you're using batch-size.
if you're using batch-count, then entire sequence would be repeated for each batch entry.

Really? I was under the impression that batch size generates multiple images at the same time, but batch count is the number of iterations you would do, i.e. size = 2, count = 10 would give you 20 images, generating 2 at a time, and size = 10, count = 2 would also give you 20 images but it would do 10 at a time? Maybe we're saying the same thing a different way.

Anyway, since I noticed you also pushed new commits from dev to master earlier today, what I am going to do later on this evening when I have time is to create a completely fresh, virgin enlistment and see if the problem still reproduces, in case maybe there is some weird configuration bug on my end at the source of this.

[vladmandic]

Maybe we're saying the same thing a different way.

yup.

virgin enlistment and see if the problem still reproduces,

ah, you're still on 10/18 release. yes, let me know.

[atomicstrawberry]

Fresh install, latest master, the problem no longer reproduces for me either.

I think that it must have been something stupid I did somewhere, but I've got no idea where or what it was I did.

I do notice now also that the image for the output after hires now also say they were generated by StableDiffusionXLPipeline not StableDiffusionXLImg2ImgPipeline, but I think seeing this before was probably a symptom of the problem.

Comparing the config.json from before and after, there's only a couple things I had changed on the old enlistment:

  "save_images_before_highres_fix": true,
  "sequential_seed": false,
  "keep_incomplete": false,
  "save_images_before_refiner": true
  "prompt_attention": "a1111",

Almost all those were things I set trying to debug the problem. None looked likely to have introduced whatever was going on, but I tried turning them on. Still can't reproduce the issue any more.

I'd say it must have been something weird going on at my end, or a bug that has already been fixed.

Thanks for looking into it.

[vladmandic]

none of those settings have impact on this issue.
related issue was resolved few weeks back, it also had impact on resolving this one.
off-topic, why use old/obsolete a1111 prompt parser?

[atomicstrawberry]

No particular reason, I was trying to reproduce something as close to A1111 as possible to compare side by side and had read the prompt attention works slightly differently in SD.Next vs A1111, and forgot to turn it back off again. Having tried it both ways the new parser I think produces better results so there's no reason to use the old one.

[klu1982]

I am posting now only because I want you to know you are not alone. I have exactly same issue.
Since I quickly found out that using T2I instead of Image tab avoid the issue, I just never bother to report it.
For me specifically, it only happened on October 17 release. September 15 or October 31 do not have this issue.

[vladmandic]

pls do report issues when you find them- that's how they get fixed