Could you please tell me what kind of prompt words are needed to achieve the painterly image harmonization effects shown in the supplementary material of the paper, and where to adjust the diffusion steps/duration?

[thenotshyshyshy]

[WindVChen]

Hi @thenotshyshyshy ,

The prompt is the same as shown in the figure (e.g., Cat Photo → Car Painting).
If you want to achieve painterly image harmonization, you need to relax the structural constraints introduced in the Content Structure Preservation module of our paper.

First, disable the Edge Detection Constraint, which corresponds to the loss defined here:

Diff-Harmonization/diff_harmon.py

Line 539 in bfbfa1b

loss_edge_all = 0.1 * loss_func(out_edge * original_mask, original_edge * original_mask) + loss_func(

Then, disable the self-attention map replacement during the early (e.g., first several or first half) diffusion denoising steps. The replacement occurs in this line:

Diff-Harmonization/diff_harmon.py

Line 578 in bfbfa1b

latents = diffusion_step(model, controller, latents, context[ind], t, guidance_scale)

more specifically, the forward-pass attention is overridden as defined here:

Diff-Harmonization/diff_harmon.py

Line 156 in bfbfa1b

attn = controller(attn, is_cross, place_in_unet)

After making these changes, it should work for painterly harmonization, though you may still need some light parameter tuning to reproduce the visual results shown in the paper.

[thenotshyshyshy]

Hello, I have set loss_edge_all to 0 and replaced the line with

latents = diffusion_step(model, EmptyControl(), latents, context[ind], t, guidance_scale)

Is this modification correct?

[WindVChen]

Hi @thenotshyshyshy ,

Sorry for the delayed reply. For loss_edge_all, yes—that part is correct. But for the latter issue, it’s not. Note that the controller is already injected into the network during inference through this line of code:

Diff-Harmonization/diff_harmon.py

Line 564 in bfbfa1b

register_attention_control(model, controller)

where the original attention computation is changed (self-attention maps are replaced).

What you likely need to do (I don’t fully remember the exact implementation since it’s been a while) is delay the replacement of the attention computation by the controller, meaning you should call register_attention_control() only after a few denoising steps. In theory, that should work.

Hope this helps.

[thenotshyshyshy]

thanks