[Bug]: API调用插件v2v，controlnet is enabled but no input image is given

[Chi-yuki]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

Have you read FAQ on README?

• I have updated WebUI and this extension to the latest version

What happened?

我通过API调用插件，进行v2v，controlnet不能正常工作

Steps to reproduce the problem

1. 启动sd webui
2. 通过以下代码调用api

import requests
import json

url = "http://127.0.0.1:7890/sdapi/v1/txt2img"

payload = {
"alwayson_scripts": {
"AnimateDiff": {
"args": [
{
"model": "mm_sd15_v3.safetensors",
"format": ["PNG", "MP4"],
"enable": True,
"video_length": 1,
"fps": 1,
"closed_loop": "N",
"batch_size": 1,
"video_path": "D:/frame",
"mask_path": "D:/mask"
}
]
},
"controlnet": {
"args": [
{
"enabled": True,
"model": "control_v11p_sd15_inpaint [ebff9138]",
"module": "inpaint_only",
"pixel_perfect": True,
"weight": 1
}
]
}
},
"batch_size": 8,
"cfg_scale": 7,
"denoising_strength": 0.7,
"height": 1280,
"prompt": "beautiful_lighting,1girl,",
"sampler_name" : "Euler a",
"seed" : -1,
"steps" : 1,
"width" : 720
}

response = requests.post(url, json=payload)

What should have happened?

正常通过controlnet控制生成动画

Commit where the problem happens

webui: 版本: v1.8.0  •  python: 3.10.6  •  torch: 2.1.2+cu121  •  xformers: 0.0.23.post1  •  gradio: 3.41.2  •  checkpoint: 86618977ac

extension: master | a81565d

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

set COMMANDLINE_ARGS=--port 7890 ^
--listen ^
--enable-insecure-extension-access ^
--xformers ^
--api

Console logs

2024-04-10 11:22:44,162 - AnimateDiff - INFO - AnimateDiff process start.
2024-04-10 11:22:44,162 - AnimateDiff - INFO - Loading motion module mm_sd15_v3.safetensors from D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\model\mm_sd15_v3.safetensors
2024-04-10 11:22:44,177 - AnimateDiff - INFO - Guessed mm_sd15_v3.safetensors architecture: MotionModuleType.AnimateDiffV3
2024-04-10 11:22:45,880 - AnimateDiff - INFO - Injecting motion module mm_sd15_v3.safetensors into SD1.5 UNet input blocks.
2024-04-10 11:22:45,880 - AnimateDiff - INFO - Injecting motion module mm_sd15_v3.safetensors into SD1.5 UNet output blocks.
2024-04-10 11:22:45,880 - AnimateDiff - INFO - Setting DDIM alpha.
2024-04-10 11:22:45,880 - AnimateDiff - INFO - Injection finished.
*** Error running before_process: D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 776, in before_process
        script.before_process(p, *script_args)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 65, in before_process
        params.set_p(p)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_ui.py", line 172, in set_p
        cn_units = get_controlnet_units(p)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_utils.py", line 66, in get_controlnet_units
        cn_unit_dataclass.batch_images = cn_unit_dict.get("batch_images", None)
    AttributeError: 'UiControlNetUnit' object has no attribute 'get'

---
Warning: field infotext in API payload not found in <modules.processing.StableDiffusionProcessingTxt2Img object at 0x0000023970F31000>.
2024-04-10 11:22:45,896 - ControlNet - INFO - unit_separate = False, style_align = False
2024-04-10 11:22:46,209 - ControlNet - INFO - Loading model: control_v11p_sd15_inpaint [ebff9138]
2024-04-10 11:22:46,427 - ControlNet - INFO - Loaded state_dict from [D:\stable-diffusion-webui\stable-diffusion-webui\models\ControlNet\control_v11p_sd15_inpaint.pth]
2024-04-10 11:22:46,427 - ControlNet - INFO - controlnet_default_config
2024-04-10 11:22:47,584 - ControlNet - INFO - ControlNet model control_v11p_sd15_inpaint [ebff9138](ControlModelType.ControlNet) loaded.
*** Error running process: D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 784, in process
        script.process(p, *script_args)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1245, in process
        self.controlnet_hack(p)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1230, in controlnet_hack
        self.controlnet_main_entry(p)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 916, in controlnet_main_entry
        input_image, resize_mode = Script.choose_input_image(p, unit, idx)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 700, in choose_input_image
        raise ValueError("controlnet is enabled but no input image is given")
    ValueError: controlnet is enabled but no input image is given

---
  0%|                                                                                            | 0/1 [00:00<?, ?it/s]*** Error executing callback cfg_denoiser_callback for D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\script_callbacks.py", line 230, in cfg_denoiser_callback
        c.callback(params)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_infv2v.py", line 102, in animatediff_on_cfg_denoiser
        ad_params.text_cond = ad_params.prompt_scheduler.multi_cond(cfg_params.text_cond, prompt_closed_loop)
    AttributeError: 'NoneType' object has no attribute 'multi_cond'

---
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.97s/it]
*** Error running postprocess_batch_list: D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 832, in postprocess_batch_list
        script.postprocess_batch_list(p, pp, *script_args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 84, in postprocess_batch_list
        params.prompt_scheduler.save_infotext_img(p)
    AttributeError: 'NoneType' object has no attribute 'save_infotext_img'

---
*** Error running postprocess: D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 816, in postprocess
        script.postprocess(p, processed, *script_args)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 94, in postprocess
        params.prompt_scheduler.save_infotext_txt(res)
    AttributeError: 'NoneType' object has no attribute 'save_infotext_txt'

---
Total progress: 100%|████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.45it/s]
Total progress: 100%|████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.45it/s]

Additional information

No response

[Chi-yuki]

当我使用 stable-diffusion-webui-forge 时，仍然报错：

webui版本

版本: f0.0.17v1.8.0rc-latest-276-g29be1da7  •  python: 3.10.6  •  torch: 2.1.2+cu121  •  xformers: 0.0.23.post1  •  gradio: 3.41.2  •  checkpoint: 86618977ac

animatediff版本

forge/master | b20f751

调用代码

import requests
import json

url = "http://127.0.0.1:7890/sdapi/v1/txt2img"

payload = {
    "alwayson_scripts": {
    "AnimateDiff": {
      "args": [
        {
          "model": "mm_sd15_v3.safetensors",
          "format": ["PNG", "MP4"],
          "enable": True,
          "video_length": 2,
          "fps": 2,
          "closed_loop": "N",
          "batch_size": 2,
          "video_path": "D:\\frame",
          "mask_path": "D:\\mask"
        }
      ]
    },
    "controlnet": {
      "args": [
        {
            "control_mode": "Balanced", 
            "enabled": True, 
            "generated_image": None, 
            "guidance_end": 1, 
            "guidance_start": 0, 
            "hr_option": "Both", 
            "image": None, 
            "input_mode": "batch", 
            "model": "control_v11p_sd15_inpaint [ebff9138]", 
            "module": "inpaint_only", 
            "pixel_perfect": True, 
            "processor_res": 0.5, 
            "resize_mode": "Crop and Resize", 
            "save_detected_map": True, 
            "threshold_a": 0.5, 
            "threshold_b": 0.5, 
            "use_preview_as_input": False, 
            "weight": 1
        }
      ]
      }
    },
    "batch_size": 8,
    "cfg_scale": 7,
    "denoising_strength": 0.7,
    "height": 1280,
    "prompt": "beautiful_lighting,1girl,",
    "sampler_name" : "Euler a",
    "seed" : -1,
    "steps" : 1,
    "width" : 720
}

response = requests.post(url, json=payload)

Console logs

2024-04-10 21:12:26,805 - AnimateDiff - INFO - AnimateDiff process start.
*** Error running before_process: D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 795, in before_process
        script.before_process(p, *script_args)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 63, in before_process
        params.set_p(p)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_ui.py", line 161, in set_p
        cn_units = get_controlnet_units(p)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_utils.py", line 62, in get_controlnet_units
        cn_units_dataclass = [ControlNetUnit.from_dict(cn_unit_dict) for cn_unit_dict in cn_units]
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_utils.py", line 62, in <listcomp>
        cn_units_dataclass = [ControlNetUnit.from_dict(cn_unit_dict) for cn_unit_dict in cn_units]
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\lib_controlnet\external_code.py", line 256, in from_dict
        **{k: v for k, v in d.items() if k in vars(ControlNetUnit)}
    AttributeError: 'ControlNetUnit' object has no attribute 'items'

---
Warning: field infotext in API payload not found in <modules.processing.StableDiffusionProcessingTxt2Img object at 0x000001FE09ADDDE0>.
2024-04-10 21:12:26,812 - ControlNet - INFO - ControlNet Input Mode: batch
*** Error running process: D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 803, in process
        script.process(p, *script_args)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 548, in process
        self.process_unit_after_click_generate(p, unit, params, *args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 298, in process_unit_after_click_generate
        input_list, resize_mode = self.get_input_data(p, unit, preprocessor, h, w)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 203, in get_input_data
        resize_mode = external_code.resize_mode_from_value(p.resize_mode)
    AttributeError: 'StableDiffusionProcessingTxt2Img' object has no attribute 'resize_mode'

---
2024-04-10 21:12:27,105 - AnimateDiff - INFO - Setting DDIM alpha.
2024-04-10 21:12:27,106 - AnimateDiff - INFO - Injecting motion module mm_sd15_v3.safetensors into SD1.5 UNet.
*** Error running process_before_every_sampling: D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 835, in process_before_every_sampling
        script.process_before_every_sampling(p, *script_args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 555, in process_before_every_sampling
        self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
    KeyError: 0

---
To load target model BaseModel
To load target model MotionWrapper
Begin to load 2 models
Reuse 1 loaded models
Reuse 1 loaded models
[Memory Management] Current Free GPU Memory (MB) =  19894.94873046875
[Memory Management] Model Memory (MB) =  0.0
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  18870.94873046875
[Memory Management] Current Free GPU Memory (MB) =  19894.94873046875
[Memory Management] Model Memory (MB) =  0.0
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  18870.94873046875
Moving model(s) has taken 0.01 seconds
  0%|                                                                                            | 0/1 [00:00<?, ?it/s]*** Error executing callback cfg_denoiser_callback for D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\script_callbacks.py", line 233, in cfg_denoiser_callback
        c.callback(params)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_infv2v.py", line 89, in animatediff_on_cfg_denoiser
        ad_params.text_cond = ad_params.prompt_scheduler.multi_cond(cfg_params.text_cond, prompt_closed_loop)
    AttributeError: 'NoneType' object has no attribute 'multi_cond'

---
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.18it/s]
*** Error running postprocess_batch_list: D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 859, in postprocess_batch_list
        script.postprocess_batch_list(p, pp, *script_args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 85, in postprocess_batch_list
        params.prompt_scheduler.save_infotext_img(p)
    AttributeError: 'NoneType' object has no attribute 'save_infotext_img'

---
*** Error running postprocess_batch_list: D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 859, in postprocess_batch_list
        script.postprocess_batch_list(p, pp, *script_args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 561, in postprocess_batch_list
        self.process_unit_after_every_sampling(p, unit, self.current_params[i], pp, *args, **kwargs)
    KeyError: 0

---
*** Error running postprocess: D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "D:\stable-diffusion-webui\stable-diffusion-webui\modules\scripts.py", line 843, in postprocess
        script.postprocess(p, processed, *script_args)
      File "D:\stable-diffusion-webui\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 90, in postprocess
        params.prompt_scheduler.save_infotext_txt(res)
    AttributeError: 'NoneType' object has no attribute 'save_infotext_txt'

---
Total progress: 100%|████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.58it/s]
Total progress: 100%|████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.59it/s]

forge/master b20f7519

[yuchen1984]

This PR should fix the problem
#480