Attribute Error when generating samples for HyImage 2.1

[Sarania]

Originally this was about my CPU Text encoder issue but I resolved that one below. The other questions are still relevant though! It just runs on a single thread whereas in the inference script it works as expected. I can get it running with --fp8_vl instead but... oof >_<. I also have the same problem as with Musubi where if I don't specify --max_data_loader_n_workers 4 it will get stuck waiting on some future that never returns.

Lastly this is just me I guess but I can't for the life of me figure out the correct way to specify optimizer args on the command line, I always get some variation of:

Traceback (most recent call last):
File "/home/blyss/projects/sd-scripts/hunyuan_image_train_network.py", line 709, in
trainer.train(args)
File "/home/blyss/projects/sd-scripts/train_network.py", line 767, in train
optimizer_name, optimizer_args, optimizer = train_util.get_optimizer(args, trainable_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/blyss/projects/sd-scripts/library/train_util.py", line 4899, in get_optimizer
key, value = arg.split("=")
^^^^^^^^^^
ValueError: too many values to unpack (expected 2)

Ways I've tried:
--optimizer_args "weight_decay=0.01 eps=(1e-30,1e-16) betas=(0.9,0.999,0.9999)"
--optimizer_args "weight_decay=0.01, eps=(1e-30,1e-16), betas=(0.9,0.999,0.9999)"
--optimizer_args "weight_decay=0.01,eps=(1e-30,1e-16),betas=(0.9,0.999,0.9999)"
--optimizer_args "[weight_decay=0.01, eps=(1e-30,1e-16), betas=(0.9,0.999,0.9999)]"

etc I just... am about to hard code it XD

Edit: See I did use came_pytorch.CAME | {'weight_decay': 0.01, 'eps': (1e-30, 1e-16), 'betas': (0.9, 0.999, 0.9999)} WITNESS MY DEFAULT KWARGS 😂

[Sarania]

Oh also, the args we added at the end for apg_start_step_ocr, apg_start_step_general, guidance_rescale_apg and guidance_rescale cause attribute errors when the trainer tries to call the inference script for samples. Prolly just needs defaults added into the args that are passed to generate_body. I'm not as familiar with sdscripts as Musubi or I'd just make a PR.

[Sarania]

Update: I fixed the core issue that caused me to create this issue by doing

accelerate launch --num_cpu_threads_per_process 16 ...

Maybe this is just expected behavior, but if it's set to the default of 1 it will restrict the CPU TE to that which is... oof XD. I'm gonna leave this issue open though because the second issue with samples remains and I'm still curious the proper way to specify optimizer args without resorting to just coding it in.

[kohya-ss]

Thank you for reporting the issue! I've fixed in #2212 , I would appreciate your confirmation.

Also, thanks for reporting --num_cpu_threads_per_process. That's a great finding. Until now, there has been almost no code in this repository that performs multithreading, but it seems that calls to LLM do. I'll update the documentation.

Here's how to specify multiple arguments: --optimizer_args "weight_decay=0.01" "eps=(1e-30,1e-16)" "betas=(0.9,0.999,0.9999)".

Since --optimizer_args is an option that takes multiple arguments, you need to enclose each argument in " just to be safe.

[Sarania]

Thank you for explaining the args I was just really lost on that lol.

And yeah when running models on CPU, as long as they are in torch.float32 or cpu autocast is enabled, then they will use heavy multithreading when processing the workload. This is the only reason it's really feasible at all to run them that way. It's honestly not a bad way to work around VRAM limitations for text encoders, VAE and the like as long as you have sufficient cooling as my overclocked 8 core / 16 thread i7 6900k regularly tops 200Watts and 85C even on 270mm water cooling due to multithreaded AVX workloads. It's the type of thing that will prove or disprove your overclock stability for sure! But a single CPU text encode is like 10-15 seconds for me and I would much rather pay that price and maintain the native precision than have to quantize, especially for training where every little dink in quality can have an additive effect on the result and I can cache the outputs!

As far as the args they look good now, one other small thing I noticed and I'm not sure if it's expected or not is that if you don't specify a seed in your sample prompt with --d you'll hit an attrib error in generate_body on seed_g.manual_seed(seed). Maybe you're expected to just specify one and then it works fine but I wanted to mention it while we were here in case it's a small oversight.

Oh my Spyder also calls out the del tokenizer_batch, text_encoder_batch, temp_shared_models_txt, conds_cache_batch on line 1004 of minimal inference in process_batch_prompts for referencing before assignment, I haven't tested it but it should likely be del tokenizer_vlm, text_encoder_vlm_batch, tokenizer_byt5, text_encoder_byt5_batch, temp_shared_models_txt, conds_cache_batch 🙂

[kohya-ss]

Thank you for the additional information. I apologize for the multiple errors. I'll check and fix them as soon as I have time.

[Sarania]

No worries, no apology necessary, I just wanted to share what I'd found since I didn't get much time to test while it was still a PR!

[kohya-ss]

Sorry too late, I've fixed.