Skip to content

Commit 9b09a9b

Browse files
zml-aizml-ai
committed
update readme
1 parent b45f49c commit 9b09a9b

3 files changed

Lines changed: 60 additions & 105 deletions

File tree

‎README.md‎

Lines changed: 11 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,9 @@ This repo contains PyTorch model definitions, pre-trained weights and inference/
2525
> [**DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generation**](https://arxiv.org/abs/2403.08857) <br>
2626
2727
## 🔥🔥🔥 News!!
28+
* Jul 15, 2024: 🚀 HunYuanDiT and Shakker.Ai have jointly launched a fine-tuning event based on the HunYuanDiT 1.2 model. By publishing a lora or fine-tuned model based on HunYuanDiT, you can earn up to $230 bonus from Shakker.Ai. See [Shakker.Ai](https://www.shakker.ai/activitys/shaker-the-world-hunyuan) for more details.
2829
* Jul 15, 2024: :tada: Update ComfyUI to support standardized workflows and compatibility with weights from t2i module and Lora training for versions 1.1/1.2, as well as those trained by Kohya or the official script. See [ComfyUI](./comfyui-hydit) for details.
30+
* Jul 15, 2024: :zap: We offer Docker environments for CUDA 11/12, allowing you to bypass complex installations and play with a single click! See [dockers](#installation-guide-for-linux) for details.
2931
* Jul 08, 2024: :tada: HYDiT-v1.2 version is released. Please check [HunyuanDiT-v1.2](https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.2) and [Distillation-v1.2](https://huggingface.co/Tencent-Hunyuan/Distillation-v1.2) for more details.
3032
* Jul 03, 2024: :tada: Kohya-hydit version now available for v1.1 and v1.2 models, with GUI for inference. Official Kohya version is under review. See [kohya](./kohya_ss-hydit) for details.
3133
* Jun 27, 2024: :art: Hunyuan-Captioner is released, providing fine-grained caption for training data. See [mllm](./mllm) for details.
@@ -423,11 +425,13 @@ All models will be automatically downloaded. For more information about the mode
423425

424426
### Full-parameter Training
425427

426-
**Requirement:** The minimum requriment is a single GPU with at least 20GB memory, but we recommend to use a GPU with about 30 GB memory to avoid host memory offloading.
427-
Additionally, we encourage users to leverage the multiple GPUs across different nodes to speed up training on large datasets.
428+
**Requirement:**
429+
1. The minimum requriment is a single GPU with at least 20GB memory, but we recommend to use a GPU with about 30 GB memory to avoid host memory offloading.
430+
2. Additionally, we encourage users to leverage the multiple GPUs across different nodes to speed up training on large datasets.
431+
428432
**Notice:**
429-
(1)Personal users can also use the light-weight Kohya to finetune the model with about 16 GB memory. Currently, we are trying to further reduce the memory usage of our industry-level framework for personal users.
430-
(2) If you have enough GPU memory, please try to remove --cpu-offloading or --gradient-checkpointing for less time costs.
433+
1. Personal users can also use the light-weight Kohya to finetune the model with about 16 GB memory. Currently, we are trying to further reduce the memory usage of our industry-level framework for personal users.
434+
2. If you have enough GPU memory, please try to remove `--cpu-offloading` or `--gradient-checkpointing` for less time costs.
431435

432436
Specifically for distributed training, you have the flexibility to control **single-node** / **multi-node** training by adjusting parameters such as `--hostfile` and `--master_addr`. For more details, see [link](https://www.deepspeed.ai/getting-started/#resource-configuration-multi-node).
433437

@@ -660,60 +664,9 @@ More details can be found in [./comfyui-hydit](comfyui-hydit/README.md)
660664

661665
### Using Kohya
662666

663-
We provide several commands to quick start LoRA Training and DreamBooth Training with Kohya:
664-
665-
```shell
666-
# Download kohya_ss GUI
667-
git clone https://github.com/bmaltais/kohya_ss.git
668-
cd kohya_ss/
669-
670-
# Download sd-scripts training backend, use dev branch
671-
git clone -b dev https://github.com/kohya-ss/sd-scripts ./sd-scripts
672-
673-
# Move the costom GUI codes to the kohya_ss GUI, and replace files with the same name
674-
cp -Rf ${HunyuanDiT}/kohya_ss-hydit/* ./
675-
676-
# Download model weights as before or link the existing model folder to kohya_ss/models.
677-
python -m pip install "huggingface_hub[cli]"
678-
# If you want to download the full model, use the following command
679-
huggingface-cli download Tencent-Hunyuan/HunyuanDiT-v1.1 --local-dir ./models/HunyuanDiT-V1.1
680-
huggingface-cli download Tencent-Hunyuan/HunyuanDiT-V1.2 --local-dir ./models/HunyuanDiT-V1.2
681-
# Or, if you want to download the fp16 pruned model
682-
huggingface-cli download KBlueLeaf/HunYuanDiT-V1.1-fp16-pruned --local-dir ./models/HunyuanDiT-V1.1-fp16-pruned
683-
684-
# After the model is downloaded, you may need to modify the file name an make sure it follows the kohya standard format:
685-
# rename the file name in t2i/ folder as shown below:
686-
# HunyuanDiT-V1.2/t2i/
687-
# - model/ -> denoiser/
688-
# - clip_text_encoder/ -> clip/
689-
# - mt5/ -> mt5/
690-
# - sdxl-vae-fp16-fix/ -> vae/
691-
# Also you may need to move tokenizer/* into clip/ folder
692-
mv HunyuanDiT-V1.2/t2i/model/ HunyuanDiT-V1.2/t2i/denoiser/
693-
mv HunyuanDiT-V1.2/t2i/clip_text_encoder/ HunyuanDiT-V1.2/t2i/clip/
694-
mv HunyuanDiT-V1.2/t2i/mt5/ HunyuanDiT-V1.2/t2i/mt5/
695-
mv HunyuanDiT-V1.2/t2i/sdxl-vae-fp16-fix/ HunyuanDiT-V1.2/t2i/vae/
696-
mv HunyuanDiT-V1.2/t2i/tokenizer/* HunyuanDiT-V1.2/t2i/clip/
697-
698-
# Install some essential python Package,
699-
conda create -n hydit-kohya python=3.10.12
700-
conda activate hydit-kohya
701-
702-
# Install some essential packages, please make sure cuda environment is installed and python version is 3.10
703-
# For cuda 12:
704-
pip install torch==2.1.2 torchvision==0.16.2 xformers==0.0.23.post1
705-
# For cuda 11:
706-
pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 xformers==0.0.23.post1+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
707-
# For cpu offloading to save GPU memory, we recommend to install Deepspeed as follows:
708-
DS_BUILD_CPU_ADAM=1 pip install deepspeed==0.14.1
709-
710-
# Install other python package
711-
pip install -r hunyuan_requirements.txt
712-
713-
# Run the Kohya_ss UI launch command
714-
python kohya_gui.py
715-
```
716-
More details can be found in [Kohya_ss README](kohya_ss-hydit/README.md)
667+
We support custom codes for kohya_ss GUI, and sd-scripts training codes for HunyuanDiT.
668+
![dreambooth](kohya_ss-hydit/img/dreambooth.png)
669+
More details can be found in [./kohya_ss-hydit](kohya_ss-hydit/README.md)
717670

718671
### Using Previous versions
719672

‎comfyui-hydit/README.md‎

Lines changed: 48 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -77,62 +77,64 @@ python main.py --listen --port 80
7777

7878
### Standard workflow (Recommended)
7979

80-
1. Preparing Model Weights
81-
82-
Use the command below to download the file to the specified folder. If already downloaded, create a symbolic link to that folder.
83-
```shell
84-
# clip encoder
85-
wget -O ${ComfyUI}/models/clip/pytorch_model.bin https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.2/blob/main/t2i/clip_text_encoder/pytorch_model.bin
86-
# mt5
87-
mkdir ${ComfyUI}/models/t5
88-
wget -O ${ComfyUI}/models/t5/pytorch_model.bin https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.2/blob/main/t2i/mt5/pytorch_model.bin
89-
# vae
90-
wget -O ${ComfyUI}/models/vae/diffusion_pytorch_model.bin https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.2/blob/main/t2i/sdxl-vae-fp16-fix/diffusion_pytorch_model.bin
91-
# base model
92-
wget -O ${ComfyUI}/models/checkpoints/pytorch_model_ema.pt https://huggingface.co/Tencent-Hunyuan/HunyuanDiT-v1.2/blob/main/t2i/model/pytorch_model_ema.pt
93-
```
94-
Put module weights trained through Kohya or the official script in `${ComfyUI}/models/checkpoints/` to switch model weights in ComfyUI.
80+
1. Preparing Model Weights
81+
82+
Download the file to the specified folder using the command below. For additional download links, visit [doc](https://github.com/Tencent/HunyuanDiT?tab=readme-ov-file#-download-pretrained-models).
83+
```shell
84+
# (Optional) download pretrain-weight
85+
huggingface-cli download Tencent-Hunyuan/HunyuanDiT-v1.2 --local-dir ${HunyuanDiT}/ckpts
86+
# clip
87+
ln -s ${HunyuanDiT}/ckpts/t2i/clip_text_encoder/pytorch_model.bin ${ComfyUI}/models/clip/pytorch_model.bin
88+
# mt5
89+
mkdir ${ComfyUI}/models/t5
90+
ln -s ${HunyuanDiT}/ckpts/t2i/mt5/pytorch_model.bin ${ComfyUI}/models/t5/pytorch_model.bin
91+
# vae
92+
ln -s ${HunyuanDiT}/ckpts/t2i/sdxl-vae-fp16-fix/diffusion_pytorch_model.bin ${ComfyUI}/models/vae/diffusion_pytorch_model.bin
93+
# base model
94+
huggingface-cli download Tencent-Hunyuan/Distillation-v1.2 pytorch_model_distill.pt --local-dir ${ComfyUI}/models/checkpoints/
95+
```
96+
Put module weights trained through Kohya or the official script in `${ComfyUI}/models/checkpoints/` to switch model weights in ComfyUI.
9597

9698
2. Preparing LoRa Weights
9799

98-
```shell
99-
# Put LoRa weights trained by Kohya in ComfyUI/models/loras
100-
cp ${HunyuanDiT}/kohya_ss/outputs/last-step{xxxx}.safetensors ${ComfyUI}/models/loras
101-
102-
# (Optional) Put LoRa weights trained by official scripts in ComfyUI/models/loras
103-
python convert_hunyuan_to_coimfyui_lora.py \
104-
--lora_path ${HunyuanDiT}/log_EXP/001-lora_porcelain_ema_rank64/checkpoints/0000100.pt/adapter_model.safetensors \
105-
--save_lora_path ${ComfyUI}/models/loras/adapter_model_convert.safetensors
106-
107-
# update the `lora.py` file
108-
cp ${ComfyUI}/custom_nodes/comfyui-hydit/lora.py ${ComfyUI}/comfy/lora.py
109-
```
100+
```shell
101+
# Put LoRa weights trained by Kohya in ComfyUI/models/loras
102+
cp ${HunyuanDiT}/kohya_ss/outputs/last-step{xxxx}.safetensors ${ComfyUI}/models/loras
103+
104+
# (Optional) Put LoRa weights trained by official scripts in ComfyUI/models/loras
105+
python custom_nodes/comfyui-hydit/convert_hunyuan_to_comfyui_lora.py \
106+
--lora_path ${HunyuanDiT}/log_EXP/001-lora_porcelain_ema_rank64/checkpoints/0000100.pt/adapter_model.safetensors \
107+
--save_lora_path ${ComfyUI}/models/loras/adapter_model_convert.safetensors
108+
109+
# update the `lora.py` file
110+
cp ${ComfyUI}/custom_nodes/comfyui-hydit/lora.py ${ComfyUI}/comfy/lora.py
111+
```
110112

111113
### Diffusers Wrapper
112114
1. Preparing Model Weights
113115

114-
```shell
115-
python -m pip install "huggingface_hub[cli]"
116-
mkdir models/hunyuan
117-
huggingface-cli download Tencent-Hunyuan/HunyuanDiT-v1.2 --local-dir ./models/hunyuan/ckpts
118-
huggingface-cli download Tencent-Hunyuan/HunyuanDiT-v1.2 t2i/model/pytorch_model_ema.pt --local-dir ./models/hunyuan/ckpts/t2i/model
119-
```
116+
```shell
117+
python -m pip install "huggingface_hub[cli]"
118+
mkdir models/hunyuan
119+
huggingface-cli download Tencent-Hunyuan/HunyuanDiT-v1.2 --local-dir ./models/hunyuan/ckpts
120+
huggingface-cli download Tencent-Hunyuan/HunyuanDiT-v1.2 t2i/model/pytorch_model_ema.pt --local-dir ./models/hunyuan/ckpts/t2i/model
121+
```
120122

121123
2. Preparing LoRa Weights
122124

123-
```shell
124-
# Put LoRa weights trained by Kohya in ComfyUI/models/loras
125-
cp ${HunyuanDiT}/kohya_ss/outputs/adapter_model.safetensors ${ComfyUI}/models/loras
126-
127-
# (Optional) Put LoRa weights trained by official scripts in ComfyUI/models/loras
128-
# The PEFT diffuser format needs to be converted into the standard ComfyUI format
129-
python convert_hunyuan_to_coimfyui_lora.py \
130-
--lora_path ${HunyuanDiT}/log_EXP/001-lora_porcelain_ema_rank64/checkpoints/0000100.pt/adapter_model.safetensors \
131-
--save_lora_path ${ComfyUI}/models/loras/adapter_model_convert.safetensors
132-
133-
# update the `lora.py` file
134-
cp ${ComfyUI}/custom_nodes/comfyui-hydit/lora.py ${ComfyUI}/comfy/lora.py
135-
```
125+
```shell
126+
# Put LoRa weights trained by Kohya in ComfyUI/models/loras
127+
cp ${HunyuanDiT}/kohya_ss/outputs/adapter_model.safetensors ${ComfyUI}/models/loras
128+
129+
# (Optional) Put LoRa weights trained by official scripts in ComfyUI/models/loras
130+
# The PEFT diffuser format needs to be converted into the standard ComfyUI format
131+
python custom_nodes/comfyui-hydit/convert_hunyuan_to_comfyui_lora.py \
132+
--lora_path ${HunyuanDiT}/log_EXP/001-lora_porcelain_ema_rank64/checkpoints/0000100.pt/adapter_model.safetensors \
133+
--save_lora_path ${ComfyUI}/models/loras/adapter_model_convert.safetensors
134+
135+
# update the `lora.py` file
136+
cp ${ComfyUI}/custom_nodes/comfyui-hydit/lora.py ${ComfyUI}/comfy/lora.py
137+
```
136138

137139
## Custom Node
138140
Below I'm trying to document all the nodes, thanks for some good work[[1]](#1)[[2]](#2).

‎hydit/train_v1.1.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
task_flag="dit_g2_full_1024p" # the task flag is used to identify folders.
2-
resume_module_root=./ckpts/t2i/model/pytorch_model_distill.pt # checkpoint root for resume
2+
resume_module_root=./ckpts/t2i/model_v1_1/pytorch_model_distill.pt # checkpoint root for resume
33
index_file=dataset/porcelain/jsons/porcelain.json # index file for dataloader
44
results_dir=./log_EXP # save root for results
55
batch_size=1 # training batch size

0 commit comments

Comments
 (0)