Ultimate RunPod Tutorial For Stable Diffusion - Automatic1111 - Data Transfers, Extensions, CivitAI

[FurkanGozukara]

Ultimate RunPod Tutorial For Stable Diffusion - Automatic1111 - Data Transfers, Extensions, CivitAI

Full tutorial: https://www.youtube.com/watch?v=QN1vdGhjcRc

Sign up RunPod: https://bit.ly/RunPodIO. This is the Grand Master tutorial for running Stable Diffusion via Web UI on RunPod cloud services. If I have been of assistance to you and you would like to show your support for my work, please consider becoming a patron on 🥰 https://www.patreon.com/SECourses

SECourses Discord To Get Full Support ⤵️

https://discord.com/servers/software-engineering-courses-secourses-772774097734074388

#RunPod discord: https://discord.gg/pJ3P2DbUUq

Colab Tutorial 1: https://youtu.be/mnCY8uM7E50

Colab Tutorial 2: https://youtu.be/kIyqAdd_i10

Automatic1111 Command Line: https://bit.ly/StartArguments

Best DreamBooth Tutorial: https://youtu.be/Bdl-jWR3Ukc

DreamBooth second tutorial: https://youtu.be/KwxNcGhHuLY

RunPodCTL GitHub: https://github.com/runpod/runpodctl

Pre-trained models repo link : https://huggingface.co/lllyasviel/ControlNet

Web UI install tutorial on PC: https://youtu.be/AZg6vzWHOTA

How To Use Different Models Automatic1111: https://youtu.be/aAyvsX-EpG4

Textual Inversion Training Tutorial: https://youtu.be/dNOpWt-epdQ

ControlNet Tutorial Video: https://youtu.be/vhqqmkTBMlU

ControlNet extension: http://bit.ly/3IxBYc6

ControlNet Model Files: https://bit.ly/CTRLNETModels

ControlNet Native Script: https://youtu.be/YJebdQ30UZQ

Upgrade xformers Commands: https://bit.ly/UPxformers

Kohya GUI: http://bit.ly/3ICvsB7

Cloud sync: http://bit.ly/40Zf44C

00:00:00 Intro

00:01:32 How to register RunPod.io and charge your credits

00:02:34 How to deploy a pod - start a server for Stable Diffusion 1.5 Automatic1111 Web UI

00:03:30 How to select deployment template for Stable Diffusion Web UI in RunPod

00:04:00 Explanation of temporary disk and persistent volume

00:04:44 Explanation of credit spending per minute for storage usage in RunPod

00:08:10 My Pods section

00:08:30 Connect to the started Pod

00:08:41 Start SD 2.1 Version Web UI Pod

00:09:25 Why pick a lesser used Pod

00:10:53 Bidding system of RunPod.io

00:13:11 Where and how to see scheduled maintenance

00:13:31 Stop Pod vs Terminate (delete) Pod

00:14:24 Where to see logs to debug and understand errors

00:15:08 Connect your Pod via a Jupyter Lab interface

00:15:16 How to change Automatic1111 Web UI command line arguments and restart it

00:17:54 First prompt in RunPod Automatic1111 Web UI

00:18:45 Where to see logs, find error logs, debug them

00:19:35 How to install DreamBooth extension of Automatic1111 Web UI

00:20:58 Where the generated images are saved

00:21:10 How to download generated images

00:21:38 How to update installed extensions

00:21:55 How to notice port error and fix it

00:23:04 How to install runpodctl latest version to transfer files very quickly between Pods and PC

00:23:55 How to download a ckpt file very fast from Hugging Face repo

00:25:10 Start DreamBooth training with best model and settings

00:30:41 How to upload your training dataset images

00:34:15 How to upload thousands of images (big data) from your computer to RunPod via runpodctl

00:34:28 How to install RunPodCTL on your Windows computer

00:35:06 How to send files from your PC to RunPod via runpodctl

00:39:38 Where to find generated checkpoints and sample images during DreamBooth training

00:41:30 How to delete non-empty folder

00:41:51 Even though xformers not selected during training, still breaks training and how to fix it

00:42:29 How to download a folder from RunPod to your PC via runpodctl very quickly

00:43:09 How to add runpodctl to environment path to use from every folder

00:47:25 How to continue/resume DreamBooth training

00:48:20 Test all training checkpoints with x/y plot to find best one

00:52:09 How to set correct command line arguments for SD 2.1

00:52:55 Where to see currently spent credits per hour

00:54:05 How to do DreamBooth training on SD 2.1 - 768 pixel version with best possible settings

00:57:42 How to generate classification images manually very fast

01:00:26 Why SD 1.5 is superior to 2.1

01:04:34 How to download custom models very fast from CivitAI

01:08:45 How to do Textual Inversion training with some optimal settings

01:13:00 Where Textual Inversion training samples and checkpoints are saved

01:14:07 How to use Textual Inversion check points

01:15:55 Move generated SD 2.1 classification images into correct folder

01:19:26 How to install and run ControlNet extension on RunPod IO

01:21:11 How to download your trained model files (ckpt) into your PC very fast via runpodctl

01:25:00 How to upgrade xformers to 0.0.17 for DreamBooth SD 2.1 training

01:26:04 How to expand runtime disk space

01:27:21 Best settings for SD 2.1 with xformers

01:31:30 What is Stable Diffusion fine tuning and how to do fine tuning with DreamBooth

01:39:20 Best settings quick recap for SD 2.1 for 24 GB VRAM

01:40:34 How to install and run Kohya GUI on RunPod

01:44:16 How to enable public Gradio link for Kohya GUI

01:44:52 How to start RunPods without GPU

01:46:53 Cloud snyching your Pod data / content

thumbnail freepik macrovector

Video Transcription

• 00:00:00 Greetings everyone.

• 00:00:01 In this video, I am going to show how to use Automatic1111 Web UI for Stable Diffusion

• 00:00:07 tasks on RunPod.io like you are using it on your computer.

• 00:00:11 I will cover many topics such as how to upload and download files quickly, how to delete

• 00:00:17 directories, how to install and run extensions, how to quickly download and use custom models,

• 00:00:23 how to do DreamBooth training on Stable Diffusion 1.5 or 2.1 versions, how to do fine tuning

• 00:00:30 via DreamBooth extension, how to do Textual Inversion training.

• 00:00:34 I will also explain how their pricing system works, how you can use bidding, how you can

• 00:00:39 transfer files from Pod to Pod or from Computer to Pod and vice versa, how you can install

• 00:00:45 custom other scripts such as famous Kohya graphical user interface.

• 00:00:50 I will also demonstrate how you can use new famous ControlNet on RunPod.io.

• 00:00:56 So why RunPod.io?

• 00:00:58 Because their system charges you based on per minute and they have great Discord support.

• 00:01:03 They are also easier to use with the tools they have.

• 00:01:07 But still, if you are interested in free cloud services for Stable Diffusion, I have two

• 00:01:12 great tutorials for Google Colab.

• 00:01:14 The first one is this one and the second one is this one.

• 00:01:18 And if you don't know how to use Automatic1111 Web UI, if you don't know what is Stable Diffusion,

• 00:01:23 what is Automatic1111 Web UI, I have great tutorial series for them.

• 00:01:27 For example, you can begin with watching video and you can check out the other videos in

• 00:01:31 this playlist.

• 00:01:32 So let's begin the Grandmaster RunPod.io tutorial by signing up a new account.

• 00:01:38 Click the sign up button.

• 00:01:39 For sign up I will use my Google account.

• 00:01:42 You can also enter your email and password if you wish.

• 00:01:45 Choose your account to sign up.

• 00:01:47 Click I have read and agreed RunPod Terms and Services.

• 00:01:51 Click Continue.

• 00:01:52 And yes, we are ready to start.

• 00:01:54 First of all, you need to charge some credits to start using the pods.

• 00:01:59 Click your balance from here as you can see in the right top menu, then it will show your

• 00:02:04 available balance.

• 00:02:06 From here you can pay with a card.

• 00:02:08 You can change the amount that you want to charge.

• 00:02:10 To have automatic payments you can add a card.

• 00:02:13 They also allow you to pay with a crypto.

• 00:02:16 Just click this icon.

• 00:02:17 They also show recent transactions, recent charges, and everything is very transparent.

• 00:02:23 OK, now I have logged in my account where I have my credits.

• 00:02:28 Now we can start using our Pods.

• 00:02:30 To do that, go to the browse servers tab in here and in here you will see the available

• 00:02:37 servers.

• 00:02:38 If you are going to do training, then I suggest you to get minimum 24 gigabytes VRAM having

• 00:02:45 server.

• 00:02:46 Because currently the latest officially released xformers is not working very well for training.

• 00:02:53 They have a nightly version that works well, but for training we won't use xformers.

• 00:03:00 And if you are not going to use xformers, then you should get minimum 24 gigabytes VRAM

• 00:03:06 having server.

• 00:03:07 I find that RTX A5000 is very decent GPU with a lower price.

• 00:03:14 As you can see, it is only 0.32 dollars per hour.

• 00:03:19 So I am going to deploy RTX A5000 GPU.

• 00:03:23 When you click the deploy icon, this interface will appear to you.

• 00:03:28 So in this interface, you should select your template.

• 00:03:31 There are many templates.

• 00:03:33 When you type Stable Diffusion, you see there are two very popular templates for Stable

• 00:03:39 Diffusion.

• 00:03:40 RunPod Stable Diffusion 1.5 and RunPod Stable Diffusion 2.1.

• 00:03:43 I will start both of them and I will start doing training both of them simultaneously.

• 00:03:49 So let's begin with RunPod Stable Diffusion 1.5 as a template.

• 00:03:54 So it will also download the official 1.5 version when it starts.

• 00:03:59 In here it shows us the other features.

• 00:04:01 They are very decent.

• 00:04:03 The temporary disk is the disk where the operating system will run.

• 00:04:08 You don't need to increase this.

• 00:04:10 And the persistent volume.

• 00:04:11 Now this is really important.

• 00:04:13 The persistent volume will stay remain as long as you don't delete your Pod.

• 00:04:20 So when you close your Pod, it will remain as it is.

• 00:04:23 It is like your hard drive.

• 00:04:24 It is persistent.

• 00:04:26 Everything you have generated, you have downloaded will remain there.

• 00:04:30 So this should be a sufficient amount of disk space based on your needs.

• 00:04:36 I am going to set it as 100 and when you set it, it will increase your minute credit spending.

• 00:04:44 So when you hover your mouse over this icon, it shows that 0.10 per gigabyte per month

• 00:04:52 for total disk on running Pods, 0.20 per gigabyte per month for volumes on exited Pods.

• 00:05:00 I know that this may be sounding confusing in the beginning, so I have prepared an example

• 00:05:06 for you which I will explain step by step.

• 00:05:09 So we have 105 gigabytes while running.

• 00:05:13 Why?

• 00:05:14 Persistent volume is 100 gigabytes and temporary disk is 5 gigabytes.

• 00:05:17 So while running, we are going to spend like this.

• 00:05:21 Let's say our Pod did run 75 minutes.

• 00:05:25 So 105 multiplied with 0.1 which is the per gigabyte price for per month.

• 00:05:33 In per month, how many days there are?

• 00:05:36 30 days.

• 00:05:37 So we are dividing it with 30 days.

• 00:05:40 In a day, how many hours there are?

• 00:05:42 24 hours.

• 00:05:43 So we are dividing it with 24 hours.

• 00:05:46 In an hour, how many minutes there are?

• 00:05:48 There are 60 minutes.

• 00:05:49 So this is the price of per minute running and since we are running 75 minutes, it is

• 00:05:56 going to take total 0.018 dollar from our credit.

• 00:06:02 You can also copy this.

• 00:06:04 Open your calculator with typing calculator in your search bar, paste it and hit enter

• 00:06:09 and you will get the result like this as you can see.

• 00:06:12 So in the below, I am giving example of Pod when it is not running.

• 00:06:18 When the Pod is not running, we are going to use 100 gigabytes persistent volume and

• 00:06:25 let's say our Pod did remain not running for two days.

• 00:06:30 So when Pod is not running, the price is for per gigabyte per month 0.20 dollars.

• 00:06:37 So since we have 100 gigabytes, 100 multiplied with 0.2, then let's delete this to not have

• 00:06:45 more confusion than in a month.

• 00:06:48 We have 30 days and we are going to use two days.

• 00:06:52 So this will be our spending.

• 00:06:54 So you can also open the calculator and copy-paste it, hit enter and you will get the price.

• 00:07:01 So this part is the price of one day offline for your 100 gigabytes having Pod.

• 00:07:09 And since it will be offline for two days, this is the credit that we are going to use.

• 00:07:13 The very important thing is that these credits will be deducted from your account per minute.

• 00:07:20 So if you keep using RunPod.io service for 10 minutes, you will be charged for 10 minutes.

• 00:07:26 So if it remains offline for 10 minutes, then you will be charged for 10 minutes.

• 00:07:31 It is not like taking your credits for per day, for per hour, or for per month.

• 00:07:37 It is using your credits for every minute.

• 00:07:40 When you hover your mouse over encrypt volume, you will see the message.

• 00:07:44 Encrypted volumes provide better data security, but will incur a performance penalty and cannot

• 00:07:49 be resized later.

• 00:07:50 So unless you need this, don't check this box.

• 00:07:54 Start Jupyter Notebook.

• 00:07:55 This will make your life much easier.

• 00:07:58 And this is the price per hour for our GPU.

• 00:08:02 So this price will be added these volume prices as well.

• 00:08:07 After you clicked deploy button, you will see an interface like this.

• 00:08:11 You can go to the My Pods section and you will see on demand community cloud is being

• 00:08:17 prepared.

• 00:08:18 When I click in here, you see it is showing me the messages of the Pod that is being prepared,

• 00:08:27 what is happening on the Pod.

• 00:08:28 And once it becomes ready, we will see connect button in here.

• 00:08:33 So it is initializing the Pod with the necessary installation and the Pod is now ready and

• 00:08:39 it is running.

• 00:08:40 Now I will start SD 2.1 version Pod simultaneously.

• 00:08:45 To do that I am clicking browse servers and when you open browse servers tab, you will

• 00:08:49 see in the right tab how much credits you are spending right now.

• 00:08:54 Because currently my other Pod is running, as you can see in My Pods tab.

• 00:09:00 So let's return back the browse servers and in here there are several options.

• 00:09:04 So you see there are one GPU Pods, two GPU Pods, large Pods, four GPU or x large Pods,

• 00:09:12 eight GPUs.

• 00:09:13 So if you need multiple GPUs, then you can filter them with this.

• 00:09:16 Also in each Pod, you will see their location, their available upload and download speeds,

• 00:09:22 their available disks and other things.

• 00:09:25 Choosing a less used Pod is better because if your previous Pod is fully used, then you

• 00:09:34 won't be able to get a GPU on that Pod.

• 00:09:37 So what happens then, then to use your existing files, you need to compose a new Pod and transfer

• 00:09:45 your files.

• 00:09:46 So availability is really important when choosing your Pod.

• 00:09:50 If you choose highly preferred Pod, then you will have lesser time to get it and it will

• 00:09:57 make things harder for you.

• 00:09:59 So based on this fact, you should choose your Pod.

• 00:10:03 So for the SD 2.1 version, I am going to pick another RTX A5000.

• 00:10:09 When you click more RTX A5000, it displays you other locations as well.

• 00:10:16 You see the upload and download speeds changes and the available space changes.

• 00:10:22 More available space probably means that it is being used lesser.

• 00:10:27 So for Canada server, it looks like it is not very much preferred this particular server.

• 00:10:33 So there is also Norway server.

• 00:10:35 You see it has great upload and download speeds.

• 00:10:38 It has decent hard drive space as well.

• 00:10:40 So it is probably also not very much used.

• 00:10:43 However, this is expensive than others.

• 00:10:46 So I think I will go with this Canada server.

• 00:10:50 Its speeds are also decent.

• 00:10:52 Click deploy.

• 00:10:54 There is one more thing as well that I need to explain.

• 00:10:57 Community cloud.

• 00:10:58 So what does community cloud mean that?

• 00:11:01 In the community cloud section, you will be able to bid for shared servers.

• 00:11:06 All of the servers are shared, but this is kind of that you bid and if someone overbids

• 00:11:11 you, they get your GPU.

• 00:11:14 So in here you see the prices will be lower.

• 00:11:17 When I click RTX A5000 select and then I click continue.

• 00:11:22 So you see currently this is selected.

• 00:11:24 RunPod Stable Diffusion 1.5.

• 00:11:27 I can also change it from this template.

• 00:11:29 Don't forget to change template.

• 00:11:32 When I click continue, you see now I am getting pricing summary and advanced.

• 00:11:36 When I click advanced, it will allow me to bid for a spot.

• 00:11:41 So you see the current bid is 0.198.

• 00:11:44 When I bid this, I will overbid the other person who has bidded lesser than this.

• 00:11:51 So I am going to get his GPU if there are no available other GPUs.

• 00:11:56 So let's say we did bid like this and we started our RunPod.

• 00:12:00 So someone else comes and bids 0.2 and they will get our GPU.

• 00:12:06 Then our pod will not have any GPU to do inference or training and our training will be also

• 00:12:13 halted.

• 00:12:14 So be careful with this.

• 00:12:15 If you are not going to do training, if you are only going to do image generation, then

• 00:12:20 you can go with this option and spend lesser.

• 00:12:23 The running disk cost and exited disk cost also slightly changes.

• 00:12:28 You can recalculate the cost.

• 00:12:30 So this is how you do bidding and this is how you use community cloud servers.

• 00:12:36 Since I am going to do training, I am going to use on demand server and I am going to

• 00:12:41 pick on demand server from here.

• 00:12:44 This Canada server.

• 00:12:46 Let's check again.

• 00:12:47 Yes, I am going to use this Canada server because it has the most available disk space.

• 00:12:52 Therefore, I am assuming that it is being used lesser than others.

• 00:12:57 Click deploy and we have selected RunPod Stable Diffusion 2.1 version.

• 00:13:01 Let's set our persistent volume as 100 GB and let's also deploy it so it will get deployed.

• 00:13:08 When I click My Pods, I will see them in here.

• 00:13:11 OK, when you go on My Pods, it is going to show you if there will be a maintenance or

• 00:13:18 not.

• 00:13:19 So you should be careful with this maintenance.

• 00:13:21 It says that it will start at this local time.

• 00:13:24 Therefore, I think I will delete this Pod.

• 00:13:27 So I will just click stop Pod and then I will delete it.

• 00:13:31 So when you stop your Pod, it will remain as it is.

• 00:13:34 However, if you click this terminate, then the Pod will be permanently deleted and you

• 00:13:39 won't be able to recover or access any of your data.

• 00:13:43 So now it is gone.

• 00:13:45 Let's go back to the browse servers tab and let's pick another server from here.

• 00:13:51 Maybe that is why it was being lesser used.

• 00:13:54 So I will pick this one.

• 00:13:56 OK, 2.1 version 100 GB.

• 00:13:59 Let's deploy.

• 00:14:00 Let's go to the My Pods and it is being deployed.

• 00:14:03 The first one we started is running.

• 00:14:06 The other one is being initialized and this is my per hour using credits right now.

• 00:14:12 OK, let's connect our first Pod.

• 00:14:14 To connect our first Pod.

• 00:14:16 I am clicking My Pods.

• 00:14:18 Let's refresh so you will see the interface as it is.

• 00:14:20 OK, I am clicking here.

• 00:14:22 It will open the interface.

• 00:14:24 When you click logs, it will show you the logs screen.

• 00:14:27 This is really important to debug the errors that you might encounter.

• 00:14:31 So it started with xformers with Workspace 1.5. emaonly CKPT file.

• 00:14:36 Actually, this is not the best CKPT file for training, so I will download the best one

• 00:14:43 and it is running on xformers 0.0.16.

• 00:14:47 This xformers is not compatible with DreamBooth training or Textual Inversion training, unfortunately,

• 00:14:54 so we won't use xformers during training and the other things are also displayed here.

• 00:14:59 When you click system logs, it will also show you the system logs.

• 00:15:02 When you click this refresh icon, it will refresh and when you click this X, it will

• 00:15:07 close it.

• 00:15:08 So let's click connect and I will connect it via Jupyter Lab, which will make our life

• 00:15:14 much easier.

• 00:15:15 OK, so our Jupyter has started like this.

• 00:15:19 The first thing that I am going to show you is how to change starting command line arguments.

• 00:15:25 To change them, I am zooming it for you to see easier.

• 00:15:28 That is webui-user.sh.

• 00:15:33 So this is the file where the command line arguments are provided.

• 00:15:38 You see it is starting with default port 3000.

• 00:15:41 It is starting with xformers.

• 00:15:43 The default CKPT is provided like this and there is a listen and enable insecure access.

• 00:15:50 So if you wonder what are these arguments are doing, there is a wiki page of Automatic1111

• 00:15:55 web UI and you can search for the commands by copying and pasting them and it will show

• 00:16:02 you launch gradio with 0000 as server name allowing to respond network requests.

• 00:16:07 Actually, I am going to also add share to be able to use it from my browser like this

• 00:16:14 and enable insecure extension access means that we will be able to install extensions.

• 00:16:19 Make sure that these commands are already enabled.

• 00:16:23 Otherwise, you won't be able to install extensions and I think we are ready.

• 00:16:28 I will also change the port to not get conflicted with any of the initial starting.

• 00:16:34 Just save.

• 00:16:35 When you save, you will see in the bottom saving completed.

• 00:16:37 Then go to the running terminals and kernels.

• 00:16:40 Shut down all of the running terminals and then go back to the file browser.

• 00:16:46 Make sure that you are inside Stable Diffusion web UI folder.

• 00:16:50 Then start the terminal.

• 00:16:51 When you start the terminal, it will start with the folder that you are currently in.

• 00:16:56 You see it is the same as the folder that we are in and in here we will use relauncher.py.

• 00:17:03 To do that just type python and I will copy paste the name relauncher.py hit enter and

• 00:17:10 it will restart our web UI with the newest set command line arguments.

• 00:17:15 We should be able to see them in here.

• 00:17:17 Yes, we are seeing dash dash port three thousand ten xformers.

• 00:17:21 So with this way you can also start multiple instances of web UI.

• 00:17:27 If you are a professional, then you can do that.

• 00:17:29 But if you are not, I don't suggest you to do that.

• 00:17:32 Now we can access it from this public URL.

• 00:17:35 This public URL is currently not secured by a password.

• 00:17:39 You can also add a password in here I think.

• 00:17:42 Let me show you.

• 00:17:43 Yes, you can also set out username and password.

• 00:17:46 However, if you are not giving this URL to anyone, then it should be safe.

• 00:17:51 As you can see, our interface is started.

• 00:17:54 Let's start with typing a simple prompt and see what happens.

• 00:17:58 OK, I have prepared my prompt.

• 00:18:01 I hit generate and in My Pods now you will see the GPU memory used is being increased.

• 00:18:07 GPU utilization will also increase as it generates the images and image is already generated.

• 00:18:14 Let's set the batch size as eight and batch count as one hundred.

• 00:18:18 And let's see how it is using our GPU.

• 00:18:21 So let's hit the refresh.

• 00:18:23 So it is showing like ten seconds ago.

• 00:18:25 OK, now you see the GPU utilization is one hundred percent.

• 00:18:29 GPU memory used is still significantly low because it is using also xformers, even though

• 00:18:37 we are generating images as batches with eight as batch size.

• 00:18:42 So in each time it will generate eight images.

• 00:18:45 So where are these files are being saved?

• 00:18:49 And how can I see if any error occurs?

• 00:18:52 You see in the My Pods, just click the logs and you will see all of the logs here.

• 00:18:58 This is really important to debug the logs.

• 00:19:00 And in here in the terminal window, you will see what is happening.

• 00:19:04 So how can you open the terminal.

• 00:19:05 To open the terminal, go to the running terminals and kernels.

• 00:19:08 And let's say I have closed the terminal.

• 00:19:11 I double click the terminal and it will show me the terminal as here.

• 00:19:16 As you can see in here.

• 00:19:17 This is equal to the terminal that we have on our computer when we are running it locally

• 00:19:23 on our computer.

• 00:19:24 This is the it per second.

• 00:19:26 However, since we are generating eight images at a time, it is actually over twenty four

• 00:19:31 it per second.

• 00:19:33 You need to multiply this with eight.

• 00:19:35 OK, let's hit the interrupt.

• 00:19:37 Now I will install the DreamBooth extension.

• 00:19:39 To do that go to the extension tab.

• 00:19:41 Go to the available hit load from.

• 00:19:44 Search DreamBooth, hit install.

• 00:19:47 Meanwhile, my two point one version terminal is also spending my time, my credit.

• 00:19:53 So I will just stop it.

• 00:19:55 So when you click stop Pod you are going to get this message, you should read it and understand

• 00:20:01 it.

• 00:20:02 OK, stopped Pod.

• 00:20:03 Basically what does it says that all of the things that is not saved on your workspace

• 00:20:09 will be lost.

• 00:20:11 So whatever you have in your workspace will be saved.

• 00:20:15 OK, let's see the status of the installation.

• 00:20:18 OK, it says that installed into workspace, Stable Diffusion, web ui extensions, SD DreamBooth

• 00:20:24 extension.

• 00:20:25 Now I will restart my terminal because when you first time install DreamBooth, you really

• 00:20:30 need to restart terminal so that it can install the necessary dependencies.

• 00:20:35 So I am going to do terminal stop, shut down all terminals.

• 00:20:39 Then I am going to Stable Diffusion Web UI folder and in here I will open a new terminal.

• 00:20:46 Same as before, I will type Python and relauncher.py and hit enter.

• 00:20:51 So the Web UI has been restarted and now we got a new link.

• 00:20:56 Let's copy and paste it.

• 00:20:58 Meanwhile, it is being loaded let's check out the generated images.

• 00:21:01 So they are saved in the outputs folder in the in the text to image images folder.

• 00:21:07 And yes, they are in here.

• 00:21:09 So how to download them?

• 00:21:10 You can download them one by one, right click and download.

• 00:21:13 Then it will download like this.

• 00:21:16 You can alternatively right click and download current folder as an archive.

• 00:21:20 It will first make archive and it will download all of the images like this.

• 00:21:25 It is a decent speed and it has downloaded all of these images.

• 00:21:31 121 files so far.

• 00:21:32 OK, the interface has been reloaded and now we are seeing the DreamBooth extension.

• 00:21:38 When we go to the extension tab, check for updates.

• 00:21:41 We should see the latest version in here.

• 00:21:44 Actually, it says that it is behind.

• 00:21:46 So let's click apply and restart UI.

• 00:21:49 And once we do that, we get an error.

• 00:21:53 It is relaunching in two seconds.

• 00:21:55 OK, when relaunching, we are getting port error because the previous one was crashed.

• 00:22:01 So what I'm going to do is: I will shut down all of the terminals.

• 00:22:05 Go back to the file browser.

• 00:22:07 In the first installation, you may encounter such errors.

• 00:22:10 Go to the webui user.sh file and change the port here and then go to the terminal tab.

• 00:22:18 Open a new terminal like this and type Python relauncher.py.

• 00:22:22 it will restart and when restarting now it is showing us the DreamBooth revision and

• 00:22:28 the SD Web UI revision like this.

• 00:22:30 I will just start training.

• 00:22:31 OK, it has been restarted.

• 00:22:33 Let's open the new URL.

• 00:22:35 OK, currently it is selected as 1.5 pruned emaonly CKPT and in the DreamBooth tab.

• 00:22:42 When we are going to generate a new training model, this is only available model.

• 00:22:48 However, 1.5 pruned CKPT is better than emaonly for training.

• 00:22:53 Therefore, I am going to download this CKPT file.

• 00:22:56 So how am I going to download it?

• 00:22:58 You see there is a download button in here.

• 00:23:01 I am right clicking and copying link address.

• 00:23:04 But before doing that, let's start a new terminal.

• 00:23:06 To do that, I am going to right click new plus icon here.

• 00:23:10 It will open a new launcher.

• 00:23:11 Hit terminal.

• 00:23:13 For fast download I am going to use RunPod CTL.

• 00:23:17 The RunPod CTL allows us to quickly download or upload files through our Pods to Pods or

• 00:23:24 from Windows to Pods and vice versa.

• 00:23:27 There are different versions.

• 00:23:29 I am going to install the Linux one on my RunPod.

• 00:23:32 So I am selecting it like this and copying it.

• 00:23:35 Then in my terminal I am pasting it with control V and I am hitting enter.

• 00:23:41 It will install the latest RunPod CTL.

• 00:23:44 After this command type RunPod CTL hit enter and you should get a message like this.

• 00:23:50 That means that it has been successfully installed or it was already installed.

• 00:23:55 Now how are we going to download this pruned CKPT file.

• 00:23:57 To download it first enter where you want to download, which is inside models inside

• 00:24:05 Stable Diffusion.

• 00:24:06 And in here where we want to download our model file, then I am going to click this

• 00:24:12 plus new launcher, launch a new terminal, and in this new terminal, this is the folder

• 00:24:18 where we are right now.

• 00:24:19 Now for downloading type wget and copy this URL paste it, hit enter and it will get downloaded

• 00:24:29 inside this folder.

• 00:24:30 By the way RunPod CTL is not necessary to download this file, but we will use it to

• 00:24:37 send data and get data from RunPod to our computer or from computer to RunPod or from

• 00:24:44 RunPod to RunPod.

• 00:24:45 This wget is a unix command and also alternative of it is available on windows as well.

• 00:24:52 So with this wget command, you can quickly download files into your RunPod folders like

• 00:24:59 this.

• 00:25:00 So you see currently it is downloading with 90 megabytes per second which is pretty decent

• 00:25:05 speed.

• 00:25:06 Okay the download has been completed and now the file is located in here.

• 00:25:10 Then what are we going to do is hit refresh button here and now I can see the 1.5 pruned

• 00:25:18 CKPT as well.

• 00:25:19 This is the way to download models from Hugging Face or wherever they are hosted.

• 00:25:24 If you can get direct link of it I will show examples.

• 00:25:28 Don't worry.

• 00:25:29 So now I will start DreamBooth training with the best possible settings.

• 00:25:33 First let's switch to 1.5 pruned CKPT.

• 00:25:36 This is not necessary but I'm not being sure that it is working as expected.

• 00:25:40 So I am making sure I have selected the target model in here as well.

• 00:25:45 So it has been loaded.

• 00:25:47 If it doesn't get loaded.

• 00:25:48 You should check the terminal window.

• 00:25:50 It is running on here.

• 00:25:52 It will show what is happening and you can also check the logs window in here.

• 00:25:57 It will show what is happening.

• 00:25:59 Okay now let's give a name to our training.

• 00:26:01 Let's say test SD 15 and check the source point.

• 00:26:05 So you see it is not seeing my latest checkpoint.

• 00:26:07 I am clicking refresh and I am checking the latest checkpoint.

• 00:26:11 This is very good to teach faces.

• 00:26:14 1.5 pruned CKPT the 512x model is selected and hit create model.

• 00:26:20 I am not changing other parameters because optimal parameters are currently selected.

• 00:26:26 These are more like experimental things or things that for more professional people.

• 00:26:31 And in the terminal you see it is downloading the necessary files right now.

• 00:26:36 That is why it is waiting.

• 00:26:37 Okay it says that checkpoint successfully extracted.

• 00:26:41 So the model has been generated.

• 00:26:42 However as you can see, the interface is frozen.

• 00:26:46 Unfortunately this is a problem of Gradio.

• 00:26:49 So what are we going to do is we will refresh reload this page and now it says no interface

• 00:26:54 is running.

• 00:26:55 It looks like the interface has been terminated unexpectedly.

• 00:27:01 And what do we see in the terminal in here in the system logs.

• 00:27:05 Okay it doesn't show anything and it doesn't show anything in here either.

• 00:27:09 So let's check out our terminals.

• 00:27:11 Terminal one which is our main terminal and yes it is not showing.

• 00:27:17 So what can we do.

• 00:27:19 We need to restart.

• 00:27:20 To restart I will shut down all terminals and I will follow the same procedure.

• 00:27:24 Open terminal.

• 00:27:26 However currently we are inside model Stable Diffusion so it won't work.

• 00:27:29 We need to move to the parent folder.

• 00:27:32 To moving parent folder.

• 00:27:33 I am closing this terminal going to the folders tab.

• 00:27:37 I am navigating like this opening a new terminal.

• 00:27:40 Python relauncher.py and in my pod current GPU memory usage is only 11 percent.

• 00:27:46 So it is good, which means that no other terminal or instance of Web UI is running.

• 00:27:52 Also there are some warning messages here.

• 00:27:55 I think we could ignore them.

• 00:27:57 Okay it has started.

• 00:27:58 I am opening this URL.

• 00:28:00 I am going DreamBooth tab and now I will select my model because I already created it and

• 00:28:06 it is selected.

• 00:28:07 Let's set up the settings.

• 00:28:09 Okay I won't pick this checkbox because it is usually causing me problems.

• 00:28:14 How many steps per image.

• 00:28:15 I am going to use 12 images and I am going to train up to 200 epochs.

• 00:28:20 I will save model for every 10 epoch.

• 00:28:24 Be careful with this because each save will take about five gigabyte space and with every

• 00:28:32 10 epoch, it is going to make 20 saves.

• 00:28:35 So it is going to take all of my hard drive.

• 00:28:38 So I think I will make this up to 180 or 160.

• 00:28:42 This should be sufficient.

• 00:28:44 If you don't know what are these parameters, how am I setting them.

• 00:28:48 I have an excellent DreamBooth tutorial on my YouTube channel.

• 00:28:53 You should watch this definitely to learn more about DreamBooth training.

• 00:28:57 Okay the batch size is one.

• 00:28:59 Gradient accumulation steps are one class batch size which will determine how many images

• 00:29:04 at a time that I want to be generated for classification images not related to training.

• 00:29:11 I will set this as 16 because this graphic card has huge VRAM, but if we get error, I

• 00:29:17 will reduce it.

• 00:29:18 Set gradients to none when zeroing.

• 00:29:20 Okay correct.

• 00:29:21 I am going to use half learning rate.

• 00:29:24 I am going to use sanity prompt as photo of ohwx man by Tomer Hanuka.

• 00:29:30 I will explain what are these for.

• 00:29:33 Actually I am explaining what are these for in this tutorial with details.

• 00:29:37 This is for checking the over trained or not.

• 00:29:41 And in here I am going to use EMA.

• 00:29:43 This will improve my training success rate and I have 24 gigabyte VRAM.

• 00:29:48 I will use 8 bit adam.

• 00:29:50 I am going to use mixed precision and I am going to use fp16 because this bf16 is not

• 00:29:57 supported by all graphic cards.

• 00:29:58 It is supported by RTX 2000 series or 3000 series.

• 00:30:03 I am not sure about this card as well.

• 00:30:05 So fp16 is our most safe option for every cards.

• 00:30:09 I am not going to use xformers.

• 00:30:11 This is important because the current xformers is not supporting the DreamBooth training

• 00:30:17 or Textual Inversion training.

• 00:30:18 It is you see, xformers 0016.

• 00:30:21 I think it will become compatible with xformers 0017 when it is officially released.

• 00:30:28 Currently nightly version is supporting as well as far as I know.

• 00:30:32 Cache latents.

• 00:30:33 Yes it will improve speed.

• 00:30:35 Train UNET.

• 00:30:36 Okay these are the optimal settings actually, so no need to change them.

• 00:30:40 And in here concepts.

• 00:30:41 Okay first we need to upload our training data set.

• 00:30:44 To do that go to the Stable Diffusion web ui folder or workspace.

• 00:30:48 Doesn't matter I will upload them to workspace.

• 00:30:51 In here create new folder training data set.

• 00:30:55 I have named the folder like this.

• 00:30:57 Enter inside folder and click upload files.

• 00:31:00 Select the files from your computer.

• 00:31:03 Since I don't have many files currently I am going to use this method and you see I

• 00:31:08 have only nine images which are pretty close shots.

• 00:31:12 No same background.

• 00:31:13 No same clothes as you can see.

• 00:31:16 I am all explaining what is a good training data set in this video and they are getting

• 00:31:21 uploaded.

• 00:31:22 We could also use runpodctl.

• 00:31:24 However, since there isn't many files, I am using this methodology for this task and our

• 00:31:31 training data set is ready.

• 00:31:33 Okay now we need to give the path of it.

• 00:31:35 To give the path of it.

• 00:31:36 Go back to the workspace like this: right click, copy path, paste it like this and put

• 00:31:42 a backslash to the beginning of it and where we want regularization images to be generated.

• 00:31:48 I am copy pasting like this and I will type classification images.

• 00:31:53 Okay filewords.

• 00:31:55 For training faces I am not using filewords.

• 00:31:57 It is more likely needed to fine-tune your model with lots of tokens and lots of good

• 00:32:05 images.

• 00:32:06 If you wonder how filewords are working in this short video.

• 00:32:09 I am explaining how file words are actually working.

• 00:32:13 So I'm just skipping file words and I am going to prompts.

• 00:32:17 So our instance prompt will be ohwx man.

• 00:32:20 Ohwx is our rare token and man is our class.

• 00:32:24 Class prompt will be photo of man since I am teaching a face of a man.

• 00:32:29 Sample prompt will be simply photo of ohwx man.

• 00:32:33 I am not going to set negative prompt or other things.

• 00:32:36 How many classification regularization images we want for per training image.

• 00:32:42 I have nine training images and I want 50 for per image.

• 00:32:46 This is actually a debated topic, how many is good is not precise.

• 00:32:52 In the official DreamBooth paper, the authors have used 200 so you can also try with 100

• 00:32:58 like this as well.

• 00:32:59 Okay then go to the saving tab, generate a ckpt file when saving during training.

• 00:33:04 So we will be able to generate checkpoints for every 10 epochs and then we will be able

• 00:33:11 to compare them to see which one of the checkpoint is performing best, which one of the checkpoint

• 00:33:19 has learned our subject best and with this way you can avoid over training.

• 00:33:26 And once you are ready, click save settings and hit train.

• 00:33:30 First it will start with generating class images.

• 00:33:32 In my pod I will see GPU utilization and memory usage.

• 00:33:35 Okay it says that exception training model no executable batch size found reached zero.

• 00:33:42 Why we got this error because we did set the classification images batch size pretty big.

• 00:33:50 If you make it like let's say six and try again.

• 00:33:54 And now I am seeing that it is generating six images at a time.

• 00:33:59 The it is pretty low actually only 12 because we need to multiply this with six and we are

• 00:34:07 seeing the images are being generated.

• 00:34:09 They will be saved in workspace, classification images directory like this: if you have previously

• 00:34:18 generated images on your computer, then you can alternatively upload them.

• 00:34:22 For uploading them I will install runpodctl on my windows.

• 00:34:28 To do that I am going to run this command on my windows powershell.

• 00:34:33 Type powershell, right click and hit enter.

• 00:34:37 Okay the installation has been completed, the runpodctl is now available on my command

• 00:34:42 prompt: let's see, runpodctl and now I am seeing it.

• 00:34:48 So I have previously generated 2400 images on my hard drive.

• 00:34:54 I am going to share this with runpodctl to download them in RunPod.

• 00:34:59 Alternatively, you can use upload methodology as well.

• 00:35:03 It also works, but for bigger files, runpodctl is better.

• 00:35:08 So for sharing the folder type runpodctl send and the folder path like this.

• 00:35:16 Getting the folder path easier, copy the folder path from here, paste it into the notepad

• 00:35:22 like this.

• 00:35:23 Put quotation mark to the beginning and end and type in your cmd.

• 00:35:29 I will show from beginning once again.

• 00:35:32 Open cmd type runpodctl send and paste the path like this and it will prepare like it.

• 00:35:39 It says that photo of man zip already exists because in another cmd window we used that.

• 00:35:46 So I need to delete this file.

• 00:35:49 Okay, this zip file is generated inside local disk c users and my username directory.

• 00:35:56 I am just going to delete it and I will run the command once again.

• 00:36:00 It will quickly prepare all of the files and now share link is generated I am copying this,

• 00:36:07 selecting it ctrl c or select it right click from here and copy, then go back to your Jupyter

• 00:36:14 Lab where your RunPod is running and in here I will make a new folder like this: ready

• 00:36:21 class.

• 00:36:22 I will enter inside ready class folder, then I will open a new terminal like this and I

• 00:36:28 will paste the command.

• 00:36:31 You see runpodctl receive the URL it has generated, hit enter.

• 00:36:36 It will connect to my computer and it will start downloading all of the files very quickly.

• 00:36:41 So this is how you can upload files from your computer to the remote RunPod.

• 00:36:48 The same thing applies to the RunPod to RunPod, so this is all vice versa.

• 00:36:53 RunPod to computer, RunPod to RunPod computer to RunPod.

• 00:36:57 You can send and receive files like this.

• 00:37:00 This is of course totally depends on my upload speed.

• 00:37:03 So when I open my task manager I see that it is using all of my available upload speed

• 00:37:09 like this.

• 00:37:10 This is pretty useful and convenient.

• 00:37:13 Instead of generating new classification images each time which uses your GPU time and consumes

• 00:37:20 your credits, you can prepare them on your computer and then quickly upload them to your

• 00:37:25 RunPod.

• 00:37:27 You can also upload them to any hosting, website, or other places that has better upload speed

• 00:37:33 and download them with the wget command as I have shown to download ckpt file.

• 00:37:41 RunPodCTL is extremely useful to upload and download files as you can see.

• 00:37:48 Okay 2400 photo of man.

• 00:37:51 The classification regularization images upload have been completed.

• 00:37:55 Now I see that it is uploaded as a zip here.

• 00:37:59 I need to extract them like oh.

• 00:38:01 It has automatically extracted as you can see after refresh.

• 00:38:05 Now they are here.

• 00:38:07 So what am I going to do is I will cancel training and I will give this folder.

• 00:38:13 So I will just skip image generation.

• 00:38:16 So it has been cancelled.

• 00:38:17 Let's give the new folder.

• 00:38:20 In concepts type here new folder name and click save settings and okay looks like the

• 00:38:27 train button is not appeared.

• 00:38:29 So what we need to do is we need to refresh reload.

• 00:38:34 Okay reloaded.

• 00:38:35 Go to the DreamBooth select the model, hit load settings, verify the settings are properly

• 00:38:41 loaded.

• 00:38:42 Okay, this is not being saved so you should uncheck it.

• 00:38:45 Okay, all settings are looking good and click train.

• 00:38:49 Now it won't generate any new classification regularization images because we already provided.

• 00:38:54 We can see that in the terminal window in here.

• 00:38:58 So you see it is processing the uploaded photo of man images.

• 00:39:02 Then it is going to cache the classification images with caching latents.

• 00:39:07 Okay, the training has started.

• 00:39:09 It has a pretty good speed as you can see.

• 00:39:13 It is supposed to do 180 epochs in less than 15 minutes.

• 00:39:17 However, this will take a little bit more time because it will generate ckpt during

• 00:39:22 the training.

• 00:39:23 We can also watch the training here.

• 00:39:25 However, you may get disconnected from gradio interface.

• 00:39:30 You can just watch the command line interface from here and know the status of the training

• 00:39:35 if that happens.

• 00:39:37 Okay, 10 epochs have been completed so it started generating the initial images as you

• 00:39:42 can see.

• 00:39:43 It also generated a checkpoint.

• 00:39:45 Where can we see the checkpoint?

• 00:39:47 Go to the workspace, go to the Stable Diffusion Web UI, go to the models folder, go to the

• 00:39:52 Stable Diffusion folder, and in here you will see our training name, go inside that folder

• 00:39:58 and now we can see the checkpoints being generated.

• 00:40:01 Then we will test each one of them with x/y plot and see how they are performing.

• 00:40:07 So if you want to see the generated samples during training, go to the models folder,

• 00:40:12 go to the DreamBooth folder, go to your training named folder, and in here you will see samples.

• 00:40:18 So these are the samples being generated during training and when you click the txt file,

• 00:40:24 you will see which prompt was used to generate this image.

• 00:40:27 When you double click the image, it will open image like this.

• 00:40:30 So far it is not like me at the moment.

• 00:40:34 When you go to the My Pods, you can see the GPU utilization and GPU memory being used.

• 00:40:39 The GPU memory is almost full because we are using EMA and we are not using xformers.

• 00:40:45 Because in the settings tab, we checked to use EMA and in the memory attention we didn't

• 00:40:49 use xformers.

• 00:40:50 And do, and these two are heavily increasing the memory usage.

• 00:40:55 Also, we didn't check the gradient checkpointing.

• 00:40:58 This also reduces the VRAM usage.

• 00:41:01 However, if you have sufficient amount of VRAM you shouldn't check this as well.

• 00:41:05 Okay, even after 130 epochs, it is still not learning even though it shows a good loss

• 00:41:11 rate.

• 00:41:12 That means that there is a bug currently with DreamBooth extension.

• 00:41:16 Therefore, I have cancelled the training.

• 00:41:18 Now I will delete the folder to open a space.

• 00:41:22 Right click folder.

• 00:41:23 Delete it.

• 00:41:24 It says that it is not empty so you can't delete it.

• 00:41:27 However, we can.

• 00:41:28 Now I will show you how to do it.

• 00:41:30 Click new, open a new terminal type rm minus r and the directory name test sd15.

• 00:41:39 It will iteratively delete all of the files and the folder.

• 00:41:42 After we refresh it is gone.

• 00:41:44 Now I will figure out the problem and show you the working settings and setup.

• 00:41:50 So I have figured out the problem and the problem was exactly as I have guessed it.

• 00:41:55 It was using xformers even though we didn't select use xformers.

• 00:42:02 In the settings, we had used memory attention default.

• 00:42:06 However, it was still using xformers.

• 00:42:09 Wo what did I do to fix this problem?

• 00:42:12 It is simple.

• 00:42:13 I have opened the Web UI user dot sh file and I have removed the dash dash xformers

• 00:42:21 from command line arguments.

• 00:42:22 I have restarted my Web UI.

• 00:42:25 Then I have composed a new training with the exactly same parameters and it did work very

• 00:42:31 well.

• 00:42:32 The training has been completed, so let's download the samples and check them out on

• 00:42:37 our computer.

• 00:42:38 To download the folder of samples, I will use runpodctl command.

• 00:42:42 So what I need to do is I will enter the samples folders.

• 00:42:47 So to do that, go to the models folder, go to the DreamBooth, go to the training folder

• 00:42:52 name so the samples are located here.

• 00:42:55 Open a new command terminal, write runpodctl send samples which is the folder name and

• 00:43:03 it will zip the samples folder and generate a receive command.

• 00:43:07 Copy it with ctrl c.

• 00:43:09 First, I need to add the path of runpodctl into my environment.

• 00:43:15 So the currently runpodctl exe is located inside my user folder.

• 00:43:21 Go to the users and your username and I will copy the runpot, yaml and runpodctl exe file.

• 00:43:27 Copy them.

• 00:43:28 Then I will make a new folder in my c drive as runpot exe.

• 00:43:33 Paste them here.

• 00:43:34 Then in the search bar search for environment, it will open, edit environment variables like

• 00:43:41 here and in here.

• 00:43:42 I am going to add a path variable for system variables, so go to the path, click edit and

• 00:43:49 in here click browse, select the folder where you have copy pasted which is inside c drive.

• 00:43:56 Runpod exe click ok, now the runpod exe is registered in my path.

• 00:44:02 Click ok, click ok click ok and now runpod exe should be available to call from everywhere.

• 00:44:09 Where I want to download.

• 00:44:10 I want to download the files inside my pictures, inside test samples.

• 00:44:16 I type cmd here.

• 00:44:18 So currently this is where I am.

• 00:44:20 Now I will copy and paste this command into my cmd window.

• 00:44:25 And yes, it is running as expected and the files are being copied into my folder.

• 00:44:33 And then they are automatically extracted with the folder name.

• 00:44:36 So in here we are able to see the generated sample images.

• 00:44:40 I can say that after 800 steps it started to resemble me and we have totally trained

• 00:44:48 it for 160 epochs, 3200 steps, we can see the examples here.

• 00:44:55 Okay, this is pretty much like me, so with good prompting I think we can get good results.

• 00:45:03 So let's try all of the checkpoints to see which one is working best.

• 00:45:07 How are we going to do that?

• 00:45:09 We are going to do that with text to image tab and in here we are going to use x/y/z

• 00:45:14 plot.

• 00:45:15 Okay, it didn't appear.

• 00:45:16 Let's refresh.

• 00:45:17 Oh, looks like our instance is closed so I will restart.

• 00:45:22 So before restarting make sure that you have closed all of the running terminals and I

• 00:45:27 will also close all of the open tabs.

• 00:45:29 Okay, all of the tabs and terminals are closed.

• 00:45:33 Okay, Web UI is restarted.

• 00:45:35 Let's open it.

• 00:45:36 Okay, now we can also see the checkpoints in here so you can test particularly one of

• 00:45:43 them.

• 00:45:44 But I am going to do xyz plot test.

• 00:45:47 But before that, let's decide our testing prompt.

• 00:45:51 So I am going to make my tests on 2200 step checkpoint.

• 00:45:58 I am going to select it from here.

• 00:46:00 First, let's see the raw prompt.

• 00:46:02 Ohwx man.

• 00:46:03 Okay, this is the raw prompt and it looks pretty decent.

• 00:46:08 This is the training data set you see.

• 00:46:11 It looks pretty decent, but it looks like have some memorization.

• 00:46:15 Actually, not exactly memorization.

• 00:46:18 The clothe is similar but not exactly same.

• 00:46:20 Okay, while doing testing, my Web UI has been killed.

• 00:46:24 So I have checked the terminal to see the message.

• 00:46:28 So you should be careful if some error happens.

• 00:46:31 Make sure to check the terminal to see what is happening in the behind the scenes and

• 00:46:36 now it is not able to restart.

• 00:46:39 Therefore, I will close all of the terminals and start with a different port.

• 00:46:46 To do that, you need to go to the terminals tab, shut down all and change the webui user.sh

• 00:46:53 file: change the port from here.

• 00:46:56 Save and restart.

• 00:46:57 Okay, I got a simple prompt like this: photo of ohwx man 1.2 emphasis: you can learn emphasis

• 00:47:05 from wiki page of Automatic1111.

• 00:47:09 Just pause the video and read here if you don't know.

• 00:47:12 And digital painting, artstation, masterpiece.

• 00:47:14 I don't have any negative prompts.

• 00:47:17 The picture is not exactly like me.

• 00:47:19 So now we are ready to do test and see if model is trained enough.

• 00:47:23 If it is not trained enough, then go to the DreamBooth tab, select the model load settings

• 00:47:29 and continue training.

• 00:47:31 It will continue training for the number of steps that you have defined in here.

• 00:47:36 Okay, I started continue training and it will start from this model revision which means

• 00:47:42 it will start from 3200 steps and it will continue to do training for number of epochs

• 00:47:50 that we have defined here.

• 00:47:52 However, my Gradio is crashed once again and I am able to see the continuing training from

• 00:47:59 here.

• 00:48:00 Now let's test the current checkpoints and see whether they are trained enough or not

• 00:48:05 and decide upon that to continue training or not.

• 00:48:08 However, since my Gradio is crashed, I have to restart the terminal because there is no

• 00:48:14 way to cancel the training right now.

• 00:48:16 Let's also have yes no way.

• 00:48:18 Okay, I did a restart.

• 00:48:20 So how are we going to test different checkpoints?

• 00:48:24 Prompt emphasis, and CFG values.

• 00:48:26 Go to the bottom, pick x/y/z plot and in here you see there are different type of parameters.

• 00:48:33 So first parameter will be checkpoint name.

• 00:48:36 When you click this icon it will paste the available checkpoints.

• 00:48:40 I am going to start picking from 1600 steps which means 80 epochs for me.

• 00:48:46 It depends on the your training dataset size and I will test the remaining as well like

• 00:48:52 this.

• 00:48:53 It is also displaying the calculated hash value.

• 00:48:56 Okay, as a second thing, I am going to test prompt strength.

• 00:49:00 To do that, I am going to use prompt s/r.

• 00:49:02 So I am going to give this a any keyword like prsr.

• 00:49:07 So the first value here will be prsr.

• 00:49:10 Then I will type the prompt strengths like 1.1, 1.2, 1.3 let's also try 1.0, 1.4 1.5,

• 00:49:20 1.6 and 1.7 okay, as a third comparison thing, I am going to test CFG value.

• 00:49:28 So for CFG values, I am going to test seven, seven point five, eight, eight point five,

• 00:49:34 nine, nine point five and ten.

• 00:49:37 If you keep minus one for seeds then you won't be able to compare them very well.

• 00:49:42 So do not check this checkbox so it will use same seed for all of the comparisons and then

• 00:49:48 when you click generate it will process all of them.

• 00:49:52 You can see the process in the command line interface.

• 00:49:56 Now meanwhile this is running I will start my 2.1 version RunPod as well.

• 00:50:02 Okay, it says that there is no available GPU for this RunPod right now so I can start it

• 00:50:09 without a GPU and transfer my files with runpodctl.

• 00:50:14 However, I do not have any files on it so I will just delete it because I didn't even

• 00:50:20 start it yet and I will start a new one.

• 00:50:22 Okay, I am going to use this one.

• 00:50:26 Select the template from here.

• 00:50:27 I will pick Stable Diffusion 2.1 version: I will start with 100 gigabytes, deploy my

• 00:50:34 pods, it is being initialized and my other Pod is currently working with this kind of

• 00:50:40 i/t.

• 00:50:41 By the way, xformers is still not enabled right now, so if you enable it, this will

• 00:50:46 become even faster.

• 00:50:47 But for training, make sure that you have disabled it.

• 00:50:50 And images are being generated in here.

• 00:50:52 We will download all of them and check all of them later.

• 00:50:56 Okay, 2.1 version is being generated and getting ready.

• 00:51:01 Okay, 2.1 is now ready.

• 00:51:03 Just click connect.

• 00:51:05 Connect to the Jupyter.

• 00:51:06 Okay, it says that it cannot connect yet so it is probably still not ready.

• 00:51:10 Let's wait.

• 00:51:11 Try again.

• 00:51:13 Okay, let's refresh the page.

• 00:51:14 Maybe the URL is incorrect.

• 00:51:17 Yes, after the refresh I think it is fixed or it is just started.

• 00:51:21 So just be patient a little bit.

• 00:51:23 It is getting loaded and yes, 2.1 version is started.

• 00:51:27 It is exactly same as the previous one.

• 00:51:31 We are editing the command line arguments here.

• 00:51:33 I will add dash dash share so I can use it as I want.

• 00:51:37 And I will also remove xformers because it is preventing training.

• 00:51:42 I will set the port as 3001.

• 00:51:45 Save it.

• 00:51:46 Then there is no open terminals.

• 00:51:49 Let's open a new launcher terminal python relauncher.py Our comparison on SD 1.5 trained

• 00:51:57 models are continuing.

• 00:51:58 Okay, 2.1 RunPod is ready.

• 00:52:02 Let's start it.

• 00:52:03 Okay, currently selected model is 2.1 version.

• 00:52:06 Let's test it.

• 00:52:07 Okay, I have written my prompt the the output resolution is 768 and 768.

• 00:52:14 Looks like we got a problem.

• 00:52:17 It says that a tensor with all NaNs was produced in Unet.

• 00:52:21 So we need to add no half command to the command line because with this graphic card, otherwise

• 00:52:28 it won't work.

• 00:52:29 So let's go back to the RunPod.

• 00:52:31 Open the webui dash user dash sh.

• 00:52:34 So for SD 2.1 version, make sure that you are using these command line arguments.

• 00:52:41 These may be necessary for some of the custom models as well.

• 00:52:44 So check the messages that you see in here.

• 00:52:47 This message should be available also in the terminal window.

• 00:52:51 Yes, you can also see the error in here as well.

• 00:52:55 So I will close the terminal and restart it.

• 00:52:58 Currently I am spending 0.669 dollars per hour.

• 00:53:03 My mode of the RunPods are running right now.

• 00:53:06 Okay, it looks like I have mistyped the dash dash precision.

• 00:53:12 So it says that argument precision expected one argument.

• 00:53:15 I will just fix it quickly.

• 00:53:17 To fix it, I am opening the file and I am setting dash dash precision as full, saving

• 00:53:24 it and restarting.

• 00:53:26 Make sure that you only have one active running terminal, otherwise other terminals will also

• 00:53:32 consume your VRAM memory.

• 00:53:35 You can also see the VRAM memory usage in your My Pods tab and you can see the logs

• 00:53:41 from here.

• 00:53:42 This is really important to debug the errors.

• 00:53:44 Okay, it is started with these command line arguments exactly like this.

• 00:53:49 Let's open the Gradio window.

• 00:53:51 Okay, let's hit generate with our written prompt and it is getting generated.

• 00:53:57 And we got our tank image.

• 00:54:00 Now I will install the extension same exactly as I have did.

• 00:54:04 Okay, 2.1 version is ready with DreamBooth now.

• 00:54:07 Go to the DreamBooth tab, make a new model I will name as test select the source checkpoint.

• 00:54:13 Uncheck 512 model.

• 00:54:14 Hit create.

• 00:54:15 When the first time you click hit create it is downloading the necessary files same as

• 00:54:19 before because this is a new RunPod so they are not connected.

• 00:54:23 This is a fresh installation and checkpoint successfully extracted so it is ready.

• 00:54:29 Okay, we didn't get any error so we can continue.

• 00:54:32 So for 2.1 version usually you need more epochs so I will set this as 300.

• 00:54:38 However, now it will also use more space.

• 00:54:42 Due to more epochs so I need to reduce save model frequency.

• 00:54:47 I think I will save it for every 20 epochs.

• 00:54:50 Batch size one, gradient accumulation one, class batch size will be four.

• 00:54:55 I am not going to set gradient checkpointing.

• 00:54:58 You can also leave it as default learning rate.

• 00:55:01 This would make it learn faster, however, it may also not learn very well or it may

• 00:55:07 get over trained quickly.

• 00:55:09 So I will make this as one.

• 00:55:10 But you can also leave it as default.

• 00:55:13 So the other things are same.

• 00:55:14 Now with 2.1 version, I don't know if 24 gigabytes will be enough without xformers when we use

• 00:55:20 EMA so I will test it.

• 00:55:23 Okay, it says let's make it like this.

• 00:55:26 Actually, we should click performance wizard so it will set the optimal ones for us.

• 00:55:32 Okay, okay, I am leaving the settings like this.

• 00:55:35 Let's also set the memory attention as default and let's see if it will work.

• 00:55:39 By the way, we also need to re-upload our training images and these training images

• 00:55:44 have to be 768 pixels because this model is 768 pixels model.

• 00:55:52 So to upload them I am following just the same things.

• 00:55:56 Here my 768 pixel images.

• 00:55:59 I'm just going to use drag and drop but you can use runpodctl as well as just I have displayed.

• 00:56:06 Okay they are ready.

• 00:56:07 So I am right clicking copy path, paste it, adding a backslash to the beginning.

• 00:56:12 Copy this and let's say class 786.

• 00:56:17 All other settings are same.

• 00:56:19 Ohwx man, photo of man, photo of ohwx man and I will use only 12 images because I want

• 00:56:28 training to start quickly but you should use bigger number.

• 00:56:32 I am checking generate ckpt during the checkpoints, click save settings, and hit train.

• 00:56:39 So it will start with generating class images.

• 00:56:42 So for each image we are generating 12.

• 00:56:44 Okay, we got an error.

• 00:56:46 Therefore, we need to decrease the class batch size.

• 00:56:49 Let's hit train again.

• 00:56:50 Okay, looks like our Gradio is killed, therefore it has to be restarted.

• 00:56:55 You may get these errors.

• 00:56:56 Okay, during restart, it is throwing error because port is still being in used.

• 00:57:01 So I am going to close the terminal, change the port and restart myself manually.

• 00:57:07 Okay, restart has been completed.

• 00:57:09 Let's go to the DreamBooth, select model load settings, just quickly verify settings.

• 00:57:14 I am unchecking this because it is usually problematic.

• 00:57:17 Class batch size is two and let's hit train.

• 00:57:21 You can also generate classification images from text to image directly yourself.

• 00:57:25 Cut the generated images and put them into a new folder.

• 00:57:28 Okay, we got error once again.

• 00:57:30 This is a memory error actually.

• 00:57:32 When we check the command line interface, we can see the memory error.

• 00:57:37 So looks like our only option is class batch size one.

• 00:57:40 Let's click train.

• 00:57:41 Okay, it is working.

• 00:57:42 However, this will be very slow.

• 00:57:44 So what am I going to do is?

• 00:57:45 I will enable xformers, manually generate from text to image and use them as classification

• 00:57:51 images which will save our time significantly.

• 00:57:55 So follow me how am I doing.

• 00:57:57 First, I will just terminate the terminal from here.

• 00:58:00 I will add dash dash xformers, change the port and restart python relauncher.py I would

• 00:58:08 also clear text to images tab so you can directly use it so I will just rename it.

• 00:58:14 It will generate a new folder for me and new app is started with xformers.

• 00:58:19 Let's open it!

• 00:58:20 So our class prompt is photo of man.

• 00:58:23 I am typing photo of man.

• 00:58:25 I am going to set the sampling steps as 30 which is a decent enough and I am leaving

• 00:58:30 all other options are same and I will use batch size as eight and how many images total

• 00:58:37 do you need?

• 00:58:38 Let's say for per training image 50 images since I have nine images, I am going to generate

• 00:58:44 480 images.

• 00:58:46 Therefore I need to set this minimum 57 and then hit generate and let's see if we will

• 00:58:53 get out of memory error.

• 00:58:55 And you see from text to image tab we are not getting out of memory error even when

• 00:59:01 the batch size is eight.

• 00:59:03 So it will very quickly generate all of these images for us much faster than using the classification

• 00:59:11 images that is being generated in the DreamBooth.

• 00:59:15 If you wonder why it is generating images like this or why we are using these kind of

• 00:59:19 images, in this video I am explaining all of them so we are keeping the underlying contextual

• 00:59:25 data of the model.

• 00:59:27 You could also use more beautiful images in your classification training data set.

• 00:59:32 However, it would break your model conceptual meaning so your model would become more biased

• 00:59:40 to the images that you have used.

• 00:59:42 Also, your face would be biased to the images that you use.

• 00:59:46 With this methodology, we are using the underlying contextual knowledge of the model and we are

• 00:59:53 trying to keep it as much as possible.

• 00:59:55 However, this is up to you.

• 00:59:58 So if you use all handsome images, all full colored, professional real images, then your

• 01:00:06 model would become more biased to them.

• 01:00:08 This is how custom models are usually made.

• 01:00:12 They are being cooked to those kind of images.

• 01:00:15 So whatever you type, you are getting beautiful images because all of the other underlying

• 01:00:20 conceptual data of the model is lost during the training.

• 01:00:25 Actually, according to the ControlNet developer, SD 2.1 version is inferior to the SD 1.5 due

• 01:00:35 to the used CLIP.

• 01:00:36 You can read this with pausing the video right now.

• 01:00:39 Okay, looks like our 1.5 version experiment has ended.

• 01:00:44 Let's go to the outputs and in here there are text to image grids and you see there

• 01:00:51 is a grid file.

• 01:00:53 35 megabytes.

• 01:00:54 Let's open it.

• 01:00:55 Actually I will download this and there is also 228 megabytes.

• 01:01:00 So for downloading let's use the runpodctl.

• 01:01:04 I am going to open a new command line in here.

• 01:01:07 Runpodctl, send text to image grids.

• 01:01:11 Hit enter and it will generate download link for us.

• 01:01:15 Go to the folder where you want to download.

• 01:01:17 I will download inside in here, type cmd, copy paste the link like this.

• 01:01:22 So it is going to download 265 megabyte grid output.

• 01:01:27 This is much faster than downloading from the Jupyter notebook.

• 01:01:31 Okay, the grid images are downloaded.

• 01:01:34 And in here this is the newest grid image that is generated.

• 01:01:39 It is over 200 megabytes, it is over 35 000 pixels and now we are able to compare different

• 01:01:47 checkpoints with different prompt emphasis and with different CFG scale.

• 01:01:53 So this is for CFG scale 7.

• 01:01:55 These are the checkpoints and these are the prompt emphasis.

• 01:02:00 Let's find the best one that we like and that is similar to us.

• 01:02:05 You see these faces are not like me but in here I am seeing faces like me.

• 01:02:12 So with prompt strength 1.4 in these checkpoints I am starting to get similar face like to

• 01:02:19 me.

• 01:02:20 I think this one is very similar to me.

• 01:02:22 So with prompt strength 1.4 for CFG scale 7 and for checkpoint 3000 steps.

• 01:02:30 Yeah I like it.

• 01:02:31 So you should also compare for yourself.

• 01:02:34 And after prompt strength 1.4 the image becomes very very bad.

• 01:02:39 So let's also look at the other CFG scales and checkpoints.

• 01:02:44 Okay now I will show you slowly what is happening from CFG scale 10 to 7 and this is the prompt

• 01:02:51 strength 1.4.

• 01:02:53 This is how the images are changing.

• 01:02:55 This would of course depend on your training data set, how it is trained and I can see

• 01:03:01 that they are not very good at all because we also didn't use any negative prompts.

• 01:03:08 Our aim here is finding the sweet spot of prompt strength and the checkpoint and the

• 01:03:16 CFG possibly.

• 01:03:18 Okay I think this model is still not trained enough.

• 01:03:22 Because with only 1.4 strength and in the 3200 steps, it is providing the best.

• 01:03:31 So therefore I will train this model even further with more steps and then do another

• 01:03:38 experiment.

• 01:03:39 However, currently we could use 1.4 strength with checkpoint 3200.

• 01:03:45 I suggest you to test no half and precision full training for SD 1.5 version as well without

• 01:03:54 xformers and compare whether it is learning better or not.

• 01:04:00 Because of the used graphic card this could be making a difference and you can test use

• 01:04:06 8bit adam or not.

• 01:04:08 You can test mixed precision no versus fp16 and bf16 so these all things could improve

• 01:04:16 your training success rate.

• 01:04:18 You should experiment with them and currently I do not have time to test all of them.

• 01:04:24 I am showing the some of the settings that are widely used but you should also experiment

• 01:04:30 with them.

• 01:04:31 Like options like this or like this or like this or like this.

• 01:04:36 Now I will show you how to download custom models from CivitAI .com and use them in your

• 01:04:43 RunPod io.

• 01:04:44 So I am going to show example of Protogen x3.4.

• 01:04:49 Right click download latest copy link, go to your RunPod io interface, Jupyter interface

• 01:04:57 and in here go to the folder where the model files are downloaded.

• 01:05:02 So in this folder which is model Stable Diffusion where you are supposed to put your model files,

• 01:05:09 open a new launcher, open launcher, type wget, paste the link and hit enter and it will start

• 01:05:16 downloading the model file.

• 01:05:18 So you see 5.6 gigabytes and you see there are no more space left on my hard drive.

• 01:05:26 What I need to do is I will delete the some of the models.

• 01:05:31 So I am going to delete some of the training checkpoints.

• 01:05:34 They are located inside models, inside Stable Diffusion, inside my training folder and in

• 01:05:41 here I am going to remove delete some of them.

• 01:05:43 You can also do a directory delete right, click and delete.

• 01:05:47 You can also select them and hit delete button on your keyboard.

• 01:05:51 Okay, I think we got now sufficient space so I will just rerun the prompt.

• 01:05:56 So to open back the latest executed command I just hit up arrow and hit enter and now

• 01:06:02 it will start downloading.

• 01:06:03 Currently it will be downloaded in this folder where we had opened this terminal.

• 01:06:10 Let's go back to there.

• 01:06:11 Models Stable Diffusion and now this file is being downloaded with the name of 4048.

• 01:06:19 Then I will rename it.

• 01:06:21 Meanwhile, 2.1 version classification regularization images are still being generated.

• 01:06:26 We can see the process in the terminal of it.

• 01:06:30 You see it has generated over 160 images so far.

• 01:06:34 Okay, it is downloading the custom model file with 50 megabytes per second.

• 01:06:39 You can also upload those files from your computer or you can download from Hugging

• 01:06:45 Face as I have shown you already.

• 01:06:48 So this is how you can download files fast on your Pod.

• 01:06:52 Okay, the file has been downloaded and saved as 4048.

• 01:06:57 I will rename right click, rename and let's say protogen x34 it is renamed.

• 01:07:05 Then let's go back to our Stable Diffusion interface.

• 01:07:08 Click, refresh folder.

• 01:07:10 It is not appearing because the model file extension is not correct.

• 01:07:14 Right Click.

• 01:07:15 And when renaming, add dot ckpt to end of it like this and then refresh again.

• 01:07:23 Okay, now we see the model here.

• 01:07:25 Let's test it.

• 01:07:26 Okay, it didn't load even though I have selected.

• 01:07:29 Let's look at the command line interface.

• 01:07:31 Okay, it says that we should add disable safe unpickle because we have downloaded it like

• 01:07:38 that.

• 01:07:39 So I will add this to the command line arguments and restart like this.

• 01:07:44 Let's also change the port.

• 01:07:46 Just close all of the terminals.

• 01:07:47 Okay, restart has been completed with disable safe unpickle.

• 01:07:51 Let's open the interface.

• 01:07:53 Okay, let's try with protogen.

• 01:07:55 Okay, we got error once again because when we download it, it is downloading safetensors

• 01:08:02 not ckpt.

• 01:08:03 Therefore, we have to rename it once again into safe tensors .safetensors like this and

• 01:08:11 try again.

• 01:08:12 Let's hit refresh.

• 01:08:13 Now there is safetensors.

• 01:08:15 Okay, it is loaded.

• 01:08:16 Let's test it and protogen is working as expected.

• 01:08:20 You see of awesome, intricate, fantastic, castle, in a forest and this is what I got.

• 01:08:25 Let's run again.

• 01:08:26 And yes, this is definitely protogen.

• 01:08:28 Let me run it on 1.5 version official as well.

• 01:08:32 Okay, 1.5 version is loaded and this is the result on 1.5 version official.

• 01:08:38 So this is how you can use custom models on RunPod io.

• 01:08:43 2.1 image generation is still going on.

• 01:08:45 Now I will show you how to do Textual Inversion training.

• 01:08:49 To do that, let's go to the train tab.

• 01:08:52 By the way before doing that, let's go to the settings and in here in training, move

• 01:08:56 VAE and CLIP to RAM when training if possible.

• 01:09:00 You can pick this option to reduce VRAM usage.

• 01:09:03 You can also turn on pin memory for data loader.

• 01:09:06 Makes training slightly faster, but it can increase memory usage.

• 01:09:09 You can also pick this depending on your machine's RAM memory.

• 01:09:13 However, since we have 24 gigabytes, I am not going to pick them.

• 01:09:17 So let's give a name as test initialization text is none.

• 01:09:21 Number of vectors is two.

• 01:09:24 You can watch my excellent how to do Stable Diffusion Textual Inversion video.

• 01:09:30 I am explaining in great details in this video and you can learn many of the things related

• 01:09:37 to the Textual Inversion from this video.

• 01:09:40 Hit create embedding and it is already created.

• 01:09:43 Let's go to the train tab, pick the embedding.

• 01:09:46 We also need to set dataset directory.

• 01:09:49 So our data set directory is like this.

• 01:09:52 We don't need classification images for Textual Inversion training.

• 01:09:56 You can reduce the learning rate or leave it as default.

• 01:10:00 You can test it.

• 01:10:01 Okay, we need a style file word for Textual Inversion.

• 01:10:06 When you watch this video, you will understand it better.

• 01:10:10 So this text file is located inside Stable Diffusion, inside Textual Inversion templates.

• 01:10:16 In here, i'm going to edit the none as as [name].

• 01:10:20 You need this otherwise it won't work.

• 01:10:22 This is the name of the Textual Inversion.

• 01:10:24 This is basically going to use the unique tokens that it generates so i'm going to pick

• 01:10:30 none from here.

• 01:10:32 My width and height are 512 pixels.

• 01:10:35 Max number of steps.

• 01:10:37 You can leave it as this because it will generate pretty small files, but since we are already

• 01:10:42 using a lot of space, I will delete my older checkpoints from DreamBooth, Stable Diffusion

• 01:10:48 Web UI inside models inside Stable Diffusion and inside test2 folder.

• 01:10:54 Okay for selecting hit left shift key, select first, then go to the very bottom while pressing

• 01:11:00 shift key hit here it will select all of them.

• 01:11:04 Then while hitting control button left control, unpick the ones that you don't want to delete

• 01:11:10 right, click and hit delete.

• 01:11:12 It will delete all these files and open a space for me.

• 01:11:16 Okay, now we are ready.

• 01:11:17 I want to check checkpoints for every 10 epochs.

• 01:11:23 How many training images I have.

• 01:11:25 I have nine training images.

• 01:11:26 Therefore, one epochs means nine steps.

• 01:11:30 Five epochs means 45 steps.

• 01:11:33 So for every five epoch I am going to make save.

• 01:11:36 I don't need this and I will pick deterministic.

• 01:11:40 This is the best option and we are ready.

• 01:11:43 Just click hit train embedding.

• 01:11:45 Okay, it has started training.

• 01:11:48 By the way currently xformers is enabled.

• 01:11:51 Therefore, I will disable it and restart again because there is a bug as I have just shown

• 01:11:58 and it is preventing good training.

• 01:12:01 Also in settings this is unchecked.

• 01:12:04 Use cross attention optimizations but still it could be using it due to a bug.

• 01:12:10 So best thing is just disabling the xformers and restarting the training.

• 01:12:16 However, looks like learning right now I think.

• 01:12:19 So probably there is no bug for this one unlike the DreamBooth.

• 01:12:24 The loss rate is also pretty low and it is pretty fast.

• 01:12:28 Okay, it already started learning my face.

• 01:12:32 Not very good but there is a resemblance as you can see and it is really really fast the

• 01:12:38 number of steps it is taking really really fast.

• 01:12:41 This is how fast it is you see.

• 01:12:44 Training Textual Inversion epochs, training speed, the i/t per second and it is learning.

• 01:12:52 However, which one will be best is needs to be checked from text to image tab from x/y

• 01:13:00 plot and as you can see it is learning.

• 01:13:03 So all these samples are being saved inside.

• 01:13:07 Let's go to the Stable Diffusion Web UI folder inside here, textual inversion, inside here

• 01:13:13 you will see the training date and inside here the name of the Textual Inversion training

• 01:13:17 inside here images and these are the images named with the epoch number.

• 01:13:24 You can check them like this, or you can download them and check all of them.

• 01:13:29 Okay, 2700 steps looks a little bit decent.

• 01:13:34 It is actually equal to 300 epochs.

• 01:13:38 Maybe it may get better over time or we may need to use more vector count, but since I

• 01:13:44 am just trying to explain, I will use this and show you how you can use this checkpoint

• 01:13:51 in your queries in your text to image tab.

• 01:13:54 First I will cancel the training.

• 01:13:55 This one also looks like a decent one.

• 01:13:59 Hit interrupt: yeah.

• 01:14:01 3240 also looking decent so it may get even better over time as we do more training, but

• 01:14:08 I don't have too much time.

• 01:14:10 Okay, so to be able to use these embeddings first, we need to copy the generated pt file

• 01:14:16 which is the checkpoint.

• 01:14:18 To do that, go to the Textual Inversion inside your main folder, go to the date that you

• 01:14:23 did training, go to the training name, go to the embeddings, and in here you will see

• 01:14:28 the dot pt files.

• 01:14:30 Pick the checkpoints that you want to test right, click, copy, then go back to the main

• 01:14:36 installation folder and in here you will see embeddings folder and paste them there like

• 01:14:42 this so it is pasted now here.

• 01:14:44 So to activate this Textual Inversion, we are going to type it like this.

• 01:14:50 By the way, there is one very important thing when you do training, it will train based

• 01:14:56 on the model selected here.

• 01:14:57 Therefore this will be most compatible with this selected model and just hit generate

• 01:15:04 and you see our face is generated trained subject.

• 01:15:07 Now we can try stylizing.

• 01:15:09 Okay, I did a simple test awesome, intricate, 3d artstation, cinematic lightning and generated

• 01:15:16 batch size as eight and these are the generated images.

• 01:15:20 So with better prompting it should be possible to get better results.

• 01:15:25 You can do same training on protogen or any other custom model as well, just check it

• 01:15:31 from here, make a new embedding and do training.

• 01:15:35 The Textual Inversion training works pretty decent on custom models as well.

• 01:15:40 However, custom models are not working very well with DreamBooth training.

• 01:15:44 Okay, so our image generation for classification data set for SD 2.1 is completed.

• 01:15:52 Now we will put them into the correct folder so all of the images are now generated inside

• 01:15:59 this folder.

• 01:16:00 How am I gonna do that?

• 01:16:01 I will right click cut, then I will go to the workspace, right, click paste and then

• 01:16:07 I will rename as class 768 version 2 like this.

• 01:16:14 Then I will go to the DreamBooth tab, I will open my test, load settings, go to the settings

• 01:16:21 and in here I will set the concept the classification data set directory as class 768 version 2

• 01:16:30 and now I have 50 images for per instance.

• 01:16:34 Okay, everything else is same.

• 01:16:36 Just save settings and hit train and let's see if we will get out of memory error or

• 01:16:41 not.

• 01:16:42 So it is preprocessing class images.

• 01:16:44 We can see the command line interface okay, uh, so it looks like the Gradio is killed

• 01:16:51 or our web app.

• 01:16:52 Therefore, we need to restart it.

• 01:16:55 By the way, we also need to disable xformers, otherwise it won't work for training.

• 01:16:59 So I am disabling xformers, saving, closing all of the terminals and starting a new instance

• 01:17:06 of the web ui.

• 01:17:08 Okay, restart is done.

• 01:17:09 You see these are the command line arguments that I have used to start 2.1 version Web

• 01:17:16 UI let's open it.

• 01:17:18 Go to the DreamBooth select model, click load settings.

• 01:17:22 Just verify settings quickly if they are correct or not.

• 01:17:25 Okay, all looking good and let's click train to see how it works.

• 01:17:30 Okay, preprocessing class.

• 01:17:32 Let's also see the cmd window from here.

• 01:17:35 Okay, you see it says nothing to generate because we already have sufficient number

• 01:17:40 of classification images in our folder 456 and we need 450 images.

• 01:17:47 So it is caching right now.

• 01:17:49 Okay, after caching it is killed once again and trying to relaunch.

• 01:17:55 Okay, we got out of memory error so we need to enable some more of the memory optimization

• 01:18:03 and I already unchecked the EMA.

• 01:18:07 Therefore, looks like we need some more optimization.

• 01:18:10 So I will pick fp16, but we are not using mixed precision so it is probably being ignored.

• 01:18:17 What else we can do for more optimization?

• 01:18:21 Gradient checkpointing yes, we can do this and let's save settings, load settings, and

• 01:18:28 hit train once again.

• 01:18:30 Okay, looks like I had to refresh load settings.

• 01:18:34 Hit train okay, yeah, it says that change in precision detected.

• 01:18:39 Please restart Web UI entirely to use new precision.

• 01:18:43 All right, so we will restart it.

• 01:18:46 Okay.

• 01:18:47 Restart is done.

• 01:18:48 Let's go to DreamBooth select model load settings and now gradient checkpointing enabled.

• 01:18:54 Use 8-bit adam fp16, memory attention default, cache latents and let's see if we will get

• 01:19:02 any error or not.

• 01:19:03 Okay, training started this time.

• 01:19:05 I hope we don't get any error during preview generation because it also uses GPU and we

• 01:19:11 can see our GPU is being used 95 percent already.

• 01:19:16 You can also see other utilization parameters here volume, container, and this is my other

• 01:19:21 running pod and this is how much I have spent and how much I am spending.

• 01:19:26 So now I will show you how to install ControlNet on SD 1.5 version.

• 01:19:33 If you don't know what is control net and how to install and use it.

• 01:19:37 I already have a great tutorial on my channel.

• 01:19:40 So this is the extension that we are going to install.

• 01:19:42 Copy the extension URL.

• 01:19:45 You can also find this in the description.

• 01:19:47 Go to the extension tabs, go to the install from URL, copy paste it, and click install.

• 01:19:53 Then once it is installed, go to the installed tab, apply and restart UI.

• 01:19:58 After we clicked it and unfortunately the Gradio is died again.

• 01:20:01 So I will relaunch it and since I am not going to do any training, I am enabling xformers

• 01:20:07 once again because it will speed up my image generation.

• 01:20:11 Okay, after restart, go to the text to image tab and in the bottom you should see ControlNet

• 01:20:16 like this.

• 01:20:17 Now we need to download ControlNet model which is hosted on Hugging Face in here.

• 01:20:24 Go to the files and versions and just download which model that you want to use.

• 01:20:29 Because each model files are like five gigabytes.

• 01:20:32 I'm going to show scribble as an example.

• 01:20:35 All others are same, exactly same and when you watch this video you will learn more about

• 01:20:40 them.

• 01:20:41 Okay right.

• 01:20:43 Click the download button, copy link path, go to your RunPod.

• 01:20:46 So these files will be put inside another folder.

• 01:20:50 Go to the extensions, go to the sd Web UI control net, go to the models.

• 01:20:55 We are going to put them inside here.

• 01:20:58 So in here I will open new launcher, open terminal wget and copy paste the link and

• 01:21:05 you see it has started downloading file from Hugging Face with an incredible speed.

• 01:21:10 Meanwhile I will show something else how you can download your trained models into your

• 01:21:15 computer.

• 01:21:17 So to download your trained DreamBooth model, go to the models, go to the Stable Diffusion,

• 01:21:23 go to the training and let's say you want to download this ckpt.

• 01:21:27 You can right click and download.

• 01:21:30 Or you can use runpodctl as we already shown multiple times.

• 01:21:34 But let's just show once again, runpodctl send the checkpoint file full name, not the

• 01:21:41 directory and it generated the download command like this: go to the download folder where

• 01:21:47 you want to download.

• 01:21:48 So let's say I want to download here.

• 01:21:50 Open cmd, right!

• 01:21:51 click, paste and hit enter and that model file will be downloaded into your computer

• 01:21:57 with a great speed like this as you can see.

• 01:22:00 It is downloading with 70 megabits per second and my maximum internet is 100 megabits per

• 01:22:06 second.

• 01:22:07 So this will of course totally depend on how other users are currently using the Pot network.

• 01:22:14 Okay, meanwhile ControlNet file is downloaded and saved in the folder.

• 01:22:19 Let's verify it.

• 01:22:20 Go to the extensions sd web ui control net inside models.

• 01:22:24 I see the pth file.

• 01:22:26 Let's go back to the ControlNet and in here.

• 01:22:29 When we refresh models we should see it.

• 01:22:33 Yes it is here and there is also pre-processor, then upload your file into this canvas that

• 01:22:40 you want to use.

• 01:22:41 I will do a scribble.

• 01:22:43 I am going to use this file.

• 01:22:46 Let's set the canvas with and height like this, also set your target resolution.

• 01:22:50 I will use the native resolution of the provided image which is 866 and 684.

• 01:22:58 Then type your prompt here and you can use the any model from here.

• 01:23:03 Let's use Protogen model so my prompt is dragon, awesome, intricate, cinematic, artstation.

• 01:23:08 Let's type some negative low,bad, worse.

• 01:23:11 Hit generate.

• 01:23:12 Okay, we didn't get the output because we didn't enable the ControlNet.

• 01:23:16 Don't forget that.

• 01:23:18 And don't forget the check scribble mode, invert colors and now it is the map it it

• 01:23:25 generated and this is the output we got.

• 01:23:28 So you can play with different prompts and different models and generate different images.

• 01:23:35 It works pretty fast and pretty correct.

• 01:23:38 Just watch this video to learn more.

• 01:23:39 Actually, I have another control net video as well which is based on the native released

• 01:23:45 scripts from the official author.

• 01:23:47 You can also watch this video to learn even more about ControlNet.

• 01:23:51 Our SD 2.1 version training is going on.

• 01:23:55 However, it looks like there are some problems because generated image is not correct.

• 01:24:00 Okay, I have done a lot of research and looks like there is no way to do SD 2.1 version

• 01:24:07 768 pixels training with DreamBooth without using xformers.

• 01:24:15 I wanted to avoid xformers during training because it reduces the quality of the training.

• 01:24:21 However, 24 gigabytes VRAM is just not enough.

• 01:24:24 So we need to downgrade the xformers version to 0.0.14 I already have an excellent tutorial

• 01:24:33 video for that for windows installation, so now I will show it on unix on RunPod.

• 01:24:40 Alternatively, you can go to the browse servers and in here you can deploy a RunPod with 48

• 01:24:49 gigabytes VRAM or 40 gigabytes VRAM.

• 01:24:52 It is up to you, but they cost more.

• 01:24:55 Therefore, we will just downgrade the xformers version.

• 01:25:00 Now, follow me very carefully to learn how to downgrade xformers on RunPod io.

• 01:25:07 First close all of the running kernels and terminals.

• 01:25:11 Then inside python 3.10 folder, start a new terminal.

• 01:25:16 First, we are going to run this command.

• 01:25:19 Pip Uninstall torch torchvision.

• 01:25:22 Paste it and hit yes and hit yes.

• 01:25:25 Okay, it is uninstalled.

• 01:25:26 Then we are going to run pip Uninstall torch audio.

• 01:25:31 Paste it.

• 01:25:32 Okay, it is done.

• 01:25:33 Then we are going to use pip Uninstall xformers.

• 01:25:36 Hit yes and it is done.

• 01:25:39 You know?

• 01:25:40 Currently I am inside workspace venv lib python 3.10.

• 01:25:44 The folder where you are currently located makes huge difference.

• 01:25:49 Make sure that you are inside the same folder.

• 01:25:51 You can also apply this to SD 1.5 version as well.

• 01:25:56 It is just same thing.

• 01:25:57 Then we are going to install torch vision.

• 01:25:59 Just copy this and paste it and hit enter.

• 01:26:02 Okay, I got error.

• 01:26:04 It says that there is no space left on the device because currently we started with five

• 01:26:11 gigabyte space for runtime.

• 01:26:14 Therefore, I will stop the pod like this.

• 01:26:17 I will edit the disk space from.

• 01:26:21 Click here.

• 01:26:22 More actions.

• 01:26:23 Click edit pod and in here in increase the container disk size.

• 01:26:27 Save it, run it, start it, and reconnect to Jupyter lab.

• 01:26:32 Enter inside the same folder venv lib, python 3.10 open terminal and make sure that you

• 01:26:41 run all of the commands once again to be sure.

• 01:26:44 Pip uninstall hit yes! if they are installed once again and pip uninstall torch audio,

• 01:26:52 then pip uninstall xformers.

• 01:26:54 Okay, it is done, then we will install this one.

• 01:26:58 As you can see, I have changed it because this is the one that is working.

• 01:27:02 Copy paste and hit.

• 01:27:03 Enter and it is going to install.

• 01:27:06 So once the full version of 0.0.17 is released, it will work with DreamBooth.

• 01:27:11 Currently this is a development version as you can see and it is installed.

• 01:27:16 Now we are ready to run our web UI as usual and it should support DreamBooth training

• 01:27:22 with xformers.

• 01:27:23 Before starting, I am going to edit xformers command line arguments minus minus xformers

• 01:27:29 and I am going to add back the full precision minus minus no half and minus minus precision

• 01:27:37 full and minus minus no half vae.

• 01:27:41 Save it, run on a different port, shut down all of the terminals start a new terminal,

• 01:27:47 relaunch the Web UI like this.

• 01:27:49 Okay, so our application is now starting with 0.0.17.dev 448 version for xformers and these

• 01:27:59 are the torch, torch vision, diffusers, and other versions.

• 01:28:03 Okay, it is started now.

• 01:28:04 Time to test whether it is working correctly or not for SD 2.1 DreamBooth training: okay,

• 01:28:12 I am loading my model, load settings and in here.

• 01:28:16 Let me show you quickly the latest settings.

• 01:28:19 So let's make the amount of time to pause between epochs zero.

• 01:28:23 I will save for every 20 epochs.

• 01:28:26 I am unchecking gradient checkpointing.

• 01:28:28 I will make learning rate as default.

• 01:28:30 Actually, let's try it.

• 01:28:32 Okay photo of ohwx man by tomer hanuka for sanity prompt and in advanced tab: now this

• 01:28:38 is important.

• 01:28:39 I will use EMA and in the mixed position, I am going to use fp16.

• 01:28:44 Some cards also supports bf16, but to be sure use fp16.

• 01:28:49 And when you hover your mouse it also says you that required when using xformers and

• 01:28:55 in here I am going to use xformers.

• 01:28:56 This is important.

• 01:28:57 Cache latents: okay, then go to the concepts tab.

• 01:29:02 They are set.

• 01:29:03 Everything is looking good and in saving, generate a ckpt file when saving during training

• 01:29:09 and hit train.

• 01:29:10 By the way, we should have clicked save settings before, but I think it is automatically saved.

• 01:29:16 If it doesn't work right away, just click save settings then hit train.

• 01:29:20 Okay, let's watch the terminal.

• 01:29:22 I hope that we won't get any more.

• 01:29:24 Uh, out of memory error.

• 01:29:26 Okay, it is killed so I will test one more time.

• 01:29:30 Refresh the Gradio, DreamBooth select model load settings.

• 01:29:35 Now this time I will set gradient checkpointing because it looks like necessary.

• 01:29:40 Fp16 use EMA and yes, everything is same and let's try again with save settings.

• 01:29:47 Train: okay, we got another error so this time I won't use EMA.

• 01:29:51 Refresh the interface DreamBooth, model load settings uncheck gradient checkpointing and

• 01:29:58 uncheck use EMA.

• 01:29:59 This is significantly increasing the VRAM usage.

• 01:30:02 Save settings hit train okay.

• 01:30:04 Finally, the training has started and now time to wait and see how well it is learning

• 01:30:10 and training.

• 01:30:12 The Gradio is still responsive.

• 01:30:13 That is very good and it is using this much of GPU memory so you see how much GPU memory

• 01:30:20 usage the EMA is increasing when we check the EMA option.

• 01:30:27 Meanwhile, SD 2.1 version training continues.

• 01:30:29 I will explain what is fine tuning with DreamBooth.

• 01:30:34 Okay, before I show how to do fine tuning.

• 01:30:37 We got an error during the SD 2.1 version training at the 400 steps which means when

• 01:30:44 it is generating a ckpt from the 20th epoch checkpoint.

• 01:30:49 Therefore, I will restart the training with one change one parameter, change load settings,

• 01:30:56 and go to the settings tab and enable gradient checkpointing.

• 01:31:01 The rest is same like this so it should just work fine this time I think.

• 01:31:07 Save settings hit train okay, this time we didn't get any error.

• 01:31:11 During SD 2.1 version training, we got sample yes, somewhat similar.

• 01:31:17 This is the first one at the 20th epoch and we got our sanity prompt as well.

• 01:31:21 This is the loss rate which is very erratic as you can see and this is the VRAM usage

• 01:31:27 like this.

• 01:31:28 Now I can start showing you fine tuning.

• 01:31:30 I have opened my 1.5 version RunPod so what is the difference of fine tuning.

• 01:31:38 In the fine tuning we are not going to use classification images and we are going to

• 01:31:44 use file words.

• 01:31:45 Fine tuning is basically using a lot of good images with proper captions and not using

• 01:31:53 any classification images.

• 01:31:54 The rest is same.

• 01:31:56 So every one of the keywords, every one of the tokens in the captions of the images will

• 01:32:02 be trained and they will become like the images that you use for fine tuning.

• 01:32:09 First of all, we need to process image files and add captions to them.

• 01:32:15 So go to the training tab, go to the preprocess images, set the source directory.

• 01:32:20 I don't have a data set for fine tuning a good data set you need a lot of images you

• 01:32:24 need so I will use my own pictures that I used for training and set a destination like

• 01:32:32 training captioned and in here use BLIP for caption and if those images are not 512 and

• 01:32:41 512 pixels.

• 01:32:42 If you are going to fine tune SD 2.1 version with 768 pixels then you need to change these

• 01:32:49 resolution as well.

• 01:32:50 You can also crop them with autofocal point crop but manually cropping them and preparing

• 01:32:57 them is better and then click preprocess.

• 01:33:00 When the first time you run it.

• 01:33:01 It will download the BLIP model from internet.

• 01:33:04 Okay, preprocessing has been completed now.

• 01:33:07 Training captioned folder is generated.

• 01:33:10 Now you see there are txt files named same as the image file.

• 01:33:16 When you open them, you will see this captioning.

• 01:33:19 So what does this mean.

• 01:33:21 In the fine tuning all of these words, these tokens will be improved by the image they

• 01:33:30 are same named.

• 01:33:32 So all of these words will be improved towards this image.

• 01:33:37 This is what is fine tuning.

• 01:33:39 Let's say you want to improve castle images, then you should have good castle images and

• 01:33:44 inside their description, you should have castle word.

• 01:33:48 And if you want to associate those pictures with other words such as beautiful, intricate,

• 01:33:53 high quality, then you should also put them.

• 01:33:56 So put here whatever the words that you want to improve in your model with related to the

• 01:34:03 picture they are associated with, and then once you prepared good captions and images

• 01:34:09 inside your folder, copy the path of the new folder, go back to your DreamBooth tab and

• 01:34:15 make this setup like this in concepts, workspace, training captioned data directory.

• 01:34:21 Now this is important.

• 01:34:23 In the prompt just type [filewords] and nothing else.

• 01:34:28 This means that whenever it is training that particular image, it will load whatever is

• 01:34:36 written inside here and replace instance prompt with it.

• 01:34:40 That's it.

• 01:34:41 So this will be equal to this prompt for this particular image that is going to train.

• 01:34:49 In class prompt we are not using any classifications or class prompt.

• 01:34:53 In the sample prompt, you can use the [filewords] to see what kind of images it is generating

• 01:34:59 and make sure that class images per instance is zero.

• 01:35:03 Because we don't want to try to keep the previous context of the model, we want its underlying

• 01:35:10 context, latent space to be improved.

• 01:35:14 And that's it everything else is same.

• 01:35:16 So for fine tuning you need a lot of good images, good quality images with good captions.

• 01:35:23 Those captions will be improved.

• 01:35:25 It will also improve the Unet of the model so it will become overall better and overall

• 01:35:30 cooked we can say.

• 01:35:33 Because if you show less number of images than it was trained on, it will lose a lot

• 01:35:38 of the contextual knowledge it has.

• 01:35:42 Therefore, these cooked custom models are not good to train your faces on them because

• 01:35:47 they don't have as much as information as these 1.5 pruned ckpt have.

• 01:35:52 For example, this model was trained on 5 billion images as far as I know of.

• 01:35:59 However, those custom models may be trained on 1000 images, one maybe 10 000 images.

• 01:36:06 So their Unet has become like those 10 000 images instead of being trained on 5 billion

• 01:36:14 images.

• 01:36:15 That is why they are so good, but they have much lesser knowledge in their underlying

• 01:36:21 context in their latent space.

• 01:36:23 So this is basically fine tuning how it is done.

• 01:36:27 If you want to be exactly same as Stable Diffusion training that the official training.

• 01:36:33 You can also remove text encoder training with setting this parameter as zero.

• 01:36:40 So with this way the tokens won't be improved.

• 01:36:44 Only Unet will be improved.

• 01:36:46 However, you don't want that for fine tuning.

• 01:36:49 This is more like using hundreds thousands of images and training from scratch your model.

• 01:36:57 So you should keep it perhaps like one and train Unet as well.

• 01:37:01 So you will train both text encoder and the Unet and improve all of those keywords together.

• 01:37:09 Hopefully I will make another very technical video about how training works, what is Unet,

• 01:37:16 what is text encoder, how they are being changed during training, and it will explain a lot

• 01:37:22 of the questions that are not very well answered in the community.

• 01:37:27 So stay subscribed.

• 01:37:29 Open notifications to not miss it.

• 01:37:31 So let's check out our 2.1 version training.

• 01:37:34 Okay, our sanity prompt already looks like lost its stylizing ability and the sample

• 01:37:41 is not also looking very good.

• 01:37:43 Uh, however, I have seen that it was learning so let's open the directory.

• 01:37:47 Okay, inside DreamBooth, inside samples, let's look at each one of the sample.

• 01:37:53 So this is the 20 epoch.

• 01:37:54 Yes, it has a resemblance.

• 01:37:56 It is not very good.

• 01:37:57 This is the 40 epoch.

• 01:37:59 Very minor resemblance.

• 01:38:01 Let's check out the sanity prompt.

• 01:38:03 The sanity prompt is much better.

• 01:38:05 So this is somewhat similar to me, but stylized in Tomer Hanuka style.

• 01:38:10 So the sanity prompt of the 60 epoch is not good at all.

• 01:38:14 It lost its stylizing.

• 01:38:17 The sample is also not very related, but this is SD 2.1 so it is harder to train and obtain

• 01:38:23 good images.

• 01:38:24 So you see this is the 80 epoch.

• 01:38:26 This is almost as like me.

• 01:38:28 Let me show you for comparison.

• 01:38:31 With 80 epoch 2.1 version, it is started to learning my face very well.

• 01:38:37 Let's check out the sanity prompt.

• 01:38:39 However, sanity prompt also lost its ability to stylize so our learning rate could be very

• 01:38:46 high.

• 01:38:47 Perhaps we should try half of it.

• 01:38:49 Based on your training data set the learning rate may change, number of steps, number of

• 01:38:55 epochs that you need to do training may change.

• 01:38:58 So it is up to you to do multiple trainings and compare how well they are working with

• 01:39:05 x/y/z plots as I have shown.

• 01:39:07 However, the training is working very well.

• 01:39:10 It is learning the subject very well so we managed to make it work very well for SD 2.1

• 01:39:17 version 768 a model.

• 01:39:21 Let me show you the parameters once again.

• 01:39:24 So I will slowly scroll down and you will be able to see all of the settings.

• 01:39:30 This totally depends on your learning rate and how many number of training images you

• 01:39:35 use.

• 01:39:36 You should also save multiple checkpoints during training and compare them: batch size

• 01:39:41 one and gradient accumulation one.

• 01:39:42 If you increase this, it will increase significantly your VRAM usage.

• 01:39:47 Also, we can't say bigger batch size is better.

• 01:39:50 It's a debated topic.

• 01:39:52 Mini batches versus full batches.

• 01:39:54 These two are checked.

• 01:39:55 Otherwise, we are getting VRAM error on 24 gigabyte.

• 01:39:58 This is my current learning rate.

• 01:40:01 This may be fast, so you may try half of it or even lower.

• 01:40:05 This is the resolution.

• 01:40:07 This is the sanity prompt to see how well it stylized.

• 01:40:10 So don't check EMA because you will get error VRAM error even when using xformers.

• 01:40:16 Use 8-bit adam, use fp16 to be sure that it is supported on your graphic card.

• 01:40:22 Use xformers, cache latents, train Unet, train text encoder and these other things are just

• 01:40:29 default.

• 01:40:30 Okay, now I will show you how to install and run Kohya Lora training Kohya GUI on RunPod.

• 01:40:37 To do that we are going to use Kohya ss linux branch.

• 01:40:41 To do that we are going to use kohya ss linux fork of the official repository of kohya ss.

• 01:40:48 This is modified to run on linux.

• 01:40:51 So first of all, we are going to clone the repository into our RunPod.

• 01:40:56 So this is my 1.5 RunPod.

• 01:40:58 I am inside workspace.

• 01:41:00 I have closed everything.

• 01:41:02 Open a new terminal, copy paste the git clone command.

• 01:41:05 It will clone into the kohya ss linux folder, then move into the kohya ss linux, type cd

• 01:41:12 ko type tab.

• 01:41:14 Hit enter and now I am inside kohya ss linux.

• 01:41:18 Then we will generate virtual environment folder with this command.

• 01:41:23 Copy it, hit enter inside this folder.

• 01:41:26 Okay, it is generated.

• 01:41:28 Let's also move it in here.

• 01:41:30 Now we will run the next command which is for activating and entering inside that virtual

• 01:41:36 folder.

• 01:41:37 Actually source venv command: copy paste it.

• 01:41:40 Hit.

• 01:41:41 Enter now you see venv here.

• 01:41:44 That means that currently actually we are running on the newly generated virtual environment

• 01:41:50 folder.

• 01:41:51 Next, we are going to install requirements.

• 01:41:53 This is only one time necessary.

• 01:41:56 The requirements file is located inside here and currently we are also inside that folder

• 01:42:01 so it should work.

• 01:42:02 The requirements installation may take some time.

• 01:42:06 These installations will not affect your other installations because everything being installed

• 01:42:12 here will be only installed inside this folder.

• 01:42:16 Okay, we got an error that says no space left on the drive so I will just.

• 01:42:21 I will just close the RunPod with stop pod and I will increase the container disk size

• 01:42:26 to 10 gigabytes.

• 01:42:28 To do that, click here, edit pod and run it once again.

• 01:42:31 Start then click connect.

• 01:42:33 Connect to Jupyter lab.

• 01:42:35 Okay, it is still being getting launched so be patient.

• 01:42:40 Okay notebook started once again so I will just delete the venv folder rm -r, venv.

• 01:42:49 So I will start from beginning.

• 01:42:51 Python minus m, venv venv then source command activate then install command.

• 01:42:58 It will install the requirements.

• 01:43:00 Okay, all requirements have been installed.

• 01:43:03 As author here noted, it requires python 3.10 and it doesn't work on 3.11 since the RunPod

• 01:43:12 runs on 3.10.9 for Stable Diffusion, it is just fine.

• 01:43:18 Then we will set accelerate config.

• 01:43:20 I am copying this pasting in here.

• 01:43:23 We are still inside that venv folder.

• 01:43:27 So now it will ask us bunch of questions.

• 01:43:29 Select this machine hit, enter select, no distributed hit enter type no to this question,

• 01:43:36 then type no to this question as well.

• 01:43:39 And type no to this question as well.

• 01:43:42 And type all for this question.

• 01:43:45 And do you wish to use fp16 or bf16?

• 01:43:49 select fp16.

• 01:43:51 It will speed up your training and also use lesser VRAM.

• 01:43:56 Okay and everything is ready.

• 01:43:58 Then we are currently activated with source command so we don't need to run this again.

• 01:44:03 We will just run this command and it should start our GUI.

• 01:44:08 Okay, it is running on localhost so therefore we need to run it on shared link.

• 01:44:15 To enable public Gradio link as we are using in Web UI.

• 01:44:19 Open the kohyagui.py go to the interface launch tab here and add this comma Share true, save

• 01:44:28 it and start it once again.

• 01:44:30 So new terminal open first.

• 01:44:33 Activate the venv like this and just run this command and now it has given us a Gradio link.

• 01:44:40 When we run it, the famous Kohya GUI is loaded and ready to do training.

• 01:44:46 The training with it is another topic and I won't cover it in this.

• 01:44:50 Now I will stop my running RunPods and when I stop at them nothing will happen.

• 01:44:55 They will just remain as they are.

• 01:44:57 I can also start them without any GPU.

• 01:45:00 So from here you can select zero and you can start your RunPod to backup your data to download

• 01:45:07 your data without using any GPUs.

• 01:45:09 So when you run them on CPU the disk cost plus it uses 0.16 dollars per hour.

• 01:45:18 So it is still costing something.

• 01:45:20 I think it is costing half of the original GPU price.

• 01:45:24 However, sometimes you may not get a GPU.

• 01:45:27 Sometimes all of the GPUs may be full on the RunPod so you will be have to run it without

• 01:45:33 a GPU.

• 01:45:35 So this is how you start it without using any GPU and there is also terminate.

• 01:45:41 When you hit terminate it will delete your RunPod permanently.

• 01:45:45 I already said this but I am saying it again.

• 01:45:49 So do not hit terminate button unless you are 100 sure because it will delete everything

• 01:45:55 on this RunPod and until you terminate and delete your RunPod it will continue using

• 01:46:02 your credits.

• 01:46:04 Currently I have two RunPods not running and it is using 0.056 dollars per hour.

• 01:46:14 So this is the cost of keeping these two RunPods on my account.

• 01:46:20 And when I delete them you will see this will get decreased.

• 01:46:23 Let's delete first one with terminate pod.

• 01:46:26 Okay and now this should get decreased.

• 01:46:29 Let's go to the my pods.

• 01:46:30 Let's refresh.

• 01:46:32 Okay now you see currently it is decreasing.

• 01:46:35 Zero point zero point twenty eight dollars per hour.

• 01:46:39 This is charged per minute by the way, not per hour and I will also delete this and it

• 01:46:45 will become zero.

• 01:46:47 And now my credits are remaining as they are until I start another pod.

• 01:46:52 There is one final thing that I want to show you cloud sync button here.

• 01:46:57 So with cloud sync you can synchronize your data in your server to these cloud services

• 01:47:04 and there is a great tutorial on the RunPod blog.

• 01:47:08 I will share this link into the description as well so you can read here and set up your

• 01:47:15 cloud storage and set a synchronization with your run pod and everything generated in your

• 01:47:22 RunPod will be synchronized with your cloud.

• 01:47:25 Also you can use the runpodctl that I have shown multiple times to download your data

• 01:47:32 to upload your data.

• 01:47:34 It is up to you that how you want to use it.

• 01:47:36 I think I have covered everything that I have mentioned in the beginning.

• 01:47:41 I hope you have enjoyed.

• 01:47:42 Please like, subscribe and leave a comment to do this tutorial also join our Discord

• 01:47:48 channel, ask any questions that you can't solve.

• 01:47:52 Also, please support us on Patreon.

• 01:47:55 It is really important.

• 01:47:56 The Patreon link and the Discord link will be in the comments and description.

• 01:48:01 All of the links we have used in this video will be in the description.

• 01:48:05 You can also find our Patreon page on our about tab of our youtube page youtube channel.

• 01:48:06 We have so far 26 patrons.

• 01:48:07 I thank them a lot thank them very much.

• 01:48:08 I hope you also become a Patreon.

• 01:48:09 Hopefully see you in another awesome video!

• 01:48:10 Thank you so much.