How to Enable Multi-GPU Inference for this Models with 4x20GB GPUs?

[ZQ1102118381]

Question:​​
Thank you for this amazing project! I’m encountering issues with inference on multiple GPUs.

​​Context:​​

• I have 4 GPUs, each with 20GB of memory.
• I want to run inference across all 4 GPUs to improve performance.

​​Ask:​​
Could you please guide me on how to modify the code to enable multi-GPU inference? Are there specific configurations or examples I should follow?

[niuwengang]

同问

[zzz5y]

hello sir，have you finish this?

[ZQ1102118381]

hello sir，have you finish this?
Hi,​​

I previously tried using ​Accelerate​ to distribute the model across multiple GPUs to achieve my goal. Unfortunately, during module partitioning, one of the modules was too large (I don’t recall the exact one), and 24GB VRAM was clearly insufficient. As a result, I had to switch to an ​80GB VRAM GPU​ for inference.

You could try refining the ​Accelerate​ approach by splitting each module into even smaller parts. However, please note that ​some operations require both inputs to reside on the same GPU, which complicates the implementation. Given the engineering effort involved, I’d recommend either:

​Upgrading to a higher-VRAM GPU, or
​Using two 48GB GPUs, which might meet your requirements.
Let me know if you need further details!

[zzz5y]

hello sir，have you finish this?
Hi,​​

I previously tried using ​Accelerate​ to distribute the model across multiple GPUs to achieve my goal. Unfortunately, during module partitioning, one of the modules was too large (I don’t recall the exact one), and 24GB VRAM was clearly insufficient. As a result, I had to switch to an ​80GB VRAM GPU​ for inference.

You could try refining the ​Accelerate​ approach by splitting each module into even smaller parts. However, please note that ​some operations require both inputs to reside on the same GPU, which complicates the implementation. Given the engineering effort involved, I’d recommend either:

​Upgrading to a higher-VRAM GPU, or​Using two 48GB GPUs, which might meet your requirements. Let me know if you need further details!

Yes I have 2 48G gpus,And i wonder how to modify the class VideoDiffusionModel.Thank you sir for your so quick response

[ZQ1102118381]

hello sir，have you finish this?
Hi,​​

I previously tried using ​Accelerate​ to distribute the model across multiple GPUs to achieve my goal. Unfortunately, during module partitioning, one of the modules was too large (I don’t recall the exact one), and 24GB VRAM was clearly insufficient. As a result, I had to switch to an ​80GB VRAM GPU​ for inference.
You could try refining the ​Accelerate​ approach by splitting each module into even smaller parts. However, please note that ​some operations require both inputs to reside on the same GPU, which complicates the implementation. Given the engineering effort involved, I’d recommend either:
​Upgrading to a higher-VRAM GPU, or​Using two 48GB GPUs, which might meet your requirements. Let me know if you need further details!

Yes I have 2 48G gpus,And i wonder how to modify the class VideoDiffusionModel.Thank you sir for your so quick response

I apologize, but this work was done two months ago, and since it wasn’t successful, I didn’t keep detailed records. From what I recall, the VideoDiffusionModel consists of modules like the encoder, decoder, and UNet (the exact names can be found in the model config).

You might want to check ​Accelerate’s documentation​ on how to partition such modules—I believe I followed a similar approach. Here’s a ​simple demo​ for reference:
🔗 https://blog.csdn.net/weixin_43457608/article/details/136071789

The basic idea is to ​place different modules on different GPUs, though residual operations (e.g., in the UNet) may need to stay on the same GPU. If this still doesn’t work, you could try ​decomposing the UNet’s convolutional layers further—but first, you’d need to print the model’s full layer structure to identify them.

[ZQ1102118381]

hello sir，have you finish this?
Hi,​​

I previously tried using ​Accelerate​ to distribute the model across multiple GPUs to achieve my goal. Unfortunately, during module partitioning, one of the modules was too large (I don’t recall the exact one), and 24GB VRAM was clearly insufficient. As a result, I had to switch to an ​80GB VRAM GPU​ for inference.
You could try refining the ​Accelerate​ approach by splitting each module into even smaller parts. However, please note that ​some operations require both inputs to reside on the same GPU, which complicates the implementation. Given the engineering effort involved, I’d recommend either:
​Upgrading to a higher-VRAM GPU, or​Using two 48GB GPUs, which might meet your requirements. Let me know if you need further details!

Yes I have 2 48G gpus,And i wonder how to modify the class VideoDiffusionModel.Thank you sir for your so quick response

Additionally,​​ it’s worth noting that ​no_split_module_class in Accelerate​ specifies the class names that ​cannot be split further. This means that as long as a module’s class name is unique, you can ​split and assign it to a specific GPU at the class level.

[zzz5y]

hello sir，have you finish this?
Hi,​​

I previously tried using ​Accelerate​ to distribute the model across multiple GPUs to achieve my goal. Unfortunately, during module partitioning, one of the modules was too large (I don’t recall the exact one), and 24GB VRAM was clearly insufficient. As a result, I had to switch to an ​80GB VRAM GPU​ for inference.
You could try refining the ​Accelerate​ approach by splitting each module into even smaller parts. However, please note that ​some operations require both inputs to reside on the same GPU, which complicates the implementation. Given the engineering effort involved, I’d recommend either:
​Upgrading to a higher-VRAM GPU, or​Using two 48GB GPUs, which might meet your requirements. Let me know if you need further details!

Yes I have 2 48G gpus,And i wonder how to modify the class VideoDiffusionModel.Thank you sir for your so quick response

I apologize, but this work was done two months ago, and since it wasn’t successful, I didn’t keep detailed records. From what I recall, the VideoDiffusionModel consists of modules like the encoder, decoder, and UNet (the exact names can be found in the model config).

You might want to check ​Accelerate’s documentation​ on how to partition such modules—I believe I followed a similar approach. Here’s a ​simple demo​ for reference: 🔗 https://blog.csdn.net/weixin_43457608/article/details/136071789

The basic idea is to ​place different modules on different GPUs, though residual operations (e.g., in the UNet) may need to stay on the same GPU. If this still doesn’t work, you could try ​decomposing the UNet’s convolutional layers further—but first, you’d need to print the model’s full layer structure to identify them.

Thank you for your so kindly and detail response!

[zzz5y]

hello sir，have you finish this?
Hi,​​

I previously tried using ​Accelerate​ to distribute the model across multiple GPUs to achieve my goal. Unfortunately, during module partitioning, one of the modules was too large (I don’t recall the exact one), and 24GB VRAM was clearly insufficient. As a result, I had to switch to an ​80GB VRAM GPU​ for inference.
You could try refining the ​Accelerate​ approach by splitting each module into even smaller parts. However, please note that ​some operations require both inputs to reside on the same GPU, which complicates the implementation. Given the engineering effort involved, I’d recommend either:
​Upgrading to a higher-VRAM GPU, or​Using two 48GB GPUs, which might meet your requirements. Let me know if you need further details!

Yes I have 2 48G gpus,And i wonder how to modify the class VideoDiffusionModel.Thank you sir for your so quick response

I apologize, but this work was done two months ago, and since it wasn’t successful, I didn’t keep detailed records. From what I recall, the VideoDiffusionModel consists of modules like the encoder, decoder, and UNet (the exact names can be found in the model config).

You might want to check ​Accelerate’s documentation​ on how to partition such modules—I believe I followed a similar approach. Here’s a ​simple demo​ for reference: 🔗 https://blog.csdn.net/weixin_43457608/article/details/136071789

The basic idea is to ​place different modules on different GPUs, though residual operations (e.g., in the UNet) may need to stay on the same GPU. If this still doesn’t work, you could try ​decomposing the UNet’s convolutional layers further—but first, you’d need to print the model’s full layer structure to identify them.

Hello sir.I have try to put different modules in different GPUs.But the calculation part and input data are still concentrated in the cuda 0.Do you have any idea where I did wrong?

[ZQ1102118381]

hello sir，have you finish this?
Hi,​​

I previously tried using ​Accelerate​ to distribute the model across multiple GPUs to achieve my goal. Unfortunately, during module partitioning, one of the modules was too large (I don’t recall the exact one), and 24GB VRAM was clearly insufficient. As a result, I had to switch to an ​80GB VRAM GPU​ for inference.
You could try refining the ​Accelerate​ approach by splitting each module into even smaller parts. However, please note that ​some operations require both inputs to reside on the same GPU, which complicates the implementation. Given the engineering effort involved, I’d recommend either:
​Upgrading to a higher-VRAM GPU, or​Using two 48GB GPUs, which might meet your requirements. Let me know if you need further details!

Yes I have 2 48G gpus,And i wonder how to modify the class VideoDiffusionModel.Thank you sir for your so quick response

I apologize, but this work was done two months ago, and since it wasn’t successful, I didn’t keep detailed records. From what I recall, the VideoDiffusionModel consists of modules like the encoder, decoder, and UNet (the exact names can be found in the model config).
You might want to check ​Accelerate’s documentation​ on how to partition such modules—I believe I followed a similar approach. Here’s a ​simple demo​ for reference: 🔗 https://blog.csdn.net/weixin_43457608/article/details/136071789
The basic idea is to ​place different modules on different GPUs, though residual operations (e.g., in the UNet) may need to stay on the same GPU. If this still doesn’t work, you could try ​decomposing the UNet’s convolutional layers further—but first, you’d need to print the model’s full layer structure to identify them.

Hello sir.I have try to put different modules in different GPUs.But the calculation part and input data are still concentrated in the cuda 0.Do you have any idea where I did wrong?

Sorry, I haven't tested this on a 48G device. Could you break down this class further? For example, directly assigning CUDA for each class.

[zzz5y]

hello sir，have you finish this?你好，先生，你吃完了吗？
Hi,​​ 你好,

I previously tried using ​Accelerate​ to distribute the model across multiple GPUs to achieve my goal. Unfortunately, during module partitioning, one of the modules was too large (I don’t recall the exact one), and 24GB VRAM was clearly insufficient. As a result, I had to switch to an ​80GB VRAM GPU​ for inference.我以前曾尝试使用Accelerate将模型分布到多个gpu上以实现我的目标。不幸的是，在模块分区期间，有一个模块太大了（我不记得具体是哪个模块了），24GB VRAM显然不够用。因此，我不得不切换到80GB的VRAM GPU进行推理。
You could try refining the ​Accelerate​ approach by splitting each module into even smaller parts. However, please note that ​some operations require both inputs to reside on the same GPU, which complicates the implementation. Given the engineering effort involved, I’d recommend either:您可以尝试通过将每个模块分成更小的部分来改进Accelerate方法。但是，请注意，有些操作需要两个输入驻留在同一个GPU上，这使实现变得复杂。考虑到所涉及的工程工作，我推荐以下两种：
​Upgrading to a higher-VRAM GPU, or​Using two 48GB GPUs, which might meet your requirements. Let me know if you need further details!升级到更高vram的GPU，或者使用两个48GB的GPU，这可能满足您的要求。如果您需要进一步的细节，请告诉我！

Yes I have 2 48G gpus,And i wonder how to modify the class VideoDiffusionModel.Thank you sir for your so quick response是的，我有2个48G的gpu，我想知道如何修改类VideoDiffusionModel。谢谢你先生这么快的回复

I apologize, but this work was done two months ago, and since it wasn’t successful, I didn’t keep detailed records. From what I recall, the VideoDiffusionModel consists of modules like the encoder, decoder, and UNet (the exact names can be found in the model config).很抱歉，这项工作是两个月前完成的，由于没有成功，所以我没有做详细的记录。据我回忆，VideoDiffusionModel由编码器、解码器和UNet等模块组成（确切的名称可以在模型配置中找到）。
You might want to check ​Accelerate’s documentation​ on how to partition such modules—I believe I followed a similar approach. Here’s a ​simple demo​ for reference: 🔗 https://blog.csdn.net/weixin_43457608/article/details/136071789您可能需要查看Accelerate关于如何划分此类模块的文档—我相信我使用了类似的方法。这里有一个简单的示例供参考：🔗https://blog.csdn.net/weixin_43457608/article/details/136071789
The basic idea is to ​place different modules on different GPUs, though residual operations (e.g., in the UNet) may need to stay on the same GPU. If this still doesn’t work, you could try ​decomposing the UNet’s convolutional layers further—but first, you’d need to print the model’s full layer structure to identify them.基本思想是将不同的模块放在不同的GPU上，尽管剩余的操作（例如，在UNet中）可能需要留在同一个GPU上。如果这仍然不起作用，你可以尝试进一步分解UNet的卷积层——但首先，你需要打印模型的完整层结构来识别它们。

Hello sir.I have try to put different modules in different GPUs.But the calculation part and input data are still concentrated in the cuda 0.Do you have any idea where I did wrong?你好，先生。我试着把不同的模块放在不同的gpu上。但是计算部分和输入数据仍然集中在cuda 0中。你知道我哪里做错了吗？

Sorry, I haven't tested this on a 48G device. Could you break down this class further? For example, directly assigning CUDA for each class.抱歉，我还没有在48G的设备上测试过。你能进一步细分这门课吗？例如，直接为每个类分配CUDA。

hello sir，have you finish this?
Hi,​​

I previously tried using ​Accelerate​ to distribute the model across multiple GPUs to achieve my goal. Unfortunately, during module partitioning, one of the modules was too large (I don’t recall the exact one), and 24GB VRAM was clearly insufficient. As a result, I had to switch to an ​80GB VRAM GPU​ for inference.
You could try refining the ​Accelerate​ approach by splitting each module into even smaller parts. However, please note that ​some operations require both inputs to reside on the same GPU, which complicates the implementation. Given the engineering effort involved, I’d recommend either:
​Upgrading to a higher-VRAM GPU, or​Using two 48GB GPUs, which might meet your requirements. Let me know if you need further details!

Yes I have 2 48G gpus,And i wonder how to modify the class VideoDiffusionModel.Thank you sir for your so quick response

I apologize, but this work was done two months ago, and since it wasn’t successful, I didn’t keep detailed records. From what I recall, the VideoDiffusionModel consists of modules like the encoder, decoder, and UNet (the exact names can be found in the model config).
You might want to check ​Accelerate’s documentation​ on how to partition such modules—I believe I followed a similar approach. Here’s a ​simple demo​ for reference: 🔗 https://blog.csdn.net/weixin_43457608/article/details/136071789
The basic idea is to ​place different modules on different GPUs, though residual operations (e.g., in the UNet) may need to stay on the same GPU. If this still doesn’t work, you could try ​decomposing the UNet’s convolutional layers further—but first, you’d need to print the model’s full layer structure to identify them.

Hello sir.I have try to put different modules in different GPUs.But the calculation part and input data are still concentrated in the cuda 0.Do you have any idea where I did wrong?

Sorry, I haven't tested this on a 48G device. Could you break down this class further? For example, directly assigning CUDA for each class.

Yes i will try further.Thank you sir