The 3D Guided Generative AI Blueprint unlocks greater control over image generation by laying out the content in Blender to guide the image layout. Users can quickly alter the look of the 3D scene using generative AI, and the image outputs can be iterated on by making simple changes in the 3D viewport - such as changing the image perspective by adjusting the camera angle in Blender. Creators can ideate on scene environments much faster using generative AI, and adjustments are made much faster due to the control offered by using the viewport as a depth map.
The blueprint produces high-quality outputs by leveraging the FLUX.dev NIM, using Black Forest Labs' state-of-the-art FLUX.dev models, and ComfyUI provides a flexible and convenient UI. The models are quantized and accelerated on NVIDIA GPUs, doubling performance and enabling this workflow to run on consumer GPUs. Sample image generation times using 30 steps at 1024x1024 resolution on a GeForce RTX 5090:
| NIM | Native (FP8) |
|---|---|
| 11 sec | 25 sec |
This blueprint is ready for non-commercial use. Contact [email protected] for commercial terms.
This blueprint supports the following NVIDIA GPUs: RTX 5090, RTX 5090 Laptop, RTX 5080, RTX 4090, RTX 4090 Laptop, RTX 4080, RTX 6000 Ada. We recommend at least 48 GB of system RAM.
The NIM Prerequisite Installer requires Microsoft User Account Control (UAC) to be enabled. UAC is enabled by default for Windows, but if it has been disabled, it must be enabled to ensure successful installation of the NIM Prerequisite Installer.
More information on Microsoft UAC can found: HERE
Download the NIM Prerequisite Installer, and run the NIMSetup.exe file, and follow the instructions in the setup dialogs. This will install the necessary system components to work with NVIDIA NIMs on your system.
You will need to reboot your computer to complete the installation.
Git is required and should be installed using winget from a command prompt:
winget install --id Git.Git
Obtain a Hugging Face API Access Token Open and command prompt and type the following:set HF_TOKENIf a value is shown for the HF_TOKEN like in the image below you can skip the steps for obtaining a Hugging Face API Access Token and proceed to Installing the Blueprint
If the command returns that the environment variable HF_TOKEN is not defined, complete the steps below.
Create a user account at https://huggingface.co/ or login to your existing account.
Select the “Settings” option on the left-hand menu.
From the Left-Hand Menu select Access Tokens
Create a new Access Token with READ permissions. (Note: If you have an existing Access Token with read permissions you may use it instead of creating a new access token)
Copy your access token.
Open a command prompt and issue the following command:
setx HF_TOKEN hf_access_token hf_access_token represents the actual access token value you created in the step above.
Once you have generated an access token you’ll need to agree to the FluxDev Non-Commercial License Agreement and acknowledge the Acceptable Use Policy by visiting: https://huggingface.co/black-forest-labs/FLUX.1-dev
Click the Agree and access repository button.
Repeat the above process to accept the license for the following FLUX model variants:
| Model | URL |
|---|---|
| FLUX.1-Canny-dev | https://huggingface.co/black-forest-labs/FLUX.1-Canny-dev |
| FLUX.1-Depth-dev | https://huggingface.co/black-forest-labs/FLUX.1-Depth-dev |
| FLUX.1-dev-onnx | https://huggingface.co/black-forest-labs/FLUX.1-dev-onnx |
| FLUX.1-Canny-dev-onnx | https://huggingface.co/black-forest-labs/FLUX.1-Canny-dev-onnx |
| FLUX.1-Depth-dev-onnx | https://huggingface.co/black-forest-labs/FLUX.1-Depth-dev-onnx |
# Installing the Blueprint: Open a new Command Prompt and cd to the location where you would like the blueprint files to be installed.
Download the Blueprint and start the installation using the following command:
git clone https://github.com/NVIDIA-AI-Blueprints/3d-guided-genai-rtx.git && cd 3d-guided-genai-rtx && setup.bat
The Setup script will install several 3rd party software packages, by installing this blueprint you agree to the individual licenses for these software packages:
- Microsoft Visual C++ 2015-2022 Redistributable Package
- Blender
- Microsoft VisualStudio 2022 BuildTools
During the setup process you will be instructed to agree to the 3rd party licenses to continue. Press "y" and enter to accept the licenses and continue the installation.
Select Yes when presented with the User Account Control dialog.
The setup installer will install ComfyUI, the ComfyUI plugin for Blender, and other components required for the blueprint.
As part of the setup process the NIM models will be downloaded and cached on the system for later use in the blueprint.
Installation may take up to 90 minutes depending on download speed.
Once complete the installation will list the ComfyUI Path, and Python Path, this information can be used to verify the settings in the Blender ComfyUI add-on.
Once installation is complete start Blender and press open Preferences from the menu: Edit >>Preferences
Select the Add-On section , and click the checkbox next to ComfyUI BlenderAI node.
Expand the ComfyUI BlenderAI node section by clicking on the >
The Add-On will attempt to automatically configure the paths for the ComfyUI and Comfy Python locations. In the ComfyUI Path and the Python Path configuration section, verify that these paths match the paths shown at the end of the blueprint installation process. Alternatively, you can click the folder icon and navigate to the installation location and select the ComfyUI folder, and the python_embedded folder in the ComfyUI installation.
From the Blender menu select File >> Open
Navigate to Documents >> Blender
Select the Guided_GenAI_BP.blend file
If necessary expand the panel in the upper left viewport by clicking on the < indicator. Alternatively move the mouse into the upper left viewport and press the “n” key on the keyboard.
Select the ComfyUI X Blender tab if needed. Click the Launch/Connect to ComfyUI button to start a local ComfyUI server instance.
It may take up to two minutes to start the ComfyUI service and establish a connection.
NOTE: The Blender system console can be opened from the Blender Menu selection Window >> Toggle System Console. The system console can help provide additional information about the ComfyUI startup process and provide updates while ComfyUI tasks are running.
Once ComfyUi has started and is ready the panel will change and a Run button will appear.
If the Run button does not appear or the Launch/Connect to ComfyUI reappears, check the system console for any error messages.
Click the Run button.
The first time the FLUX NIM is utilized it will need to download models from NVIDIA NGC and setup the FLUX NIM container, this process can take up to 20 minutes or more depending on the connection speed.
There will be an initial connection delay when first connecting to the NIM during a session which may take between 2-5 minutes.
By default the sample workflow will use the viewport scene combined with the following prompt to generate an image that matches both the overall look of the 3D scene, and the text prompt:
“a stunning professional photo of a quaint village in the mountains in winter, a snow covered fountain is visible in the center of the town square”
You can change the output, by changing either the text prompt, the 3D viewport information or both. NOTE: When generating output, some parameter must be changed before it’s possible to generate a new output, either the 3D scene information, prompt, or some parameter. If nothing has been changed the workflow will not process a new image.
The ComfyUI Connector panel is linked to the Input Text Node, you can change the prompt information here.
In the prompt input area, add some additional information to the end of the existing text to change the output, for example try any of the following:
“At sunset”
“At night”
“In the rain”
With the mouse in the upper left viewport press SHIFT + ~ to enter navigation mode. You can fly though the scene using the WASD keys and using the mouse to point in a direction. The E and F keys raise and lower the camera. Navigate the scene to find different camera angles.
Click on the fountain object and click delete on the keyboard to remove the fountain.
In the lower left area of the screen grab the boat object and drag it into the upper left viewport to the general location where the fountain was previously.
Replace the entire prompt with one of these:
“A stunning profession photo of a modern luxury boat in the canals of Venice, classical buildings, at sunset”
“An abandoned boat sitting between rows of warehouses”
Change the output path in the SaveImage node to point to a location on your system where you would like to save generated images.
Closing Blender after the FLUX NIM has been loaded may leave the NIM running in the background which can unneccesarily consume GPU and System resources, best practice is to stop the NIM when it is no longer needed, or before exiting Blender.
Steps to stop the NIM:
- In the node tree, locate the LoadNIMNode node
- Switch the operation setting from Start to Stop
- Click Run to run the workflow
- This will cause an error in the NIMFLUXNode as the NIM is no longer available to run the workflow.
- If a NIM isn't stopped via the Stop method of the LoadNIMNode it may be necessary to manually stop the NIM via the command console
- Open a windows command console and type:
-
wsl podman ps -a
- This will show the current active NIM containers
- Note the name of the container FLUX_DEPTH
- Issue the following command:
-
wsl podman stop FLUX_DEPTH
- You can re-run the wsl podman ps -a command to verify that the FLUX_NIM is no longer running.
If errors occur when working with the NIM it may be necessary to restart the ComfyUI Server. To restart ComfyUI, place your mouse cursor in the ComfyUI node graph area and press “N” to display the panel.
Click the 
icon to stop ComfyUI.
Click the 
icon again to restart ComfyUI, or click the Launch/Connect to ComfyUI button.
Re-run the workflow.