Skip to content

README.md

Latest commit

 

History

History
164 lines (118 loc) · 5.64 KB

README.md

File metadata and controls

164 lines (118 loc) · 5.64 KB
Raw

Steer3D Data Engine

Installation

All scripts (env_setup.sh, run_pipeline.sh, etc.) require the Current Working Directory (CWD) to match the script's location. You might need to run cd dataengine.

Prerequisites:

  • Ampere/Ada/Hopper GPU with 48GB+ VRAM
  • Anaconda on Linux

Setup:

First, decide on a data root directory where all pipeline outputs will be stored:

export DATA_ROOT=/path/to/your/data

Then install all environments:

./env_setup.sh $DATA_ROOT

This creates conda environments for each component and downloads required model checkpoints.

API key and model access:

  • Place your OpenAI key in llm/openai_key.txt
  • You might need to login to HuggingFace and agree to its terms to use RMBG-2.0

Running the Pipeline

1. Initialize Shards

python3 utils/init_shards.py <objaverse_list.json> <seed_start_end> $DATA_ROOT

Creates a shard directory and initializes prompts.json with randomly sampled Objaverse UIDs.

Parameters:

  • objaverse_list.json: Path to Objaverse UIDs list (e.g., objaverse_320k.json downloaded by env_setup.sh)
  • seed_start_end: Format seed_beg_end where UIDs are sampled with seed, then sliced from index beg to end (inclusive)
  • $DATA_ROOT: Root directory for data storage

Example:

# Sample 1000 assets (indices 0-999) with seed 42
conda run -n utils python3 utils/init_shards.py $DATA_ROOT/objaverse/objaverse_320k.json 42_0_999 $DATA_ROOT

# Next 1000 assets (indices 1000-1999) with same seed
conda run -n utils python3 utils/init_shards.py $DATA_ROOT/objaverse/objaverse_320k.json 42_1000_1999 $DATA_ROOT

This creates directories:

$DATA_ROOT/42_0_999/
├── prompts.json
├── original_2d/
├── edited_2d/
├── original_3d/
├── edited_3d/
├── llm_log/
└── latents/

2. Execute Pipeline

./run_pipeline.sh $DATA_ROOT <shard1> [shard2 shard3 ...]

Example:

./run_pipeline.sh $DATA_ROOT 42_0_999 42_1000_1999

Processes all specified shards sequentially through the full pipeline.

3. Export Data

conda run -n utils python3 utils/export_data.py $DATA_ROOT <output.tar> \
    --source_types=png,latents,meshes \
    --target_types=png,latents,meshes \
    --hash_fanout

Please see utils/README.md for detail on how to use the export script.

Pipeline

1. Render Original → render_mesh/

Renders single-view 2D images from Objaverse meshes.

2. Generate Instructions → llm/

Uses GPT to generate diverse editing instructions from rendered images. Updates prompts.json with the generated instructions. (see llm/README.md for schema).

3. 2D Editing → step1x/

Applies editing instructions to 2D images. Processes each input UID with its corresponding instruction from prompts.json to generate edited images. Generates a random UID for each edited image.

4. Remove Backgrounds → birefnet/

Segment objects for 3D reconstruction.

5. 3D Reconstruction → hunyuan/

Reconstructs both original and edited 3D meshes from single-view transparent images using Hunyuan3D.

6. Render Reconstructed → render_mesh/

Renders reconstructed 3D meshes from multiple viewpoints for quality evaluation in subsequent stages.

7. DreamSim Filter → dreamsim/

Evaluates 3D reconstruction quality by computing perceptual similarity between input 2D images and rendered 3D views. Results stored in recon_result.json (see dreamsim/README.md for schema).

8. VLM Verification → llm/

Two-stage VLM verification of editing correctness:

  • Stage 1: Identifies visual differences between original and edited 3D renders
  • Stage 2: Verifies differences match the editing instruction

Only evaluates pairs that passed DreamSim filtering. Outputs verdicts to prompts_result.json (see llm/README.md for schema).

9. Build Metadata → utils/

Aggregates filtering results and builds metadata.csv containing only examples that passed both Dreamsim and VLM filters.

10. Encode Latents → trellis/

Encodes 3D assets (meshes + textures) into structured latent representations using TRELLIS.

Output Structure

$DATA_ROOT/
└── {shard}/
    ├── prompts.json              # Editing instructions (see llm/README.md)
    ├── prompts_result.json       # VLM verdicts (see llm/README.md)
    ├── metadata.csv              # Training metadata (passed examples only)
    ├── original_2d/              # Rendered Objaverse views
    │   ├── {uid}.jpg
    │   └── {uid}.png
    ├── edited_2d/                # 2D edited images
    │   ├── {uid}.jpg
    │   └── {uid}.png
    ├── original_3d/              # Reconstructed original meshes
    │   ├── recon_result.json     # Dreamsim scores (see dreamsim/README.md)
    │   └── {uid}/
    │       ├── mesh.glb
    │       ├── mesh_textured.glb
    │       ├── views/            # 4 canonical views
    │       └── views_tilted/     # 4 tilted views
    ├── edited_3d/                # Reconstructed edited meshes
    │   ├── recon_result.json
    │   └── {uid}/...
    ├── llm_log/
    │   └── editing_check.json    # VLM conversation logs
    └── latents/                  # TRELLIS latent encodings
        └── latents/
            └── dinov2_vitl14_reg_slat_enc_swin8_B_64l8_fp16/
                └── {uid}.npz