Issues with Recommended Generation to Physics Simulation Workflow

[penghaohsu]

Hi, thanks for the great work and open-sourcing this amazing project!

I’m currently trying to follow the “Recommended Generation to Physics Simulation Workflow” described in the README, but I’ve encountered some issues that I hope to get clarification on.

1. Error with pytorch-lightning:

First of all, I use the following scripts to run the main.py using the checkpoints and metadata provided in this repo:

poetry run python main.py +name=pantry_drake load=data/checkpoints/pantry_shelf.ckpt \
  dataset.processed_scene_data_path=data/metadatas/pantry_shelf.json \
  dataset.model_path_vec_len=15 \
  dataset.max_num_objects_per_scene=61 \
  experiment.tasks=[test] \
  algorithm.test.num_directives_to_generate=5

and the code crashes with the following error:

MisconfigurationException: `test_dataloader` must be implemented to be used with the Lightning Trainer

Although _build_test_loader() is defined in exp_base.py, it is not passed explicitly into the Trainer.test() call. This causes Lightning to fallback to calling self.algo.test_dataloader(), which is not defined in SceneDiffuserBase class, thus raising MisconfigurationException.

At first, I suspected that this issue might be caused by a version mismatch of PyTorch Lightning. However, after checking, I confirmed that my installed lightning version matches the one specified in the repository (e.g., via pyproject.toml or requirements.txt), so the error is not due to API changes across versions.

I was wondering if there’s a preferred way in this codebase to wire that up properly for the test stage? Thanks in advance!

2. Text-conditioned Generation + Simulation?

I’m also wondering whether text-conditioned sampling can be used directly in the physics simulation pipeline.
For instance, can I simply pass the label prompt like this?

poetry run python main.py +name=pantry_drake load=data/checkpoints/pantry_shelf.ckpt \
  dataset.processed_scene_data_path=data/metadatas/pantry_shelf.json \
  dataset.model_path_vec_len=15 \
  dataset.max_num_objects_per_scene=61 \
  experiment.tasks=[test] \
  algorithm.test.num_directives_to_generate=5 \
  algorithm.classifier_free_guidance.weight=0.8 \
  algorithm.classifier_free_guidance.sampling.labels="'text input'"

Will this activate the text-conditioned sampling path during testing, or do I need to modify the experiment/task setup?

3. Dataset Requirement for "Recommended Generation to Physics Simulation Workflow"?

Do I need to pre-download the full dataset for it to work properly? Or is it sufficient to only have the .json metadata file as done in the “sampling without dataset access” example?

Thanks again for sharing such a comprehensive and well-engineered codebase. I’m enjoying exploring it!
Looking forward to your insights.

[nepfaff]

Hey, thank you for your interest in the project! Glad to hear that you like the codebase.

1. Error with pytorch-lightning:

Note that you will need to pass an actual dataset instead of just the metadata when running main.py. The reason is that test mode in main.py computes both a test loss using the dataset and runs sampling.

You can use our released datasets for that by specifying the HF ID:

poetry run python main.py +name=pantry_drake load=data/checkpoints/pantry_shelf.ckpt \
  dataset.processed_scene_data_path=nepfaff/steerable-scene-generation-pantry-shelf \
  dataset.model_path_vec_len=15 \
  dataset.max_num_objects_per_scene=61 \
  experiment.tasks=[test] \
  algorithm.test.num_directives_to_generate=5

2. Text-conditioned Generation + Simulation?

Yes, that will activate the text-conditioned sampling. Text-conditioning is always activated (see config), but we default to a guidance weight of -1, resulting in unconditional samples.
You might want to set masking_prob=0 for inference.

3. Dataset Requirement for "Recommended Generation to Physics Simulation Workflow"?

You will need to download the full dataset (as per the first answer). You could avoid requiring the full dataset by porting the export logic to the sample_and_render.py script. This script only runs inference and thus doesn't require the full dataset. The directive export logic is here (would just need to call that function from the script).

As a warning, let me know if you have issues with the directive export result. In the past, I had some issues where the exported rotations were slightly different to the rotated ones, leading to physical violations. I'd be happy to have another look at that functionality if you observe something similar. For the paper results, I exported as dict and then used that in my proof-of-concept simulation pipeline.

[penghaohsu]

Just a quick update: after switching to the full dataset, I was able to successfully run the script without hitting the test_dataloader error — thanks again for the clarification!

That said, I’m currently encountering a different issue:
During test execution, it seems that loading the Testing Dataloader causes the RAM usage to spike until it fully saturates, which leads to the process freezing midway. I’m still investigating whether there are any config parameters (like dataset loading strategy, number of workers) that I could adjust to mitigate this. I’ll report back once I’ve found something reproducible or conclusive.

Also, once I get the full pipeline running smoothly, I’ll take a close look at the directive export results — especially to check whether the rotation issues you mentioned might appear on my end too. If I notice any similar behavior, I’ll follow up here right away!

Thanks again for your quick and helpful reply!

[nepfaff]

Glad that you got further this time!

Your screenshot looks good. I've personally not faced such an issue, but I never tested on a machine with <96GB of RAM. You could try to reduce the test split further, reduce the test batch size, dataset workers.

Using a profiler might also be helpful. The dataset shouldn't be loaded into memory, but I think HuggingFace needs to load an index or similar. Let me know if none of these work, and I can see if I can replicate somehow.

[penghaohsu]

Thanks a lot for the suggestions — adjusting the dataset num_workers and reducing the test batch_size did the trick! The script now runs smoothly without memory issues.

Now that I’ve moved on to text-conditioned generation, I’ve noticed that the output scenes often don’t match the semantics of my input text descriptions. For example, when I use the prompt:

"A scene with a shelf, three tea bottles, and 5 apples."

The generated scenes sometimes contain objects that don’t match the description.

I’m wondering if there’s something I might have missed in the config that affects classifier-free guidance at inference.

To help debug, I’m attaching:

1. My current scene_diffuser_base.yaml configuration

classifier_free_guidance:
  use: True
  txt_encoder: "bert" # "t5", "clip", "bert"
  txt_encoder_size: "base" # Affects T5 and bert
  txt_encoder_coarse: null # Can be null
  txt_encoder_coarse_size: "" # Affects T5 and bert
  max_length: 110 # Used for T5 and bert, max number of input tokens
  masking_prob: 0 # Probability of masking the text conditioning
  masking_prob_coarse: 0 # Probability of masking the coarse text conditioning
  weight: 0 # w=0 is conditional, w=-1 is unconditional, w>0 is CFG
  sampling: # Test-time sampling
    use_data_labels: False # Use data labels for sampling
    # Labels to use for sampling. Only used if `use_data_labels` is False.
    labels: "A scene with 10 objects." # str | list[str]

2. The full CLI command I used to launch the run

poetry run python main.py +name=pantry_drake  load=data/checkpoints/pantry_shelf.ckpt \
 dataset.processed_scene_data_path=/home/Desktop/steerable_dataset/steerable-scene-generation-pantry-shelf \
 dataset.model_path_vec_len=15 \
 dataset.max_num_objects_per_scene=61 \
 experiment.tasks=[test] \
 algorithm.test.num_directives_to_generate=3 \
 algorithm.classifier_free_guidance.sampling.labels="'A scene with a shelf, three tea bottles, and 5 apples.'"

I’d greatly appreciate it if you could help verify whether the current config looks correct for text-conditioned generation, or whether further tuning is recommended.

[nepfaff]

That config does look correct. What does "often" mean? Could it be within the APF metrics from the paper? Note that the metrics are computed using the prompts present in the dataset. Hence, custom prompts might have slightly lower APF, depending on how out of distribution they are.

I'm on vacation, but I can also play with your config once I'm back in ~2 weeks.