[Question] Poor results when changing action prompt from "flying" to "walking" or "roaring" in demo

[ChambinLee]

Description

While testing the provided demo code, I observed that the algorithm works well for the prompt "The object is flying", producing a smooth and visually plausible animation. However, when I replace "flying" with other action prompts like "walking" or "roaring", the generated results become significantly worse—either the motion is unnatural, the object deforms incorrectly, or it barely moves at all.

Expected Behavior

The algorithm should generalize to different action prompts beyond "flying". At minimum, "walking" or "roaring" should produce recognizable motion patterns, even if not perfectly realistic.

Questions

1. Is the current model overfitted to the "flying" action used in the demo?
2. Would fine-tuning on other actions help, or is the model architecture fundamentally limited to certain motion types?

Any insights or suggestions would be greatly appreciated! Happy to provide more details if needed.

[JarrentWu1031]

Hello! Your observation is correct. I just tested the case you mentioned, and indeed the “dragon” example under the mentioned prompts does not perform as well as expected. Below I will explain why this happens and offer practical suggestions.

[JarrentWu1031]

Why it happens?

• Data scale：As noted in the paper, the released model is trained on 16-frame dynamic mesh sequences with fewer than 4096 vertices. The total number of clips is about 330k, and the number of unique static mesh IDs is under 30k. For a base generative model, this is a relatively small amount.
• Caption quality: We currently use Qwen-VL-2.5 for all captioning, with a simple pipeline—feed a front-view rendered video for captioning and keep captions concise and motion-centric. However, we found that for background-free 3D object renders, the caption quality of VLMs (e.g., GPT-4o, Gemini, Qwen-VL) is noticeably worse than for real-world videos.
• Long-tail object/action combinations: Our training data (from Objaverse 1.0 and DT4D) lacks some rare object–action pairs. In your “dragon roaring” example, most instances in Objaverse 1.0 and DT4D are bipedal creatures; for winged characters, the wings typically flap, which under Qwen-VL-2.5 labeling may bias captions toward “flying” or “waving wings.” This issue can be seen as a combination of the first two points.

[JarrentWu1031]

Tips to get better results with the current AnimateAnyMesh release.

• Try different seeds: For the same prompt and object, outputs can vary significantly across seeds. Some seeds may produce results you find satisfactory.
• Adjust guidance: If a given prompt yields very small motion, try increasing --guidance_scale (e.g., to 5.0).
• Adjust the number of sampled trajs: Another way to amplify the motion is: setting --num_traj to be smaller than --max_length // 8. For example, for a mesh with 20,000 vertices, we default to sampling 20000 // 8 = 2500 trajectories; you can try setting --num_traj to 2000 or 1500. (Note: Sampling too few trajectories can degrade shape preservation. We train with an 8× downsampling; test-time mismatch may lead to shape distortion.)
• If multiple seeds and trajectory counts still fail: It likely indicates that captions similar to your prompt are rare or OOD problem. In that case, please try rephrasing or changing the prompt. All results shown in the paper and demo can be reproduced by the current model with suitable prompt/seed choices; in our tests we typically tried about three prompt/seed combinations per example.

[JarrentWu1031]

Why we haven’t achieved better performance yet, and what’s next?

• Before the ICCV submission, the available dynamic mesh data we collect—especially unique IDs—was limited to only tens of thousands, and due to training cost we could use only a subset (bounded by vertex count). We even resorted to overlapping segment splits and reversed sequences to increase diversity, which clearly cannot fully meet the data scale needed for strong generative performance. By analogy to video generation, reaching very strong results requires substantially more data than we currently have.
• Despite this, we believe the proposed pipeline has strong potential and will scale with more data. We are actively expanding the dataset and developing captioning methods better suited for dynamic meshes. We are optimistic that with sufficient data and improved annotations, the capabilities of the vertex-based paradigm will scale up significantly. We're working on it!