Our approach can enhance output motion quality by adding contextually relevant text signals or by translating high-level motion descriptions to a set of low-level motion descriptions. In the following, we show 3 examples for qualitative improvements of the generated motion and one example for a degradation, where the LLM omitted important information. Note that we subsequently built a quality check stage to minimize the occurrence of such refinement failures.
| Ground Truth | Unrefined (MoMask) | Refined (Ours) |
|---|---|---|
a person performs a typical broadjump |
a person performs a typical broadjump |
The person bends their arms and crouches down preparing for a jump, then extend their arms back as they propel themselves forward with their legs. |
a figure winds up for the pitch |
a figure winds up for the pitch |
The figure pulls back their arms in preparation for throwing something. |
person is performing a golf motion |
person is performing a golf motion |
Imitating a golf swing, the person assumes a stance and clasps their hand together in a golf grip, leans forward to simulate a put motion that swings from left to right. |
a person slowly walks in a counter clockwise circle |
a person slowly walks in a counter clockwise circle |
The person's arms are relaxed and swing gently with each step. |
Our approach can also enhance motion output quality for high-level motions not contained in the HumanML3D training data by utilizing the additional world knowledge of LLMs.
| Unrefined (MoMask) | Refined (Ours) |
|---|---|
A man does the floss dance. |
A man moves his arms in parallel and always sideways in an alternating motion so that one arm is always behind and one in front of his body. The hands are always at hip level and his hips move in sync with his arms. |
A person does the running man dance. |
A person imitates slowly running in place with a very rhythmic motion. The head bumps in sync with the feet and the arms are supporting the running motion. |
Details
This repo relies on submodules (MoMask). Pull the whole repo with
git clone --recurse-submodules https://github.com/mkiefferus/DigitalHumans
External repos are found in the folder external_repos
We are using two different conda environments due to dependency conflicts, one for MoMask and one for Text Enhancement.
cd external_repos/momask-codes
conda env create -f environment.yml
conda activate momask
pip install git+https://github.com/openai/CLIP.git
conda env create -f environment_enhance.yml
conda activate enhance
cd external_repos/momask-codes
conda create -n "momask" python=3.7.13
conda activate momask
pip install -r requirements.txt
these are the most important packages:
- spacy
- torch
- tqdm
- openai
Furthermore, you will need to download the en_core_web_sm model:
python -m spacy download en_core_web_sm
conda create -n "enhance" python=3.9.18
pip install -r requirements_enhance.txt
Disclaimer: this section below is the setup section from MoMask. Please follow the link for further details.
bash prepare/download_models.sh
For evaluation only.
bash prepare/download_evaluator.sh
bash prepare/download_glove.sh
To address the download error related to gdown: "Cannot retrieve the public link of the file. You may need to change the permission to 'Anyone with the link', or have had many accesses". A potential solution is to run pip install --upgrade --no-cache-dir gdown, as suggested on wkentaro/gdown#43. This should help resolve the issue.
Visit [Google Drive] to download the models and evaluators mannually.
You have two options here:
- Skip getting data, if you just want to generate motions using own descriptions.
- Get full data, if you want to re-train and evaluate the model.
- (if eligible) refined texts can be shared upon request
(a). Full data (text + motion)
HumanML3D - Follow the instruction in HumanML3D, then copy the result dataset to our repository:
cp -r ../HumanML3D/HumanML3D ./dataset/HumanML3D
NOTE the dataset folder is located in external_repos/momask-codes/dataset.
KIT-Download from HumanML3D, then place result in ./dataset/KIT-ML
This project relies on LLMs for text refinement. Accessing these LLMs is done via the OpenAI client. When working with local language models, skip this part.
When working with OpenAI models (GPT3.5-turbo, GPT4o, ...):
Please create an OPENAI API Token and export it as a global variable to your system. OPENAI_API_KEY = ".."
Follow the instructions given in "Step 2 - Set up your API key for all projects (recommended)" in the OpenAI API Documentation to configure your OpenAI API access.
Details
Remember to switch to our environment for text refinement
conda activate enhance
Use the text_enhance.py script to refine motion descriptions.
Specify the system prompt (to be found in the folder 'prompts') by its filename. Here are some example commands:
- Text refinement using GPT-3.5 Turbo for the first 10 text files, here with the limb-specific refinement strategy
python text_enhance.py --system_prompt limb_specific_v4 --use_example -r --verbose --early_stopping 10 - Prompt enhancement by similarity search for the first 10 text files
python text_enhance.py --prompt_adaptation similarity -r --verbose --early_stopping 10
-v, --verbose: verbose-s, --early_stopping: early stopping - stop refinement after x steps for testing purposes
Text Refinement
-pa, --prompt_adaptation: available options: similarity, regular-sp, --system_prompt: name of the system prompt to be given to the prompt adaptation model, see folder 'prompts' for options--continue_previous: continue refining texts from a specific folder--refine_all_samples: refine whole dataset, not only test-set (default)--use_example: add additional context with assistant/user example pairs--samples_text_file: text file specifying samples to refine, by default test.txt--use_cross_sample_information: treat sample text file as one motion (ignores batch size)--use_llama: use llama instead of GPT-3.5 Turbo (default). Requires to also provide--llama_key <key>--batch_size: specify how many files to batch to speed up refinement (not recommended, likely leads to less detailed added information and more format errors)
Quality control
-ror-d: delete or replace with original refined files if they do not meet the quality control
Details
Remember to switch to the MoMask environment for training
conda activate momask
Use the t2m_train_eval.py script to manage the evaluation and training of different text-to-motion models. The script provides various options for training specific models, resuming training, and evaluating models.
The text folder you specify with texts_folder_name should be located in 'external_repos/momask-codes/data/t2m'.
Find our pretrained MoMask models here.
- Train Masked Transformer Model end-to-end
python t2m_train_eval.py --train_mask --texts_folder_name folder_name - Train Residual Transformer Model end-to-end
python t2m_train_eval.py --train_res --texts_folder_name folder_name - Evaluate All Metrics
python t2m_train_eval.py --eval_all_metrics --texts_folder_name folder_name - Evaluate Single Samples
python t2m_train_eval.py --eval_single_samples --texts_folder_name folder_name
The model folders should all be located in 'external_repos/momask-codes/checkpoints/t2m'
--res_name: Specify the Residual Transformer model to evaluate. Defaults to the original MoMask model.--mask_name: Specify the Masked Transformer model to evaluate. Defaults to the original MoMask model.--texts_folder_name: The name of the folder containing the texts to be used for training or evaluation.-tm, --train_mask: Set if you want to train the Masked Transformer end-to-end-tr, --train_res: Set if you want to train the Residual Transformer end-to-end--eval_single_samples: Whether to generate a multimodal distance score for each sample in the dataset.-v,--verbose: Output information to the console (True) or the logfile (False).-r,--resume_training: Resume training that was stopped before.
Details
To render SMPL animations and images in our paper, we used the MotionGPT repository. In the visualization section of their README, they detail the necessary steps to set up a pipeline that can first fit SMPL meshes from NPY files produced by MoMask and subsequently produce animations using Blender.
Details
Remember to switch to our environment for results analysis
conda activate enhance
This repo provides analysis scripts for post-processing under result_analysis.
Disclaimer These scripts are not part of the original pipeline and were solely used to identify trends to optimise the text refinement and training further.
BART_text_classifier.ipynb leverages the bart-large-mnli model for zero-shot-classification.
The candidate_labels variable holds a list with labels. The default confidence threshold is set to THRESHOLD = 0.87
Provide the link to the text samples folder unter test_txt_path.
This is a multiclass classifier. It will label the data with the provided labels.
This script has the option to output the results to text files for further processing.
filter_test_dataset.ipynb filters test data based on keywords rather than using a classifier to label the test data. It currently tries to identify high level motion descriptions but also contains extensions to identify emotions, adjectives and limbs.
Adjust the PROJECT_ROOT_DIR and the path variables.
This script has the option to output the results to text files for further processing.
Refined texts produced by LLMs often include semantic errors (hallucinated information, removal of relevant information). semantic_check.py aims to find these mistakes and filter them out.
An LLM (default: llama3) is fed the original motion description and the refined description and asked if the texts are roughly equivalent.
Use
python result_analysis/semantic_check.py --data path/to/texts/folder -r
--model: define model to be used (supports 'llama3' and 'gpt-3.5-turbo')-r: replace faulty prompt refinements with original texts-v: verbose
single_sample_score_analysis.ipynb compares the performance of the original text files vs the performance of the refined text files.
It also shows the top 10 improved motion descriptions and the top 10 degradations with respect to the original data.
Adjust the original and altered variables, providing the paths to the two datasets respectively.