Text2HOI

Official code of Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction in CVPR 2024
Arxiv paper. / Project page.

Data

Dataset	Resource
H2O	Download
GRAB	Download
ARCTIC	Download
ARCTIC Text Description	Download
MANO	Download
Object Pickle File	Download

Preprocessing GRAB object

GRAB objects have so many vertiecs. So we reduce the number of vertices to 4,000.

python preprocessing_grab_object.py

Folder Tree for Demo

data
├─ h2o
│  ├─ obj.pkl
│  └─ object
│     ├─ book
│     ├─ cappuccino
│     ├─ chips
│     ├─ cocoa
│     ├─ espresso
│     ├─ lotion
│     ├─ milk
│     └─ spray
|
├─ grab
│  ├─ obj.pkl
│  ├─ processed_object_meshes
│  └─ contact_meshes
│     ├─ airplane.ply
│     ├─ alarmclock.ply
│     ├─ apple.ply
│     ├─ banana.ply
│     ...
|
├─ arctic
│  ├─ obj.pkl
│  └─ downloads/data/meta/object_vtemplates
│     ├─ box
│     ├─ capsulemachine
│     ├─ espressomachine
│     ├─ ketchup
│     ...
│
└─ mano
   └─ mano_v1_2
      └─ models
         ├─ info.txt
         ├─ LICENSE.txt
         ├─ MANO_LEFT.pkl
         └─ MANO_RIGHT.pkl

Checkpoints

Download

checkpoints
├─ h2o
├─ grab
└─ arctic

Installation

source scripts/install.sh

Demo

source scripts/demo.sh

Train

Folder Tree for Preprocessing

data
├─ h2o
│  ├─ object
│  ├─ subject1
│  ├─ subject2
│  └─ subject3
|
├─ grab
│  ├─ contact_meshes
│  ├─ processed_object_meshes
│  ├─ s1
│  ├─ s2
│  ├─ ...
│  └─ s10
│
├─ arctic
│  └─ downloads
│     └─ data
│        ├─ raw_seqs
|        ├─ description   # (Download available in the 'Data - Arctic' section above or in the 'For Future Work' section below.)
│        └─ meta
│           └─ object_vtemplates
│
└─ mano
   └─ mano_v1_2
      └─ models
         ├─ info.txt
         ├─ LICENSE.txt
         ├─ MANO_LEFT.pkl
         └─ MANO_RIGHT.pkl

Data preparation (preprocessing)

python preprocessing.py

Alternatively, you can download the preprocessed data files, Download.

data
├─ h2o
│  ├─ balance_weights.pkl
│  ├─ data.npz
│  ├─ obj.pkl
│  ├─ text.json
│  ├─ text_count.json
│  └─ text_length.json
|
├─ grab
│  ├─ balance_weights.pkl
│  ├─ data.npz
│  ├─ obj.pkl
│  ├─ text.json
│  ├─ text_count.json
│  └─ text_length.json
│
└─ arctic
   ├─ balance_weights.pkl
   ├─ data.npz
   ├─ obj.pkl
   ├─ text.json
   ├─ text_count.json
   └─ text_length.json

Train Text2HOI

source scripts/train/train_texthom.sh

Train Refiner

source scripts/train/train_refiner.sh

For Future Work

Text description for Arctic.