Skip to content

Commit 874db09

Browse files
TastyDucksTastyDucks
committed
Update README with hand bases
1 parent cbae0fd commit 874db09

File tree

2 files changed

+113
-86
lines changed

2 files changed

+113
-86
lines changed

‎README.md‎

Lines changed: 113 additions & 86 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,107 @@
11
# Spectacles-2-Unitree Coordination Server
22

3-
Connects Snapchat Spectacles Lens client to a Unitree G1 robot client for XR teleop.
3+
Connects Snapchat Spectacles Lens client to a Unitree G1 robot client for real-time XR teleop.
4+
5+
## Usage
6+
7+
### Deployment
8+
9+
#### Coordination Server
10+
11+
Container (AMD64): `docker pull ghcr.io/tastyducks/spectacles-2-unitree-server.server:latest`
12+
13+
Deploy anywhere that hosts containers with a public IP address behind TLS.
14+
15+
> **IMPORTANT**: The server should have the environment variable `DASHBOARD_PASSWORD` set to something with a decent amount of entropy. The default web UI password is `admin`.
16+
17+
#### Unitree G1 Client
18+
19+
Container (ARM64): `docker pull ghcr.io/tastyducks/spectacles-2-unitree-server.client:latest`
20+
21+
There are two ways to run this:
22+
23+
1. **No Unitree G1:** Run the container on your own machine and pass `--mock` as an argument:
24+
```bash
25+
docker run -p 7000:7000 -it ghcr.io/tastyducks/spectacles-2-unitree-server.client:latest --mock --server wss://SERVER_HOST/ws
26+
```
27+
You'll still be able to connect to the client and see the simulated hand tracking.
28+
29+
2. **With a Unitree G1:** Install the [NVIDIA container toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/1.17.5/install-guide.html#with-apt-ubuntu-debian) on the robot's NVIDIA Jetson Orin. Then, copy over [run-robot-client.sh](./unitree-client/run-robot-client.sh) and run it.
30+
31+
<br/>
32+
33+
> **IMPORTANT**: To view the simulated images in the Spectacles Lens, open the meshcat UI in your browser after the Unitree client starts: <http://UNITREE_CLIENT_ADDRESS:7000/static/>.
34+
35+
### Development
36+
37+
This repo includes a [devcontainer](https://containers.dev/) that has the Github CLI and `act` (a local Github actions runner) installed.
38+
39+
- Github authentication: To use `act`, you need to authenticate with Github. Run:
40+
```bash
41+
gh auth login -s repo,gist,read:org,write:packages,read:packages,delete:packages
42+
```
43+
The package permissions are needed for `act` to write to the Github package registry.
44+
- Host mounting: the host's Docker socket (on MacOS and Linux) is mounted into the container workspace.
45+
46+
#### Setup
47+
48+
1. Install Docker or a similar container engine like Orbstack. On MacOS with Homebrew:
49+
```bash
50+
brew install docker # or brew install --cask orbstack
51+
```
52+
2. If using VS Code, install the [Remote Development extension pack](vscode:extension/ms-vscode-remote.vscode-remote-extensionpack). If you're using Cursor, you can grab that extension by following [these instructions](https://www.cursor.com/en/how-to-install-extension).
53+
3. Reopen this project in a container: <kbd>⌘</kbd>+<kbd>Shift</kbd>+<kbd>P</kbd> -> "Dev Containers: Rebuild and Reopen in Container"
54+
4. Fetch dependencies for the server:
55+
```bash
56+
uv sync
57+
```
58+
5. Fetch dependencies for the Unitree G1 client:
59+
```bash
60+
cd unitree-client \
61+
&& source /opt/conda/etc/profile.d/conda.sh \
62+
&& conda activate unitree-client \
63+
&& conda env update -f environment.yml
64+
```
65+
66+
Snapchat Spectacles will refuse WebSocket connections on `localhost`, so the server **must** be run at a public IP address with a valid (not self-signed) SSL certificate. I used [Railway](https://railway.com) to automate deployments from the `main` branch while developing.
67+
68+
The Unitree client can be run locally for testing in "mock" mode. This will allow it to run without access to the robot, and will simply print out commands as they are received:
69+
```bash
70+
# Be sure to activate the conda venv first.
71+
cd unitree-client && source /opt/conda/etc/profile.d/conda.sh && conda activate unitree-client \
72+
&& python main.py --mock --server wss://SERVER_HOST/ws
73+
```
74+
75+
#### Troubleshooting
76+
77+
If you see a message like `*** buffer overflow detected ***: terminated`, or `waiting for dds`, it probably means the C++ bindings for the Unitree code are attempting to connect to the robot and failing.
78+
79+
In development (working off the robot), be sure to pass `--mock` when running the Unitree client.
480

581
## Architecture
682

783
The server uses `aiohttp` and `jinja2` to serve a simple web interface for pairing together Spectacles and Unitree G1 clients on a first-come-first-serve basis and monitoring messages. The dashboard is served at the root `/`, and the WebSocket server is served at `/ws`.
884

9-
The server also handles inverse kinematics calculations between Spectacles' wrist and hand keypoints and the robot's URDF model.
85+
Clients are *not authenticated* and messages are passed transparently between the two clients without modification.
1086

11-
Clients are not authenticated currently, and messages are passed transparently between the two clients without modification.
87+
Both the robot client and the Spectacles lens maintain a persistent WebSocket connection to the server.
1288

13-
### Spectacles-space and Unitree-space transformations
89+
The Unitree client runs inverse kinematics calculations via [Pinocchio](https://github.com/stack-of-tasks/pinocchio), transforming the Spectacles hand tracking data into the robot's URDF basis.
90+
91+
### Spectacles-space and Unitree-space Transformations
1492

1593
Recorded with [version 0.10.0 of the Spectacles Interaction Kit][SIK-0.10.0]
1694

17-
The **spectacles world basis** is centered on the user's head (X right, Y up, Z back), and uses a right-handed coordinate system.
95+
The **spectacles world basis** is centered on the user's head (X right, Y up, Z back) and uses a right-handed coordinate system.
1896

1997
Reference pose: hold left and right hands in front of the face, palms facing toward the face, and thumbs pointing outward.
2098

99+
- Right vector: Green
100+
- Up vector: Red
101+
- Back vector: Blue
102+
103+
![Reference pose perspective](hands.jpeg)
104+
21105
The Spectacles has non-uniform joint bases.
22106
If the entries share a symbol, they use the same basis:
23107

@@ -42,124 +126,67 @@ For reference, review the Spectacles [landmark names].
42126
- Hand: left, right
43127
- Landmarks: wrist, pinky-0, pinky-1, pinky-2, ring-0, ring-1, ring-2, middle-0, middle-1, middle-2, index-0, index-1, index-2
44128
- X right (index to pinky on the left hand, pinky to index on the right hand)
45-
- Y up (wrist to fingertips)
129+
- Y up (fingertips to wrist)
46130
- Z back (palm to back of hand).
47131

132+
**The rest of these are described relative to "X".**
133+
48134
#### P
49135

50136
- Hand: left, right
51137
- Landmarks: pinky-3
52-
- X
53-
- Y
54-
- Z
138+
- X (X.X inverted) (pinky to index on the left hand, index to pinky on the right hand)
139+
- Y (X.Z) (palm to back of hand)
140+
- Z (X.Y) (fingertips to wrist)
55141

56142
#### R
57143

58144
- Hand: left, right
59145
- Landmarks: ring-3
60-
- X
61-
- Y
62-
- Z
146+
- X (X.X) (index to pinky on the left hand, pinky to index on the right hand)
147+
- Y (X.Z) (palm to back of hand)
148+
- Z (X.Y inverted) (wrist to fingertips)
63149

64150
#### M
65151

66152
- Hand: left, right
67153
- Landmarks: middle-3
68-
- X
69-
- Y
70-
- Z
154+
- X (X.X) (index to pinky on the left hand, pinky to index on the right hand)
155+
- Y (X.Z inverted) (back of hand to palm)
156+
- Z (X.Y) (fingertips to wrist)
71157

72158
#### A
73159

74160
- Hand: left
75161
- Landmarks: thumb-3
76-
- X
77-
- Y
78-
- Z
162+
- X ???
163+
- Y ???
164+
- Z ???
79165

80166
#### J
81167

82168
- Hand: left
83169
- Landmarks: thumb-0, thumb-1, thumb-2
84-
- X
85-
- Y
86-
- Z
170+
- X ???
171+
- Y ???
172+
- Z ???
87173

88174
#### B
89175

90176
- Hand: right
91177
- Landmarks: thumb-3
92-
- X
93-
- Y
94-
- Z
178+
- X ???
179+
- Y ???
180+
- Z ???
95181

96182
#### K
97183

98184
- Hand: right
99185
- Landmarks: thumb-0, thumb-1, thumb-2
100-
- X
101-
- Y
102-
- Z
103-
104-
## Usage
105-
106-
### Development
107-
108-
1. Install Docker or a similar container engine like Orbstack. On MacOS with Homebrew:
109-
```sh
110-
brew install docker # or brew install --cask orbstack
111-
```
112-
2. If using VS Code, install the [Remote Development extension pack](vscode:extension/ms-vscode-remote.vscode-remote-extensionpack). If you're using Cursor, you can grab that extension by following [these instructions](https://www.cursor.com/en/how-to-install-extension).
113-
3. Reopen this project in a container: <kbd>⌘</kbd>+<kbd>Shift</kbd>+<kbd>P</kbd> -> "Dev Containers: Rebuild and Reopen in Container"
114-
4. Fetch dependencies for the server:
115-
```sh
116-
uv sync
117-
```
118-
5. Fetch dependencies for the Unitree G1 client:
119-
```sh
120-
cd unitree-client && uv sync
121-
```
122-
123-
Snapchat Spectacles will refuse localhost websocket connections, so the server **must** be run at a public IP address with a valid (not self-signed) SSL certificate. I used [Railway](https://railway.com) to automate deployments from the `main` branch.
124-
125-
The unitree client can be run locally for testing in "mock" mode. This will allow it to run without access to the robot, and will simply print out commands as they are received:
126-
```sh
127-
cd unitree-client && uv run . -- --mock --server wss://SERVER_HOST/ws
128-
```
129-
130-
131-
This repo includes a devcontainer that has the Github CLI and `act`, a local Github actions runner.
132-
133-
- Github authentication: To use `act`, you need to authenticate with Github. Run:
134-
```bash
135-
gh auth login -s repo,gist,read:org,write:packages,read:packages,delete:packages
136-
```
137-
The package permissions are needed for `act` to write to the Github package registry.
138-
139-
If you want to run just a single action, you can use the `--job` flag:
140-
```bash
141-
act --job build-and-push
142-
```
143-
- Host mounting: the host's Docker socket (assuming MacOS and Linux) is mounted into the container workspace.
144-
145-
#### Troubleshooting
146-
147-
If you see a message like `*** buffer overflow detected ***: terminated`, it probably means the C++ bindings for the Unitree code are attempting to connect to the robot and failing.
148-
149-
In development (working off the robot), be sure to pass `--mock` when running the Unitree client.
150-
151-
### Deployment
152-
153-
- Build the coordination server targeting `linux/amd64` (necessary for `robotpkg-py318-pinocchio`):
154-
```sh
155-
docker buildx build --platform linux/amd64 -t coordination-server .
156-
```
157-
You can do this inside the devcontainer as it mounts the Docker socket from the host via `/var/run/docker.sock`.
158-
> **NOTE**: If you are using Orbstack on an M-series Mac, you'll need to disable Rosetta while building the server's Docker image locally: Orbstack > Settings > System > Use Rosetta to run Intel code (uncheck this).
159-
160-
WIP. Basically, get the Unitree G1 client running on the robot's computer with access to the required dependencies, and the server running in the `Dockerfile` container on a public IP address with a valid SSL certificate. Port `80` should be exposed in the container and mapped to 443 for ingress and egress.
186+
- X ???
187+
- Y ???
188+
- Z ???
161189

162-
> **IMPORTANT**: The server should have the environment variable `DASHBOARD_PASSWORD` set to something with a decent amount of entropy. The default password is `admin`.
163190

164191
[landmark names]: https://developers.snap.com/lens-studio/api/lens-scripting/enums/Packages_SpectaclesInteractionKit_Providers_HandInputData_LandmarkNames.LandmarkName.html
165192
[SIK-0.10.0]: https://developers.snap.com/spectacles/spectacles-frameworks/spectacles-interaction-kit/release-notes#v0100

‎hands.jpeg‎

156 KB
Loading

0 commit comments

Comments
 (0)