Skip to content

Commit 9ac0c5f

Browse files
FoamyGuyFoamyGuy
authored
Merge pull request #3244 from FoamyGuy/embodiment_kit
add embodiment kit files
2 parents 57c5412 + 43b2c6f commit 9ac0c5f

172 files changed

Lines changed: 2374 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Embodiment_Kit/agent_skill/embodiment-kit/SKILL.md‎

Lines changed: 237 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,237 @@
1+
---
2+
name: embodiment-kit
3+
description: Interact with a physical embodiment kit microcontroller located in the user's room. Send commands to control a display (faces, messages, prompts), NeoPixel lights, a piezo buzzer, and a vibration motor — and to read sensors (temperature, humidity, pressure, ambient lux, microphone, accelerometer). Use to answer questions about the physical environment and to express status, get the human's attention, or communicate visually/audibly/haptically in the room.
4+
metadata:
5+
version: "1.1"
6+
requires: ["python3", "either (ADAFRUIT_AIO_USERNAME + ADAFRUIT_AIO_KEY env vars) or (EMBODIMENT_KIT_HOST env var)"]
7+
---
8+
9+
# Embodiment Kit Skill
10+
11+
This skill lets you read sensors from and drive actuators on a physical embodiment kit microcontroller located in the user's room. Every command round-trips and returns a JSON ack containing the current sensor readings and output state — and, for actuator commands, a `proof` block with before/after sensor readings that confirm the action had a physical effect.
12+
13+
## When to Use This Skill
14+
15+
Use this skill when you need to:
16+
17+
- **Sense the room**: answer questions about the current temperature, humidity, pressure, ambient light, sound level, or device orientation.
18+
- **Express into the room**: signal status, completion, attention, or mood using the display, lights, buzzer, or vibration motor.
19+
- **Interact with the human**: display a message they should read, or pose a multiple-choice question and wait for a button press.
20+
- **Verify physical effects**: confirm an actuator actually fired by reading the `proof` block in the ack.
21+
22+
Choose actuators thoughtfully — these produce real light, sound, and motion in the human's room. Avoid firing them gratuitously or when the human may be sleeping/focused unless they've asked for it.
23+
24+
## Requirements
25+
26+
- Python virtual environment with `requests` installed.
27+
- If using the Adafruit IO transport (see below), also need installed:
28+
- `adafruit-circuitpython-adafruitio`
29+
- `adafruit-circuitpython-connectionmanager`
30+
- `adafruit-circuitpython-minimqtt`
31+
- One of the two transports below must be configured via environment variables.
32+
- CLI script at `<skills_directory>/embodiment-kit/scripts/embodiment_cli.py`.
33+
34+
### Transports
35+
36+
The CLI supports two ways of reaching the device. Set one (the CLI prefers HTTP when both are present):
37+
38+
- **Direct HTTP (recommended for same-network use)**: set `EMBODIMENT_KIT_HOST` to the device's URL or hostname, e.g. `http://192.168.1.189:5000` or `embodiment.local:5000`. A scheme is optional; `http://` is assumed if missing. Port `5000` is default. The CLI POSTs the command JSON to `<host>/embodiment_kit` and reads the ack from the HTTP response — no MQTT broker or Adafruit account needed. This is the lowest-latency option and works fully offline from the public internet.
39+
- **Adafruit IO MQTT (for remote / cross-network use)**: set both `ADAFRUIT_AIO_USERNAME` and `ADAFRUIT_AIO_KEY`. The CLI publishes the command on the `embodiment.client-to-mcu` feed and waits for the ack on `embodiment.mcu-to-client`. Use this when you cannot reach the device directly on the local network.
40+
41+
If neither transport's env vars are set, the CLI exits with an error before sending anything.
42+
43+
## How to Run a Command
44+
45+
General form:
46+
47+
```bash
48+
python <skills_directory>/embodiment-kit/scripts/embodiment_cli.py <command> [key=value ...] [-t TIMEOUT] [-q]
49+
```
50+
51+
- `key=value` arguments are parsed as JSON when possible. Integers like `0xff8800` are parsed as hex. Strings with spaces should be quoted at the shell level (`message="Hello world"`).
52+
- `-q` / `--quiet` suppresses connection logging on stderr, leaving stdout as pure JSON suitable for piping into `jq` or `json.loads`.
53+
- Default timeout is 10s (22s for `show_prompt`). Override with `-t SECONDS`.
54+
- `--commandlist` prints the full command reference from `embodiment_cli_help.md`.
55+
56+
Every successful response is a single line of JSON on stdout, parseable directly:
57+
58+
```json
59+
{
60+
"state": {
61+
"sensors": { "...": "current readings" },
62+
"outputs": { "neopixels": {}, "display": "..." }
63+
},
64+
"metadata": { "type": "ack", "proof": {} },
65+
"command": { "name": "...", "uuid": "...", "arguments": {} }
66+
}
67+
```
68+
69+
The `metadata.proof` field is only present on actuator commands. To consume the output programmatically, pipe through `jq` or read it with `json.loads`:
70+
71+
```bash
72+
python embodiment_cli.py get_data -q | jq '.state.sensors.temperature'
73+
python embodiment_cli.py get_data -q | python -c "import sys, json; print(json.load(sys.stdin)['state']['sensors']['temperature'])"
74+
```
75+
76+
## Sensors (always returned in every ack)
77+
78+
Under `state.sensors`:
79+
80+
- `temperature` — degrees C
81+
- `humidity` — percent
82+
- `pressure` — hPa
83+
- `lux` — ambient light
84+
- `pdm_mic` — normalized RMS sound magnitude
85+
- `accelerometer` — string `"x=<x>, y=<y>, z=<z> G"`, averaged over 10 samples. A device sitting flat reads roughly `z≈9.8`.
86+
87+
### `get_data`
88+
Returns the standard ack with no side effects. Use to answer questions about the room's current state.
89+
90+
```bash
91+
python embodiment_cli.py get_data
92+
```
93+
94+
## Display Face Commands
95+
96+
### Preset Faces
97+
- `show_happy_face`
98+
- `show_sad_face`
99+
- `show_angry_face`
100+
- `show_confused_face`
101+
- `show_sleepy_face`
102+
103+
Shows the specified preset face. Optional `background_color` (24-bit RGB int, default `0x88ddff`).
104+
105+
```bash
106+
python embodiment_cli.py show_happy_face
107+
python embodiment_cli.py show_happy_face background_color=0xffcc00
108+
python embodiment_cli.py show_sleepy_face
109+
python embodiment_cli.py show_sleepy_face background_color=0xffcc00
110+
```
111+
112+
### `show_custom_face`
113+
Build a face from layer choices. First call `list_custom_face_options` to see valid filenames for each layer (`eyebrows`, `eyes`, `blush`, `mouth`). Each layer is a `[filename, y_offset]` pair. The y_offset is relative to the center of the screen. Negative values move the sprite up, positive values move it down. The display height is only 135px, offset values should be within the range -50 to 50 in order to keep them visible.
114+
115+
```bash
116+
python embodiment_cli.py list_custom_face_options
117+
python embodiment_cli.py show_custom_face options='{"eyes":["round_eyes_happy.png",-20],"mouth":["mouth_smiling.png",60]}'
118+
```
119+
120+
## Display Message Commands
121+
122+
### `show_message` — important formatting rules
123+
Hides the face and displays text.
124+
125+
**Display constraints — the screen fits at most 5 rows of 19 characters per row.** Plan messages around this:
126+
127+
- Keep each line ≤ 19 characters. Longer lines will be cut off.
128+
- Use at most 5 lines total. Additional lines will not render.
129+
- Use `\n` (literal backslash-n in the shell argument) to break lines manually — do NOT rely on auto-wrapping.
130+
- Prefer short, scannable phrasing over paragraphs. Abbreviate when needed.
131+
132+
```bash
133+
python embodiment_cli.py show_message message="Build passed\nReady to deploy"
134+
python embodiment_cli.py show_message message="Tests: 42/42 OK\nLint: clean\nCoverage: 87%"
135+
```
136+
137+
Bad (too long per line, will be truncated):
138+
139+
```bash
140+
# DON'T: 23 chars on line 1, no newlines — will overflow / wrap unpredictably
141+
python embodiment_cli.py show_message message="The deployment finished successfully a moment ago"
142+
```
143+
144+
### `show_prompt` — important formatting rules
145+
Displays a message and waits for the human to press one of three hardware buttons (`D0`, `D1`, `D2`) or time out.
146+
147+
**Same display constraints as `show_message`: 5 rows × 19 chars.** Use them to your advantage by structuring the prompt as a question followed by labeled options on their own lines.
148+
149+
Recommended structure: question on line 1, then up to three options on the remaining lines, each labeled with the button index that selects it (`0)`, `1)`, `2)`):
150+
151+
```bash
152+
python embodiment_cli.py show_prompt message="Deploy now?\n0) Yes\n1) No\n2) Later" timeout=30
153+
python embodiment_cli.py show_prompt message="Mood?\n0) Good\n1) OK\n2) Tired"
154+
```
155+
156+
Arguments:
157+
- `message` (string, default `"Press a button"`).
158+
- `timeout` (seconds, default `20`). If you set this, the CLI's own ack-timeout (`-t`) is auto-bumped to `timeout + 2`. If you override both, `-t` must be strictly greater than `timeout`.
159+
160+
The ack's `command.response.prompt_result` will be one of:
161+
- `"btn D0 pressed"` → option 0 chosen
162+
- `"btn D1 pressed"` → option 1 chosen
163+
- `"btn D2 pressed"` → option 2 chosen
164+
- `"timed out"` → no press before timeout
165+
166+
Map the button → option by the labels you wrote in the message. After a press the display briefly shows `"Thank you"`; on timeout it shows `"Prompt timed out"`.
167+
168+
Tips:
169+
- Only three buttons exist — never offer more than three options.
170+
- Keep option labels very short (e.g. `Yes`, `No`, `Later`) so the `N) label` line fits in 19 chars.
171+
- If asking a yes/no question, you can use 2 options and leave the third button unused.
172+
- Pick a `timeout` long enough that the human can read and decide, but not so long that the prompt blocks the conversation indefinitely (20–60s is usually right).
173+
174+
## Light Commands (NeoPixel strip)
175+
176+
### `lights_on`
177+
Turns on the strip. `color` is a 24-bit RGB int (default magenta `0xff00ff`). `brightness` is 0.0–1.0 (default is the current brightness).
178+
179+
```bash
180+
python embodiment_cli.py lights_on color=0x00ff00 brightness=0.4
181+
```
182+
183+
Proof block: `idle_lux_level` vs `on_lux_level` (lux should rise).
184+
185+
### `lights_off`
186+
Fills the strip with black.
187+
188+
```bash
189+
python embodiment_cli.py lights_off
190+
```
191+
192+
Proof block: `idle_lux_level` vs `off_lux_level` (lux should drop).
193+
194+
## Sound Command
195+
196+
### `play_tone`
197+
Plays a tone on the piezo buzzer.
198+
199+
- `frequency` (Hz, default `440`)
200+
- `duty_cycle` (default `2**15` ≈ 50%)
201+
- `duration` (seconds, default `0.5`, **max `3`**)
202+
203+
```bash
204+
python embodiment_cli.py play_tone frequency=523 duration=0.8
205+
```
206+
207+
Proof block: `idle_sound_level` vs `playing_sound_level` (mic RMS should rise).
208+
209+
## Haptic Command
210+
211+
### `vibrate`
212+
Runs the vibration motor for ~0.75 s. No arguments.
213+
214+
```bash
215+
python embodiment_cli.py vibrate
216+
```
217+
218+
Proof block: `idle_z_acceleration` vs `vibrating_z_acceleration` (Z-axis acceleration usually decreases during vibration).
219+
220+
## Patterns for Expressing Status
221+
222+
Some useful patterns for using actuators meaningfully:
223+
224+
- **Task complete, low-key**: `show_message message="Done"` or a brief `play_tone frequency=880 duration=0.2`.
225+
- **Task complete, celebratory**: `show_happy_face` + `lights_on color=0x00ff00 brightness=0.5`, then `lights_off` a few seconds later.
226+
- **Need attention**: `vibrate` + `show_message` describing what you need.
227+
- **Error / warning**: `lights_on color=0xff0000 brightness=0.7` + `show_message message="Build failed\nsee terminal"`.
228+
- **Ask the human a question**: `show_prompt` with a short question and 2–3 labeled options.
229+
- **Ambient status indicator**: `lights_on` with a color encoding state (e.g. blue=working, green=done, red=blocked), turn off when state ends.
230+
231+
Always pair attention-grabbing actuators (sound, vibration, bright lights) with a display message that explains *why* you grabbed attention, so the human knows what to do next. Clean up after yourself — turn lights off when the signal is no longer relevant.
232+
233+
## Reference Files
234+
235+
- `embodiment_cli.py` — the CLI itself.
236+
- `embodiment_cli_help.md` — the canonical command reference (also printable via `--commandlist`).
237+
- `embodiment_message_handler.py`, `code_mcu_mqtt.py`, `code_mcu_httpserver.py` — MCU-side firmware (for context, not invoked from this skill).

‎Embodiment_Kit/agent_skill/embodiment-kit/scripts/code_mcu_httpserver.py‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# SPDX-FileCopyrightText: 2021 ladyada for Adafruit Industries
2+
# SPDX-License-Identifier: MIT
3+
import json
4+
from os import getenv
5+
6+
import board
7+
import supervisor
8+
import wifi
9+
import socketpool
10+
import rtc
11+
import digitalio
12+
import audiobusio
13+
import pwmio
14+
15+
from adafruit_bme280 import basic as adafruit_bme280
16+
from adafruit_debouncer import Debouncer
17+
import adafruit_veml7700
18+
import adafruit_lis3dh
19+
import adafruit_drv2605
20+
import neopixel
21+
import adafruit_ntp
22+
from embodiment_message_handler import EmbodimentMessageHandler
23+
from adafruit_httpserver import Request, Response, Server, POST
24+
25+
# Get WiFi details and Adafruit IO keys, ensure these are setup in settings.toml
26+
# (visit io.adafruit.com if you need to create an account, or if you need your Adafruit IO key.)
27+
ssid = getenv("CIRCUITPY_WIFI_SSID")
28+
password = getenv("CIRCUITPY_WIFI_PASSWORD")
29+
30+
### WiFi ###
31+
if not wifi.radio.connected:
32+
print(f"Connecting to {ssid}")
33+
wifi.radio.connect(ssid, password)
34+
print(f"Connected to {ssid}!")
35+
36+
# update system time
37+
pool = socketpool.SocketPool(wifi.radio)
38+
ntp = adafruit_ntp.NTP(pool, tz_offset=0, cache_seconds=3600)
39+
cur_time = rtc.RTC()
40+
cur_time.datetime = ntp.datetime
41+
42+
### Initialize hardware components ###
43+
bme280 = adafruit_bme280.Adafruit_BME280_I2C(board.I2C())
44+
veml7700 = adafruit_veml7700.VEML7700(board.I2C())
45+
mic = audiobusio.PDMIn(board.D6, board.D5, sample_rate=16000, bit_depth=16)
46+
buzzer = pwmio.PWMOut(board.D9, variable_frequency=True)
47+
48+
lis3dh = adafruit_lis3dh.LIS3DH_I2C(board.I2C())
49+
lis3dh.data_rate = adafruit_lis3dh.DATARATE_1344_HZ
50+
51+
rgb_strip = neopixel.NeoPixel(board.D10, 8, brightness=0.3, auto_write=True)
52+
53+
drv = adafruit_drv2605.DRV2605(board.I2C())
54+
drv.sequence[0] = adafruit_drv2605.Effect(15) # Set the effect on slot 0.
55+
56+
embodiment_config = {
57+
"sensors": [
58+
{"type": "temperature", "sensor": bme280, "units": "C"},
59+
{"type": "pressure", "sensor": bme280, "units": "hPa"},
60+
{"type": "humidity", "sensor": bme280, "units": "%"},
61+
{"type": "lux", "sensor": veml7700, "units": "lux", "property": "autolux"},
62+
{"type": "pdm_mic", "sensor": mic, "units": "normalized_rms"},
63+
{
64+
"type": "accelerometer",
65+
"sensor": lis3dh,
66+
"units": "G",
67+
},
68+
],
69+
"buttons": {},
70+
"piezo_buzzer": buzzer,
71+
"vibration_driver": drv,
72+
"neopixels": rgb_strip,
73+
"display": supervisor.runtime.display,
74+
}
75+
76+
pins = [(board.D0, "D0"), (board.D1, "D1"), (board.D2, "D2")]
77+
for pin_i in range(len(pins)):
78+
pin = pins[pin_i]
79+
dio = digitalio.DigitalInOut(pin[0])
80+
dio.direction = digitalio.Direction.INPUT
81+
# Pins D1 and D2 use different PULL from pin D0
82+
if pin_i == 0:
83+
dio.pull = digitalio.Pull.UP
84+
else:
85+
dio.pull = digitalio.Pull.DOWN
86+
btn = Debouncer(dio)
87+
embodiment_config["buttons"][pin[1]] = btn
88+
89+
embodiment_message_handler = EmbodimentMessageHandler(embodiment_config)
90+
91+
display = supervisor.runtime.display
92+
display.root_group = embodiment_message_handler.main_group
93+
94+
# Set up HTTPServer
95+
server = Server(pool, "/static", debug=True)
96+
server.start(str(wifi.radio.ipv4_address), 5000)
97+
98+
99+
@server.route("/embodiment_kit", POST)
100+
def embodiment_kit(request: Request):
101+
print("getting data")
102+
data = request.json()
103+
print("data: ", data)
104+
if "messages" in data:
105+
resp_obj = embodiment_message_handler.handle_messages(data["messages"])
106+
return Response(request, json.dumps(resp_obj))
107+
return Response(request, json.dumps({"error": "no messages in request data"}))
108+
109+
110+
while True:
111+
server.poll()

0 commit comments

Comments
 (0)