Add GPT-6 cabinet-opening skill with Innate proxy access - #737
Draft
axelpey wants to merge 71 commits into
Draft
Conversation
axelpey
force-pushed
the
codex/pull-held-handle
branch
from
August 30, 2026 22:40
6e8567f to
e196309
Compare
axelpey
changed the title
… inside Finds an open cabinet by prompt, parks in front of the middle of the opening, raises the arm to shelf height and sends four wrist-camera views — ahead, left, right, down. It does not judge the contents: the four frames go to the agent, which reads them. The frames travel as skill feedback, not as a SkillOutput evidence image. That is the only path that puts more than one picture in front of the agent: feedback becomes an event carrying an image, and the agent takes the newest four event images per turn (_MAX_EVENT_IMAGES). So the scan is exactly four views and the result attaches no image of its own — a fifth would push the first view out of context. Two things the hardware taught, both about not sweeping the arm through what it came to look at: The peek goes UP first, then IN. It used to stage through ZERO, which holds the gripper at (0.31, 0.21) — further out and lower than the peek at (0.24, 0.34) — so the arm swung forward at shelf height and kicked what was standing there before it ever rose. Now it lifts at a 0.12 m reach, inside the robot's own footprint, and only then swings out at constant height. Teardown mirrors it: REST from inside the cabinet would swing the gripper down and forward through the shelf, so an arm that got past the front tucks back in before folding. The run ends by backing off 0.10 m, and tucks before it drives, because reversing with the gripper still inside drags it along the shelf. The pan is a swing about the base axis, not a wrist yaw. Five joints leave the gripper's yaw as j1 and nothing else, so a pose whose yaw disagrees with its own bearing (atan2(y, x)) asks j1 to be in two places and the IK answers with a compromise; here yaw and bearing are the same number by construction. Reach also shrinks with height — 0.387 m out at z=0.21, 0.289 at z=0.34 — so peek_reach and peek_z are tuned as a pair. approach.py gains NAV_ARM, moved out of pick_any_object: the pose that folds the arm clear of the head camera belongs with the approach every floor skill shares, not with one of them. It also gains Debug, a best-effort JSON telemetry publisher on /brain/approach_debug that every FloorApproach now feeds from its localize and follow stages. Nothing in it may raise — a tuning instrument must not be able to fail a run.
Ported from feat/drop-in-box-debug, where it was the instrument the floor approach was tuned on. It could not be cherry-picked: that branch's approach.py is the older subclass design, so the publisher had to be reworked onto the collaborator one (approach.Debug, in the previous commit). Subscribes to /brain/approach_debug and draws the detection, the back-projected floor point and the live flow-tracked pixel over the frame the vision model actually saw, beside a base_link plan view. The sweet box draws as a rectangle, not a square: range and bearing carry their own tolerance, because one image row near the park is centimetres of range but sub-millimetre of bearing. Debug surface, not product: it is one page plus a route and a rail entry, so it comes back out with rm -rf webapp/js/approach git checkout main -- webapp/js/router.js webapp/js/railLayout.js webapp/tests/railLayout.test.js leaving only a telemetry topic nobody subscribes to. The rail entry changes the rail order, so railLayout.test.js moves with it — the source branch just left that test red.
axelpey
force-pushed
the
codex/pull-held-handle
branch
from
September 4, 2026 23:42
722ab8d to
a407e93
Compare
axelpey
changed the title
axelpey
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
innate-os/open_cabinet_with_gptto acquire and open the lower kitchen cabinet from head/wrist images and measured robot state. GPT-6 Astra selects one bounded action per observation using the Responses API with priority service and low reasoning. The prompt guides a substantial outward pull before following the hinge arc.Managed robots use their existing
INNATE_SERVICE_KEYthrough Innate's OpenAI proxy, so no provider key needs to be installed on the robot.OPENAI_API_KEYis an optional standalone-development fallback when service credentials are absent; proxy failures do not silently switch accounts. Missing credentials fail before motion.The skill uses level IK, fresh images/effort feedback, measured motion checks, cumulative travel limits and cancellation-aware waiting. Cleanup stops arm/base and preserves the grip. Supporting changes add consistent arm-effort units, timestamped telemetry, opt-in skill debug export, and the hinged simulator cabinet with its asset packaging. Tests live in the existing ROS CI test directory.
Validation:
mars_armrebuilt; both C++ effort-conversion tests passed.OPENAI_API_KEYunset, including reasoning/tool-result history. These used recorded observations and executed no motion.A fresh full grasp/opening run and physical-robot validation remain outside this verification. Success is model-reported visual evidence after release. Setup, bounds and provenance:
docs/experiments/gpt6-cabinet.md.