Start with conversational missions and add held-object throwing - #779
Draft
axelpey wants to merge 8 commits into
Draft
Start with conversational missions and add held-object throwing#779axelpey wants to merge 8 commits into
axelpey wants to merge 8 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First arrival offers three prepared missions: put away a LEGO brick, find the Backrooms exit, or cross the street. IntroAgent guides ordinary conversation using map-matched spatial memories. Only the physical mission result or explicit Skip unlocks the interface; reopening resumes the same attempt. Completed/skipped first runs persist on the stable broker origin. Help is a separate, responsive interface tour, and completing/skipping a first mission suppresses the redundant Challenges introduction.
Adds ThrowObject for a small object already held, with synchronized gripper release and grasp checks before the committed swing. Preserves wrist roll, guides the agent to approach within short-toss range, and tells it to finish explicitly requested action sequences. Pickup retention is checked after carry. Misses remain recoverable through chat.
Stacked on draft #721 (based on #757). Includes OS environment bridge #776; paired broker controls/completion persistence are in innate-cloud#110. The local combined preview also includes CTA change innate-cloud#107. Durable LEGO pickup is a separate dependency: local integration 2af6ae82e borrowed the validated Astra candidate based on 0a19eff and OpenAI transport from #772; those files are NOT part of this PR. Successful pickup→approach→throw evidence is preserved at local 7ed351fc3. The pickup speed task is still revising its candidate, so one final compatibility run is pending before the combined work is ready.
Validation: all nine webapp test files pass; focused challenge/memory/intersection physics tests pass; three ROS-backed throw tests pass; viewer environment tests, build and typecheck pass. Full webapp typecheck still has existing errors outside these changes. Live desktop/mobile chooser, conversational retry, reopen during pickup, Skip, keyboard handling and passive Help verified. All three missions passed real physical judges. The cleanup retry completed pickup→30 cm approach→throw and settled the brick inside the box. A fresh first-run street pass revealed the UI automatically; completion survived a fresh iframe origin, without a second introduction. Tests also caught a mistaken agent success claim: it did not unlock the interface, and normal chat correction continued the mission. Agent distance/destination judgments can still need user correction. No physical robot testing or deployment.