Marco structured output nav in sight

[axelpey]

No description provided.

[greptile-apps]

Greptile Summary

This PR revises native Gemini structured output and navigation-in-sight behavior, externalizes the vision system prompt, adds pose information to pre-action history, and changes retained image-history proportions.

• Adds a workaround for schemas representing primitives with no inputs.
• Moves static vision instructions into system_prompt.md.
• Changes navigate_in_sight to structured spatial-indicator and target inputs, adjusts candidate distances, and adds point-selection prompting.
• Records robot coordinates with pre-action images and renders headings in degrees.
• Refreshes the checked-in multimodal-history artifact and README setup content.

Confidence Score: 3/5

The PR should not merge until coordinate normalization and the structured spatial_indicator contract are fixed, because supported coordinate values can break visual processing and schema-valid navigation output can be rejected at runtime.

Two changed execution boundaries can fail concretely: coordinate strings accepted by the input model now undergo unsupported arithmetic, and unrestricted structured-output strings are checked against a case-sensitive navigation enum only after generation.

Files Needing Attention: src/agents/native_gemini_vision_agent.py, src/history/history.py, src/primitives/navigate_in_sight.py, README.md

Important Files Changed

Filename	Overview
src/agents/native_gemini_vision_agent.py	Externalizes the system prompt and converts current headings to degrees, but arithmetic is unsafe for string-valued coordinates allowed by the input model.
src/agents/native_gemini_schema_builder.py	Adds and removes a dummy field so no-input primitive schemas avoid empty objects.
src/primitives/navigate_in_sight.py	Reworks navigation inputs and point-selection prompting; strict values are not represented in the structured schema, and the textual example lacks its reference image.
src/brain_utils/navigation_handler.py	Passes the new in-sight inputs and corrects one navigation-memory safety branch's tuple return.
src/history/history.py	Adds pose annotations to selected pre-action images but assumes coordinates are already numeric.
src/history/types.py	Extends history entries with optional robot-coordinate dictionaries.
src/brain.py	Changes retained image-history proportions and attaches coordinates to pre-action images.
README.md	Adds an unexplained environment-specific benchmark SSH command to local setup instructions.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Robot image payload] --> B[Validate coordinate keys]
    B --> C[Store pre-action image and pose]
    C --> D[Build multimodal history]
    B --> E[Build Gemini user context]
    D --> E
    F[system_prompt.md] --> G[Format system instruction]
    H[Available primitives] --> I[Generate structured response schema]
    E --> J[Gemini visual decision]
    G --> J
    I --> J
    J --> K{Next primitive}
    K -->|navigate_in_sight| L[Read spatial indicator and target]
    L --> M[Select safe visual candidate]
    M --> N[Convert to navigate_to_position]
    K -->|Other primitive| O[Normal Brain task flow]

Loading

_{Reviews (1): Last reviewed commit: "WIP" | Re-trigger Greptile}

[greptile-apps]

Coordinate strings break processing

MultimodalVisionAgentInput.robot_coords permits string values, and image validation only checks that theta exists. A payload such as {"theta": "1.57"} therefore reaches this arithmetic and raises TypeError, preventing the visual decision from being assembled. Stored pre-action coordinates have the same problem because x, y, and theta are numerically formatted without normalization. Validate coordinates as finite numbers when they enter the system, or convert supported numeric strings before using them.

Knowledge Base Used:

• Conversation history and summarization
• Gemini vision execution

[greptile-apps]

Schema permits rejected indicators

The generated structured-output schema treats spatial_indicator as an unrestricted string, but this code accepts only five case-sensitive spellings. A schema-valid response such as "Front of the" or "right of the" is therefore canceled instead of producing a navigation action. Encode the allowed values in the schema, or normalize casing consistently before validation.

Knowledge Base Used:

• Agent response schemas and fast answers
• Navigation strategies

[greptile-apps]

Example lacks visual context

This point-selection example is inserted without its referenced image, while point numbers are assigned anew from the safe candidates in each current scene. The claim that point 7 is best therefore has no visible relationship to the example target and can bias the model toward an arbitrary point in the current image. Include the annotated example image and its candidate layout, or remove the hard-coded point number.

Knowledge Base Used: Navigation strategies

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Internal SSH command is unexplained

This command appears directly after the local memory-command example and hard-codes an internal project, zone, and instance. Contributors can mistake it for a required setup step even though it is unusable without access to that environment. Remove it or place it in a clearly labeled benchmark or deployment section with prerequisites and configurable placeholders.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!