Skip to content

Commit a18917c

Browse files
AndrewBMadisonAndrewBMadison
committed
Merge main into AndrewDevelopment
Resolved conflicts by: - Using main's reasoning_trace.py (more complete file-based TraceStore) - Adding backwards compatibility aliases for old PromptInspector API - Using main's enhanced behavior.py with SpatialMemory support - Using main's inspect_agent.py CLI tool with better features - Combining exports in __init__.py for full backwards compatibility The merge brings in: - Enhanced memory systems (SpatialMemory, improved RAG) - Better reasoning trace infrastructure with JSONL storage - New tools (inspect_memory, navigation, observe_inspector) - Updated hazard and visibility tracking systems - Comprehensive documentation updates 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 parents adba8cf + d7dd93f commit a18917c

62 files changed

Lines changed: 11015 additions & 1032 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎configs/models.yaml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,52 @@ models:
8484
file: "Meta-Llama-3-8B-Instruct.Q8_0.gguf"
8585
sha256: null
8686

87+
# Qwen2.5 models - excellent for structured JSON output
88+
qwen2.5-7b-instruct:
89+
huggingface_id: "bartowski/Qwen2.5-7B-Instruct-GGUF"
90+
description: "Excellent structured output and JSON, great for agents"
91+
size_class: "medium"
92+
formats:
93+
gguf:
94+
q4_k_m:
95+
file: "Qwen2.5-7B-Instruct-Q4_K_M.gguf"
96+
sha256: null
97+
q5_k_m:
98+
file: "Qwen2.5-7B-Instruct-Q5_K_M.gguf"
99+
sha256: null
100+
q8_0:
101+
file: "Qwen2.5-7B-Instruct-Q8_0.gguf"
102+
sha256: null
103+
104+
qwen2.5-14b-instruct:
105+
huggingface_id: "bartowski/Qwen2.5-14B-Instruct-GGUF"
106+
description: "Higher quality Qwen, excellent reasoning and JSON output"
107+
size_class: "large"
108+
formats:
109+
gguf:
110+
q4_k_m:
111+
file: "Qwen2.5-14B-Instruct-Q4_K_M.gguf"
112+
sha256: null
113+
q5_k_m:
114+
file: "Qwen2.5-14B-Instruct-Q5_K_M.gguf"
115+
sha256: null
116+
q8_0:
117+
file: "Qwen2.5-14B-Instruct-Q8_0.gguf"
118+
sha256: null
119+
120+
qwen2.5-32b-instruct:
121+
huggingface_id: "bartowski/Qwen2.5-32B-Instruct-GGUF"
122+
description: "Top tier Qwen, best quality for complex reasoning"
123+
size_class: "xlarge"
124+
formats:
125+
gguf:
126+
q4_k_m:
127+
file: "Qwen2.5-32B-Instruct-Q4_K_M.gguf"
128+
sha256: null
129+
q5_k_m:
130+
file: "Qwen2.5-32B-Instruct-Q5_K_M.gguf"
131+
sha256: null
132+
87133
# Larger models for high quality
88134
llama-2-13b-chat:
89135
huggingface_id: "TheBloke/Llama-2-13B-Chat-GGUF"

‎docs/backlog_items.md‎

Lines changed: 255 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -942,6 +942,96 @@ Advanced learners need to inspect and debug agent memory contents. Currently the
942942

943943
---
944944

945+
#### B-35: SpatialMemory Integration - World Map for Agents 🔄 IN PROGRESS
946+
**Priority**: High
947+
**Component**: Memory / Agent Runtime
948+
**Size**: M
949+
**Status**: Core implementation complete, prompt integration done
950+
951+
**Problem Statement**:
952+
Agents can only see objects currently in their line-of-sight. Once an object goes out of view, the agent forgets it existed. This prevents agents from:
953+
- Navigating back to known resource locations
954+
- Planning paths around remembered hazards
955+
- Building a mental model of the world
956+
957+
**Goal**: Give agents a persistent "world map" that remembers all objects they've seen, even when out of line-of-sight.
958+
959+
**Implementation Tasks**:
960+
961+
**Core Memory System** (Complete):
962+
- [x] Create `WorldObject` dataclass in `schemas.py` for tracked objects
963+
- [x] Create `SpatialMemory` class with grid-based spatial indexing
964+
- [x] Implement `update_from_observation()` to extract and store objects
965+
- [x] Implement `query_near_position()` for proximity queries
966+
- [x] Implement `query_by_type()` for type-based filtering
967+
- [x] Implement `mark_collected()` / `mark_destroyed()` for status tracking
968+
- [x] Implement `summarize()` for LLM context generation
969+
- [x] Add optional semantic search layer (FAISS + sentence-transformers)
970+
- [x] Create unit tests (`test_spatial_memory.py`)
971+
972+
**Framework Integration** (Complete):
973+
- [x] Add `world_map` property to `AgentBehavior` base class (lazy initialized)
974+
- [x] Add `_update_world_map()` method called by framework each tick
975+
- [x] Add automatic update call in `ipc/server.py` before `decide()`
976+
- [x] Add `mark_collected()` helper method to `AgentBehavior`
977+
- [x] Update `on_episode_start()` to clear world map
978+
- [x] Export `SpatialMemory` from `agent_runtime.memory`
979+
980+
**Prompt Integration** (Complete):
981+
- [x] Update `LocalLLMBehavior._build_prompt()` to include remembered objects
982+
- [x] Update `LLMForager._build_context()` to include remembered objects
983+
- [x] Filter out currently visible objects to avoid duplication
984+
- [x] Sort remembered objects by distance from current position
985+
- [x] Include staleness info (ticks since last seen)
986+
987+
**Documentation** (Complete):
988+
- [x] Create `docs/memory_architecture.md` with design philosophy
989+
- [x] Update `docs/memory_system.md` with SpatialMemory docs
990+
- [x] Document common pitfalls (vectors vs structured storage)
991+
992+
**Future Enhancements** (Pending):
993+
- [ ] Add confidence decay based on staleness
994+
- [ ] Add persistence (save/load world map across sessions)
995+
- [ ] Add visualization tools for debugging world map contents
996+
- [ ] Add integration with pathfinding (avoid remembered hazards)
997+
998+
**Architecture Notes**:
999+
1000+
Storage is **in-memory, transient** (cleared each episode):
1001+
```python
1002+
self._objects: dict[str, WorldObject] = {} # name -> object
1003+
self._spatial_grid: dict[tuple, set[str]] = {} # grid cell -> object names
1004+
```
1005+
1006+
Design follows "vector memory for meaning, structured memory for state":
1007+
- Spatial queries are **deterministic** (not fuzzy similarity)
1008+
- O(1) grid-based lookups
1009+
- Optional semantic layer for queries like "food to collect"
1010+
1011+
**Files Changed**:
1012+
| File | Changes |
1013+
|------|---------|
1014+
| `python/agent_runtime/schemas.py` | Added `WorldObject` dataclass |
1015+
| `python/agent_runtime/memory/spatial.py` | New `SpatialMemory` class |
1016+
| `python/agent_runtime/memory/__init__.py` | Export `SpatialMemory` |
1017+
| `python/agent_runtime/behavior.py` | Added `world_map` property |
1018+
| `python/ipc/server.py` | Auto-update world map before decide() |
1019+
| `python/agent_runtime/local_llm_behavior.py` | Include in prompt |
1020+
| `python/user_agents/examples/llm_forager.py` | Include in prompt |
1021+
| `docs/memory_architecture.md` | Design philosophy doc |
1022+
| `docs/memory_system.md` | API documentation |
1023+
| `python/test_spatial_memory.py` | Unit tests |
1024+
1025+
**Acceptance Criteria**:
1026+
- [x] Agents remember objects after they go out of line-of-sight
1027+
- [x] LLM prompts include "Remembered Objects (out of sight)" section
1028+
- [x] Collected resources are filtered out of queries
1029+
- [x] World map cleared at episode start
1030+
- [x] No external dependencies for core functionality (FAISS optional)
1031+
- [x] Tests pass
1032+
1033+
---
1034+
9451035
#### B-33: Tier 3 Reasoning Trace System
9461036
**Priority**: Medium
9471037
**Component**: Agent Runtime
@@ -997,11 +1087,172 @@ Advanced learners want agents that learn from experience. They need hooks to ref
9971087

9981088
---
9991089

1090+
#### B-36a: Physics-Based Movement - Phase 1: Collision Detection
1091+
**Priority**: Medium
1092+
**Component**: Godot / Agent
1093+
**Size**: M
1094+
**Design Doc**: [docs/design/physics_based_movement.md](design/physics_based_movement.md)
1095+
1096+
**Problem Statement**:
1097+
Agents currently move by directly setting `global_position`, passing through all obstacles (trees, walls, hazards). This creates unrealistic behavior and removes spatial navigation challenges.
1098+
1099+
**Goal**: Implement physics-based collision so agents are blocked by solid obstacles.
1100+
1101+
**Implementation Tasks**:
1102+
1103+
**Godot Changes**:
1104+
- [ ] Change `BaseAgent` from `extends Node3D` to `extends CharacterBody3D`
1105+
- [ ] Update `SimpleAgent._process()` to `_physics_process()` with `move_and_slide()`
1106+
- [ ] Add `CollisionShape3D` (CapsuleShape3D) to agent scene
1107+
- [ ] Add collision detection callback `_on_collision()`
1108+
- [ ] Set up collision layers (Agents=4, Obstacles=2, Hazards=3)
1109+
1110+
**Obstacle Setup**:
1111+
- [ ] Add `StaticBody3D` + `CollisionShape3D` to tree prefabs
1112+
- [ ] Add collision to rocks, walls in foraging scene
1113+
- [ ] Configure collision masks correctly
1114+
1115+
**Hazard Behavior**:
1116+
- [ ] Fire: `Area3D` - agent passes through, takes damage while inside
1117+
- [ ] Pit: `Area3D` - traps agent for N ticks, continuous damage
1118+
- [ ] Add `take_damage()` method to BaseAgent
1119+
1120+
**Files Changed**:
1121+
| File | Changes |
1122+
|------|---------|
1123+
| `scripts/base_agent.gd` | `extends Node3D` → `extends CharacterBody3D` |
1124+
| `scripts/simple_agent.gd` | Physics-based movement |
1125+
| `scenes/agents/simple_agent.tscn` | Add CollisionShape3D |
1126+
| `scenes/foraging.tscn` | Add collision to obstacles |
1127+
| `scenes/prefabs/tree.tscn` | Add StaticBody3D |
1128+
1129+
**Acceptance Criteria**:
1130+
- [ ] Agent cannot walk through trees/walls
1131+
- [ ] Agent slides along obstacles (doesn't stick)
1132+
- [ ] Agent can walk into fire (takes damage per tick)
1133+
- [ ] Agent gets trapped in pit
1134+
- [ ] Existing scenes still function
1135+
1136+
**Blocked By**: None
1137+
**Blocks**: B-36b
1138+
1139+
---
1140+
1141+
#### B-36b: Physics-Based Movement - Phase 2: Experience Memory
1142+
**Priority**: Medium
1143+
**Component**: Python / Memory
1144+
**Size**: M
1145+
**Depends On**: B-36a
1146+
**Design Doc**: [docs/design/physics_based_movement.md](design/physics_based_movement.md)
1147+
1148+
**Problem Statement**:
1149+
When agents collide with obstacles or take damage, they have no way to remember and learn from these experiences. The LLM makes the same mistakes repeatedly.
1150+
1151+
**Goal**: Store collision and damage events in memory so the LLM can learn from experience.
1152+
1153+
**Implementation Tasks**:
1154+
1155+
**Python Schema**:
1156+
- [ ] Add `ExperienceEvent` dataclass to `schemas.py`
1157+
- [ ] Fields: tick, event_type, description, position, object_name, damage_taken
1158+
1159+
**SpatialMemory Extension**:
1160+
- [ ] Add `_experiences: list[ExperienceEvent]` storage
1161+
- [ ] Add `record_experience(event)` method
1162+
- [ ] Add `get_recent_experiences(limit)` method
1163+
- [ ] Store collision locations as "obstacle" WorldObjects
1164+
1165+
**IPC Protocol**:
1166+
- [ ] Extend tool result to include `blocked`, `blocked_by`, `blocked_at`
1167+
- [ ] Add damage event reporting from Godot to Python
1168+
- [ ] Add trap event reporting
1169+
1170+
**Prompt Integration**:
1171+
- [ ] Add "Recent Experiences" section to LLM prompt
1172+
- [ ] Format: "Tick 5: Movement blocked by Tree_003 at (5.2, 0, 3.1)"
1173+
- [ ] Format: "Tick 8: Took 10 damage from Fire_001"
1174+
- [ ] Add "Known Obstacles" section from collision memory
1175+
1176+
**Godot Reporting**:
1177+
- [ ] Report collisions via IPC when `move_and_slide()` hits obstacle
1178+
- [ ] Report damage events from fire/pit hazards
1179+
- [ ] Include object name and position in reports
1180+
1181+
**Files Changed**:
1182+
| File | Changes |
1183+
|------|---------|
1184+
| `python/agent_runtime/schemas.py` | Add `ExperienceEvent` |
1185+
| `python/agent_runtime/memory/spatial.py` | Experience storage |
1186+
| `python/agent_runtime/local_llm_behavior.py` | Prompt integration |
1187+
| `python/ipc/server.py` | Handle experience events |
1188+
| `scripts/simple_agent.gd` | Report collisions |
1189+
| `scripts/hazards/fire.gd` | Report damage |
1190+
| `scripts/hazards/pit.gd` | Report trap events |
1191+
1192+
**Acceptance Criteria**:
1193+
- [ ] Collision events logged in Python
1194+
- [ ] Damage events logged in Python
1195+
- [ ] Experiences appear in LLM prompt
1196+
- [ ] Agent avoids previously-collided locations
1197+
- [ ] Experience memory cleared on episode start
1198+
1199+
**Blocked By**: B-36a
1200+
**Blocks**: B-36c (optional)
1201+
1202+
---
1203+
1204+
#### B-36c: Physics-Based Movement - Phase 3: Pathfinding (Optional)
1205+
**Priority**: Low
1206+
**Component**: Godot / Navigation
1207+
**Size**: L
1208+
**Depends On**: B-36b
1209+
**Design Doc**: [docs/design/physics_based_movement.md](design/physics_based_movement.md)
1210+
1211+
**Problem Statement**:
1212+
If the LLM struggles with spatial navigation, we may want automated pathfinding as a fallback or comparison baseline.
1213+
1214+
**Goal**: Add optional pathfinding using Godot's NavigationAgent3D.
1215+
1216+
**Implementation Tasks**:
1217+
1218+
**Navigation Setup**:
1219+
- [ ] Add `NavigationRegion3D` to foraging scene
1220+
- [ ] Bake navigation mesh excluding obstacles
1221+
- [ ] Add `NavigationAgent3D` to SimpleAgent
1222+
1223+
**Navigation Tool**:
1224+
- [ ] Add `navigate_to` tool that uses pathfinding
1225+
- [ ] Keep `move_to` as direct movement for comparison
1226+
- [ ] Tool finds path around obstacles automatically
1227+
1228+
**Stuck Detection**:
1229+
- [ ] Detect when agent hasn't moved significantly for N ticks
1230+
- [ ] Suggest using `navigate_to` when stuck
1231+
- [ ] Optional: Auto-switch to pathfinding on repeated failures
1232+
1233+
**Files Changed**:
1234+
| File | Changes |
1235+
|------|---------|
1236+
| `scenes/foraging.tscn` | Add NavigationRegion3D |
1237+
| `scripts/simple_agent.gd` | NavigationAgent3D support |
1238+
| `python/tools/navigation.py` | New navigate_to tool |
1239+
1240+
**Acceptance Criteria**:
1241+
- [ ] `navigate_to` finds paths around obstacles
1242+
- [ ] Navigation mesh properly excludes trees/walls
1243+
- [ ] Stuck detection identifies when agent is trapped
1244+
- [ ] LLM can choose between direct movement and pathfinding
1245+
1246+
**Blocked By**: B-36b
1247+
**Blocks**: None
1248+
1249+
---
1250+
10001251
## Total Backlog Summary
10011252

1002-
- **High Priority**: 8 items
1003-
- **Medium Priority**: 18 items
1004-
- **Low Priority**: 8 items
1005-
- **Total**: 34 items
1253+
- **High Priority**: 9 items (1 in progress)
1254+
- **Medium Priority**: 20 items (+2 from physics movement)
1255+
- **Low Priority**: 9 items (+1 from physics movement)
1256+
- **Total**: 38 items (+3 from physics movement)
10061257

10071258
**Estimated Timeline**: 6-12 months for all items with 2 developers

0 commit comments

Comments
 (0)