feat: various enhancements

.gitignore

@@ -20,6 +20,7 @@ imgui.ini
 anomaly_images/
 package_condition_images/
 models/
+runs/
 *.bak0
 sim/user/
 backtrace.log

docker/compose.yaml

@@ -25,6 +25,20 @@
 #   - GPU + NPU exposed via the devices below (amdxdna kernel driver loaded on host)
 #   - model weights present in ../models (pixi run download-models), bind-mounted in
 
+# Shared runtime for the demo image (image tag, GPU/NPU devices, memlock, host
+# networking). Both `demo` (interactive) and `demo-trace` (one-shot) merge it in.
+x-demo-runtime: &demo-runtime
+  image: robotecai/agentic-mobile-manipulator:latest
+  devices:
+    - /dev/kfd:/dev/kfd            # AMD GPU compute (ROCm)
+    - /dev/dri:/dev/dri            # AMD GPU render (Vulkan)
+    - /dev/accel/accel0:/dev/accel/accel0  # Ryzen AI NPU (amdxdna)
+  ulimits:
+    memlock:
+      soft: -1
+      hard: -1
+  network_mode: host
+
 services:
   # ── Base: RoboStack ROS 2 + built workspace ────────────────────────────────
   ros2:
@@ -44,7 +58,7 @@ services:
 
   # ── The demo: + agents, local inference engines; runs the whole stack ───────
   demo:
-    image: robotecai/agentic-mobile-manipulator:latest
+    <<: *demo-runtime
     build:
       context: ..
       dockerfile: docker/Dockerfile.demo
@@ -64,17 +78,31 @@ services:
       # Keep NPU (FastFlowLM) weights inside the mounted models/ dir instead of
       # flm's default ~/.config/flm, so they persist via the bind mount.
       - FLM_MODEL_PATH=/root/MobileManipulatorDemo/models/flm
-    devices:
-      - /dev/kfd:/dev/kfd            # AMD GPU compute (ROCm)
-      - /dev/dri:/dev/dri            # AMD GPU render (Vulkan)
-      - /dev/accel/accel0:/dev/accel/accel0  # Ryzen AI NPU (amdxdna)
-    ulimits:
-      memlock:
-        soft: -1
-        hard: -1
     volumes:
       - /tmp/.X11-unix:/tmp/.X11-unix
       - ${XAUTHORITY:-$HOME/.Xauthority}:/root/.Xauthority
       - ../config.toml:/root/MobileManipulatorDemo/config.toml
       - ../models:/root/MobileManipulatorDemo/models
-    network_mode: host
+
+  # ── One-shot trace run: start the stack, send one task, dump the agent ───────
+  # conversation to ../runs/, then exit. Reuses the image built by `demo`.
+  # No AMM_KEEP_ALIVE, so demo.sh returns and demo_trace.sh drives the lifecycle.
+  #   xhost +local:root
+  #   docker compose -f docker/compose.yaml run --rm demo-trace
+  # Override the task with:  -e TASK="Do Housekeeping of rack J01"
+  demo-trace:
+    <<: *demo-runtime
+    entrypoint: ["pixi", "run", "demo-trace"]
+    environment:
+      - DISPLAY=${DISPLAY}
+      - QT_X11_NO_MITSHM=1
+      - OPENAI_API_KEY=${OPENAI_API_KEY:-}
+      - ROS_DOMAIN_ID=${ROS_DOMAIN_ID:-42}
+      - FLM_MODEL_PATH=/root/MobileManipulatorDemo/models/flm
+    volumes:
+      - /tmp/.X11-unix:/tmp/.X11-unix
+      - ${XAUTHORITY:-$HOME/.Xauthority}:/root/.Xauthority
+      - ../config.toml:/root/MobileManipulatorDemo/config.toml
+      - ../models:/root/MobileManipulatorDemo/models
+      # Trace output lands here on the host (root-owned; chown if needed).
+      - ../runs:/root/MobileManipulatorDemo/runs

docs/demo.md

@@ -2,13 +2,13 @@
 
 Demo video: [https://vimeo.com/1128259690/68cf9430d1](https://vimeo.com/1128259690/68cf9430d1)
 
-This demo showcases the Robotnik Kairos+ mobile manipulator operating in our warehouse facility. The full ROS 2 stack coordinates navigation, manipulation, and combined tasks, and the agent executes them end-to-end without manual intervention.
+This demo runs the Robotnik Kairos+ mobile manipulator in a warehouse facility. The full ROS 2 stack coordinates navigation, manipulation, and combined tasks, and the agent executes them end-to-end without manual intervention.
 ![Various objects](images/kairos.jpg)
 
-The warehouse layout mirrors the production-ready configuration, giving the robot ample space to navigate, avoid obstacles, and complete missions. The scene resembles a live operation floor, so the agent runs in conditions that match real deployments.
+The warehouse layout mirrors a production configuration, with space for navigation, obstacle avoidance, and manipulation tasks.
 ![Various objects](images/scene.jpg)
 
-We staged a diverse set of objects and anomalies - including spills, misplaced boxes, and damaged packaging - to validate perception, navigation resilience, and adaptive planning.
+The scene includes a diverse set of objects and anomalies: spills, misplaced boxes, and damaged packaging.
 
 ![Various objects](images/objects.jpg)
 

docs/dependencies.md

@@ -19,23 +19,20 @@
 
 ## ROS 2
 
-| Package Name                      | Repository URL                                                           |
-| --------------------------------- | ------------------------------------------------------------------------ |
-| rai_interfaces                    | https://github.com/RobotecAI/rai_interfaces.git                          |
-| Universal_Robots_ROS2_Description | https://github.com/UniversalRobots/Universal_Robots_ROS2_Description.git |
-| control_msgs                      | https://github.com/ros-controls/control_msgs.git                         |
-| control_toolbox                   | https://github.com/ros-controls/control_toolbox.git                      |
-| kinematics_interface              | https://github.com/ros-controls/kinematics_interface.git                 |
-| moveit2                           | https://github.com/moveit/moveit2.git                                    |
-| moveit_msgs                       | https://github.com/moveit/moveit_msgs.git                                |
-| realtime_tools                    | https://github.com/ros-controls/realtime_tools.git                       |
-| robotnik_common                   | https://github.com/RobotnikAutomation/robotnik_common.git                |
-| robotnik_description              | https://github.com/RobotnikAutomation/robotnik_description.git           |
-| robotnik_sensors                  | https://github.com/RobotnikAutomation/robotnik_sensors.git               |
-| ros2_control                      | https://github.com/ros-controls/ros2_control.git                         |
-| ros2_controllers                  | https://github.com/ros-controls/ros2_controllers                         |
-| srdfdom                           | https://github.com/moveit/srdfdom.git                                    |
-| simulation_interfaces             | https://github.com/ros-simulation/simulation_interfaces.git              |
+Most ROS 2 dependencies (MoveIt 2, Nav2, `ros2_control`, `control_msgs`,
+`simulation_interfaces`, and the rest) come from the
+[RoboStack](https://robostack.github.io) `robostack-jazzy` conda channel and are
+installed automatically by pixi — see `[dependencies]` in `pixi.toml`. They are no
+longer vcs-imported or built from source.
+
+These packages are built from source (see `ros2_ws.repos`):
+
+| Package Name         | Repository URL                                                 |
+| -------------------- | -------------------------------------------------------------- |
+| robotnik_common      | https://github.com/RobotnikAutomation/robotnik_common.git      |
+| robotnik_description | https://github.com/RobotnikAutomation/robotnik_description.git |
+| robotnik_sensors     | https://github.com/RobotnikAutomation/robotnik_sensors.git     |
+| rai_interfaces       | https://github.com/RobotecAI/rai_interfaces.git                |
 
 ## Python
 

docs/operating.md

@@ -0,0 +1,76 @@
+# Operating the Demo
+
+This page explains how to drive the demo from the HMI once everything is running.
+If the stack is not up yet, start it first. See [Running the demo](./running.md)
+for a local run, or the [Quickstart](./quickstart.md) for Docker. Then come back
+here.
+
+Populate the scene first, wait for the objects to finish spawning, then start a
+mission.
+
+## The HMI at a glance
+
+The HMI window has three tabs:
+
+- **Control**: populate the scene, start missions, emergency stop, manual driving.
+- **Status**: system tiles (CPU, GPU, NPU, RAM, Nav2, MoveIt2, agents) and logs.
+- **Mission**: the mission observer, robot cameras, the map, the live agent task
+  list, and the VLM output.
+
+The window opens on the **Mission** tab. Click over to the **Control** tab to set
+things up.
+
+## Step 1: Populate the scene first
+
+Populate the scene with objects before starting a mission. On the **Control** tab,
+find the **Simulation Scenarios** group and click one of the scene recipes:
+
+- **Standard**: the usual starting point. Fills the racks with boxes, places
+  return packages on the tables.
+- **Housekeeping**: a similar fill recipe.
+- **Anomalies**: adds a few out-of-place objects and a stack of barrels, so the
+  inspection and safety agents have something to find.
+- **Cleanup**: resets the simulation and clears the spawned objects. Use this to
+  start over.
+
+Spawning is not instant. It can take up to about 30 seconds, and you can watch the
+objects appear one by one in the O3DE simulation window (or on the top view camera
+on the **Mission** tab). Wait until spawning has finished before moving on.
+
+If you click a scene button and get a "Service not available" warning, the scene
+agent is not running yet. Make sure the agents are up (see
+[Running the demo](./running.md)) and try again.
+
+## Step 2: Start a mission
+
+Once the scene is populated, use the **On Demand predefined tasks** group on the
+**Control** tab to give the robot a mission:
+
+- **Sort returns**: the robot sorts the return packages waiting on the tables based on their condition.
+- **Housekeeping**: the robot tidies a rack. Each click targets the next rack in sequence, so pressing it repeatedly walks through the warehouse rack by rack.
+- **Prepare shipment**: first tick the items you want (CPU, GPU, pipes, hammers,
+  nails, motherboard) in the shipment configuration, then click **Prepare
+  shipment**. The robot collects the selected items.
+
+The mission is handed to the orchestrator, which breaks it into steps and drives
+the robot. You do not need to wait for one mission to fully finish before queueing
+another; the orchestrator keeps a task queue.
+
+## Monitoring progress
+
+While a mission runs, the **Mission** tab shows:
+
+- The robot cameras (base and wrist, plus a top view).
+- The map, with the robot pose and its current plan.
+- The **Agent Task** panel, showing the current action, the current task, and the
+  queued and completed steps.
+- The VLM output, showing what the vision models report.
+
+The **Status** tab shows live system utilization and the health of Nav2, MoveIt2,
+and the agents, which is useful when something looks stuck.
+
+## Stopping and manual control
+
+- **STOP!** on the **Control** tab triggers an emergency stop.
+- **Manual Control** on the **Control** tab drives the base directly with the
+  on-screen teleop buttons.

docs/quickstart.md

@@ -25,15 +25,15 @@ cd agentic-mobile-manipulator
 docker compose -f docker/compose.yaml build
 ```
 
-This builds three cacheable layers — `…-ros2` (RoboStack ROS 2 + workspace) → `…-o3de` (O3DE SDK + simulation) → `…demo` (Python agents + llama.cpp + FastFlowLM). The O3DE layer is large, so the first build takes a while; later rebuilds reuse the cached bases.
+This builds three cacheable layers — `…-ros2` (RoboStack ROS 2 + workspace) → `…-o3de` (O3DE SDK + simulation) → `…demo` (Python agents + llama.cpp + FastFlowLM). The O3DE layer is large; later rebuilds reuse the cached bases.
 
 **Or pull the prebuilt images:**
 
 ```bash
 docker compose -f docker/compose.yaml pull
 ```
 
-If you pull, remember to pass `--no-build` when starting the demo (see step 4).
+If you pull, pass `--no-build` when starting the demo (see step 4).
 
 ## 3. Download the model weights
 
@@ -45,9 +45,6 @@ docker compose -f docker/compose.yaml run --rm --entrypoint "" demo pixi run dow
 
 This writes the llama.cpp GGUFs to `models/` and the NPU model to `models/flm/` (via `flm pull`). It only fetches what's missing, so it's safe to re-run.
 
-> [!TIP]
-> If you have pixi installed natively you can instead run `FLM_MODEL_PATH="$PWD/models/flm" pixi run download-models`.
-
 ## 4. Run the demo
 
 Allow local connections to your X server, then start the `demo` service:
@@ -58,16 +55,35 @@ docker compose -f docker/compose.yaml up demo          # if you built (step 2)
 docker compose -f docker/compose.yaml up --no-build demo   # if you pulled (step 2)
 ```
 
-> [!IMPORTANT] > `docker compose up` rebuilds any service that has a `build:` section — even when the image is already present locally. So after `pull`, a bare `up demo` ignores the pulled image and rebuilds `ros2 → o3de → demo`. Pass `--no-build` to run the pulled image as-is.
+> [!IMPORTANT]
+> The `docker compose up` command rebuilds any service that has a `build:` section, even when the image is already present locally. So after `pull`, a bare `up demo` ignores the pulled image and rebuilds `ros2 → o3de → demo`. Pass `--no-build` to run the pulled image as-is.
 
 ## What to expect
 
 ![Demo Windows](demo_windows.jpg)
 
-1. **Simulation Window**: the O3DE warehouse environment. Use _Control → Simulation Scenarios_ to spawn objects, and _Control → On demand predefined tasks_ to run tasks — or publish your own to the `/user_tasks` topic.
-2. **HMI**: a window/terminal showing the Human-Machine Interface.
+1. **Simulation Window**: the O3DE warehouse environment. Use _Control → Simulation Scenarios_ to spawn objects, and _Control → On Demand predefined tasks_ to run tasks — or publish your own to the `/user_tasks` topic.
+2. **HMI**: a window showing the Human-Machine Interface.
 3. **Agents**: the autonomous agents run in the background, driving the robot.
 
+See how to operate the demo [Operating](operating.md)
+
+## Verify the install (one-shot trace run)
+
+Instead of the interactive `demo`, the `demo-trace` service runs the whole stack,
+sends one task to the orchestrator, saves the agent conversation trace, and exits:
+
+```bash
+xhost +local:root
+docker compose -f docker/compose.yaml run --rm demo-trace
+```
+
+The trace is written to `runs/<timestamp>/` on the host (bind-mounted from the
+container; files are root-owned, `chown` if needed). Override the task with
+`-e TASK="Do Housekeeping of rack J01"`. See
+[Verify your installation](setup_single_machine.md#verify-your-installation) for
+the full list of knobs and the Langfuse option.
+
 ## Inspecting a component
 
 Each component runs in its own tmux session inside the container. Attach to one (e.g. the simulation) from the host:

docs/repo_overview.md

@@ -47,7 +47,7 @@ The system is built on a multi-agent architecture orchestrated by a "MegaMind" a
 
 This file is the control center for your model backend. You can define:
 
-- **LLM/VLM Endpoints**: Switch between local models or cloud providers (e.g., OpenAI).
+- **LLM/VLM Endpoints**: Define the model, backend, and port each agent uses.
 - **Model Parameters**: Adjust reasoning capabilities or temperature for specific agents.
 - **RAG Settings**: Configure the embedding models used by the Safety Agent.
 

docs/running.md

@@ -1,11 +1,48 @@
 # Running the Demo
 
+## What to expect
+
+![Demo Windows](demo_windows.jpg)
+
+1. **Simulation Window**: the O3DE warehouse environment. Use _Control → Simulation Scenarios_ to spawn objects, and _Control → On Demand predefined tasks_ to run tasks — or publish your own to the `/user_tasks` topic.
+2. **HMI**: a window showing the Human-Machine Interface.
+3. **Agents**: the autonomous agents run in the background, driving the robot.
+
+See how to operate the demo [Operating](operating.md)
+
 ## Run all at once
 
 ```shell
-pixi run -e single-pc-gpu demo
+pixi run -e single-pc-gpu-and-npu demo
+```
+
+> [!TIP]
+> If you have a system-installed ROS 2, pass the `--clean-env` flag to `pixi run`, or use Docker for clean isolation. Otherwise the demo may fail to start due to an API/ABI mismatch.
+
+## tmux sessions
+
+Each component runs in its own detached tmux session named
+`agentic-mobile-manipulator-<component>` (`-sim`, `-stack`, `-llm-servers`,
+`-agents`, `-hmi`). `pixi run demo` starts all of them in the background and prints
+the attach commands. Running a single task such as `pixi run sim` attaches you to
+its session directly.
+
+Attach to a session from another terminal (swap the suffix for any component):
+
+```shell
+tmux attach -t agentic-mobile-manipulator-sim
+```
+
+Inside tmux, `Ctrl-b s` lists and switches sessions and `Ctrl-b d` detaches. Stop
+every session with:
+
+```shell
+pixi run kill
 ```
 
+To run a component in the foreground instead (no tmux), use its `-fg` variant:
+`pixi run sim-fg`, `pixi run stack-fg`, or `pixi run hmi-fg`.
+
 ## Run separately
 
 ### O3DE
@@ -34,27 +71,26 @@ SSOT and launches each server on its backend — `gpu`/`cpu` via llama.cpp, `npu
 via FastFlowLM — in a tmux grid:
 
 ```shell
-pixi run -e single-pc-gpu inference        # launch all local endpoints (tmux grid)
-pixi run -e single-pc-gpu smoke-test       # health-check them
+pixi run -e single-pc-gpu-and-npu inference        # launch all local endpoints (tmux grid)
+pixi run -e single-pc-gpu-and-npu smoke-test       # health-check them
 ```
 
 To start a single endpoint (e.g. for debugging), launch it by its endpoint name:
 
 ```shell
-pixi run -e single-pc-gpu serve-llm        # endpoints.main_llm   → port 8080
-pixi run -e single-pc-gpu serve-vlm-safety       # endpoints.vlm_safety     → port 8081 (NPU)
-pixi run -e single-pc-gpu serve-vlm-inspection   # endpoints.vlm_inspection → port 8084 (GPU)
-pixi run -e single-pc-gpu serve-embedding  # endpoints.embeddings → port 8082
-pixi run -e single-pc-gpu serve-reranker   # endpoints.reranker   → port 8083
+pixi run -e single-pc-gpu-and-npu serve-llm        # endpoints.main_llm   → port 8080
+pixi run -e single-pc-gpu-and-npu serve-vlm-safety       # endpoints.vlm_safety     → port 8081 (NPU)
+pixi run -e single-pc-gpu-and-npu serve-vlm-inspection   # endpoints.vlm_inspection → port 8084 (GPU)
+pixi run -e single-pc-gpu-and-npu serve-embedding  # endpoints.embeddings → port 8082
+pixi run -e single-pc-gpu-and-npu serve-reranker   # endpoints.reranker   → port 8083
 ```
 
 To change a weight path, backend, or port, edit the endpoint in `config.toml`
-(e.g. set `model_path`, or flip the VLM `backend` between `gpu` and `npu`) — there
-are no per-server env vars anymore. Preview the resolved launch commands without
-starting anything:
+(e.g. set `model_path`, or flip the VLM `backend` between `gpu` and `npu`).
+Preview the resolved launch commands without starting anything:
 
 ```shell
-pixi run -e single-pc-gpu inference --print
+pixi run -e single-pc-gpu-and-npu inference --print
 ```
 
 ### Agents