Skip to content

setup_single_machine.md

Latest commit

 

History

History
217 lines (154 loc) · 8.08 KB

setup_single_machine.md

File metadata and controls

217 lines (154 loc) · 8.08 KB
Raw

Setup

This repository is compatible with the following system:

  • System: Ubuntu 24.04
  • ROS 2: Jazzy, provided by pixi via RoboStack (no separate ROS 2 install needed)
  • Python: 3.12

Building the Project

Prerequisites

Clone the Repository

git clone https://github.com/RobotecAI/agentic-mobile-manipulator.git
cd agentic-mobile-manipulator

Install System Dependencies

These packages must be installed via apt before using pixi:

sudo apt update
sudo apt install git git-lfs python3-vcstool ninja-build \
    cmake libstdc++-12-dev clang \
    libglu1-mesa-dev libxcb-randr0-dev libxcb-xinerama0 libxcb-xinput0 \
    libxcb-xinput-dev libxcb-xfixes0-dev libxcb-xkb-dev libxkbcommon-dev \
    libxkbcommon-x11-dev libfontconfig1-dev libpcre2-16-0 zlib1g-dev \
    mesa-common-dev libunwind-dev libzstd-dev tix

Install pixi

pixi orchestrates all build steps and sets environment variables automatically.

curl -fsSL https://pixi.sh/install.sh | sh

Restart your shell or run source ~/.bashrc after installation.

Build Everything

pixi run -e single-pc-gpu-and-npu setup

This single command runs the full build pipeline in the correct order:

Step pixi task What it does
1 clone vcs import + git lfs pull for gems and ROS 2 ws
2 install-o3de Install the O3DE engine
3 fetch-gems Clone o3de-extras locally, validate all gem paths
4 build-ros2 colcon build (deps provided by RoboStack)
5 build-sim CMake configure + Ninja build (GameLauncher)
6 sync uv sync installs the Python dependencies
7 build-llama Build llama.cpp with the Vulkan backend (GPU)
8 build-fastflowlm Build FastFlowLM for the AMD Ryzen AI NPU backend
9 find-runnables List the built runnables (GameLauncher, llama.cpp, FastFlowLM)

Prefer a GPU-only machine (no AMD Ryzen AI NPU)? Use pixi run -e single-pc-gpu setup instead — it skips step 8. You must also set [endpoints.vlm_safety] backend = "gpu" in config.toml; otherwise inference routes that endpoint to the NPU.

Local Inference

Setup already checks out the inference submodules and builds both backends — llama.cpp (Vulkan, GPU) and FastFlowLM (NPU) — as part of the pipeline above. To serve models locally you still need to download the weights:

pixi run -e single-pc-gpu-and-npu download-models # downloads every weight referenced in config.toml (gguf via wget, NPU tags via `flm pull`); or grab them manually below

config.toml is the single source of truth for inference: each [endpoints.*] table fixes a model's backend, port, and weights, and the agents reference those endpoints by name. pixi run inference launches them all. See Running the demo.

NPU backend (AMD Ryzen™ AI)

The single-pc-gpu-and-npu setup above builds FastFlowLM (step 8), so NPU endpoints are served without extra configuration. The RobotecAI/FastFlowLM fork (pinned as a submodule) includes GBNF grammar-constrained sampling, so the NPU path can produce the structured/JSON output the agents rely on. Which endpoints run on the NPU is driven by backend = "npu" entries in config.toml — the NPU VLM endpoint serves a FastFlowLM vision tag (default gemma3:4b); see [endpoints.vlm_safety].

Building FastFlowLM only needs the XRT/amdxdna dev headers, but serving on the NPU requires an AMD Ryzen AI processor with the amdxdna driver loaded. On an AMD machine without the NPU, use the GPU-only single-pc-gpu setup and switch [endpoints.vlm_safety] to backend = "gpu" in config.toml, so every endpoint runs on llama.cpp.

Download Models

For every GGUF-backed model in config.toml, download the file and place it in $DEMO_ROOT/models/:

The NPU vlm_safety model (gemma3:4b) has no GGUF; flm pull downloads it for you.

Using cloud models instead (low-VRAM machines)

If the machine cannot serve the chat/vision models locally, switch the agents to OpenAI-hosted models via cloud_config.toml:

cp cloud_config.toml config.toml
export OPENAI_API_KEY=sk-...

Your OpenAI account must have access to the models it names (gpt-5-mini, gpt-5-nano) — edit the [endpoints.*] model fields to use different ones.

Everything reads config.toml, so the usual commands work unchanged: download-models now fetches only the two RAG GGUFs (embedding + reranker, the only endpoints still served locally) and pixi run inference launches just those two llama.cpp servers, skipping the remote backend = "openai" endpoints. The NPU is not used in this configuration, so the GPU-only single-pc-gpu environment suffices.

Verify your installation

Once the build is done and the weights are downloaded, run one command to exercise the whole stack end to end:

pixi run -e single-pc-gpu-and-npu demo-trace

This brings up the full demo (sim, stack, inference, agents, HMI), populates the scene, sends one task to the orchestrator (ship one CPU), waits for it to finish, saves the agent trace, and shuts everything down. A successful run ends with:

  ● Task complete (marker).

=== Trace saved ===
  runs/<timestamp>
    log.txt         human-readable conversation (orchestrator + subagents)
    trace.jsonl     one JSON record per event
    agents_pane.log raw agents tmux output
    manifest.txt    task + timestamps

Open runs/<timestamp>/log.txt to read the orchestrator and subagent conversation for the task.

Environment knobs:

  • TASK: the task string sent to the orchestrator (default: ship one CPU)
  • MAX_WAIT: hard cap in seconds on task execution (default: 900)
  • IDLE: treat this many seconds of trace inactivity as done (default: 180)
  • SKIP_SCENE=1: skip scene population
  • TRACE_DIR: output directory (default: runs/<timestamp>)

On a GPU-only box, use -e single-pc-gpu and set [endpoints.vlm_safety] backend = "gpu" in config.toml first (see above).

The conversation trace is written on any agent run (default-on, to runs/<timestamp>/); demo-trace just automates a single task plus teardown. Set AMM_TRACE=0 to disable it.

Richer traces with Langfuse (optional)

The orchestrator is already instrumented with a Langfuse callback. Point it at a Langfuse instance (self-hosted or cloud) to get a full browsable trace, including nested subagent spans and token usage, alongside the local runs/ files:

export LANGFUSE_PUBLIC_KEY=pk-...
export LANGFUSE_SECRET_KEY=sk-...
export LANGFUSE_HOST=http://localhost:3000   # your Langfuse server

Without these keys the callback stays inactive and only the local runs/ trace is written.

Developer Setup

Conventional Commits

We use conventional commits to ensure that the commit messages are consistent and follow a specific format. Read more about conventional commits here.

Pre-commit

pixi run -e dev pre-commit-install

To run hooks manually:

pixi run -e dev lint

Next Steps