Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
ce355c7
feat(telemetry): complete metrics instrumentation with cost tracking …
l33t0 Feb 21, 2026
cfa4602
Route MiniMax through OpenAI-compatible endpoint
Marenz Feb 20, 2026
fcfe290
Log empty Anthropic responses at warn level
Marenz Feb 20, 2026
07b6b3a
fix: harden 6 security vulnerabilities (injection, CORS, env leak, sh…
PyRo1121 Feb 22, 2026
5bfa2bd
fix: redact secrets from Debug output, bound cortex channel, block sy…
PyRo1121 Feb 22, 2026
81c3c06
fix: block dangerous env vars in exec tool, validate memory associati…
PyRo1121 Feb 22, 2026
4c281d9
fix: harden leak detection against base64, hex, and URL-encoded secrets
PyRo1121 Feb 22, 2026
bad87d3
fix: replace panic/expect with proper error propagation in startup an…
PyRo1121 Feb 22, 2026
19f5806
feat(ui): make AgentTabs horizontally scrollable on mobile
Feb 22, 2026
b6171e1
feat(web): memory management in web ui
skulldogged Feb 22, 2026
d692153
fix ui jank upon adding/deleting memories
skulldogged Feb 22, 2026
7ce6c16
feat: update Gemini model support with latest Google models
adryserage Feb 22, 2026
cc4c52a
fix: resolve compilation errors in browser tool and API server
adryserage Feb 22, 2026
33e83ac
feat: support ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN, and SPACEBOT_…
adryserage Feb 22, 2026
a358a2b
fix: remove VOLUME keyword from Dockerfile
adryserage Feb 22, 2026
c4f2d5b
fix: add cache mount IDs to Dockerfile for Railway compatibility
adryserage Feb 22, 2026
3cba979
fix: remove BuildKit cache mounts for Railway compatibility
adryserage Feb 22, 2026
a8fc4bf
Update .gitignore
adryserage Feb 22, 2026
6601a22
fix: prevent sidebar from being excluded from flex layout
Feb 22, 2026
dbb0aed
fix: prevent iOS auto-zoom on input focus at mobile viewport widths
Feb 22, 2026
bf9078e
fix: use !important to ensure mobile input font-size overrides Tailwind
Feb 22, 2026
721c90b
Add SettingSectionNav shared component
Feb 22, 2026
a47f7ed
Use SettingSectionNav in Settings, add responsive wrapper
Feb 22, 2026
198e423
Use SettingSectionNav in AgentConfig, add responsive wrapper
Feb 22, 2026
aee2bae
feat(web): ui/ux cleanup
skulldogged Feb 22, 2026
e785441
Merge commit 'refs/pull/102/head' of https://github.com/spacedriveapp…
bilawalriaz Feb 22, 2026
a374570
Merge commit 'refs/pull/119/head' of https://github.com/spacedriveapp…
bilawalriaz Feb 22, 2026
be161f4
Merge commit 'refs/pull/84/head' of https://github.com/spacedriveapp/…
bilawalriaz Feb 22, 2026
f72f135
Merge commit 'refs/pull/121/head' of https://github.com/spacedriveapp…
bilawalriaz Feb 22, 2026
4c4c767
Merge commit 'refs/pull/125/head' of https://github.com/spacedriveapp…
bilawalriaz Feb 22, 2026
29bbd95
Merge commit 'refs/pull/126/head' of https://github.com/spacedriveapp…
bilawalriaz Feb 22, 2026
88a7870
Merge commit 'refs/pull/127/head' of https://github.com/spacedriveapp…
bilawalriaz Feb 22, 2026
2867151
Merge commit 'refs/pull/134/head' of https://github.com/spacedriveapp…
bilawalriaz Feb 22, 2026
452e2e4
Merge commit 'refs/pull/143/head' of https://github.com/spacedriveapp…
bilawalriaz Feb 22, 2026
9b61b6f
Fix build errors from upstream merges
bilawalriaz Feb 22, 2026
63aa38e
fix cron timezone handling and delete unregister
jamiepine Feb 22, 2026
bda32d2
Add multi-agent communication graph
jamiepine Feb 22, 2026
326482a
Strip instance.db, back links with config
jamiepine Feb 22, 2026
ac7ecb2
Merge remote-tracking branch 'upstream/main' into upstream-sync
bilawalriaz Feb 22, 2026
1adcd13
Merge commit 'refs/pull/149/head' of https://github.com/spacedriveapp…
bilawalriaz Feb 22, 2026
f2d5ee1
Merge commit 'refs/pull/150/head' of https://github.com/spacedriveapp…
bilawalriaz Feb 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,13 @@ interface/dist/
.idea
list/
agents/

# Claude Code - Fichiers propriétaires (auto-généré)
.claude/
CLAUDE.md
.claude-context
.claude-memory
.claude-decisions
docs/phases/
docs/specs/
PROJECT-STATUS.md
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ fastembed = "4"

# Encoding
base64 = "0.22"
hex = "0.4"

# Logging and tracing
tracing = "0.1"
Expand Down Expand Up @@ -64,6 +65,7 @@ uuid = { version = "1.15", features = ["v4", "serde"] }

# Time handling
chrono = { version = "0.4", features = ["serde"] }
chrono-tz = "0.10"

# Regular expressions (for leak detection)
regex = "1.11"
Expand Down
19 changes: 4 additions & 15 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# syntax=docker/dockerfile:1.6
# ---- Builder stage ----
# Compiles the React frontend and the Rust binary with the frontend embedded.
FROM rust:bookworm AS builder
Expand All @@ -22,20 +21,15 @@ WORKDIR /build
# 1. Fetch and cache Rust dependencies.
# cargo fetch needs a valid target, so we create stubs that get replaced later.
COPY Cargo.toml Cargo.lock ./
RUN --mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/usr/local/cargo/git \
--mount=type=cache,target=/build/target \
mkdir src && echo "fn main() {}" > src/main.rs && touch src/lib.rs \
RUN mkdir src && echo "fn main() {}" > src/main.rs && touch src/lib.rs \
&& cargo build --release \
&& rm -rf src

# 2. Build the frontend.
COPY interface/package.json interface/
RUN --mount=type=cache,target=/root/.bun/install/cache \
cd interface && bun install
RUN cd interface && bun install
COPY interface/ interface/
RUN --mount=type=cache,target=/root/.bun/install/cache \
cd interface && bun run build
RUN cd interface && bun run build

# 3. Copy source and compile the real binary.
# build.rs runs the frontend build (already done above, node_modules present).
Expand All @@ -45,10 +39,7 @@ COPY build.rs ./
COPY prompts/ prompts/
COPY migrations/ migrations/
COPY src/ src/
RUN --mount=type=cache,target=/usr/local/cargo/registry \
--mount=type=cache,target=/usr/local/cargo/git \
--mount=type=cache,target=/build/target \
SPACEBOT_SKIP_FRONTEND_BUILD=1 cargo build --release \
RUN SPACEBOT_SKIP_FRONTEND_BUILD=1 cargo build --release \
&& mv /build/target/release/spacebot /usr/local/bin/spacebot \
&& cargo clean -p spacebot --release --target-dir /build/target

Expand All @@ -71,8 +62,6 @@ ENV SPACEBOT_DIR=/data
ENV SPACEBOT_DEPLOYMENT=docker
EXPOSE 19898 18789

VOLUME /data

HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD curl -f http://localhost:19898/api/health || exit 1

Expand Down
125 changes: 97 additions & 28 deletions METRICS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Metrics Reference

Comprehensive reference for Spacebot's Prometheus metrics. For quick-start setup, see `docs/metrics.md`.
Comprehensive reference for Spacebot's Prometheus metrics. For quick-start setup, see `docs/metrics.md`. For the published docs, see the metrics page on [docs.spacebot.sh](https://docs.spacebot.sh).

## Feature Gate

Expand All @@ -27,9 +27,7 @@ All metrics are prefixed with `spacebot_`. The registry uses a private `promethe
| Instrumented in | `src/llm/model.rs` — `SpacebotModel::completion()` |
| Description | Total LLM completion requests (one per `completion()` call, including retries and fallbacks). |

**Cardinality:** `agents × models × tiers`. Currently `agent_id` and `tier` are hardcoded to `"unknown"` because `SpacebotModel` doesn't carry process context. Effective cardinality is just the number of distinct model names (typically 5–15). Once agent context is threaded through, expect `agents(1–5) × models(5–15) × tiers(5)` = 25–375 series.

**Known limitation:** Labels `agent_id` and `tier` are always `"unknown"`. The `SpacebotHook` has these values but can't be used here without structural changes.
**Cardinality:** `agents × models × tiers`. With agent context wired, expect `agents(1–5) × models(5–15) × tiers(5)` = 25–375 series.

#### `spacebot_tool_calls_total`

Expand All @@ -40,7 +38,7 @@ All metrics are prefixed with `spacebot_`. The registry uses a private `promethe
| Instrumented in | `src/hooks/spacebot.rs` — `SpacebotHook::on_tool_result()` |
| Description | Total tool calls executed across all processes. Incremented after each tool call completes (success or failure). |

**Cardinality:** `agents × tools`. With 1–5 agents and ~20 tool names, expect 20–100 series. Tool names are a bounded set defined in `src/tools/`.
**Cardinality:** `agents × tools`. With 1–5 agents and ~20 tool names, expect 20–100 series.

#### `spacebot_memory_reads_total`

Expand All @@ -62,6 +60,52 @@ All metrics are prefixed with `spacebot_`. The registry uses a private `promethe

**Cardinality:** 1 series.

#### `spacebot_llm_tokens_total`

| Field | Value |
|-------|-------|
| Type | `IntCounterVec` |
| Labels | `agent_id`, `model`, `tier`, `direction` |
| Instrumented in | `src/llm/model.rs` — `SpacebotModel::completion()` |
| Description | Total LLM tokens consumed. `direction` is one of `input`, `output`, or `cached_input`. |

**Cardinality:** `agents × models × tiers × 3`. Expect 75–1125 series.

#### `spacebot_llm_estimated_cost_dollars`

| Field | Value |
|-------|-------|
| Type | `CounterVec` (f64) |
| Labels | `agent_id`, `model`, `tier` |
| Instrumented in | `src/llm/model.rs` — `SpacebotModel::completion()` |
| Description | Estimated LLM cost in USD. Uses a built-in pricing table (`src/llm/pricing.rs`). |

**Cardinality:** Same as `spacebot_llm_requests_total`.

**Note:** Costs are best-effort estimates. The pricing table covers major models (Claude 4/3.5/3, GPT-4o, o-series, Gemini, DeepSeek) with a conservative fallback for unknown models ($3/M input, $15/M output).

#### `spacebot_process_errors_total`

| Field | Value |
|-------|-------|
| Type | `IntCounterVec` |
| Labels | `agent_id`, `process_type`, `error_type` |
| Instrumented in | `src/llm/model.rs` — `SpacebotModel::completion()` error paths |
| Description | Process errors by type. `error_type` classifies the failure (timeout, rate_limit, auth, server, provider, unknown). |

**Cardinality:** `agents × process_types × error_types`. Expect 15–75 series.

#### `spacebot_memory_updates_total`

| Field | Value |
|-------|-------|
| Type | `IntCounterVec` |
| Labels | `agent_id`, `operation` |
| Instrumented in | `src/memory/store.rs` (save/delete), `src/tools/memory_save.rs`, `src/tools/memory_delete.rs` (forget) |
| Description | Memory mutation operations. `operation` is one of `save`, `delete`, or `forget`. |

**Cardinality:** `agents × operations(3)`. Expect 3–15 series.

### Histograms

#### `spacebot_llm_request_duration_seconds`
Expand All @@ -70,16 +114,12 @@ All metrics are prefixed with `spacebot_`. The registry uses a private `promethe
|-------|-------|
| Type | `HistogramVec` |
| Labels | `agent_id`, `model`, `tier` |
| Buckets | 0.1, 0.25, 0.5, 1, 2.5, 5, 10 |
| Buckets | 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 15, 30, 60, 120 |
| Instrumented in | `src/llm/model.rs` — `SpacebotModel::completion()` |
| Description | End-to-end LLM request duration in seconds. Includes retry loops and fallback chain traversal. |

**Cardinality:** Same as `spacebot_llm_requests_total` (per-bucket overhead is fixed, not per-series).

**Known limitation:** Buckets max out at 10s. LLM requests with retries and fallbacks routinely exceed 10s (15–60s is common). Everything above 10s collapses into the +Inf bucket, losing resolution. A future fix should extend buckets to cover `[..., 15, 30, 60, 120]`.

**What the timer measures:** The timer wraps the entire `completion()` method body, including all retry attempts on the primary model and the full fallback chain. This measures user-perceived latency, not individual provider call latency.

#### `spacebot_tool_call_duration_seconds`

| Field | Value |
Expand All @@ -91,7 +131,19 @@ All metrics are prefixed with `spacebot_`. The registry uses a private `promethe

**Cardinality:** 1 series.

**Implementation note:** Duration is tracked via a `LazyLock<Mutex<HashMap<String, Instant>>>` static keyed by Rig's internal call ID. The timer starts in `on_tool_call` and is consumed in `on_tool_result`. If a tool call starts but the agent terminates before `on_tool_result` fires (e.g. leak detection terminates the agent), the timer entry remains in the map. These orphaned entries are small (String + Instant) and bounded by concurrent tool calls, so this is not a practical concern.
**Implementation note:** Duration is tracked via a `LazyLock<Mutex<HashMap<String, Instant>>>` static keyed by Rig's internal call ID. If a tool call starts but the agent terminates before `on_tool_result` fires (e.g. leak detection), the timer entry remains — bounded by concurrent tool calls, not a practical concern.

#### `spacebot_worker_duration_seconds`

| Field | Value |
|-------|-------|
| Type | `HistogramVec` |
| Labels | `agent_id`, `worker_type` |
| Buckets | 1, 5, 10, 30, 60, 120, 300, 600, 1800 |
| Instrumented in | `src/agent/channel.rs` — `spawn_worker_task()` |
| Description | Worker lifetime duration in seconds from spawn to completion. |

**Cardinality:** `agents × worker_types`. Currently `worker_type` is `"builtin"` — expect 1–5 series.

### Gauges

Expand All @@ -102,40 +154,55 @@ All metrics are prefixed with `spacebot_`. The registry uses a private `promethe
| Type | `IntGaugeVec` |
| Labels | `agent_id` |
| Instrumented in | `src/agent/channel.rs` — `spawn_worker_task()` |
| Description | Currently active workers. Incremented when a worker task is spawned, decremented when it completes (success or failure). Covers both builtin Rig workers and OpenCode workers. |
| Description | Currently active workers. Incremented when a worker task is spawned, decremented when it completes. |

**Cardinality:** Number of agents (typically 1–5).
**Cardinality:** Number of agents (1–5).

#### `spacebot_memory_entry_count`

| Field | Value |
|-------|-------|
| Type | `IntGaugeVec` |
| Labels | `agent_id` |
| Instrumented in | **Not instrumented.** Defined in registry but not wired to any call site. |
| Description | Intended to track total memory entries per agent. |
| Instrumented in | `src/memory/store.rs` — `save()` (inc) and `delete()` (dec) |
| Description | Approximate memory entry count per agent. Tracks net saves minus deletes — starts at 0 on process start, not the actual database count. |

**Cardinality:** Number of agents (typically 1–5). Currently always 0.
**Cardinality:** Number of agents (1–5).

**Status:** Requires periodic store queries or integration into `MemoryStore::save()` / `MemoryStore::delete()` to maintain an accurate count. Not blocked for merge — the metric is registered but idle.
**Note:** This gauge tracks deltas from process start, not the absolute database count. On restart it resets to 0. For the true count, query the database directly.

## Total Cardinality
#### `spacebot_active_branches`

With the current instrumentation (hardcoded `"unknown"` labels on LLM metrics):
| Field | Value |
|-------|-------|
| Type | `IntGaugeVec` |
| Labels | `agent_id` |
| Instrumented in | `src/agent/channel.rs` — branch spawn (inc) and completion (dec) |
| Description | Currently active branches per agent. |

**Cardinality:** Number of agents (1–5).

## Total Cardinality

| Metric | Series estimate |
|--------|-----------------|
| `llm_requests_total` | ~10 (distinct models) |
| `tool_calls_total` | ~20–100 (agents × tools) |
| `llm_requests_total` | ~25–375 |
| `llm_tokens_total` | ~75–1125 |
| `llm_estimated_cost_dollars` | ~25–375 |
| `tool_calls_total` | ~20–100 |
| `memory_reads_total` | 1 |
| `memory_writes_total` | 1 |
| `llm_request_duration_seconds` | ~10 (distinct models) |
| `llm_request_duration_seconds` | ~25–375 |
| `tool_call_duration_seconds` | 1 |
| `active_workers` | ~1–5 (agents) |
| `memory_entry_count` | 0 (not instrumented) |
| **Total** | **~45–130** |
| `worker_duration_seconds` | ~1–5 |
| `active_workers` | ~1–5 |
| `active_branches` | ~1–5 |
| `memory_entry_count` | ~1–5 |
| `process_errors_total` | ~15–75 |
| `memory_updates_total` | ~3–15 |
| **Total** | **~195–2465** |

This is well within safe operating range for any Prometheus deployment.
Well within safe operating range for any Prometheus deployment.

## Feature Gate Consistency

Expand All @@ -147,9 +214,11 @@ Every instrumentation call site uses `#[cfg(feature = "metrics")]` at the statem
| `src/main.rs` | `#[cfg(feature = "metrics")] let _metrics_handle = ...` |
| `src/llm/model.rs` | `#[cfg(feature = "metrics")] let start` + `#[cfg(feature = "metrics")] { ... }` |
| `src/hooks/spacebot.rs` | `#[cfg(feature = "metrics")] static TOOL_CALL_TIMERS` + 2 blocks |
| `src/tools/memory_save.rs` | `#[cfg(feature = "metrics")] crate::telemetry::Metrics::global()...` |
| `src/tools/memory_save.rs` | `#[cfg(feature = "metrics")] { ... }` |
| `src/tools/memory_recall.rs` | `#[cfg(feature = "metrics")] crate::telemetry::Metrics::global()...` |
| `src/agent/channel.rs` | `#[cfg(feature = "metrics")] ...` (×2, inc + dec) |
| `src/tools/memory_delete.rs` | `#[cfg(feature = "metrics")] crate::telemetry::Metrics::global()...` |
| `src/memory/store.rs` | `#[cfg(feature = "metrics")] if _result...` + `#[cfg(feature = "metrics")] { ... }` |
| `src/agent/channel.rs` | `#[cfg(feature = "metrics")]` (×4, branches + workers) |
| `Cargo.toml` | `prometheus = { version = "0.13", optional = true }`, `metrics = ["dep:prometheus"]` |

All consistent. No path references `crate::telemetry` without a `cfg` gate.
Expand Down
15 changes: 14 additions & 1 deletion docs/content/docs/(configuration)/config.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ max_turns = 5 # max LLM turns per channel message
context_window = 128000 # context window size in tokens
history_backfill_count = 50 # messages to fetch from platform on new channel
worker_log_mode = "errors_only" # "errors_only", "all_separate", or "all_combined"
cron_timezone = "UTC" # optional default timezone for cron active hours

# Model routing per process type.
[defaults.routing]
Expand Down Expand Up @@ -105,6 +106,7 @@ screenshot_dir = "/path/to/screenshots" # optional, defaults to data_dir/screens
id = "main"
default = true
workspace = "/custom/workspace/path" # optional, defaults to ~/.spacebot/agents/{id}/workspace
cron_timezone = "America/Los_Angeles" # optional per-agent cron timezone override

# Per-agent routing overrides (merges with defaults).
[agents.routing]
Expand Down Expand Up @@ -180,7 +182,7 @@ This creates a single "main" agent with default settings.
For any setting, the resolution chain is:

```
agent-level override > [defaults] section > hardcoded default
agent-level override > [defaults] section > env var fallback (if supported) > hardcoded default
```

An agent with no overrides inherits everything from `[defaults]`. An agent with partial overrides gets those values from its own config and everything else from defaults. See [Agents](/docs/agents) for how agent config merging works.
Expand Down Expand Up @@ -394,6 +396,7 @@ At least one provider (legacy key or custom provider) must be configured.
| `context_window` | integer | 128000 | Context window size in tokens |
| `history_backfill_count` | integer | 50 | Messages to fetch from platform on new channel |
| `worker_log_mode` | string | `"errors_only"` | Worker log persistence: `"errors_only"`, `"all_separate"`, or `"all_combined"` |
| `cron_timezone` | string | None | Default timezone for cron active-hours evaluation (IANA name like `UTC` or `America/New_York`) |

### `[defaults.routing]`

Expand Down Expand Up @@ -481,6 +484,7 @@ Thresholds are fractions of `context_window`.
| `id` | string | **required** | Agent identifier |
| `default` | bool | false | Whether this is the default agent |
| `workspace` | string | `~/.spacebot/agents/{id}/workspace` | Custom workspace path |
| `cron_timezone` | string | inherits | Per-agent timezone override for cron active-hours evaluation |
| `max_concurrent_branches` | integer | inherits | Override instance default |
| `max_turns` | integer | inherits | Override instance default |
| `context_window` | integer | inherits | Override instance default |
Expand All @@ -499,6 +503,15 @@ Agent-specific routing is set via `[agents.routing]` with the same keys as `[def
| `active_end_hour` | integer | None | End of active hours window |
| `enabled` | bool | true | Whether this cron job is active |

Cron timezone precedence is:

1. `agents.cron_timezone`
2. `defaults.cron_timezone`
3. `SPACEBOT_CRON_TIMEZONE`
4. server local timezone

If a configured timezone is invalid, Spacebot logs a warning and falls back to server local time.

### `[messaging.discord]`

| Key | Type | Default | Description |
Expand Down
2 changes: 1 addition & 1 deletion docs/content/docs/(deployment)/meta.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"title": "Deployment",
"pages": ["hosted", "roadmap"]
"pages": ["hosted", "metrics", "roadmap"]
}
Loading
Loading