Ship platform logs to Axiom via Vector (#240)

* ship platform logs to Axiom via Vector with cross-service request_id

Adds internal/obslog: JSON slog with a stable host envelope (service,
service_id, cell_id, region, hostname, host_ip, version) and per-request
fields (request_id, sandbox_id, worker_id) propagated via context. The
package installs itself as slog.Default AND redirects stdlib log.Printf
through slog so existing log call sites emit JSON automatically.

Wires obslog into both control plane (cmd/server, internal/controlplane,
internal/api/router) and worker (cmd/worker, internal/worker), replacing
echo's middleware.Logger with obslog.EchoMiddleware. internal/proxy
forwards X-Request-Id in the Director so worker log lines share the same
id as the control plane line for proxied requests.

internal/config adds OPENSANDBOX_CELL_ID and OPENSANDBOX_HOST_IP; CellID
defaults to "<region>-default" when unset.

deploy/vector/ ships Vector configs that read journald (worker, dev-host)
or docker_logs (control-plane), parse JSON, enrich non-JSON lines (kernel,
systemd) with the host envelope from env, and forward to a NEW Axiom
dataset oc-platform-logs — kept separate from the existing customer
oc-sandbox-logs dataset for cost / retention / blast-radius reasons.
install.sh accepts roles {worker, control-plane, dev-host}.

deploy/azure/deploy-azure-dev.sh: extend rsync excludes so local
dev-env-secrets-* and dev-vector-token-* files don't end up on the VM.

.gitignore: cover deploy/azure/.dev-vector-token-*.

Tests in internal/obslog cover the host envelope, context propagation,
the Echo middleware round-trip (including a forwarded X-Request-Id
appearing on both handler and access-log lines), and the link-local
filter in detectHostIP.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* ship Vector via Packer + deploy-server: prod gets logs without a follow-up PR

Wires Vector install into both deploy paths so merging this PR doesn't
leave a "Vector still needs deploying" gap.

- deploy/vector/populate-vector-env.{sh,service}: systemd oneshot that
  runs Before=vector.service. Reads SECRETS_VAULT_NAME from worker.env
  (or server.env), fetches the platform-logs ingest token from Azure
  Key Vault via the VM's managed identity (IMDS → AAD → KV REST), and
  writes /etc/opensandbox/vector.env so Vector picks it up. Token never
  appears in any image or workflow artifact. The script exits 0 on any
  failure path (no IMDS, no vault, missing secret) so a logging
  credential problem doesn't break the worker boot.

- deploy/vector/install.sh: installs Vector + drops the role config +
  installs the populator unit + wires the systemd drop-in with the env
  files (worker.env, server.env, vector.env). Idempotent. PACKER_BUILD=1
  skips the systemctl start so the AMI capture doesn't trip over a
  started service.

- deploy/vector/control-plane.yaml: rewrote from docker_logs to journald.
  Current prod CP runs the server binary directly under systemd (per
  deploy-server.yml), not in Docker.

- deploy/packer/worker-ami.pkr.hcl: new provisioner step that extracts
  /tmp/packer-vector-ctx.tar.gz (created by CI in build-worker-ami.yml)
  and runs install.sh worker with PACKER_BUILD=1. New worker AMIs come
  out with Vector pre-installed and pre-configured; populator runs at
  first boot to fetch the token.

- .github/workflows/build-worker-ami.yml: pre-tars deploy/vector/ to
  /tmp/packer-vector-ctx.tar.gz before invoking Packer.

- .github/workflows/deploy-server.yml: bundles deploy/vector/ as
  bin/vector-deploy.tar.gz, uploads to blob storage alongside the
  server binary, and runs install.sh control-plane on each CP host via
  az vm run-command (idempotent — refreshes the config on every deploy).

Operator prerequisite: create a `shared-axiom-platform-ingest-token`
secret in each prod KV. Documented in the PR description's env-vars
checklist.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* vector: rename CELL_ID/HOST_IP to OPENCOMPUTER_*; fetch dataset from KV too

Two reviewer-requested changes:

- Rename OPENSANDBOX_CELL_ID → OPENCOMPUTER_CELL_ID and
  OPENSANDBOX_HOST_IP → OPENCOMPUTER_HOST_IP. New fields use the
  product-named prefix; existing OPENSANDBOX_* fields untouched.
  Touch: config.go (env var read), the three vector configs (VRL refs),
  populate-vector-env.sh (env file write), install.sh (auto-detect),
  vector.env.example.

- AXIOM_PLATFORM_DATASET is now also fetched from Key Vault as
  `shared-axiom-platform-dataset` alongside the token. No default value
  baked into the configs — a missing dataset secret fails Vector
  healthcheck (loud) instead of silently shipping to a presumed default.
  populate-vector-env.sh fetches both secrets in one pass with a shared
  IMDS-acquired AAD token.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* keyvault: register shared-axiom-platform-{ingest-token,dataset}

Vector reads these from /etc/opensandbox/vector.env via the dedicated
populate-vector-env.service, but secretMapping is the documented source
of truth for "what shared-* secrets does this deployment need in KV".
Adding the entries so:
  - operators have one place to audit required KV secrets
  - the Go binary loads them into its own env at startup (side-effect),
    so future admin endpoints / health views can surface platform-stream
    config without a separate KV fetch

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: motatoes

.github/workflows/build-worker-ami.yml

@@ -94,6 +94,14 @@ jobs:
             deploy/ec2/build-rootfs-docker.sh \
             scripts/claude-agent-wrapper/
 
+      - name: Package Vector configs
+        # Bundled separately so Packer's file provisioner has a known
+        # tarball path (it can't reliably upload an arbitrary directory).
+        # install.sh on the builder extracts and installs Vector + the
+        # KV-token populator into the AMI.
+        run: |
+          tar czf /tmp/packer-vector-ctx.tar.gz -C deploy vector
+
       - name: Azure Login
         uses: azure/login@v2
         with:

.github/workflows/deploy-server.yml

@@ -50,6 +50,13 @@ jobs:
       - name: Package web assets
         run: tar czf bin/web-dist.tar.gz -C web dist
 
+      - name: Package Vector configs
+        # Bundle deploy/vector/ for distribution to control-plane hosts.
+        # install.sh is idempotent: skips Vector install if already present,
+        # always refreshes config + restarts. Workers get the same files via
+        # the Packer image bake (build-worker-ami.yml).
+        run: tar czf bin/vector-deploy.tar.gz -C deploy vector
+
       - name: Azure Login
         uses: azure/login@v2
         with:
@@ -75,6 +82,14 @@ jobs:
             --file bin/web-dist.tar.gz \
             --overwrite true -o none
 
+          az storage blob upload \
+            --account-name ${{ secrets.AZURE_STORAGE_ACCOUNT }} \
+            --account-key "${{ secrets.AZURE_STORAGE_KEY }}" \
+            --container-name checkpoints \
+            --name deploy/vector-deploy.tar.gz \
+            --file bin/vector-deploy.tar.gz \
+            --overwrite true -o none
+
           echo "Artifacts uploaded to blob storage"
 
       - name: Discover control planes
@@ -120,6 +135,12 @@ jobs:
                   --container-name checkpoints --name deploy/web-dist.tar.gz \
                   --file /tmp/web-dist.tar.gz --overwrite true -o none
 
+                az storage blob download \
+                  --account-name ${{ secrets.AZURE_STORAGE_ACCOUNT }} \
+                  --account-key '${{ secrets.AZURE_STORAGE_KEY }}' \
+                  --container-name checkpoints --name deploy/vector-deploy.tar.gz \
+                  --file /tmp/vector-deploy.tar.gz --overwrite true -o none
+
                 # Stop, replace binary, extract frontend
                 systemctl stop opensandbox-server
                 sleep 1
@@ -136,6 +157,17 @@ jobs:
                 fi
 
                 systemctl start opensandbox-server
+
+                # Install/refresh Vector + the KV token populator. install.sh
+                # is idempotent (skips Vector install if already present,
+                # always refreshes config + reloads). populate-vector-env
+                # fetches AXIOM_PLATFORM_TOKEN from KV via the VM's managed
+                # identity at every boot (and on this restart).
+                mkdir -p /tmp/vector-deploy
+                tar xzf /tmp/vector-deploy.tar.gz -C /tmp/vector-deploy
+                bash /tmp/vector-deploy/vector/install.sh control-plane
+                rm -rf /tmp/vector-deploy /tmp/vector-deploy.tar.gz
+
                 echo 'Deploy complete'
               " -o none
 

.gitignore

@@ -65,4 +65,5 @@ delme
 deploy/azure/.dev-env-state-*
 deploy/azure/.dev-env-secrets*
 !deploy/azure/.dev-env-secrets.example
+deploy/azure/.dev-vector-token-*
 /server

cmd/server/main.go

@@ -5,6 +5,7 @@ import (
 	"encoding/base64"
 	"fmt"
 	"log"
+	"log/slog"
 	"os"
 	"strconv"
 	"strings"
@@ -22,6 +23,7 @@ import (
 	"github.com/opensandbox/opensandbox/internal/controlplane"
 	"github.com/opensandbox/opensandbox/internal/crypto"
 	"github.com/opensandbox/opensandbox/internal/db"
+	"github.com/opensandbox/opensandbox/internal/obslog"
 	"github.com/opensandbox/opensandbox/internal/observability"
 	"github.com/opensandbox/opensandbox/internal/proxy"
 	"github.com/opensandbox/opensandbox/internal/sandbox"
@@ -42,6 +44,21 @@ func main() {
 		log.Fatalf("failed to load config: %v", err)
 	}
 
+	// Structured logging (JSON to stdout, host envelope baked in). Installs
+	// itself as slog.Default AND redirects stdlib log.Printf through slog so
+	// existing log call sites emit JSON automatically. Vector reads stdout
+	// from the Docker logging driver and ships to Axiom.
+	cpHostname, _ := os.Hostname()
+	obslog.Init(obslog.HostFields{
+		Service:   obslog.ServiceControlPlane,
+		ServiceID: cpHostname, // hostname distinguishes HA replicas
+		CellID:    cfg.CellID,
+		Region:    cfg.Region,
+		Hostname:  cpHostname,
+		HostIP:    cfg.HostIP,
+		Version:   ServerVersion,
+	}, slog.LevelInfo)
+
 	// Sentry error reporting — no-op if OPENSANDBOX_SENTRY_DSN is unset.
 	flushSentry := observability.Init(cfg, "control-plane", ServerVersion)
 	defer flushSentry()

cmd/worker/main.go

@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"io"
 	"log"
+	"log/slog"
 	"os"
 	"os/signal"
 	"path/filepath"
@@ -19,6 +20,7 @@ import (
 	"github.com/opensandbox/opensandbox/internal/db"
 	"github.com/opensandbox/opensandbox/internal/metrics"
 	"github.com/opensandbox/opensandbox/internal/observability"
+	"github.com/opensandbox/opensandbox/internal/obslog"
 	"github.com/opensandbox/opensandbox/internal/proxy"
 	qm "github.com/opensandbox/opensandbox/internal/qemu"
 	"github.com/opensandbox/opensandbox/internal/sandbox"
@@ -66,6 +68,21 @@ func main() {
 		log.Fatalf("failed to load config: %v", err)
 	}
 
+	// Structured logging (JSON to stdout/journald, host envelope baked in).
+	// Installs itself as slog.Default AND redirects stdlib log.Printf through
+	// slog so existing log call sites emit JSON automatically. Vector on the
+	// host reads journald and ships to Axiom.
+	workerHostname, _ := os.Hostname()
+	obslog.Init(obslog.HostFields{
+		Service:   obslog.ServiceWorker,
+		ServiceID: cfg.WorkerID,
+		CellID:    cfg.CellID,
+		Region:    cfg.Region,
+		Hostname:  workerHostname,
+		HostIP:    cfg.HostIP,
+		Version:   WorkerVersion,
+	}, slog.LevelInfo)
+
 	// Sentry error reporting — no-op if OPENSANDBOX_SENTRY_DSN is unset.
 	flushSentry := observability.Init(cfg, "worker", WorkerVersion)
 	defer flushSentry()

deploy/azure/deploy-azure-dev.sh

@@ -125,7 +125,9 @@ SETUP_DISK
     log "Syncing codebase..."
     rsync -az --progress \
         --exclude '.git' --exclude 'bin/' --exclude 'node_modules/' \
-        --exclude '.dev-env-state*' --exclude '*.ext4' --exclude 'vendor/' \
+        --exclude '.dev-env-state*' --exclude '.dev-env-secrets*' \
+        --exclude '.dev-vector-token-*' \
+        --exclude '*.ext4' --exclude 'vendor/' \
         "$REPO_ROOT/" "${AZURE_ADMIN_USER}@${VM_PUBLIC_IP}:~/opensandbox/"
 
     log "Running host setup..."
@@ -190,7 +192,9 @@ cmd_deploy() {
     log "Syncing code..."
     rsync -az --progress \
         --exclude '.git' --exclude 'bin/' --exclude 'node_modules/' \
-        --exclude '.dev-env-state*' --exclude '*.ext4' --exclude 'vendor/' \
+        --exclude '.dev-env-state*' --exclude '.dev-env-secrets*' \
+        --exclude '.dev-vector-token-*' \
+        --exclude '*.ext4' --exclude 'vendor/' \
         "$REPO_ROOT/" "${AZURE_ADMIN_USER}@${VM_PUBLIC_IP}:~/opensandbox/"
 
     # Build binaries on instance

deploy/packer/worker-ami.pkr.hcl

@@ -320,6 +320,36 @@ build {
     ]
   }
 
+  # 4.5. Install Vector + the KV-token-populator for platform-logs shipping.
+  #
+  # Vector is enabled but NOT started (Packer captures the image before
+  # systemd has run). At first boot:
+  #   1. cloud-init writes /etc/opensandbox/worker.env
+  #   2. populate-vector-env.service fires (Before=vector.service), reads
+  #      SECRETS_VAULT_NAME from worker.env, fetches AXIOM_PLATFORM_TOKEN
+  #      from KV via the VM's managed identity, writes /etc/opensandbox/vector.env
+  #   3. vector.service starts, reads both env files, ships to oc-platform-logs
+  #
+  # PACKER_BUILD=1 tells install.sh to skip `systemctl start` — systemd in a
+  # baking image is offline.
+  #
+  # CI is expected to pre-tar deploy/vector/ at /tmp/packer-vector-ctx.tar.gz
+  # (see .github/workflows/build-worker-ami.yml).
+  provisioner "file" {
+    source      = "/tmp/packer-vector-ctx.tar.gz"
+    destination = "/tmp/vector-ctx.tar.gz"
+  }
+  provisioner "shell" {
+    execute_command = "chmod +x {{ .Path }}; {{ .Vars }} sudo -E bash '{{ .Path }}'"
+    environment_vars = ["PACKER_BUILD=1"]
+    inline = [
+      "mkdir -p /tmp/vector-ctx",
+      "tar xzf /tmp/vector-ctx.tar.gz -C /tmp/vector-ctx",
+      "cd /tmp/vector-ctx/vector && PACKER_BUILD=1 bash install.sh worker",
+      "rm -rf /tmp/vector-ctx /tmp/vector-ctx.tar.gz",
+    ]
+  }
+
   # 4b. Archive base image to blob storage keyed by goldenVersion so that old
   #     checkpoints referencing this base can be rebased even after workers roll.
   provisioner "shell" {

deploy/vector/control-plane.yaml

@@ -0,0 +1,52 @@
+# Vector config for the control-plane host (Azure prod: systemd binary,
+# not Docker). Reads opensandbox-server.service from journald, parses JSON
+# log lines (emitted by internal/obslog), enriches non-JSON lines with the
+# host envelope from env, and ships to oc-platform-logs.
+#
+# Install path: /etc/vector/vector.yaml
+# Env file:    /etc/opensandbox/vector.env (populated by
+#              populate-vector-env.service from Key Vault at boot)
+
+data_dir: /var/lib/vector
+
+sources:
+  control_plane_journald:
+    type: journald
+    include_units:
+      - opensandbox-server.service
+    # Default current_boot_only=true is fine — Vector tracks its own cursor
+    # in data_dir, so restarts resume cleanly without replaying the journal.
+
+transforms:
+  control_plane_parse:
+    type: remap
+    inputs:
+      - control_plane_journald
+    source: |
+      parsed, err = parse_json(.message)
+      if err == null && is_object(parsed) {
+        . = merge(., object!(parsed))
+        # Drop stringified duplicate; non-JSON lines (panics, systemd boot)
+        # keep .message as the readable content.
+        del(.message)
+      }
+      if !exists(.service) { .service = "control-plane" }
+      if !exists(.service_id) { .service_id, _ = get_hostname() }
+      if !exists(.cell_id) { .cell_id = "${OPENCOMPUTER_CELL_ID:-unknown}" }
+      if !exists(.region) { .region = "${OPENSANDBOX_REGION:-unknown}" }
+      if !exists(.host_ip) { .host_ip = "${OPENCOMPUTER_HOST_IP:-unknown}" }
+      if !exists(.hostname) { .hostname, _ = get_hostname() }
+
+sinks:
+  axiom_platform:
+    type: axiom
+    inputs:
+      - control_plane_parse
+    dataset: "${AXIOM_PLATFORM_DATASET}"
+    token: "${AXIOM_PLATFORM_TOKEN}"
+    buffer:
+      type: disk
+      max_size: 268435488
+      when_full: drop_newest
+    healthcheck:
+      enabled: true

deploy/vector/dev-host.yaml

@@ -0,0 +1,87 @@
+# Vector config for a dev host that runs BOTH opensandbox-server AND
+# opensandbox-worker as systemd services on the same VM (the layout produced
+# by deploy/azure/deploy-azure-dev.sh).
+#
+# Production hosts use worker.yaml or control-plane.yaml — one role per host.
+# This config exists so the request-id join can be validated end-to-end on a
+# single dev VM before standing up a multi-VM dev cluster.
+#
+# The journald source captures both units in one stream; the parse transform
+# tags `service` based on which unit emitted each line.
+#
+# Install path: /etc/vector/vector.yaml
+# Env file:    /etc/opensandbox/vector.env (provides AXIOM_PLATFORM_TOKEN)
+
+data_dir: /var/lib/vector
+
+sources:
+  dev_journald:
+    type: journald
+    include_units:
+      - opensandbox-server.service
+      - opensandbox-worker.service
+    # `current_boot_only: false` is rejected on systemd 250-257 (Vector
+    # limitation). Default (true) is fine — Vector tracks its own cursor in
+    # data_dir, so restarts pick up where they left off rather than replaying.
+
+transforms:
+  dev_parse:
+    type: remap
+    inputs:
+      - dev_journald
+    source: |
+      parsed, err = parse_json(.message)
+      if err == null && is_object(parsed) {
+        . = merge(., object!(parsed))
+        # The structured fields (msg, level, request_id, ...) are now at
+        # root. Drop the stringified copy so events don't duplicate the
+        # payload in Axiom. Non-JSON lines (panics, kernel) fall through
+        # this branch and keep .message as the readable content.
+        del(.message)
+      }
+      # Tag service based on the systemd unit. obslog.Init already sets
+      # `service` inside JSON log lines, so this only fires for non-JSON
+      # output (panics, plain stderr, systemd boot messages).
+      if !exists(.service) {
+        unit = string(.SYSTEMD_UNIT) ?? ""
+        if unit == "opensandbox-server.service" {
+          .service = "control-plane"
+        } else if unit == "opensandbox-worker.service" {
+          .service = "worker"
+        } else {
+          .service = "unknown"
+        }
+      }
+      if !exists(.service_id) {
+        # Worker exports OPENSANDBOX_WORKER_ID; control plane has no
+        # equivalent on a binary-deployed dev host — fall back to hostname.
+        unit = string(.SYSTEMD_UNIT) ?? ""
+        if unit == "opensandbox-worker.service" {
+          .service_id = "${OPENSANDBOX_WORKER_ID:-unknown}"
+        } else {
+          .service_id, _ = get_hostname()
+        }
+      }
+      if !exists(.cell_id) { .cell_id = "${OPENCOMPUTER_CELL_ID:-unknown}" }
+      if !exists(.region) { .region = "${OPENSANDBOX_REGION:-unknown}" }
+      if !exists(.host_ip) { .host_ip = "${OPENCOMPUTER_HOST_IP:-unknown}" }
+      if !exists(.hostname) { .hostname, _ = get_hostname() }
+      # Stamp every event with environment so dev traffic is filterable inside
+      # a shared dataset. Only the dev-host config sets this — production
+      # worker.yaml / control-plane.yaml deliberately omit it so prod events
+      # don't claim the dev tag.
+      .environment = "dev"
+
+sinks:
+  axiom_platform:
+    type: axiom
+    inputs:
+      - dev_parse
+    dataset: "${AXIOM_PLATFORM_DATASET}"
+    token: "${AXIOM_PLATFORM_TOKEN}"
+    buffer:
+      type: disk
+      max_size: 268435488
+      when_full: drop_newest
+    healthcheck:
+      enabled: true