Observabillity and Testing

Author: spectral-mesh

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+- **Observability:** New scripts **`scripts/simulate_mesh_ingest.sh`**, **`simulate_mesh_grafana.sh`**, **`simulate_capture_demo.sh`**, **`simulate_edge_scan.sh`**; Grafana dashboards **`monitoring/grafana/dashboards/spectral-mesh.json`** (cleartext/ingest wording + metric semantics row) and **`spectral-edge.json`** (**Job**/**Instance** variables, **HTTP 429** rate on throughput panel, simulation note). **`monitoring/README.md`** documents the script matrix.
 - **Docs:** Refresh **Windows/macOS** descriptions across **`README.md`**, **`docs/EDGE_VS_MESH.md`**, **`docs/EDGE.md`**, **`docs/POLICY_MESH_AND_EDGE.md`**, **`docs/ENTERPRISE_SELF_HOSTED.md`**, **`docs/SAAS_ARCHITECTURE.md`**, **`docs/INFOSEC_SUMMARY.md`**, **`docs/BACKGROUND_OF_INVENTION.md`**, and **`docs/INVENTION_DISCLOSURE_OUTLINE_US.md`** to match **ingest**, **hookwire**, optional **hooks**, and **edge** vs host rolling-buffer behavior (replacing outdated “capture stub” wording).
 - **Windows OpenSSL discovery:** **`-windows-openssl-discovery-interval`** polls module lists via Toolhelp32 and logs **`windows_openssl_module_mapped`** (`internal/capture/winssl`).
 - **macOS Endpoint Security (opt-in build):** **`-darwin-es-mmap-watch`** with **`go build -tags spectral_es`** (CGO) subscribes to **`ES_EVENT_TYPE_NOTIFY_MMAP`** and logs **`es_openssl_mmap`** when mapped paths look like OpenSSL (`internal/capture/esmmap`, **`internal/capture/platformhint`**). Default macOS binaries omit ES linkage; use an SDK-equipped toolchain to link **EndpointSecurity.framework**.

README.md

@@ -366,9 +366,13 @@ See [`openshift/README.md`](openshift/README.md) and [`openshift/daemonset.yml`]
 | `docs/DEPLOY_GCP_GKE.md` | **GKE + internal load balancer** + Artifact Registry |
 | `docs/DEPLOY_AZURE_AKS.md` | **AKS + internal load balancer** + ACR |
 | `packaging/systemd/` | Example systemd unit |
-| `scripts/simulate_mesh.sh` | Outbound HTTPS simulation for **spectral-mesh** (curl / OpenSSL) |
+| `scripts/simulate_mesh.sh` | Outbound HTTPS simulation for **spectral-mesh** (curl / OpenSSL, Linux eBPF) |
+| `scripts/simulate_mesh_ingest.sh` | **POST `/v1/ingest/chunk`** simulation (any OS; needs **`-capture-ingest-addr`**) |
+| `scripts/simulate_mesh_grafana.sh` | Ingest burst (+ optional split chunks) for **Grafana** mesh dashboard |
+| `scripts/simulate_capture_demo.sh` | Wrapper for **`cmd/spectral-capture-demo`** |
 | `scripts/simulate_edge.sh` | HTTP POST simulation for **spectral-edge** |
-| `scripts/simulate_edge_grafana.sh` | Mixed traffic to populate **Grafana** / Prometheus **`spectral_edge_*`** metrics |
+| `scripts/simulate_edge_scan.sh` | **POST `/v1/scan`** only (handler latency in Grafana) |
+| `scripts/simulate_edge_grafana.sh` | Mixed traffic for **Grafana** **`spectral_edge_*`** metrics |
 | `scripts/flood_https.sh` | HTTPS curl flood without API key (general testing) |
 | `scripts/flood_openai.sh` | Optional OpenAI API flood (curl); requires API key |
 | `api/spectral-edge.openapi.yaml` | OpenAPI 3 description of core **spectral-edge** HTTP routes |
@@ -450,7 +454,11 @@ Tune `MODEL` and `MAX_TOKENS` (see script header) to balance cost vs. response s
 | Script | Purpose |
 |--------|---------|
 | [`scripts/simulate_mesh.sh`](scripts/simulate_mesh.sh) | Outbound HTTPS via **`curl`** (OpenSSL) so **`spectral-mesh`** can observe **`SSL_write`** on Linux. Optional `TRIGGER_POLICY_MATCH=1` includes **`Project Ethos`** in the POST body for built-in policy smoke tests. |
+| [`scripts/simulate_mesh_ingest.sh`](scripts/simulate_mesh_ingest.sh) | **`POST /v1/ingest/chunk`** with **base64** payloads; works on **Linux / Windows / macOS** when mesh runs with **`-capture-ingest-addr`**. Optional **`SPLIT_CHUNK=1`** splits **`Project Eth`** / **`os`** across two posts (rolling-buffer demo). |
+| [`scripts/simulate_mesh_grafana.sh`](scripts/simulate_mesh_grafana.sh) | Calls **`simulate_mesh_ingest.sh`** with higher counts + optional split-chunk burst for **`monitoring/`** Grafana; optional **`LINUX_EBPF_SIM=1`** on Linux also runs a short **`simulate_mesh.sh`**. |
+| [`scripts/simulate_capture_demo.sh`](scripts/simulate_capture_demo.sh) | Runs **`go run ./cmd/spectral-capture-demo`** (split chunks via ingest API). |
 | [`scripts/simulate_edge.sh`](scripts/simulate_edge.sh) | HTTP **`POST`s to `spectral-edge`** (same policy / `policy_alert` shape as a TLS-terminated path). Requires **`spectral-edge`** listening (e.g. `make edge && ./spectral-edge -listen 127.0.0.1:8080`). |
+| [`scripts/simulate_edge_scan.sh`](scripts/simulate_edge_scan.sh) | **`POST /v1/scan`** only — moves **`spectral_edge_http_request_duration_seconds{handler="scan"}`** without incrementing **`spectral_edge_http_requests_total`**. |
 | [`scripts/load_edge_smoke.sh`](scripts/load_edge_smoke.sh) | Load against **`spectral-edge`** (defaults to **`/v1/scan`**); uses **`hey`** if installed, else parallel **`curl`** (`xargs -P`, same idea as **`simulate_edge.sh`**). |
 | [`scripts/simulate_edge_grafana.sh`](scripts/simulate_edge_grafana.sh) | Proxy + **`/v1/scan`** traffic so **Grafana “Spectral edge”** panels (throughput, alerts, handler latency) move; see script header. |
 | [`scripts/build_spectral_edge_release.sh`](scripts/build_spectral_edge_release.sh) | Build static **release tarballs** (same artifacts as **[`release-edge.yml`](.github/workflows/release-edge.yml)**). |

monitoring/README.md

@@ -1,6 +1,6 @@
 # Prometheus + Grafana (local observability)
 
-Docker Compose runs **Prometheus** (scrapes **spectral-mesh** and **spectral-edge** on the host) and **Grafana** (dashboards provisioned from this folder). The **Spectral mesh** dashboard assumes a **Linux** host sensor with live TLS capture; **Windows/macOS** builds expose the same metric names but ringbuf/uprobe series may stay at zero until capture is integrated ([`README.md`](../README.md#platforms)).
+Docker Compose runs **Prometheus** (scrapes **spectral-mesh** and **spectral-edge** on the host) and **Grafana** (dashboards provisioned from this folder). The **Spectral mesh** dashboard’s **cleartext chunk** panels track **`spectral_mesh_ringbuf_events_total`**, which increments on **every** `HandleChunk` — **Linux eBPF** or **pluggable feeds** (ingest, hook socket, stdin JSONL). **Ringbuf drop/read-error** panels are **Linux BPF–path** metrics and often stay near zero on **Windows/macOS** or **ingest-only** tests ([`README.md`](../README.md#platforms)).
 
 ## Ports
 
@@ -44,7 +44,19 @@ Example **Prometheus alert rules**: [`prometheus/spectral-edge-rules.yml`](prome
 
    If you use **`-metrics-listen 127.0.0.1:9092`**, add a second scrape job (or change targets) so Prometheus hits **`:9092`** for **`/metrics`**, not **`-listen`**.
 
-   For a quick smoke test without the full mesh, drive edge only (see **[`scripts/simulate_edge.sh`](../scripts/simulate_edge.sh)**). Optional load: **[`scripts/load_edge_smoke.sh`](../scripts/load_edge_smoke.sh)**. For **Grafana** time series, **[`scripts/simulate_edge_grafana.sh`](../scripts/simulate_edge_grafana.sh)** sends **proxy-path** traffic (not only **`/v1/scan`**) so **`spectral_edge_http_requests_total`** updates.
+   **Simulation scripts** (repo root `scripts/`):
+
+   | Script | Purpose |
+   |--------|---------|
+   | [`simulate_mesh.sh`](../scripts/simulate_mesh.sh) | Linux **eBPF** path: HTTPS via **curl** → OpenSSL `SSL_write` |
+   | [`simulate_mesh_ingest.sh`](../scripts/simulate_mesh_ingest.sh) | **POST `/v1/ingest/chunk`** (any OS); bumps same chunk counter as hooks |
+   | [`simulate_mesh_grafana.sh`](../scripts/simulate_mesh_grafana.sh) | Ingest burst + optional **split-chunk** demo for **Grafana** |
+   | [`simulate_capture_demo.sh`](../scripts/simulate_capture_demo.sh) | **`go run ./cmd/spectral-capture-demo`** wrapper |
+   | [`simulate_edge.sh`](../scripts/simulate_edge.sh) | Edge **proxy-style** POSTs |
+   | [`simulate_edge_scan.sh`](../scripts/simulate_edge_scan.sh) | Edge **`/v1/scan`** only (**handler=scan** latency) |
+   | [`simulate_edge_grafana.sh`](../scripts/simulate_edge_grafana.sh) | Mixed proxy + scan load for dashboards |
+
+   Optional: **[`scripts/load_edge_smoke.sh`](../scripts/load_edge_smoke.sh)**. Mesh **ingest** example: `spectral-mesh -metrics-addr :9090 -capture-ingest-addr 127.0.0.1:9091` then **`simulate_mesh_ingest.sh`** or **`simulate_mesh_grafana.sh`**.
 
    If you use another port, TLS, or a **GHCR** image from **[`release-edge`](../.github/workflows/release-edge.yml)**, edit **[`prometheus/prometheus.yml`](prometheus/prometheus.yml)** under the `spectral-edge` job (`targets`, `scheme`, `tls_config`, etc.).
 
@@ -79,9 +91,9 @@ If scraping stays **DOWN**:
 
 [`grafana/dashboards/spectral-mesh.json`](grafana/dashboards/spectral-mesh.json) (host sensor) and [`grafana/dashboards/spectral-edge.json`](grafana/dashboards/spectral-edge.json) (edge, including latency panels) are provisioned automatically. Re-import manually after edits: **Dashboards → New → Import → Upload JSON**.
 
-The **Spectral mesh** dashboard includes **Job** and **Instance** template variables (multi-select, **All** = `.*`), row groupings (**Health**, **TLS capture & ringbuf**, **Uprobes & policy**, **BPF maps & rolling buffer**), a **health** stat row (aggregated error/drop rates and chunk throughput), **dashboard links** to **Spectral edge**, **Prometheus annotation** queries for firing **`SpectralMesh*`** alerts, **5m rates** on ringbuf drops (plus cumulative on the same chart), and **1h increase** helpers on policy reload alongside 5m rates.
+The **Spectral mesh** dashboard includes **Job** and **Instance** template variables (multi-select, **All** = `.*`), row groupings (**Health**, **Cleartext ingress & chunk throughput**, **Uprobes & policy**, **BPF maps & rolling buffer**, **Metric semantics**), a **health** stat row (aggregated error/drop rates and chunk throughput), **dashboard links** to **Spectral edge**, **Prometheus annotation** queries for firing **`SpectralMesh*`** alerts, **5m rates** on ringbuf drops (plus cumulative on the same chart), **1h increase** helpers on policy reload alongside 5m rates, and a **Capture paths vs counters** note (ingest/hook vs BPF-only panels).
 
-**Edge metrics** include counters (requests, alerts, truncations, allowlist/dedupe/rate-limit suppressions, policy reloads, upstream errors, gzip decode, generated **X-Request-Id**, response-scan alerts when enabled) and histograms (request duration by handler, policy scan duration, upstream round-trip). The **Spectral edge** Grafana dashboard charts throughput, **optional features** (gunzip, request IDs, response policy alerts), p50/p95 latency, suppression rates, upstream errors, and SIGHUP reload activity. See **[`docs/EDGE.md`](../docs/EDGE.md)**.
+**Edge metrics** include counters (requests, alerts, truncations, **HTTP 429** rate-limit path, allowlist/dedupe/rate-limit suppressions, policy reloads, upstream errors, gzip decode, generated **X-Request-Id**, response-scan alerts when enabled) and histograms (request duration by handler, policy scan duration, upstream round-trip). The **Spectral edge** Grafana dashboard adds **Job** / **Instance** variables, charts throughput (including **HTTP rate limited /s** when **`-http-ratelimit-rps`** is used), **optional features** (gunzip, request IDs, response policy alerts), p50/p95 latency, suppression rates, upstream errors, SIGHUP reload activity, and a **Simulation scripts** note. See **[`docs/EDGE.md`](../docs/EDGE.md)**.
 
 ## Stopping