Skip to content

Commit e49a6c1

Browse files
author
Sovereign-OS
committed
Security: constant-time API key comparison; tests: validation, telemetry; README: metrics, Redis, security note
Made-with: Cursor
1 parent ba7fab3 commit e49a6c1

File tree

9 files changed

+1334
-160
lines changed

9 files changed

+1334
-160
lines changed

README.md

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@
2121

2222
# Sovereign-OS
2323

24+
**宪法优先的 AI 编排底座:一条 Charter 定义身份与预算,CEO 规划、CFO 把关、Auditor 判分、Ledger 不可篡改。** [快速开始](#-quick-start) · [Quick Start](#-quick-start)
25+
2426
**One command. One Charter. A digital corporation that thinks, spends, and answers for every token.**
2527

2628
Not another chatbot. Not another “AI agent framework.” Sovereign-OS is the **constitution-first substrate**: one YAML defines who the entity is, what it may spend, and how success is measured. The CEO plans. The CFO gates. The Auditor judges. **The Ledger never lies.**
@@ -56,7 +58,9 @@ python -m sovereign_os.web.app
5658
# Open http://localhost:8000 — run missions, approve jobs, watch balance & token usage.
5759
```
5860

59-
**Want to charge for work?** [3-step guide →](docs/QUICKSTART.md) (Stripe + one LLM key; 13 built-in workers out of the box.)
61+
**Want to charge for work?** [3-step guide →](docs/QUICKSTART.md) (Stripe + one LLM key; 16 built-in workers out of the box.)
62+
63+
**One-click deploy:** [DEPLOY.md](docs/DEPLOY.md) — Docker Compose, volumes, graceful shutdown.
6064

6165
---
6266

@@ -107,12 +111,18 @@ Phases 1–6a done (governance, ledger, MCP, audit trail, Stripe, webhook). Phas
107111

108112
## 🚀 Features
109113

110-
- **📦 13 built-in workers** — summarize, research, reply, write_article, solve_problem, write_email, write_post, meeting_minutes, translate, rewrite_polish, collect_info, extract_structured, spec_writer. No code; configure Stripe + one LLM key and run paid jobs. [QUICKSTART](docs/QUICKSTART.md)
114+
- **📦 16 built-in workers** — summarize, research, reply, write_article, solve_problem, write_email, write_post, meeting_minutes, translate, rewrite_polish, collect_info, extract_structured, spec_writer, **assistant_chat**, **code_assistant**, **code_review**. No code; configure Stripe + one LLM key and run paid jobs. [QUICKSTART](docs/QUICKSTART.md)
111115
- **🔄 Multi-model** — Strategist and workers can use different backends (e.g. GPT-4o for planning, cheaper models for execution).
112116
- **🔐 SovereignAuth** — RBAC by TrustScore. READ_FILES, WRITE_FILES, SPEND_USD, CALL_API gated; agents earn capabilities.
113117
- **🌐 Web Dashboard (24/7)** — Run missions, job queue, approve/retry, health, token usage, audit trail. Optional ingest from URL; Stripe charges; webhook on completion.
114118
- **🔌 MCP native** — Plug into the same tool graph as the rest of the ecosystem.
115-
- **📊 Observability** — OpenTelemetry, Prometheus metrics, verifiable audit trail with `proof_hash`.
119+
- **📊 Observability** — OpenTelemetry, Prometheus metrics (`GET /metrics`: job counters, duration, queue gauges), verifiable audit trail with `proof_hash`.
120+
- **🔒 Security** — API key via env; constant-time comparison; job input validation; optional IP whitelist and rate limit. [CONFIG](docs/CONFIG.md).
121+
122+
<p align="center">
123+
<a href="docs/dashboard.png"><img src="docs/dashboard.png" alt="Dashboard" width="720" style="max-width:100%"/></a>
124+
</p>
125+
<sub align="center">Add <code>docs/dashboard.png</code> for a Dashboard screenshot; link and size above are ready.</sub>
116126

117127
---
118128

@@ -125,7 +135,7 @@ sovereign_os/
125135
├── governance/ # CEO (Strategist) + CFO (Treasury) + Engine
126136
├── agents/ # Workers, Registry, SovereignAuth
127137
├── auditor/ # ReviewEngine, AuditReport (proof_hash)
128-
├── jobs/ # JobStore (SQLite queue)
138+
├── jobs/ # JobStore (SQLite), RedisJobStore (Redis queue)
129139
├── ingest/ # Poll URL → enqueue jobs
130140
├── web/ # FastAPI dashboard, /api/jobs, /health, Stripe webhook
131141
└── ui/ # Textual TUI (optional)
@@ -146,6 +156,12 @@ tests/ # pytest
146156
| [Monetization](docs/MONETIZATION.md) | Job queue, Stripe, approval, compliance, human-out-of-loop. |
147157
| [Audit proof](docs/AUDIT_PROOF.md) | Verifiable trail, `proof_hash`, integrity check. |
148158
| [Optimization roadmap](docs/OPTIMIZATION_ROADMAP.md) | Next steps: reliability, observability, security, scale. |
159+
| [Future development plan](docs/FUTURE_DEVELOPMENT.md) | Wave-by-wave plan: stability, UX/deploy, workers/scale, community. |
160+
| [Deploy (Docker)](docs/DEPLOY.md) | One-click deploy, volumes, health, graceful shutdown. |
161+
| [Backup](docs/BACKUP.md) | Back up job DB and ledger for disaster recovery. |
162+
| [Multi-instance](docs/MULTI_INSTANCE.md) | Concurrency, Redis queue for multi-instance workers, scaling. |
163+
| [Good first issues](docs/GOOD_FIRST_ISSUES.md) | Contribution ideas: docs, tests, examples. |
164+
| [Release process](docs/RELEASE.md) | How to cut a release and update CHANGELOG. |
149165

150166
---
151167

docs/CONFIG.md

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Configuration & environment variables
2+
3+
All optional unless noted. Set in the shell, `.env`, or your process manager (e.g. Docker Compose).
4+
5+
## Core
6+
7+
| Variable | Description |
8+
|----------|-------------|
9+
| `SOVEREIGN_LEDGER_PATH` | Path to the ledger JSONL file (append-only USD/token log). If unset, ledger is in-memory. |
10+
| `SOVEREIGN_JOB_DB` | Path to the SQLite job queue DB (e.g. `jobs.db`). Used only when `REDIS_URL` is not set. If both unset, jobs are in-memory. |
11+
| `REDIS_URL` | When set, use Redis as job store and shared approved queue (multi-instance workers). Install with `pip install -e ".[deploy]"`. |
12+
| `SOVEREIGN_CHARTER` | Default charter name shown in the Web UI (e.g. `Default`). |
13+
14+
## Web UI & API
15+
16+
| Variable | Description |
17+
|----------|-------------|
18+
| `SOVEREIGN_API_KEY` | If set, `POST /api/jobs` requires `X-API-Key` or `Authorization: Bearer <key>`. |
19+
| `SOVEREIGN_JOB_RATE_LIMIT_PER_MIN` | If set (e.g. `60`), limits how many `POST /api/jobs` requests each client IP can make per minute; returns 429 when exceeded. |
20+
| `SOVEREIGN_JOB_MAX_RETRIES` | Max retries for a single job via `POST /api/jobs/{id}/retry` (default `2`). Only `failed` / `payment_failed` jobs are retryable. |
21+
| `SOVEREIGN_JOB_WORKER_CONCURRENCY` | Max jobs run in parallel by the worker (default `1`). Set to `2` or more for higher throughput. |
22+
| `SOVEREIGN_JOB_IP_WHITELIST` | Optional. Comma-separated IPs allowed to call `POST /api/jobs` and `POST /api/jobs/batch`; others get 403. |
23+
| (none) | Web UI: `GET /` (dashboard), `GET /health`, `GET /docs` (FastAPI Swagger). Dashboard shows “Auto-approve ON” / “Compliance auto ON” when those env vars are set. |
24+
25+
**`GET /health` response** (for operators): `status`, `ledger`, `redis`, `config_warnings`, `jobs_total`, `jobs_pending`, `jobs_running`, `last_job_completed_at` (Unix timestamp of last completed/failed job), `auto_approve_jobs`, `compliance_auto_proceed`. **Production hints in config_warnings:** when `STRIPE_API_KEY` contains `sk_live_` or when Stripe is set but `SOVEREIGN_API_KEY` is not, a warning is added. Use `config_warnings` to detect incomplete or unsafe production setup.
26+
27+
**Job creation:** `POST /api/jobs` validates `goal` length (max 20 000 chars), `amount_cents` (0–1 000 000), and `callback_url` (must be valid http(s) URL). Rate limit: set `SOVEREIGN_JOB_RATE_LIMIT_PER_MIN` (per client IP). Optional body: `priority` (int; higher runs first), `run_after_ts` or `run_after_sec`. Failed/payment_failed jobs can be retried via `POST /api/jobs/{id}/retry` up to `SOVEREIGN_JOB_MAX_RETRIES` (default 2). **Metrics:** `GET /metrics` returns Prometheus text (sovereign_jobs_completed_total, sovereign_job_duration_seconds, sovereign_jobs_pending/running).
28+
29+
## 24/7 & ingestion
30+
31+
The Web process is designed for **24/7** operation: a background thread continuously picks approved jobs from the queue (every few seconds), and when `SOVEREIGN_INGEST_URL` is set, another thread polls that URL for new work. Run the process under Docker (`restart: unless-stopped`), systemd, or another process manager so it stays up and restarts on failure. Use `SOVEREIGN_JOB_DB` and `SOVEREIGN_LEDGER_PATH` so jobs and ledger persist across restarts.
32+
33+
| Variable | Description |
34+
|----------|-------------|
35+
| `SOVEREIGN_INGEST_URL` | JSON URL to poll for new jobs. Response: array or `{ "jobs": [...] }` with `goal`, optional `charter`, `amount_cents`, `currency`. |
36+
| `SOVEREIGN_INGEST_INTERVAL_SEC` | Polling interval in seconds (default `60`). |
37+
| `SOVEREIGN_INGEST_DEDUP_SEC` | If set (e.g. `300`), the ingest poller will not enqueue a job when one with the same `goal` and `amount_cents` was already created within this many seconds. Reduces duplicate jobs from repeated polling. |
38+
39+
## Audit trail (verifiable)
40+
41+
| Variable | Description |
42+
|----------|-------------|
43+
| `SOVEREIGN_AUDIT_TRAIL_PATH` | Path to JSONL file where each `AuditReport` is appended (with `proof_hash`). Enables `GET /api/audit_trail`. |
44+
45+
## Compliance (Phase 6b)
46+
47+
| Variable | Description |
48+
|----------|-------------|
49+
| `SOVEREIGN_COMPLIANCE_SPEND_THRESHOLD_CENTS` | If set to a positive number, the Web UI engine uses `ThresholdComplianceHook`: when a task’s estimated cost (cents) is ≥ this value, the job is put back to `pending` with “Human approval required” until a second approval. See [MONETIZATION.md](MONETIZATION.md) and [PHASE6.md](PHASE6.md). |
50+
| `SOVEREIGN_COMPLIANCE_AUTO_PROCEED` | When set to `true` (or `1`/`yes`), the CFO does **not** require human approval when the compliance hook returns “request human approval” for spend above the threshold; the task is allowed to proceed (human-out-of-loop for high spend). Use only when you accept the risk. |
51+
52+
## Human out of loop
53+
54+
| Variable | Description |
55+
|----------|-------------|
56+
| `SOVEREIGN_AUTO_APPROVE_JOBS` | When set to `true` (or `1`/`yes`), every new job (from `POST /api/jobs` or ingest) is immediately set to `approved` so the worker runs it without a human clicking Approve. Full **human-out-of-loop**: ingest → auto-approve → run → charge → webhook. |
57+
| `SOVEREIGN_COMPLIANCE_AUTO_PROCEED` | See Compliance above. Together with `SOVEREIGN_AUTO_APPROVE_JOBS`, both gates can be automated. |
58+
59+
## Job completion webhook (delivery / proactive contact)
60+
61+
| Variable | Description |
62+
|----------|-------------|
63+
| `SOVEREIGN_WEBHOOK_URL` | When a job reaches `completed` or `payment_failed`, a POST is sent to this URL with a JSON payload (job_id, status, goal, amount_cents, result_summary, audit_score, etc.). Optional. |
64+
| `SOVEREIGN_WEBHOOK_SECRET` | If set, the request body is signed with HMAC-SHA256 and sent in header `X-Sovereign-Signature: sha256=<hex>`. Receivers should verify this. |
65+
| `SOVEREIGN_WEBHOOK_LOG_PATH` | If set (default `data/webhook_log.jsonl` when logging is enabled), failed webhook deliveries (after all retries) are appended as one JSONL line per failure (url, job_id, status, error, ts). |
66+
| Per-job `callback_url` | In `POST /api/jobs` body you can pass `callback_url`; when that job completes, the webhook is sent to this URL instead of (or in addition to) the global `SOVEREIGN_WEBHOOK_URL`. |
67+
| (payload) | Each job gets a `request_id` (trace id) at creation; it is included in the webhook payload when set, for correlating logs and delivery. |
68+
69+
Payload format and retries: see [OPEN_SOURCE_READY_PLAN.md](OPEN_SOURCE_READY_PLAN.md) (Webhook 载荷规范). Typical use: your backend receives the webhook and then notifies the customer (email, Slack, etc.).
70+
71+
## Payments (Stripe)
72+
73+
| Variable | Description |
74+
|----------|-------------|
75+
| `STRIPE_API_KEY` | Stripe secret key for charges. |
76+
| `STRIPE_WEBHOOK_SECRET` | Webhook signing secret; if set, `POST /api/webhooks/stripe` verifies `Stripe-Signature`. |
77+
78+
## Infrastructure
79+
80+
| Variable | Description |
81+
|----------|-------------|
82+
| `REDIS_URL` | Redis connection URL. If set, `GET /health` checks Redis; used for optional state/cache. |
83+
84+
## LLM & optional extras
85+
86+
- **LLM:** Configured via `sovereign_os.llm.providers` (e.g. `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`). See `[llm]` optional dependency. If only `ANTHROPIC_API_KEY` is set (no `OPENAI_API_KEY`), the default provider is `anthropic` with model `claude-3-5-sonnet-20241022`; no need to set `SOVEREIGN_LLM_PROVIDER` for Anthropic-only use.
87+
- **Memory (ChromaDB):** Optional; see `[memory]` and `sovereign_os.memory.manager`.
88+
- **Telemetry:** Optional OpenTelemetry/Prometheus; see `[telemetry]` and `sovereign_os.telemetry.tracer`.
89+
90+
## Example (Docker Compose)
91+
92+
```yaml
93+
environment:
94+
SOVEREIGN_LEDGER_PATH: /app/data/ledger.jsonl
95+
SOVEREIGN_JOB_DB: /app/data/jobs.db
96+
SOVEREIGN_AUDIT_TRAIL_PATH: /app/data/audit.jsonl
97+
SOVEREIGN_API_KEY: "${SOVEREIGN_API_KEY}"
98+
REDIS_URL: redis://redis:6379/0
99+
```
100+
101+
## CLI
102+
103+
- **Charter path:** Required for `sovereign run` (e.g. `-c charter.example.yaml`). Exits with error if the file is missing.
104+
- **Ledger:** `sovereign run --ledger /path/to/ledger.jsonl` or `SOVEREIGN_LEDGER_PATH` to persist ledger.
105+
- **Audit trail:** `sovereign run --audit-trail /path/to/audit.jsonl` or `SOVEREIGN_AUDIT_TRAIL_PATH` to append AuditReports.
106+
- **Version:** `sovereign --version` prints the package version.

docs/MULTI_INSTANCE.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Multi-instance and horizontal scaling
2+
3+
By default, the job queue is stored in **SQLite** (`SOVEREIGN_JOB_DB`). A single Web process runs the job worker; with `SOVEREIGN_JOB_WORKER_CONCURRENCY` you can run several jobs in parallel on that same process.
4+
5+
## Single instance (current)
6+
7+
- One Web process = one writer to the SQLite DB.
8+
- **Concurrency:** Set `SOVEREIGN_JOB_WORKER_CONCURRENCY=2` (or higher) to run multiple jobs in parallel on that instance.
9+
- **Backup:** See [BACKUP.md](BACKUP.md) for backing up the job DB and ledger.
10+
11+
## Multiple instances (Redis queue)
12+
13+
When **`REDIS_URL`** is set, the app uses **Redis** as the job store and approved queue. Any number of Web processes can run; workers claim the next approved job via `BRPOP` on `sovereign:queue:approved`. Install with `pip install -e ".[deploy]"` (adds `redis`). Jobs are stored in Redis hashes; approval pushes the job id to the shared list so any instance can pop and run it.
14+
15+
| Env | Effect |
16+
|-----|--------|
17+
| `REDIS_URL` | Use Redis for job storage and approved queue; enables multi-instance workers. |
18+
| `SOVEREIGN_JOB_DB` | Used only when `REDIS_URL` is not set; SQLite path for single-instance persistence. |
19+
20+
## Single worker + multiple API servers (SQLite)
21+
22+
To run **multiple Web processes** with SQLite (no Redis), use a **single worker process** that has `SOVEREIGN_JOB_DB` and the worker loop, and one or more **API-only** processes (no worker) that share the same DB path over a shared volume. Only the single worker should write. Prefer Redis or one instance with `SOVEREIGN_JOB_WORKER_CONCURRENCY` for simplicity.

0 commit comments

Comments
 (0)