feat: Chaos E2E suite, service-coverage repair, one-click installer & deploy hardening - #2
Merged
Merged
Conversation
Implements 8 failure-injection scenarios to verify AADS resilience: CM-01 agent_killed_mid_execution — failed_retryable, no lock leak CM-02 executor_crashed_mid_step — container restart → DB-driven recovery CM-03 loki_unavailable — Layer 1 graceful degradation CM-04 litellm_down — Layer 2 no schema-2.0 plan emitted CM-05 approval_expired_before_exec — Gate rejects with 403 CM-06 concurrent_repair_lock — injected lock → policy.blocked:node_locked CM-07 snapshot_missing — blocked:snapshot_failed CM-08 network_partition (iptables) — failed_retryable, no lock leak Scheduling: Fisher-Yates shuffle with --seed N for reproducible CI replays, or a safe dependency-ordered full suite by default. Also adds prereqs and admin-key-file existence checks at startup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…shot guard CM-01: sleep 3 after execute is a race — on fast VMs the entire execution completes before the kill. Kill the agent BEFORE calling execute so the executor's first HTTP call reliably hits a dead agent → failed_retryable. CM-06: SQL query used non-existent column `target_node_id`; the correct column in agent_nodes is `node_id`. Was silently returning empty string → SKIP on every run. CM-07 (Option A): remove `mkdir -p` guard in ensure_known_good_snapshot. Previously, the wrapper created the snapshot dir before checking for pre-existing files, then passed `nginx -t` on valid-but-stopped nginx and rebuilt the snapshot from broken state. Now the wrapper checks for pre-existing snapshot files first and exits 42 (→ HTTP 409 → executor blocked) if none exist, preventing execution against unknown baselines. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ection CM-07 runs immediately after CM-04 (LiteLLM restart). The LLM warm-up latency caused wait_new_plan to time out before Layer 2 generated a plan. Add a 60s readiness gate: check that diagnosis_reports or anomaly_logs show recent activity before breaking nginx and starting the timer. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extends AADS beyond nginx to support PostgreSQL monitoring and
automated repair via the same four-layer architecture pattern.
pi-agent catalog (7 new commands):
Probes: postgresql.status, postgresql.connection_test, postgresql.config_test
Actions: postgresql.restart, postgresql.reload,
postgresql.ensure_config_snapshot, postgresql.restore_known_good_config
Wrapper scripts (5 new):
aads-postgresql-restart / reload / config-test
aads-postgresql-ensure-config-snapshot — blocks if no pre-existing snapshot
aads-postgresql-restore-config — backs up current config before
applying snapshot, rolls back if configtest fails
pi-agent main.py:
- Add PG_SNAPSHOT_DIR, postgresql journal unit default
- service_status() generic probe for any systemd service
- postgresql_connection_test() via pg_isready
- PostgreSQL wrappers included in health_problems() check
Layer 2 system prompt:
- Add PostgreSQL service knowledge section: failure modes,
repair action mapping, when NOT to repair (disk full, data corruption)
E2E tests (scripts/lab/e2e-service-repair.sh):
SR-PG-01: postgresql stopped → AADS restarts → active ✓
SR-PG-02: bad config injected → AADS restores known-good → active ✓
SR-PG-03: no snapshot → execution blocked:snapshot_failed ✓
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…MariaDB
Phase 2 — Redis (5 wrappers + 7 catalog commands):
Probes: redis.status, redis.ping, redis.config_test
Actions: redis.restart, redis.reload,
redis.ensure_config_snapshot, redis.restore_known_good_config
Note: reload is implemented as graceful restart (Redis lacks SIGHUP config reload)
Phase 3 — Docker container (2 wrappers + 3 catalog commands):
Probes: docker.container_status
Actions: docker.container_restart, docker.container_start
Security: double-gated allowlist — validated in pi-agent (arg_allowlist) AND
in the wrapper (AADS_DOCKER_ALLOWED_CONTAINERS env). Sudoers uses wildcard
arg but wrapper enforces the allowlist at OS level.
Phase 4 — MySQL / MariaDB (5 wrappers + 7 catalog commands):
Probes: mysql.status, mysql.connection_test, mysql.config_test
Actions: mysql.restart, mysql.reload,
mysql.ensure_config_snapshot, mysql.restore_known_good_config
Auto-detects mysql vs mariadb service name and config path.
pi-agent main.py:
- DOCKER_ALLOWED_CONTAINERS from env
- service_status() reused for redis/mysql probes
- redis_ping(), mysql_connection_test(), docker_container_status()
- docker actions pass container name as argv[1] to wrapper
Layer 2 system prompt:
- Redis, Docker container, MySQL/MariaDB failure → action mapping added
- Safety rules: no Redis flush, no MySQL repair on InnoDB corruption
e2e-service-repair.sh:
- SR-RD-01/02: Redis stopped / bad config
- SR-DC-01/02: container stopped / not-allowlisted container → policy.blocked
- SR-MY-01/02: MySQL stopped / bad config
- --service redis|docker|mysql flags for targeted runs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…is/MySQL
Root cause: Layer 2's post-process function only injected node_agent steps
for nginx. For PostgreSQL/Redis/MySQL anomalies it returned an empty-steps
plan → schema 1.0 fallback → Gate rejected approval with "no executable steps".
Two additional fixes bundled (deployed to running infra, not in .env.lab git):
- LAYER1_LOKI_QUERY changed from {source="target-nginx"} to
{node_id="128d7819..."} so Layer 1 queries ALL services on the node
- target Alloy config updated to watch PG/Redis/MySQL log files
_ensure_lab_node_agent_steps now detects postgresql/postgres, redis,
mysql/mariadb keywords and injects the appropriate node_agent catalog steps:
status probe → restart or restore_config → connection/ping verify
_expected_outcome_for / _verification_for updated with all new service mappings.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ation Two Pydantic min_length=1 validation errors crashed _build_root_cause_report: 1. affected_service: cluster.containers may contain empty strings when anomalies come from file-based log sources without a container label (e.g. syslog entries, PostgreSQL/Redis/MySQL log files). Added fallback chain: valid containers → non-system services → any service → "unknown". 2. root_cause: action_plan.root_cause and context_analysis may both be empty strings when the LLM returns a partial response. Added default fallback string so validation always passes. Both failures threw exceptions caught by the outer except block, creating fallback plans with action_plan=None that the Gate API returns 404 for, causing the E2E test script to exit after the info "plan: ..." line. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…d unit ProtectSystem=strict makes filesystem read-only for aads-agent. Without ReadWritePaths entries, wrapper scripts writing to /etc/postgresql, /etc/redis, /etc/mysql fail with 'Read-only file system'. Added: /etc/postgresql, /etc/redis, /etc/mysql (config write) and /var/log/postgresql, /var/log/redis, /var/log/mysql (log read). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…adOnlyPaths Three ownership bugs in restore-config wrappers: - postgresql.conf restored as root:aads-agent (0640) → postgres user couldn't read - redis.conf restored as root:aads-agent → redis user couldn't read - mysql config restored without ownership reset Fix: explicit chown+chmod after cp to restore service-native ownership: postgresql.conf → postgres:postgres 0600 redis.conf → redis:redis 0640 my.cnf → root:root 0644 Also use 'su -s /bin/sh postgres -c postgres -t' for pg config test instead of pg_ctlcluster which requires data dir access under ProtectSystem=strict. systemd unit: add ReadOnlyPaths for data directories (/var/lib/postgresql, /var/lib/redis, /var/lib/mysql) so pg_ctlcluster can read cluster metadata when called as fallback configtest. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…, config-error detection Three fixes to make PG/Redis/MySQL repair plans actually executable: 1. final_verification was hardcoded to nginx.http_check — a Redis/PG repair would restart the service successfully (step_verified) but then fail the overall execution on the nginx curl. Now _final_verification_for() picks postgresql.connection_test / redis.ping / mysql.connection_test by service. 2. pre_execution_snapshot was hardcoded to nginx.ensure_known_good_snapshot, so the 'no known-good baseline' guard checked the wrong service. Now _snapshot_command_for() routes to the matching ensure_config_snapshot. This also fixes SR-PG-03 (snapshot-missing must block). 3. is_config_error matched generic 'error'/'failed', routing a plain service stop to the destructive restore path. Tightened to specific config-failure signatures (invalid line, syntax error, can't open config, our injected chaos markers, etc.) so a stop → restart and only real config errors → restore. Also: postgresql-restore-config wrapper now validates via systemctl restart + is-active (PostgreSQL has no offline configtest; the broken 'postgres -t' path always failed because the binary isn't on PATH). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Summarizes Phase 1-4 status, current test failures + root causes, the manual deploy procedure (the main gap), target VM state, and prioritized TODO. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- AADS-System-Overview.md: Three-Agent + Gate architecture, the four components, 32-command catalog, security model, four-layer repair pattern, lab environment. - AADS-Testing.md: Chaos E2E (CM-01..08, all pass) and Service-Coverage E2E (SR-*) scenario tables, results, and the full bug-fix history of both suites. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add §2.0 master component table mapping every directory and every running container to its layer (L0-L4 / Gate / target agent / data), with run location (controller vs target) and role. Expand each component subsection with layer label, compose service name, run location, and actual source files: - 2.2 Layer 0: Alloy/Loki/log-archiver/auditd config breakdown (was missing) - 2.3 Layer 1: clarify layer1-filter (active poll) vs ingester (passive sink) - 2.4 Layer 2: full submodule table (consumer/aggregator/reasoner/llm/schemas/L3) - 2.7 DB: tables grouped by owning layer Clarify three common confusions: layer0-collector is config-only; Layer 3 is merged into Layer 2 (_run_layer3); layer3-remediation/ dir is superseded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On-Device Agent (curl|bash): - dist/install-agent.sh: interactive full installer — apt deps, downloads agent payload from server, runs pi-agent/install/install.sh, registers node via /api/agents/register, installs Alloy log forwarder - dist/build-agent-payload.sh: packages pi-agent/ into aads-agent.tgz - pi-agent/alloy/config.alloy.tmpl: Alloy config template with __NODE_ID__/__LOKI_URL__ placeholders filled at install time Server Docker bundle (prebuilt registry images): - docker-compose.prod.yaml: mirrors dev compose but pulls ghcr.io/bs10081/aads-* images; mounts ./dist into dashboard for self-serve agent installer downloads - dist/bootstrap-server.sh: interactive server bootstrap — generates secrets, writes .env, builds agent payload, compose pull+up, verifies DB, prints ready-to-paste agent install command - .github/workflows/build-images.yml: builds and pushes 6 service images to GHCR on tag push; attaches agent payload to releases Supporting changes: - dashboard/app.py: add /install-agent.sh and /aads-agent.tgz static routes so targets download payload from the server without external net - .env.example: add AADS_IMAGE_REGISTRY/TAG keys for prod compose - .gitignore: stop ignoring dist/ scripts; only ignore dist/aads-agent.tgz - DEPLOYMENT.md: rewrite for one-click server-first → agent flow - docs/lab-deploy-plan.md: track lab deploy plan (untracked file) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
scripts/lab/demo-break.sh runs directly on the target node (no SSH needed). Supports an interactive menu or a single scenario as a CLI argument: nginx-bad-config / nginx-stopped mysql-bad-config / mysql-stopped redis-bad-config / redis-stopped pg-bad-config / pg-stopped all-stopped / restore-all Each scenario first restores the service to a clean baseline (using AADS snapshots when available), then injects the fault so demos always start from a known-good state. Writes to systemd logger and service-specific error logs so Layer 1 picks up the anomaly within 30-60 s. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If nginx / MySQL / Redis / PostgreSQL are not installed on the target, demo-break.sh now installs them via apt before running the scenario. Each ensure_*() also triggers the corresponding AADS snapshot wrapper (if the agent is installed) so the known-good baseline is captured immediately after installation. Also: restore-all and all-stopped now ensure all services are installed first, and the interactive menu notes that auto-install is available. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bootstrap-server.sh now prints two agent install variants: - HTTPS: pulls install-agent.sh from raw.githubusercontent.com and aads-agent.tgz from GitHub Releases latest/download — no server port needed, works from any internet-connected target - HTTP: self-hosted from the dashboard at :5000, air-gapped friendly DEPLOYMENT.md updated with both variants and usage notes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dist/install-server.sh — curl|bash entry point for the server: Downloads aads-server.tgz from GitHub Releases, extracts to ~/aads (or /opt/aads for root), then runs bootstrap-server.sh. Supports AADS_INSTALL_DIR and AADS_RELEASE_BASE_URL overrides. dist/build-server-bundle.sh — packages all server-side files into aads-server.tgz: docker-compose.prod.yaml, .env.example, config dirs (litellm, prometheus, grafana, layer0-collector, migrations), dist/ scripts, pi-agent/, and the prebuilt aads-agent.tgz so the dashboard can serve self-hosted agent installs out of the box. CI: release-assets job now also builds and attaches aads-server.tgz and install-server.sh alongside the existing agent assets. DEPLOYMENT.md: server section now leads with the HTTPS one-liner. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
raw.githubusercontent.com/main/ returns 404 when files only exist on a non-default branch. GitHub Release assets (/releases/latest/download/) are stable regardless of branch and already verified accessible (200). Updated in: dist/bootstrap-server.sh, DEPLOYMENT.md. All curl one-liners now point to releases/latest/download/*. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GHCR packages default to private even for public repos, causing
'unauthorized' errors on fresh servers. bootstrap-server.sh now:
1. Tries docker compose pull from GHCR (fast, uses prebuilt images)
2. On unauthorized error: shallow-clones the public repo, copies the
already-generated .env, and runs docker compose up --build locally
3. All other pull errors still die with the original message
The fallback builds take ~5 min on first run but require no GHCR auth.
Users can permanently fix this by making the 6 GHCR packages public at
https://github.com/bs10081?tab=packages
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
layer1-filter image is ~2.8 GB (includes PyTorch) and can hit transient EOF/network interrupts on slow or unstable connections. The pull step now retries up to 5 times with a 5s delay between attempts, showing the last error line on each failure. Auth errors (unauthorized/denied) still break immediately and fall through to the local-build fallback. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Repeated CloudFront EOFs on multi-GB image blobs were exhausting the
default retry budget. Root cause: re-running 'docker compose pull'
discards a partially-downloaded blob, so retrying the whole pull never
makes progress on a single large blob that drops mid-transfer.
Three lower-level fixes:
1. /etc/docker/daemon.json max-download-attempts=10 (merged via jq,
preserves existing settings like DNS) — Docker's own per-blob retry
inside a single pull, which DOES resume blob attempts
2. COMPOSE_PARALLEL_LIMIT=1 — serialize pulls so all bandwidth goes to
one blob, finishing it inside the connection's stable window
3. Bump script-level pull retries 5 -> 8
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/etc/nginx and /var/log/nginx lacked the '-' prefix, causing systemd to fail with status=226/NAMESPACE when nginx is not installed on the target. All service paths are now suppressed-if-missing so the agent starts regardless of which services are present on the host. Also remove 'After=nginx.service' — the agent doesn't depend on nginx being active, only on the network being online. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously the installer ran systemctl status (informational), then silently polled the HTTP API for 30 s before dying with a vague message. On status=226/NAMESPACE or similar failures the user had no actionable output and had to investigate manually. Now: after install.sh exits, wait up to 15 s for aads-agent.service to enter 'active'; if it doesn't, print the last 30 journal lines and die immediately with a clear message. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Builds on the merged Three-Agent + Gate architecture (#1) with 26 new commits covering chaos end-to-end testing, multi-service repair coverage, a one-click installer, and deployment hardening for unstable networks.
Highlights
Chaos E2E test suite
feat(chaos-e2e): add Chaos E2E suite (CM-01 through CM-08)fix(chaos-e2e): fix CM-01 timing race, CM-06 wrong column, CM-07 snapshot guardfix(chaos-e2e): add pipeline readiness wait in CM-07 before fault injectionService-coverage repair (PostgreSQL, Redis, Docker, MySQL/MariaDB)
feat(service-coverage): Phase 1 — PostgreSQL detection and repairfeat(service-coverage): Phase 2/3/4 — Redis, Docker container, MySQL/MariaDBfix(service-coverage): service-aware final verification, snapshot cmd, config-error detectionfix(service-coverage): extend_ensure_lab_node_agent_stepsfor PG/Redis/MySQLfix(layer2): fix emptyaffected_serviceandroot_causePydantic validationpi-agent hardening
fix(pi-agent): add PostgreSQL/Redis/MySQL to ReadWritePaths in systemd unitfix(pi-agent): restore correct ownership after config restore, add ReadOnlyPathsfix(agent): make all ReadWritePaths optional with '-' prefixInstaller & deployment
feat: one-click installer and server Docker bundle; server curl|bash installer and bundle builderfeat: HTTPS install command for public repo; runner v1 manual nginx demofeat(demo): on-target fault injection script; auto-install missing services before fault injectionfix(installer): fail fast with journal output when service won't startfix: retry docker pull up to 5 times, fallback to local build on unauthorized GHCR, GitHub Release asset URLs, harden image pulls for unstable networksDocs
Stats
26 commits · 117 files changed
🤖 Generated with Claude Code