Skip to content

Commit 137aff5

Browse files
Subhajit dasSubhajit das
Subhajit das
authored and
Subhajit das
committed
docs: add engineering spec, claims, reproducibility docs
- ENGINEERING_SPEC.md - system design and architecture - CLAIMS_AND_EVIDENCE.md - feature claims with evidence - REPRODUCIBILITY.md - reproducible results documentation - CHINESE_PROVIDER_RELIABILITY.md - Chinese LLM provider reliability - RELEASE_CHECKLIST.md - release process
1 parent 759aa43 commit 137aff5

5 files changed

Lines changed: 245 additions & 0 deletions

File tree

‎docs/CHINESE_PROVIDER_RELIABILITY.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Chinese Provider Reliability Playbook
2+
3+
This playbook captures retry/fail-fast guidance for Chinese LLM providers used by A3M.
4+
5+
## Why this exists
6+
7+
A generic `429 => retry` rule is often too coarse. Some Chinese-provider 429/4xx states are account/quota policy states that should fail fast instead of burning retries.
8+
9+
## A3M policy
10+
11+
- Retry:
12+
- transient network failures (`ECONNRESET`, `ETIMEDOUT`, 5xx)
13+
- throttling with temporary overload semantics
14+
- 429 with explicit `Retry-After` timing
15+
- Fail fast (non-retryable):
16+
- auth/policy/billing/account states (401/402/403)
17+
- account abnormal / access terminated
18+
- hard quota exhaustion messaging (hour/week/month quota exhausted, org TPD exceeded)
19+
20+
## Implemented in code
21+
22+
- `src/routing/providerRetry.ts`
23+
- `isPermanentProviderStateError(...)` now guards retries.
24+
25+
## Validation
26+
27+
- `eval/run_fault_injection.js`
28+
- Includes `no_retry_on_chinese_quota_account_errors` scenario.
29+
30+
## Operational guidance
31+
32+
- Track provider-specific errors by normalized category:
33+
- `transient`, `rate_limit_transient`, `quota_hard`, `auth`, `account_policy`, `server`
34+
- Use this categorization to:
35+
- reduce wasted retries
36+
- improve fallback quality
37+
- produce cleaner incident dashboards

‎docs/CLAIMS_AND_EVIDENCE.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# Claims and Evidence
2+
3+
This file maps product claims to reproducible evidence in-repo.
4+
5+
## Routing correctness and stability
6+
7+
- Claim: routing behavior is stable across releases.
8+
- Evidence:
9+
- `eval/benchmark_dataset.jsonl`
10+
- `eval/golden_routes.json`
11+
- `npm run eval:golden`
12+
13+
## Routing quality thresholds
14+
15+
- Claim: routing meets minimum quality bars for complexity/flags/domain/provider type.
16+
- Evidence:
17+
- `eval/thresholds.json`
18+
- `eval/run_eval.js`
19+
- `npm run eval:routing`
20+
- Output artifact: `eval/results/latest.json`
21+
22+
## Reliability under failure
23+
24+
- Claim: retry/circuit-breaker/fallback logic works under failure scenarios.
25+
- Evidence:
26+
- `eval/run_fault_injection.js`
27+
- `eval/fault_injection_thresholds.json`
28+
- `npm run eval:faults`
29+
- Output artifact: `eval/results/fault_injection_latest.json`
30+
31+
## Test coverage enforcement
32+
33+
- Claim: changes are tested in CI before merge.
34+
- Evidence:
35+
- `.github/workflows/ci.yml`
36+
- Includes:
37+
- `npm test`
38+
- `npm run eval:routing`
39+
- `npm run eval:golden`
40+
- `npm run eval:faults`
41+
- `npm run eval:shadow`
42+
- `npm run test:py`
43+
44+
## Baseline governance
45+
46+
- Baseline file:
47+
- `eval/baselines/main.json`
48+
- Rule:
49+
- Update baseline only when behavior change is intentional.
50+
- PR must explain what changed and why.
51+
52+
## Experiment traceability
53+
54+
- Claim: evaluation outcomes are auditable over time.
55+
- Evidence:
56+
- `eval/lib/experiment_registry.js`
57+
- `eval/experiments.jsonl` (local append-only run log)
58+
- each eval runner appends run metadata + decision.

‎docs/ENGINEERING_SPEC.md‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# A3M Engineering Spec (Canonical)
2+
3+
This is the canonical engineering behavior spec for A3M Router.
4+
Marketing and launch content are non-canonical; if there is a conflict, this file wins.
5+
6+
## Core Routing Contract
7+
8+
- Input: `routeQuery(prompt: string, available_models?: string[], budget_multiplier?: number)`
9+
- Output:
10+
- `primary_model`
11+
- `fallback_models`
12+
- `confidence`
13+
- `estimated_cost`
14+
- `estimated_latency_ms`
15+
- `features` (complexity + flags + domain)
16+
- `provider_type`
17+
18+
## Reliability Components
19+
20+
- Retry handling:
21+
- `ProviderRetryHandler` supports transient retries, backoff+jitter, and rate-limit handling.
22+
- Health management:
23+
- `ProviderHealthManager` maintains rolling health and circuit breaker states.
24+
- Circuit breaker opens after configured consecutive failures.
25+
- Fallback chain:
26+
- Health-sorted fallback ordering with unavailable providers pushed down.
27+
28+
## Guardrails
29+
30+
- Input and output checks implemented in `src/security/guardrails.ts`.
31+
- Includes prompt injection scoring, PII detection/redaction, and output validation hooks.
32+
33+
## Cost/Budget
34+
35+
- Budget enforcement and spend tracking:
36+
- `src/cost/budgetEnforcer.ts`
37+
- `src/cost/costTracker.ts`
38+
39+
## Proxy Server
40+
41+
- OpenAI-compatible endpoints implemented in `src/server/proxyServer.ts`.
42+
- Expected behavior:
43+
- Model resolution through mapper + router
44+
- Provider call with fallback behavior
45+
- Usage/cost logging for requests
46+
47+
## Validation Gates (Required)
48+
49+
- Node test suite: `npm test`
50+
- Python tests: `npm run test:py`
51+
- Routing eval: `npm run eval:routing`
52+
- Golden routing regression: `npm run eval:golden`
53+
- Fault injection reliability: `npm run eval:faults`
54+
55+
All gates above must pass for release readiness.

‎docs/RELEASE_CHECKLIST.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Release Checklist
2+
3+
Use this checklist before tagging a release.
4+
5+
## Mandatory quality gates
6+
7+
- [ ] `npm test` passes
8+
- [ ] `npm run test:py` passes
9+
- [ ] `npm run eval:routing` passes
10+
- [ ] `npm run eval:golden` passes
11+
- [ ] `npm run eval:faults` passes
12+
- [ ] `npm run eval:report` passes
13+
14+
## Evidence artifacts reviewed
15+
16+
- [ ] `eval/results/latest.json` reviewed for routing summary
17+
- [ ] `eval/results/fault_injection_latest.json` reviewed for reliability scenarios
18+
- [ ] `eval/results/shadow_latest.json` reviewed for divergence/cost deltas
19+
- [ ] `eval/results/report_latest.md` attached to release review
20+
- [ ] Any baseline change in `eval/baselines/main.json` is intentional and explained
21+
22+
## Documentation consistency
23+
24+
- [ ] `docs/ENGINEERING_SPEC.md` reflects current behavior
25+
- [ ] `docs/CLAIMS_AND_EVIDENCE.md` mappings are still valid
26+
- [ ] Public claims do not exceed available evidence
27+
28+
## Release hygiene
29+
30+
- [ ] Version bump completed
31+
- [ ] Changelog updated
32+
- [ ] CI green on release commit

‎docs/REPRODUCIBILITY.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Reproducibility Contract
2+
3+
This document defines the reproducible evaluation contract for A3M Router.
4+
5+
## Environment
6+
7+
- Node: `>=18` (CI uses Node 22)
8+
- Python: `3.12` (for Python tests)
9+
- Install:
10+
- `npm ci`
11+
- `python3 -m pip install pytest pytest-asyncio`
12+
13+
## Required commands
14+
15+
Run in repository root:
16+
17+
```bash
18+
npm test
19+
npm run test:py
20+
npm run eval:routing
21+
npm run eval:golden
22+
npm run eval:faults
23+
npm run eval:shadow
24+
npm run eval:report
25+
```
26+
27+
## Deterministic inputs
28+
29+
- Core regression dataset:
30+
- `eval/benchmark_dataset.jsonl`
31+
- Golden snapshot:
32+
- `eval/golden_routes.json`
33+
- Thresholds:
34+
- `eval/thresholds.json`
35+
- `eval/fault_injection_thresholds.json`
36+
37+
## Artifacts generated
38+
39+
- `eval/results/latest.json`
40+
- `eval/results/fault_injection_latest.json`
41+
- `eval/results/shadow_latest.json`
42+
- `eval/results/report_latest.md`
43+
44+
## Experiment registry
45+
46+
- Every eval run appends to:
47+
- `eval/experiments.jsonl` (local artifact)
48+
- Record includes:
49+
- timestamp
50+
- commit (if available)
51+
- experiment id
52+
- dataset version
53+
- metrics
54+
- decision
55+
56+
## Baseline update policy
57+
58+
- Baseline file: `eval/baselines/main.json`
59+
- Only update baseline when behavior change is intentional.
60+
- PR must explain:
61+
- what changed
62+
- why baseline needs update
63+
- expected impact on cost/quality/reliability

0 commit comments

Comments
 (0)