Skip to content

Commit 02253db

Browse files
Das-rebelDas-rebel
committed
refactor(Results): Restructure Results section as comprehensive academic-style presentation
- Results section now contains ALL evaluations: RouterArena, MMR-Bench, Local Evaluation, Official Baseline Status - Moved Local Evaluation from Getting Started into Results (proper academic structure) - Added explicit interpretation of RouterArena Score, Robustness, Abnormal entries - Added MMR-Bench as a standalone subsection with its own results table - Added methodology paragraph to Local Evaluation explaining query labeling protocol - Replaced verbose tier accuracy text with structured Tier Accuracy Breakdown table - Added Cost and Latency table with interpretation of +236ms overhead - Added CLI subsection back to Getting Started for discoverability - Section flow: Results → Architecture → Provider Coverage → Getting Started → API → Config
1 parent 72ac530 commit 02253db

1 file changed

Lines changed: 72 additions & 31 deletions

File tree

‎README.md‎

Lines changed: 72 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ A3M Router is a stateless proxy that routes LLM requests across 47+ providers us
1010

1111
### RouterArena (ICLR 2025)
1212

13-
RouterArena evaluates LLM routers on query-level routing decisions against ground-truth model selections. A3M was evaluated in the official RouterArena benchmark suite.
13+
RouterArena evaluates LLM routers on query-level routing decisions against ground-truth model selections. A3M was evaluated in the official RouterArena benchmark suite across 8,400 queries, covering diverse domains and complexity levels.
1414

1515
| Metric | Value |
1616
|--------|-------|
@@ -21,21 +21,80 @@ RouterArena evaluates LLM routers on query-level routing decisions against groun
2121
| Abnormal entries | 0 |
2222
| Total queries evaluated | 8,400 |
2323

24+
**Score** is RouterArena's composite metric combining routing accuracy, robustness, and cost efficiency. **Robustness = 1.0000** means every response was valid (no null outputs, no timeouts, no malformed responses). **Abnormal entries = 0** confirms no routing decisions produced degenerate outputs.
25+
26+
2427
**Reference:** RouteWorks/RouterArena#144 (merged, premium-tier evaluation)
2528

29+
#### RouterArena Leaderboard
30+
31+
![RouterArena Leaderboard](assets/chart-routerena-leaderboard.svg)
32+
33+
### MMR-Bench (ArXiv 2026)
34+
35+
MMR-Bench evaluates multimodal routing performance across diverse LLM tasks. A3M was adopted as an official baseline.
36+
37+
| Metric | Value |
38+
|--------|-------|
39+
| Exact tier match | 67% |
40+
| Cost savings vs all-premium | 63.5% |
41+
| Robustness | 0.86 |
42+
43+
**Reference:** Hunter-Wrynn/MMR-Bench#4 (merged)
44+
45+
### Local Evaluation (n=200, no API key required)
46+
47+
The local benchmark uses a held-out set of 200 queries labeled by complexity tier (free / cheap / mid / premium). Tier assignments were determined by estimating the minimum model capability required to answer each query correctly. Routing decisions are compared against these ground-truth labels.
48+
49+
| Metric | Value |
50+
|--------|-------|
51+
| Exact tier match | 67% (134/200) |
52+
| Within 1 tier | 96% (192/200) |
53+
| Cost savings vs all-premium | 62.9% |
54+
55+
#### Tier Accuracy Breakdown
56+
57+
| Tier | Exact match | Errors | Primary error pattern |
58+
|------|-------------|--------|---------------------|
59+
| Free (n=50) | 96% (48/50) | 2 | Upward to cheap (2) |
60+
| Cheap (n=60) | 75% (45/60) | 15 | Upward to free (13) |
61+
| Mid (n=50) | 36% (18/50) | 32 | Downward to cheap (22) |
62+
| Premium (n=40) | 57.5% (23/40) | 17 | Downward to mid (11) |
63+
64+
Mid-tier queries are the primary source of errors. The keyword-based signal approach has limited discriminative power for queries that sit at the boundary between simple and complex — for example, queries requiring domain expertise but no multi-step reasoning. However, the 96% within-1-tier rate means these errors rarely produce a severe capability mismatch: a mid query routed to cheap still reaches a mid-capability model in most cases.
65+
66+
![Routing Accuracy by Tier](assets/chart-accuracy-by-tier.svg)
67+
68+
![Confusion Matrix — Predicted vs Actual Tier](assets/chart-confusion-matrix.svg)
69+
70+
#### Cost and Latency
71+
72+
| Metric | Value |
73+
|--------|-------|
74+
| Cost per 1K tokens (RouterArena) | $0.0768 |
75+
| Cost savings vs all-premium (MMR-Bench) | 63.5% |
76+
| A3M Auto routing overhead vs direct | +236ms |
77+
| A3M Forced routing overhead vs direct | +96ms |
78+
79+
The +236ms overhead for auto routing is dominated by the routing decision itself (+140ms) and proxy forwarding (+96ms), not network latency to the target provider. The total latency (374ms end-to-end for Groq) is within typical LLM response times and does not add perceptible delay for interactive use.
80+
81+
![Cost Comparison](assets/chart-cost-comparison.svg)
82+
83+
![Latency Overhead](assets/chart-latency-overhead.svg)
84+
85+
86+
---
87+
2688
### Official Baseline Status
2789

28-
| Benchmark | Status | Notes |
29-
|-----------|--------|-------|
30-
| RouterArena (ICLR 2025) | Baseline merged — PR#144 | Premium tier: 96.77% accuracy, $0.0768/1K |
31-
| RouterArena free tier (ICLR 2025) | Baseline submitted — PR#152 | Free tier evaluation pending |
32-
| RouterEval (EMNLP 2025) | Baseline merged | MilkThink-Lab/RouterEval#4 |
90+
| Benchmark | Status | Reference |
91+
|-----------|--------|------------|
92+
| RouterArena premium tier (ICLR 2025) | Baseline merged | RouteWorks/RouterArena#144 |
3393
| MMR-Bench (ArXiv 2026) | Baseline merged | Hunter-Wrynn/MMR-Bench#4 |
34-
| LLMRouterBench (ACL 2026) | Baseline submitted | ynulihao/LLMRouterBench#3 |
35-
36-
### RouterArena Leaderboard
94+
| RouterEval (EMNLP 2025) | Baseline merged | MilkThink-Lab/RouterEval#4 |
95+
| RouterArena free tier (ICLR 2025) | Submitted | RouteWorks/RouterArena#152 |
96+
| LLMRouterBench (ACL 2026) | Submitted | ynulihao/LLMRouterBench#3 |
3797

38-
![RouterArena Leaderboard](assets/chart-routerena-leaderboard.svg)
3998

4099
---
41100

@@ -186,29 +245,11 @@ const features = router.analyze("Review this contract for liability clauses");
186245
### CLI
187246

188247
```bash
189-
npx a3m-router route "Explain quantum computing" # returns routing decision
190-
npx a3m-router benchmark # run local accuracy test
191-
npx a3m-router health # provider health status
248+
npx a3m-router route "Explain quantum computing" # returns routing decision and tier
249+
npx a3m-router benchmark # run local accuracy test (n=200)
250+
npx a3m-router health # provider health status and latency
192251
```
193252

194-
### Local Benchmark (n=200, no API key required)
195-
196-
| Metric | Value |
197-
|--------|-------|
198-
| Exact tier match | 67% |
199-
| Within 1 tier | 96% |
200-
| Cost savings vs all-premium | 62.9% |
201-
202-
**Tier accuracy breakdown:** Free queries are routed correctly 96% of the time (48/50). Cheap queries achieve 75% exact match (45/60), with the primary confusion being an upward bias — cheap queries routed to free tier (13 cases) rather than premium. Mid-tier accuracy is 36% (18/50), representing the primary weakness of the heuristic approach: mid-complexity queries are harder to classify precisely with keyword-based signals alone. Premium queries achieve 57.5% exact match (23/40), with most errors being under-routing to mid-tier (11 cases). The 96% within-1-tier rate means errors rarely result in a capability mismatch — a misrouted mid query still typically reaches a mid-capability model.
203-
204-
![Routing Accuracy by Tier](assets/chart-accuracy-by-tier.svg)
205-
206-
![Confusion Matrix — Predicted vs Actual Tier](assets/chart-confusion-matrix.svg)
207-
208-
![Cost Comparison](assets/chart-cost-comparison.svg)
209-
210-
![Latency Overhead](assets/chart-latency-overhead.svg)
211-
212253
---
213254

214255
## API Reference

0 commit comments

Comments
 (0)