Skip to content

Commit 8ebb163

Browse files
krandderkrandder
committed
auto-qa(harness): step 45 — scenario #41 candles-504-gateway-timeout (/companies row complete 16/16)
Adds 41-candles-504-gateway-timeout.scenario.mjs — fills the 2nd cell of the new "gateway timeout 504" matrix row (companion to #40 on registry side). REGISTRY happy + CANDLES 504 + Content-Type: text/html + nginx-style HTML body (no JSON, no Retry-After). Asserts the carousel renders the event card but the price degrades to "0.00 SDAI" — same terminal UX as #3 (502) and #37 (429) but DIFFERENT control flow. Distinct from #3 and #37: - vs #3 (502 + JSON): #3's body parses cleanly, consumer hits .catch on structured envelope. #41's body is HTML — .json() throws SyntaxError BEFORE the consumer's status-check runs. Different bug-class entirely. - vs #37 (429 + Retry-After + JSON): #37 has explicit Retry-After contract clients SHOULD respect. #41 has no Retry-After (504s typically lack one). Consumer that conflates them ignores Retry-After on 429 OR fails to retry on 504 — invisible regression class. Distinct from #40 (same row, registry side): #40 takes the carousel-data-source out (registry empty → "No organizations found"); #41 keeps it healthy but takes the per-card price-fetch out — exercises 2nd-tier data-fetch failure with 504+HTML control flow. Bug-shapes captured: per-pool fetcher crashes on JSON.parse(html) throwing SyntaxError, per-pool fetcher renders raw HTML body in price card, per-pool fetcher treats 504 as success with empty data (silent broken state), per-pool fetcher hammers candles upstream with retries (no Retry-After to respect — worse than #37), card hangs on LoadingSpinner forever (parse error before loading-cleanup), bulk-prefetch vs per-pool fallback race wins/loses unpredictably. MATRIX MILESTONE: /companies completes the new 8th row → 16/16 cells filled across all 8 failure-mode rows × 2 endpoints. /markets/[address] still 14/16 (next two slices will fill the market-page side of the new row). Live re-validation: smoke 80/80 (pin floor bumped from 15 to 16 on /companies), scenario #41 passes in 2.3s on first run, catalog regenerated (41 scenarios).
1 parent fb7eccd commit 8ebb163

4 files changed

Lines changed: 267 additions & 11 deletions

File tree

‎auto-qa/harness/PROGRESS.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2313,6 +2313,102 @@ Phase 6+7 scenarios (4 cases, chromium + Next.js) ✓ ~5s
23132313
cross-layer reconciliation. Remaining: cross-layer
23142314
reconciliations + cross-run monotonicity.
23152315
2316+
- **slice 45-scenario-41-candles-504-gateway-timeout
2317+
(Phase 7 chaos library — completes new row on
2318+
/companies)** (this iteration, on the interface
2319+
side) — fills the 2nd cell of the new "gateway
2320+
timeout 504" matrix row. Companion to #40
2321+
(registry-504): same failure mode on the symmetric
2322+
endpoint. Where #40 leaves the carousel unable to
2323+
mount any cards (registry empty → no events to
2324+
render), #41 keeps registry healthy so the carousel
2325+
mounts — but the per-pool price-fetch hits the
2326+
gateway-timeout candles endpoint and falls through
2327+
to the "0.00 SDAI" fallback shape (same terminal
2328+
UX as #03 / #37 but distinct control flow).
2329+
2330+
* **The scenario** —
2331+
`41-candles-504-gateway-timeout` on /companies.
2332+
REGISTRY happy (carousel mounts our pool-bearing
2333+
proposal) + CANDLES responds 504 +
2334+
`Content-Type: text/html` + nginx-style HTML
2335+
504 page (no JSON, no Retry-After). Asserts the
2336+
event card renders AND the price degrades to
2337+
"0.00 SDAI".
2338+
2339+
* **Bug-shapes captured** (NEW vs #03 and #37):
2340+
- Per-pool fetcher CRASHES on `JSON.parse(html)`
2341+
throwing SyntaxError (body is HTML, .json()
2342+
throws BEFORE the consumer's status-check or
2343+
error-envelope-handler runs — distinct from
2344+
#03's `.catch` on structured error envelope)
2345+
- Per-pool fetcher renders the HTML body raw in
2346+
the price card (consumer falls back to
2347+
`.text()` and renders the LB error page
2348+
literally — leaks infra error to UX surface)
2349+
- Per-pool fetcher treats 504 as IF the request
2350+
succeeded with empty data (silent broken
2351+
state — no telemetry, no retry, no
2352+
user-visible signal)
2353+
- Per-pool fetcher IMMEDIATELY retries in tight
2354+
loop (no exponential backoff; WORSE than #37
2355+
because there's no Retry-After to respect —
2356+
thundering-herd with no contract to follow)
2357+
- Card hangs on LoadingSpinner forever (.then
2358+
fires but parse error throws before
2359+
loading=false setter runs)
2360+
- Bulk-prefetch races per-pool fallback under
2361+
partial 504 (one branch parses, the other
2362+
doesn't — race-conditional render outcome)
2363+
2364+
* **Distinct from #40 (same row, registry side)**:
2365+
- #40 takes the carousel-data-source out
2366+
(registry empty → no event cards mount →
2367+
page hits "No organizations found")
2368+
- #41 keeps the carousel data source healthy
2369+
(event cards mount) but takes the per-card
2370+
price-fetch out — exercises the SECOND-tier
2371+
data-fetch failure path with the same
2372+
504+HTML control flow
2373+
2374+
* **Live re-validation**:
2375+
- Smoke tests: 80/80 (matrix-smoke pin floor
2376+
bumped from 15 to 16 on /companies; no
2377+
other infra changes)
2378+
- Scenario #41 itself: passed in 2.3s on
2379+
first run
2380+
- Catalog regenerated: 41 scenarios (was 40)
2381+
- Matrix script: /companies now 16/16 (8th
2382+
row complete on /companies);
2383+
/markets/[address] stays at 14/16
2384+
2385+
* **Chaos coverage matrix on /companies after
2386+
this slice (16/16 COMPLETE — all 8 rows × 2
2387+
endpoints, /companies fully covered for the
2388+
expanded 8-row failure-mode set)**:
2389+
| failure mode | registry | candles |
2390+
|---------------------|----------|---------|
2391+
| hard 502 | #02 | #03 |
2392+
| partial response | #22 | #04 |
2393+
| empty 200 | #05 | #21 |
2394+
| malformed body | #07 | #08 |
2395+
| per-row corrupt | #09 | #23 |
2396+
| slow valid resp | #19 | #20 |
2397+
| rate-limited 429 | #36 | #37 |
2398+
| gateway timeout 504 | #40 | #41 ★ |
2399+
2400+
* **What's next**: matrix expansion proceeds to
2401+
/markets/[address] cells of the new row —
2402+
slice 46 fills market-page-registry-504
2403+
(mirror of #40 on market page); slice 47
2404+
fills market-page-candles-504 (mirror of #41).
2405+
After 47, the 8-row matrix reaches 32/32 cells
2406+
across both pages — full parity restored at
2407+
the new expanded row count. Beyond that:
2408+
open another failure-mode row, expand to a
2409+
new page, or pivot to cross-layer DOM↔API
2410+
invariant work.
2411+
23162412
- **slice 44-scenario-40-registry-504-gateway-timeout
23172413
(Phase 7 chaos library — NEW failure-mode row)**
23182414
(this iteration, on the interface side) — opens

‎auto-qa/harness/scenarios/41-candles-504-gateway-timeout.scenario.mjs‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
/**
2+
* 41-candles-504-gateway-timeout.scenario.mjs — chaos: HTTP 504
3+
* Gateway Timeout from the CANDLES side on /companies, with an
4+
* HTML body (NOT JSON) — the realistic shape that load balancers
5+
* (AWS ALB, nginx, Cloudflare, GCP HTTPS LB) emit by default
6+
* when the upstream origin fails to respond within the LB's
7+
* idle timeout.
8+
*
9+
* Companion to #40 (registry-504-gateway-timeout): same failure
10+
* mode on the symmetric endpoint. Where #40 leaves the carousel
11+
* unable to mount any cards (registry empty → no events to
12+
* render), #41 keeps registry healthy so the carousel mounts
13+
* — but the per-pool price-fetch hits the gateway-timeout
14+
* candles endpoint and falls through to the same "0.00 SDAI"
15+
* fallback shape as #03 (hard 502) and #37 (rate-limited 429).
16+
*
17+
* Distinct from #03 (502 + JSON body, same fallback UX):
18+
* - #03 fetch resolves with `!response.ok` AND a JSON-shape
19+
* body — `.json()` parses cleanly so the consumer sees
20+
* the structured error envelope and hits the documented
21+
* `.catch` branch.
22+
* - #41 fetch resolves with `!response.ok` AND a non-JSON
23+
* HTML body — `.json()` THROWS SyntaxError BEFORE the
24+
* consumer's status-check or error-envelope-handler runs,
25+
* so the bug surface is the parse-error path, not the
26+
* status-error path.
27+
*
28+
* Distinct from #37 (rate-limited 429 + JSON body, same
29+
* fallback UX):
30+
* - #37 status 429 has explicit `Retry-After` header — the
31+
* contract is "wait then retry"; consumers SHOULD respect
32+
* it but often don't (thundering-herd shape).
33+
* - #41 status 504 has NO Retry-After (LBs don't typically
34+
* emit one for gateway timeouts) — the contract is "we
35+
* gave up waiting; you decide" — consumers should retry
36+
* once or twice with jitter and then give up. A consumer
37+
* that conflates 429 and 504 would either ignore
38+
* Retry-After on 429 OR fail to retry on 504 — different
39+
* bug shapes than the body-parse-error path #41 primarily
40+
* guards.
41+
*
42+
* Distinct from #40 (same failure mode, registry side):
43+
* - #40 takes the carousel-data-source out (registry empty
44+
* → no event cards mount → page hits "No organizations
45+
* found"); #41 keeps the carousel data source healthy
46+
* (event cards mount) but takes the per-card price-fetch
47+
* out — exercises the SECOND-tier data-fetch failure
48+
* path with the same 504+HTML control flow.
49+
*
50+
* Real-world parallel: production candles GraphQL endpoint
51+
* fronted by an LB with a 60s timeout. Origin Postgres has a
52+
* slow query (e.g., aggregate query against a partitioned
53+
* candles table that's missing an index). Some pool-price
54+
* requests time out at the LB → 504; the page must NOT crash,
55+
* must NOT show stale prices, and must NOT spin forever.
56+
*
57+
* Bug-shapes guarded:
58+
* - Per-pool fetcher CRASHES because `response.json()`
59+
* throws on the HTML body (SyntaxError → uncaught
60+
* promise rejection → React error boundary in the price
61+
* card → "Card error" placeholder)
62+
* - Per-pool fetcher renders the HTML body raw in the
63+
* price card (consumer falls back to `response.text()`
64+
* and renders the LB error page literally — leaks infra
65+
* error to UX surface)
66+
* - Per-pool fetcher treats 504 as IF the request
67+
* succeeded with empty data (silent broken state — no
68+
* telemetry, no retry, no user-visible signal)
69+
* - Per-pool fetcher IMMEDIATELY retries in a tight loop
70+
* (no exponential backoff, hammers origin while it's
71+
* already overloaded; thundering-herd shape worse than
72+
* #37 because there's no Retry-After to respect)
73+
* - Card hangs on LoadingSpinner forever — the .then
74+
* handler fires (fetch resolves on 5xx), but the
75+
* downstream consumer doesn't transition `loading=false`
76+
* because the JSON-parse error throws before the
77+
* loading-cleanup code runs
78+
* - Bulk-prefetch races per-pool fallback: one branch
79+
* gets 504 + parse error, the other gets a different
80+
* state, and which one wins determines whether the
81+
* page renders the price or the fallback (race
82+
* condition under partial gateway-timeout)
83+
*
84+
* Strategy: respond 504 + `Content-Type: text/html` + a
85+
* generic nginx-style HTML 504 page. No `Retry-After` (504s
86+
* typically lack one).
87+
*
88+
* Asserts /companies still renders the event card (registry
89+
* intact) but the price degrades to "0.00 SDAI" — same
90+
* terminal UX as #03 (502) and #37 (429) but DIFFERENT
91+
* control flow (status-error + parse-error vs status-error
92+
* + valid-error-envelope vs status-error + Retry-After +
93+
* valid-error-envelope).
94+
*/
95+
96+
import { expect } from '@playwright/test';
97+
98+
import {
99+
REGISTRY_GRAPHQL_URL,
100+
CANDLES_GRAPHQL_URL,
101+
makeGraphqlMockHandler,
102+
fakePoolBearingProposal,
103+
} from '../fixtures/api-mocks.mjs';
104+
105+
const HTML_504_BODY = `<!DOCTYPE html>
106+
<html>
107+
<head><title>504 Gateway Timeout</title></head>
108+
<body>
109+
<center><h1>504 Gateway Timeout</h1></center>
110+
<hr><center>nginx (chaos: candles-504-gateway-timeout)</center>
111+
</body>
112+
</html>
113+
`;
114+
115+
export default {
116+
name: '41-candles-504-gateway-timeout',
117+
description: 'REGISTRY healthy + CANDLES GraphQL responds 504 + Content-Type: text/html + HTML body (no JSON). Asserts the carousel still renders the event card but the price degrades to "0.00 SDAI" — same terminal UX as #03 (502) and #37 (429) but DIFFERENT control flow (status-error + JSON-parse-error vs status-error + valid-error-envelope vs status-error + Retry-After). Fills the 2nd cell of the new "gateway timeout 504" matrix row on /companies.',
118+
bugShape: 'per-pool fetcher crashes on JSON.parse(html) throwing SyntaxError / per-pool fetcher renders raw HTML body in price card / per-pool fetcher treats 504 as success with empty data / per-pool fetcher hammers candles upstream with retries (no backoff, no Retry-After to respect — worse than #37) / card hangs on LoadingSpinner forever (parse error before loading-cleanup) / bulk-prefetch vs per-pool fallback race wins/loses unpredictably under partial 504',
119+
route: '/companies',
120+
121+
mocks: {
122+
// REGISTRY happy path: carousel renders our pool-bearing
123+
// proposal so a card actually mounts. Same as #03/#20/#37.
124+
[REGISTRY_GRAPHQL_URL]: makeGraphqlMockHandler({
125+
proposals: [fakePoolBearingProposal({})],
126+
}),
127+
// CANDLES gateway-timeout: 504 + HTML body. Both bulk
128+
// prefetch AND per-pool fallback hit this same endpoint,
129+
// so both paths see the parse-error path on the HTML.
130+
[CANDLES_GRAPHQL_URL]: async (route) => {
131+
await route.fulfill({
132+
status: 504,
133+
contentType: 'text/html',
134+
body: HTML_504_BODY,
135+
});
136+
},
137+
},
138+
139+
assertions: [
140+
// Carousel rendered our event (REGISTRY healthy).
141+
async (page) => {
142+
await expect(
143+
page.getByText('HARNESS-PROBE-EVENT-001').first(),
144+
).toBeVisible({ timeout: 30_000 });
145+
},
146+
// Same terminal-state as #03 (hard 502) and #37 (429): the
147+
// price formatter ends up at the "no price" fallback string
148+
// (`\`0.00 ${baseTokenSymbol}\``). Distinct CONTROL FLOW
149+
// from both: 504+HTML hits the JSON-parse-error path,
150+
// distinct from 502+JSON (.catch on status) and 429+JSON
151+
// (.catch on Retry-After-aware status).
152+
async (page) => {
153+
await expect(
154+
page.getByText('0.00 SDAI').first(),
155+
).toBeVisible({ timeout: 30_000 });
156+
},
157+
],
158+
159+
timeout: 60_000,
160+
};

‎auto-qa/harness/scenarios/SCENARIOS.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ auto-discovers every `*.scenario.mjs` in this directory and emits one
1010
Playwright test per row. See [ADR-002](../docs/ADR-002-scenario-format.md)
1111
for the format definition.
1212

13-
Total scenarios: **40**
13+
Total scenarios: **41**
1414

1515
| # | File | Bug shape | Route | Description |
1616
|----|------------------------------------|-------------------------------------------------------------|----------------|-------------|
@@ -54,3 +54,4 @@ Total scenarios: **40**
5454
| 38 | `38-market-page-registry-rate-limited.scenario.mjs` | page crashes on 429 from registry (treats response.json() as valid registry data) / page-shell hangs forever (429 doesn't trigger loading=false) / page hammers registry with retries (no Retry-After respect; thundering-herd) / raw "rate limited" error rendered in proposal title or chart placeholder / "Market Not Found" false-positive from 429 wrong-code-path collapse | `/markets/0x45e1064348fD8A407D6D1F59Fc64B05F633b28FC` | REGISTRY responds 429 + Retry-After: 1 + JSON-shape error body + CANDLES happy on /markets/<probe>. Asserts the page-shell still mounts (Trading Pair + wallet shorthand visible) — proves the static MARKETS_CONFIG entry is sufficient even when registry is rate-limited. Distinct from #24 (5xx → .catch) and #36 (same failure mode, different page contract). Catches: 429 treated as valid data downstream crash, forever-loading from 429 not triggering loading=false, retry-storm without Retry-After respect, error message leak to UI, "Market Not Found" wrong-code-path collapse. |
5555
| 39 | `39-market-page-candles-rate-limited.scenario.mjs` | chart panel crashes on 429 (treats response.json() as valid candles data; downstream crash on pools.find/.length) / chart panel hangs forever (429 doesn't trigger loading=false; .catch handles 5xx not 4xx) / chart-fetch hammers candles upstream with retries (no Retry-After; worse than /companies because chart refetches on hover/zoom/time-window-change) / per-pool spot-price renders raw "rate limited" / trading panel preview crashes from candles-derived feed returning error envelope (NaN math) / whole-page crash from missing chart error boundary (collateral damage) | `/markets/0x45e1064348fD8A407D6D1F59Fc64B05F633b28FC` | REGISTRY happy + CANDLES responds 429 + Retry-After: 1 + JSON-shape error body on /markets/<probe>. Asserts the page-shell still mounts (Trading Pair + wallet shorthand visible) — proves a candles-side rate-limit doesn't cascade to a hung/crashed page-shell. Distinct from #25 (5xx → .catch on same page), #38 (registry-side 429 on same page), and #37 (same failure mode, different page). CLOSES the market-page chaos matrix to 14/14, achieving full parity with /companies (28/28 cells across both pages). |
5656
| 40 | `40-registry-504-gateway-timeout.scenario.mjs` | page crashes on 504 from JSON.parse(html) throwing SyntaxError / page renders raw HTML body in org list / page treats 504 as success with empty data (silent broken state) / page immediately retries in tight loop without backoff / page hangs in loading forever (parse error throws before loading-cleanup) / "Application error" from missing top-level error boundary (collateral collapse to whole page) | `/companies` | REGISTRY GraphQL responds 504 + Content-Type: text/html + HTML body (no JSON). Asserts /companies degrades to "No organizations found" — same terminal UX as #02 (502) but DIFFERENT control flow (status-error + JSON-parse-error vs status-error + valid-error-envelope). Opens a NEW failure-mode row "gateway timeout 504" — first cell of the new row. |
57+
| 41 | `41-candles-504-gateway-timeout.scenario.mjs` | per-pool fetcher crashes on JSON.parse(html) throwing SyntaxError / per-pool fetcher renders raw HTML body in price card / per-pool fetcher treats 504 as success with empty data / per-pool fetcher hammers candles upstream with retries (no backoff, no Retry-After to respect — worse than #37) / card hangs on LoadingSpinner forever (parse error before loading-cleanup) / bulk-prefetch vs per-pool fallback race wins/loses unpredictably under partial 504 | `/companies` | REGISTRY healthy + CANDLES GraphQL responds 504 + Content-Type: text/html + HTML body (no JSON). Asserts the carousel still renders the event card but the price degrades to "0.00 SDAI" — same terminal UX as #03 (502) and #37 (429) but DIFFERENT control flow (status-error + JSON-parse-error vs status-error + valid-error-envelope vs status-error + Retry-After). Fills the 2nd cell of the new "gateway timeout 504" matrix row on /companies. |

‎auto-qa/harness/tests/smoke-scenarios-chaos-matrix.test.mjs‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -58,17 +58,16 @@ test('scenarios-chaos-matrix CLI — prints per-page chaos coverage matrix', ()
5858

5959
test('scenarios-chaos-matrix CLI — both pages have most chaos cells filled', () => {
6060
// Pins current state of the harness. With 8 failure-mode rows
61-
// × 2 endpoint columns = 16 cells per page. /companies has
62-
// all 7 prior modes + gateway-timeout registry = 15/16;
63-
// /markets/[address] has all 7 prior modes only = 14/16 (the
64-
// gateway-timeout row is opening on /companies first; market
65-
// page cells fill in subsequent slices). As new chaos
66-
// scenarios fill cells, bump the floor numerator here. The
67-
// test EXISTS so a regression that DROPS a cell (e.g.,
68-
// deleting a scenario without renaming the file) surfaces
69-
// immediately.
61+
// × 2 endpoint columns = 16 cells per page. /companies now
62+
// has all 8 modes complete (gateway-timeout row both cells
63+
// filled) = 16/16; /markets/[address] has the 7 prior modes
64+
// only = 14/16 (the gateway-timeout row's market-page cells
65+
// fill in subsequent slices). As new chaos scenarios fill
66+
// cells, bump the floor numerator here. The test EXISTS so a
67+
// regression that DROPS a cell (e.g., deleting a scenario
68+
// without renaming the file) surfaces immediately.
7069
const r = spawnSync('node', [SCRIPT], { encoding: 'utf8' });
7170
assert.equal(r.status, 0);
72-
assert.match(r.stdout, /Page: \/companies — (1[5-9]|[2-9]\d)\/16 cells filled/);
71+
assert.match(r.stdout, /Page: \/companies — (16|1[7-9]|[2-9]\d)\/16 cells filled/);
7372
assert.match(r.stdout, /Page: \/markets\/\[address\] — (1[4-9]|[2-9]\d)\/16 cells filled/);
7473
});

0 commit comments

Comments
 (0)