Skip to content

Commit 0b7ff37

Browse files
iliia-veseloviliia-veselov
authored
feat: Accept all as an alias for * on the structure endpoint (#76)
1 parent c1e1ca7 commit 0b7ff37

7 files changed

Lines changed: 557 additions & 0 deletions

File tree

‎docs/designs/024-structure-all-keyword-alias/DESIGN.md‎

Lines changed: 357 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,357 @@
1+
# Design 024: `all` keyword alias for `*` on the structure endpoint
2+
3+
**Status:** draft (2026-05-21). Awaiting review.
4+
5+
## Context
6+
7+
The proxy exposes an SDMX 3.0 REST API. In SDMX 3.0 the wildcard
8+
placeholder in the path is `*` (e.g. `/structure/dataflow/*/*/*`); in
9+
SDMX 2.1 the same role was played by `all`. The proxy already handles
10+
the *outbound* direction -- when routing to an SDMX 2.1 registry,
11+
`QueryTranslatorImpl.getVersionSpecificQueryId` rewrites `*` to `all`
12+
before sending. The *inbound* direction is unhandled: a client request
13+
with `all` in any slot is treated as a literal artefact name.
14+
15+
Clients in practice send `all` because:
16+
17+
- They copy-paste from SDMX 2.1 documentation, of which there is far
18+
more than 3.0 documentation.
19+
- They migrate from a 2.1 client library where `all` was the documented
20+
wildcard form.
21+
- They are tooling-generated and the tool defaults to 2.1 syntax.
22+
23+
In every case the proxy currently silently returns the wrong thing --
24+
an empty / 404-style result for the literal "all" artefact.
25+
26+
## Problem
27+
28+
`/structure/{type}/all/all/all` and similar mixed forms (e.g.
29+
`/structure/dataflow/BIS/MY_FLOW/all`) currently route as specific-
30+
artefact queries against an artefact literally named "all". The
31+
existing wildcard / fan-out paths only trigger on the exact string
32+
`*`.
33+
34+
In particular:
35+
36+
- The 501 rejection at `QueryTranslatorImpl.translateStructureQuery`
37+
only checks `"*".equals(agencyId)`, so a request with `agencyId="all"`
38+
bypasses it and goes through standard single-registry routing.
39+
- The fan-out branch in `SdmxStructure30Controller.getResources` only
40+
checks `"*".equals(agencyId)`, so `all` never engages fan-out even
41+
when the toggle is on.
42+
43+
## Non-goals
44+
45+
- **Data and availability endpoints.** SDMX 3.0 data/availability use
46+
the `c[]` filter model and position-based keys; `all` has no role
47+
there. Keep scope to the structure endpoint to match SDMX 3.0
48+
semantics on the surfaces that need it.
49+
- **Case-insensitive matching.** The SDMX 2.1 spec uses the exact
50+
lowercase form `all`. We match that. `ALL`, `All`, etc. continue to
51+
be treated as literal artefact names.
52+
- **A separate config flag.** This is a normalization, not a
53+
feature -- accepting `all` is always-on and does not need a toggle.
54+
The fan-out toggle (design 023) still gates the wildcard-agency
55+
behaviour after normalization.
56+
- **Reverse rewrite in responses.** Responses keep whatever IDs the
57+
upstream registry returned. We do not rewrite `*` back to `all` on
58+
the way out.
59+
60+
## Current Architecture
61+
62+
Request flow today, ignoring fan-out:
63+
64+
```
65+
SdmxStructure30Controller.getResources(agencyId, resourceId, version, ...)
66+
[fan-out branch: "*".equals(agencyId) && structureFanOutEnabled]
67+
-> queryTranslator.translateWildcardStructureFanOut(...)
68+
[else]
69+
-> queryTranslator.translateStructureQuery(...)
70+
[throws 501 iff "*".equals(agencyId) || agencyId.contains(",")]
71+
-> selectRegistryAndVersion -> single TranslatedStructureQuery
72+
-> getVersionSpecificQueryId: "*" -> "all" for SDMX 2.1 registries
73+
-> adapterRouter.getStructures(query)
74+
```
75+
76+
The `"*"` literal appears at three branch points:
77+
78+
1. `SdmxStructure30Controller.getResources` -- fan-out gate.
79+
2. `QueryTranslatorImpl.translateStructureQuery` -- 501 gate.
80+
3. `QueryTranslatorImpl.getVersionSpecificQueryId` -- outbound rewrite
81+
for SDMX 2.1 registries.
82+
83+
## Solution
84+
85+
The wildcard vocabulary (`*` vs `all`) is SDMX wire-syntax knowledge;
86+
that belongs to the translator. The rule lives in `QueryTranslator`,
87+
and no other component ever sees the literal string `all`.
88+
89+
1. **`QueryTranslator` gains one helper method**: `String normalizePathSlot(String)`.
90+
It returns `*` if the input is `all`, otherwise the input unchanged.
91+
2. **Inside the two translator entry methods** (`translateStructureQuery`
92+
and `translateWildcardStructureFanOut`), normalise each path-slot
93+
parameter at the top. Defensive; the call is idempotent so it costs
94+
nothing when the caller already normalised.
95+
3. **The structure controller normalises its three path locals once at
96+
the top of `getResources`** via `queryTranslator.normalizePathSlot(...)`,
97+
then uses those normalised locals for everything downstream -- the
98+
fan-out branch check, the cache-key computation, and the translator
99+
call. Because the locals are already `*` by the time the cache key
100+
is built, `/structure/.../*/*/*` and `/structure/.../all/all/all`
101+
share one fan-out cache entry.
102+
103+
The controller learns *that* `all` exists only to the extent that it
104+
calls one translator method named `normalizePathSlot`. It never
105+
inspects the literal string `all`; the existing `"*".equals(agencyId)`
106+
fan-out check stays as-is because the local has already been rewritten.
107+
108+
### Mixed forms work for free
109+
110+
A request like `/structure/dataflow/BIS/MY_FLOW/all` enters the
111+
controller, `normalizePathSlot` rewrites the `version` local from
112+
`all` to `*`, the controller takes the single-registry branch
113+
(`"*".equals("BIS")` is false), and `translateStructureQuery` sees
114+
`agencyId="BIS", resourceId="MY_FLOW", version="*"`. Routing picks
115+
the BIS registry; the existing `getVersionSpecificQueryId` rewrites
116+
`*` back to `all` on the outbound leg when BIS happens to be SDMX
117+
2.1. Net round-trip: `all -> * -> all` for that registry, matching
118+
what the client sent.
119+
120+
## Implementation Plan
121+
122+
### Step 1: Add `normalizePathSlot` to the translator interface
123+
124+
**File:** `sdmx-proxy/src/main/java/com/epam/sdmxproxy/services/translator/QueryTranslator.java`
125+
126+
```java
127+
/**
128+
* Normalises a structure-endpoint path slot (one of {@code agencyId},
129+
* {@code resourceId}, {@code version}). The SDMX 2.1 wildcard
130+
* keyword {@code all} is rewritten to the SDMX 3.0 form {@code *};
131+
* every other value is returned unchanged.
132+
* <p>
133+
* Exact lowercase {@code all} only -- {@code ALL}, {@code All}, etc.
134+
* are treated as literal artefact IDs (see design 024).
135+
*
136+
* @param slot path-slot value as it arrived from the client
137+
* @return canonical form ({@code *} for wildcard, otherwise the input)
138+
*/
139+
String normalizePathSlot(String slot);
140+
```
141+
142+
**File:** `sdmx-proxy/src/main/java/com/epam/sdmxproxy/services/translator/QueryTranslatorImpl.java`
143+
144+
```java
145+
@Override
146+
public String normalizePathSlot(String slot) {
147+
return SDMX_21_ALL_WILDCARD.equals(slot) ? SDMX_30_ALL_WILDCARD : slot;
148+
}
149+
```
150+
151+
The constants `SDMX_21_ALL_WILDCARD = "all"` and
152+
`SDMX_30_ALL_WILDCARD = "*"` already exist on the class; no new
153+
literals.
154+
155+
### Step 2: Defensive normalisation inside translator entry methods
156+
157+
**File:** `sdmx-proxy/src/main/java/com/epam/sdmxproxy/services/translator/QueryTranslatorImpl.java`
158+
159+
Apply `normalizePathSlot` at the top of every translator method that
160+
accepts path slots from external callers. The call is idempotent so
161+
it costs nothing when the caller already normalised; this is a
162+
belt-and-suspenders guarantee that no downstream code ever sees
163+
`all`.
164+
165+
```java
166+
@Override
167+
public TranslatedStructureQuery translateStructureQuery(
168+
String structureType,
169+
String agencyId,
170+
String resourceId,
171+
String version,
172+
String references,
173+
String detail,
174+
String acceptHeader,
175+
String sourceArtefactUrn
176+
) {
177+
agencyId = normalizePathSlot(agencyId);
178+
resourceId = normalizePathSlot(resourceId);
179+
version = normalizePathSlot(version);
180+
181+
if ("*".equals(agencyId) || (agencyId != null && agencyId.contains(","))) {
182+
throw new UnsupportedAgencyWildcardException(/* ... */);
183+
}
184+
// ... rest unchanged
185+
}
186+
187+
@Override
188+
public List<TranslatedStructureQuery> translateWildcardStructureFanOut(
189+
String structureType,
190+
String resourceId,
191+
String version,
192+
String references,
193+
String detail,
194+
String acceptHeader
195+
) {
196+
resourceId = normalizePathSlot(resourceId);
197+
version = normalizePathSlot(version);
198+
// ... rest unchanged
199+
}
200+
```
201+
202+
`translateStructureQueryForAgencySchemaDiscovery` does not take
203+
client-supplied path slots (it hard-codes `*`), so no change there.
204+
Data and availability translator methods are out of scope per
205+
Non-goals.
206+
207+
### Step 3: Controller normalises locals up front
208+
209+
**File:** `sdmx-proxy/src/main/java/com/epam/sdmxproxy/controller/SdmxStructure30Controller.java`
210+
211+
At the top of `getResources`, normalise the three path locals via the
212+
translator helper. Everything that follows -- fan-out branch check,
213+
cache-key computation, translator call -- uses the canonical values.
214+
215+
```java
216+
@Override
217+
public ResponseEntity<StreamingResponseBody> getResources(
218+
@PathVariable("structureType") String structureType,
219+
@PathVariable("agencyId") String agencyId,
220+
@PathVariable("resourceId") String resourceId,
221+
@PathVariable("version") String version,
222+
// ...
223+
) {
224+
agencyId = queryTranslator.normalizePathSlot(agencyId);
225+
resourceId = queryTranslator.normalizePathSlot(resourceId);
226+
version = queryTranslator.normalizePathSlot(version);
227+
228+
logRequestUrl(accept, STRUCTURE);
229+
// ... existing logic unchanged; the locals are now guaranteed canonical
230+
}
231+
```
232+
233+
**Key points:**
234+
235+
- The existing fan-out branch `"*".equals(agencyId) && config.isStructureFanOutEnabled()`
236+
stays as-is -- by the time it runs, `agencyId` is either `*` or a
237+
literal agency name, never `all`.
238+
- `fanOutResponse` builds the cache key from the now-canonical
239+
`resourceId` and `version` locals, so `/structure/.../*/*/*` and
240+
`/structure/.../all/all/all` produce the **same** cache key.
241+
- `logRequestUrl` reads from `HttpServletRequest.getRequestURL()`, not
242+
from our locals, so the log line preserves the client's original
243+
form for debugging.
244+
245+
### Step 4: No other changes
246+
247+
The 501 gate (`"*".equals(agencyId) || agencyId.contains(",")`),
248+
`getVersionSpecificQueryId`, and `CacheKeyGenerator.generateFanOutResponseKey`
249+
all see canonical inputs and continue to work without modification.
250+
251+
## No Changes Required
252+
253+
These files need **no modification**:
254+
255+
- **`AgencyRoutingService`** -- only invoked for non-wildcard agency
256+
IDs.
257+
- **`CacheKeyGenerator`** -- key components are passed through as
258+
strings; the normalisation already happened upstream.
259+
- **`getVersionSpecificQueryId`** -- already converts `*` to `all` on
260+
the outbound leg for SDMX 2.1 registries. Untouched.
261+
- **Data / availability controllers** -- out of scope (see Non-goals).
262+
263+
## Files Affected
264+
265+
| File | Change Type | Description |
266+
|------|-------------|-------------|
267+
| `sdmx-proxy/src/main/java/com/epam/sdmxproxy/services/translator/QueryTranslator.java` | Modified | Add `normalizePathSlot(String)` |
268+
| `sdmx-proxy/src/main/java/com/epam/sdmxproxy/services/translator/QueryTranslatorImpl.java` | Modified | Implement `normalizePathSlot`; normalise at entry of `translateStructureQuery` and `translateWildcardStructureFanOut` |
269+
| `sdmx-proxy/src/main/java/com/epam/sdmxproxy/controller/SdmxStructure30Controller.java` | Modified | Normalise the three path locals via `queryTranslator.normalizePathSlot(...)` at the top of `getResources` |
270+
| `sdmx-proxy/src/test/java/com/epam/sdmxproxy/services/translator/QueryTranslatorImplTest.java` | Modified | Tests for `normalizePathSlot` and for the defensive normalisation in each translator entry method |
271+
| `sdmx-proxy/src/test/java/com/epam/sdmxproxy/controller/SdmxStructure30ControllerTest.java` | Modified | Tests that `agencyId="all"` engages the fan-out branch when the toggle is on, and that `/all/all/all` and `/*/*/*` produce the same cache key |
272+
| `sdmx-proxy-e2e/src/test/java/com/epam/sdmxproxy/e2e/tests/StructureFanOutE2ETest.java` | Modified | One parameterised case using `all` instead of `*` to confirm the alias works end-to-end |
273+
274+
## Edge Cases
275+
276+
1. **`all` in only one slot.** `/structure/dataflow/BIS/MY_FLOW/all`
277+
normalises to `BIS / MY_FLOW / *`. Routes to the BIS registry, asks
278+
for "any version of MY_FLOW", and the outbound rewrite re-emits
279+
`all` to BIS if BIS is 2.1.
280+
2. **`all` agency with fan-out off.** Normalises to `*`; existing 501
281+
gate fires. Same response as `/structure/.../*/*/*` with toggle
282+
off.
283+
3. **`all` agency with fan-out on.** Normalises to `*`; existing
284+
fan-out branch engages.
285+
4. **Case variants (`ALL`, `All`).** Not normalised. Treated as
286+
literal artefact names. Documented in Non-goals; can be revisited
287+
if real clients send other casings.
288+
5. **Mixed `all,BIS` comma-separated lists.** Comma-separated still
289+
rejected by the existing 501 gate after normalisation (the
290+
normaliser only matches the exact string `all`, not any token
291+
inside a comma list).
292+
6. **Cache aliasing.** `/structure/dataflow/*/*/*` and
293+
`/structure/dataflow/all/all/all` share **one** fan-out cache
294+
entry: the controller's `normalizePathSlot` calls run before the
295+
key is computed, so both forms produce identical key inputs. Same
296+
story for the per-registry parsed-structures cache (it is keyed
297+
off the post-translation `TranslatedStructureQuery`, where the
298+
defensive normalisation has already happened).
299+
300+
## Verification
301+
302+
### Unit Tests
303+
304+
**File:** `sdmx-proxy/src/test/java/com/epam/sdmxproxy/services/translator/QueryTranslatorImplTest.java`
305+
306+
| Test Method | Description |
307+
|-------------|-------------|
308+
| `normalizePathSlot_rewritesAllToStar()` | Returns `*` for input `all`. |
309+
| `normalizePathSlot_passesThroughStar()` | Returns `*` for input `*`. |
310+
| `normalizePathSlot_passesThroughLiteral()` | Returns `BIS` for input `BIS`. |
311+
| `normalizePathSlot_caseVariantsTreatedAsLiterals()` | Returns `ALL` / `All` unchanged. |
312+
| `normalizePathSlot_passesThroughCommaList()` | Returns `all,IMF` unchanged (substring match is not done; the 501 gate handles comma lists). |
313+
| `translateStructureQuery_allInAgencySlot_throws501LikeWildcard()` | Calling with `agencyId="all"` hits the same 501 gate as `agencyId="*"` (the translator normalises before checking). |
314+
| `translateStructureQuery_allInResourceOrVersion_carriesWildcardInStructure()` | `agencyId="BIS", resourceId="all", version="all"` -- the resulting `TranslatedStructureQuery.structure` carries `*` in those slots. |
315+
| `translateWildcardStructureFanOut_allInResourceAndVersionNormalised()` | Calling fan-out with `resourceId="all", version="all"` produces per-leg queries with `*` in both slots. |
316+
317+
**File:** `sdmx-proxy/src/test/java/com/epam/sdmxproxy/controller/SdmxStructure30ControllerTest.java`
318+
319+
| Test Method | Description |
320+
|-------------|-------------|
321+
| `allAgency_toggleOn_engagesFanOut()` | Wire the translator mock so `normalizePathSlot("all")` returns `*`; assert the controller invokes `translateWildcardStructureFanOut` (not single-agency translate). |
322+
| `allAgency_toggleOff_routesToSingleAgencyTranslatorWhichThrows501()` | `agencyId="all"` with toggle off behaves like `agencyId="*"` with toggle off. |
323+
| `allInResourceAndVersion_toggleOn_cacheKeyMatchesStarForm()` | Capture the cache key passed to `cacheService.getReadyResponse` for `/all/all/all` and `/*/*/*`; assert they are equal. |
324+
325+
### Manual Verification
326+
327+
With fan-out enabled:
328+
329+
```bash
330+
# Canonical 3.0 form
331+
curl -s -H "Accept: application/vnd.sdmx.structure+json;version=2.0.0" \
332+
http://localhost:8050/statgpt/sdmx-proxy/api/v0/sdmx/3.0/structure/dataflow/*/*/*
333+
334+
# 2.1 alias form -- expect identical response
335+
curl -s -H "Accept: application/vnd.sdmx.structure+json;version=2.0.0" \
336+
http://localhost:8050/statgpt/sdmx-proxy/api/v0/sdmx/3.0/structure/dataflow/all/all/all
337+
338+
# Mixed form -- single registry, "any version of MY_FLOW"
339+
curl -s -H "Accept: application/vnd.sdmx.structure+json;version=2.0.0" \
340+
http://localhost:8050/statgpt/sdmx-proxy/api/v0/sdmx/3.0/structure/dataflow/BIS/MY_FLOW/all
341+
```
342+
343+
### E2E Tests
344+
345+
Extend `StructureFanOutE2ETest` (toggle-on suite from design 023) with
346+
one parameterised case that uses `all/all/all` instead of `*/*/*`,
347+
asserting the same HTTP 200 + non-empty body contract. One alias case
348+
is sufficient -- the unit tests above cover the slot matrix; the E2E
349+
case only needs to prove the normaliser runs in the deployed jar.
350+
351+
## SDMX Standard References
352+
353+
- SDMX 2.1 REST: `all` is the documented wildcard for `agencyId` /
354+
`resourceId` / `version` in path queries.
355+
- SDMX 3.0 REST: `*` replaces `all`; the spec does not require
356+
servers to keep accepting `all`, but doing so is a low-cost
357+
compatibility courtesy.

‎sdmx-proxy-e2e/src/test/java/com/epam/sdmxproxy/e2e/tests/StructureFanOutE2ETest.java‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
import org.junit.jupiter.api.BeforeAll;
1111
import org.junit.jupiter.api.DisplayName;
1212
import org.junit.jupiter.api.Tag;
13+
import org.junit.jupiter.api.Test;
1314
import org.junit.jupiter.api.TestInstance;
1415
import org.junit.jupiter.params.ParameterizedTest;
1516
import org.junit.jupiter.params.provider.Arguments;
@@ -95,4 +96,20 @@ void wildcardAgencyReturnsMergedResponse(String structureType, String acceptHead
9596
.as("Fan-out response body must not be empty (type=%s)", structureType)
9697
.isGreaterThan(0);
9798
}
99+
100+
@Test
101+
@DisplayName("`all` keyword in agency/resource/version path slots engages fan-out (design 024)")
102+
void allKeywordAliasEngagesFanOut() {
103+
String path = BASE_PATH + "/sdmx/3.0/structure/dataflow/all/all/all?detail=full";
104+
String acceptHeader = "application/vnd.sdmx.structure+json;version=2.0.0";
105+
106+
Response response = restClient.getResponseWithAccept(path, acceptHeader);
107+
108+
assertThat(response.getStatusCode())
109+
.as("`all/all/all` must engage fan-out and return HTTP 200, identically to `*/*/*`")
110+
.isEqualTo(200);
111+
assertThat(response.getBody().asByteArray().length)
112+
.as("Fan-out response body via the `all` alias must not be empty")
113+
.isGreaterThan(0);
114+
}
98115
}

‎sdmx-proxy/src/main/java/com/epam/sdmxproxy/controller/SdmxStructure30Controller.java‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,10 @@ public ResponseEntity<StreamingResponseBody> getResources(
5858
accept = ACCEPT_HEADER_FALLBACK;
5959
}
6060

61+
agencyId = queryTranslator.normalizePathSlot(agencyId);
62+
resourceId = queryTranslator.normalizePathSlot(resourceId);
63+
version = queryTranslator.normalizePathSlot(version);
64+
6165
ProxyConfiguration config = configurationProvider.getConfiguration();
6266
if (WILDCARD_AGENCY.equals(agencyId) && config.isStructureFanOutEnabled()) {
6367
return fanOutResponse(structureType, resourceId, version, references, detail, accept, config);

0 commit comments

Comments
 (0)