Commit f95f305
authored
feat(opensearchapi,v5preview,osgen): partial-failure errors, operation classifier, default router, and union decode overhaul
Consolidates the gh-816 burn-down: a partial-failure error model with
fine-grained masking, an HTTP-layer operation classifier, blocking
node discovery with default-router injection, a generator overhaul
(idiomatic naming, single-pass union decode, request-body union
constructors), and the v5preview regeneration plus integration
coverage and user docs that go with it.
## Partial-failure error model
OpenSearch returns HTTP 200 for partial successes (bulk item failures,
shard failures, single-doc replica failures), forcing callers to
double-check responses after `err == nil`. New typed errors expose
these through the standard Go `if err != nil` idiom; both `(resp, err)`
are non-nil on partial failure and the response is fully populated.
- Adds `PartialBulkError`, `PartialSearchError`, `ShardFailureError`,
and helpers `IsPartialFailure`, `ToleratePartialFailures`,
`RequireSuccessRate` for threshold-based tolerance.
- Replaces the initial `Config.ReturnQueryErrors` boolean with
`internal/errmask.ErrorMask`, a 15-bit field where each bit masks
one wrapper schema in the proposed `x-error-responses` OpenAPI
extension (`BulkItems`, `SearchShards`, `WriteShards`,
`BroadcastShards`, `NodeFailures`, `BulkByScrollFailures`,
`TaskFailures`, `MultiSearchItems`, `MultiDocItems`,
`Snapshot{Create,Get}ShardFailures`, `SimulateDocFailures`,
`RankEvalFailures`, `IngestionShardFailures`, `PitNodeFailures`).
A set bit suppresses that category; the zero value reports every
category.
- Callers configure policy via `Config.Errors = errmask.BulkItems |
errmask.SearchShards` or `OPENSEARCH_GO_ERROR_MASK` with `+/-`
tokens (e.g. `+all,-bulk_items`).
Lifecycle:
- v4 default `errmask.All` preserves existing behavior.
`Config.ReturnQueryErrors=true` is honored as a deprecated alias
for `errmask.None`.
- v5 default flips to `errmask.None` (safe by default).
- v6 removes `Config.Errors` / `OPENSEARCH_GO_ERROR_MASK`;
behavior is unconditionally `errmask.None`.
Spec side: `opensearch-openapi.yaml` is patched with 15
`_common.errors___<Wrapper>` schemas and 115 operations gain an
`x-error-responses` annotation, mirroring the upstream proposal in
`opensearch-api-specification`. The local patch goes away cleanly
once that PR lands and we re-bundle from source.
Generator side: `cmd/osgen` reads `x-error-responses` into
`ir.Operation.ErrorWrappers`; `cmd/osgen/errwrap` supplies a fallback
for plugin operations the spec does not yet annotate. The dispatch
fragment carries a data-driven `wrappers` map of `{Template, Applies}`,
so generated code stays compilable when annotations land before the
underlying response schema models the relevant field.
## OperationClassifier and bit-packed OperationID
Adds a zero-allocation HTTP method+path classifier (reusing the
existing `routeTrie`) that maps requests to structured
`OperationID` values. Enables transparent metrics, tracing, and
access-control middleware at the `http.RoundTripper` layer without
per-operation wrappers.
- `OperationID` is a bit-packed `int64` encoding R/W flag, category,
and minor operation. `IsWrite`, `Category`, `Minor` support
efficient bitwise filtering. `String()` returns Prometheus-friendly
labels.
- `OperationClassifier` is built from the canonical route table and
is safe for concurrent use. Returns `OpOther` for unrecognized
patterns.
- `OperationID` field added to `trieLeaf`/`trieMatch`, `OpID()` to
`Route`, `.Op()` to `RouteBuilder`. All 124 routes are tagged.
## Blocking DiscoverNodes and default Router injection
- `DiscoverNodes` now waits for an in-flight discovery to complete
(or for the context to be cancelled) instead of returning `nil`
immediately, letting callers block until topology data is
available after client construction.
- `DiscoverNodesOnStart` becomes `*bool`. Auto-enabled when
`OPENSEARCH_GO_ROUTER=true` and the caller did not set it.
- `v5preview/opensearchapi.NewClient` and `NewDefaultClient` inject
`opensearchtransport.NewDefaultRouter` when `config.Client.Router`
is `nil`, opting v5preview clients into role-aware dispatch,
RTT-based scoring, AIMD congestion-window, and shard-cost
weighting by default.
- `OPENSEARCH_GO_ROUTER` is a symmetric override: a falsy value
(`false`/`0`) suppresses injection so `Router` stays `nil`,
matching v4 behavior. Caller-provided Routers are preserved.
- `internal/envvars.Falsy(name)` distinguishes "unset" from
"explicitly opted out" so the v5preview rule reads as
`!envvars.Falsy(envvars.Router)` without re-implementing parse
logic.
## Generator: idiomatic naming, request-body unions, single-pass decode
Naming rewrites at PascalCase boundaries:
| Spec form | Idiomatic Go |
|---------------|-----------------|
| `Msearch` | `MSearch` |
| `Mget` | `MGet` |
| `Mtermvectors`| `MTermVectors` |
| `Termvectors` | `TermVectors` |
| `Forcemerge` | `ForceMerge` |
| `Response` | `Resp` |
The `Response -> Resp` rule additionally requires the trailing
character to be uppercase, so standalone `SearchResponse` (a spec
wrapper) does not collide with the operation-level `<Op>Resp` name.
Hand-written types renamed for v4/v5 consistency:
`BulkResponseItem -> BulkRespItem`, `ErrorResponseBase ->
ErrorRespBase`, `MsearchErrors -> MSearchErrors`,
`MsearchTemplateErrors -> MSearchTemplateErrors`.
Request-body unions:
- `splitUnionsFromSiblings` partitions request-body subtree unions
(e.g. `ReindexSourceSort`) so they are routed to `UnionFragment`
instead of being emitted as empty structs.
- Plumbs `Op + Registry` into `UnionFragment` so plugin-package
unions qualify cross-package branches as
`opensearchapi.FieldSort`.
- Each generated discriminated union gains
`New<Union>From<Branch>(v <branch type>) <Union>` constructors
per branch and a `SetRaw(json.RawMessage)` typed escape hatch.
`SetRaw` clears the typed branch so `MarshalJSON` returns the
raw bytes verbatim.
Single-pass union decode:
- Case A (merged): object unions with one permissive primary branch
plus discriminated branch(es) (mget, msearch, indices-open). The
primary is embedded and the common case decodes in a single
`json.Unmarshal`; each discriminated branch is detected by the
presence of its distinguishing key and decoded only when matched.
Drops the `build.HasJSONKeys` map probe and per-item raw copy.
- Case B (lazy `As<T>()`): aggregation/suggest result unions carry
no wire discriminator, so they cannot be auto-selected.
`UnmarshalJSON` only retains the raw bytes; generated
`As<ConcreteType>()` accessors decode on demand.
- Unions fitting neither (reindex bodies, plugin-defined task
status) keep the existing try-each decoder; the classifier logs
once per union name when it declines.
- All union `UnmarshalJSON` aliases the owned response buffer
(`u.raw = data`) rather than copying it; `RawJSON()` documents
the borrowed-buffer contract.
Measured impact on mget (1000 docs): decode allocations down ~2.7x
(~43k -> ~16k allocs/op), time down ~1.7x (4.2ms -> 2.5ms).
Bulk wrapper template now resolves its walked element type from the
IR (via `bulkInnerItemType`) instead of hardcoding `BulkRespItem`,
so future spec or naming changes propagate automatically.
## v5preview regeneration
Mechanical output of `cmd/osgen` against the patched spec:
- `clients_gen.go` declares `errors errmask.ErrorMask` on `Client`
and `clientInit` takes the mask as a second arg.
- 19 operation dispatch files emit per-wrapper post-`do()` blocks
that return typed errors when the corresponding `errmask` bit is
unset and the wire data carries a partial failure
(`BulkItems`, `SearchShards`, `WriteShards`, `MultiSearchItems`).
- Reserved-but-unemitted wrappers (`BroadcastShards`, `NodeFailures`,
`BulkByScrollFailures`, `TaskFailures`, `MultiDocItems`,
`Snapshot{Create,Get}ShardFailures`, `SimulateDocFailures`,
`RankEvalFailures`, `IngestionShardFailures`, `PitNodeFailures`)
carry annotations but no emission until detection logic lands.
- Operations whose typed response shape lacks the field path a
wrapper needs (`CreateResp` lacks `_shards`; msearch union items
lack a top-level `Shards`) are silently skipped by the
`Applies` guard; emission starts automatically once the response
types catch up.
- Idiomatic-naming pass touches every generated type containing
`Msearch`, `Mget`, `Mtermvectors`, `Termvectors`, `Forcemerge`,
or compound `*Response*` substrings.
- Single-pass union decode applied to mget/msearch/indices-open
success|error items and `As<T>()` accessors generated for
aggregation and suggest result unions.
## Integration coverage and CI hardening
v5preview integration tests drive real requests and assert decoded
shape per server version:
- aggregation: lazy `As<T>()` accessors (terms, date_histogram,
stats, avg, sum, min, max, value_count, cardinality).
- mget: merged success|error decode (`GetResult` found/not-found
vs `MGetMultiGetError`).
- msearch: merged success|error decode
(`MSearchMultiSearchItem` vs `ErrorRespBase`), the
first-byte-switch `SearchHitsMetadataTotal` union, and the
`MultiSearchItemError` partial-failure surface.
CI matrix changes:
- Skip multi-node clusters on OpenSearch <=2.17.x to avoid the
node-join/node-left coordinator race
(`opensearch-project/OpenSearch#15521`, backported via #16118
on the 2.x line). Affected versions run with
`OPENSEARCH_NODE_COUNT=1`.
- `.github/workflows/test-compatibility.yml` adds a per-entry
`node_count` matrix field: `1` for 1.3.20 through 2.17.1, `3`
for 2.18.0 and later. Comment cites the upstream PRs so the
cutoff is greppable.
- `Makefile cluster.docker-up` respects `OPENSEARCH_NODE_COUNT`
and passes `--scale opensearch-nodeN=0` to docker compose.
Defaults to 3 for local development.
- Widen the existing 2.1.0-only shard-routing test skip to all
versions below 2.2.0 (security plugin
`java.io.OptionalDataException` from non-thread-safe `User`
serialization, fixed by
`opensearch-project/security#1970`).
- Generated-code emit/path test fixtures converted from raw
`"GET"`/`"POST"` to `net/http http.Method*` constants for
consistency with the rest of the suite.
- `testutil.PollUpdate()` decouples jitter calculation from
runtime backoff execution to fix a flake.
## Dependency bumps
- `github.com/aws/aws-sdk-go-v2/config` 1.32.18 -> 1.32.20
- `github.com/aws/aws-sdk-go-v2/credentials` 1.19.17 -> 1.19.19
- `github.com/getkin/kin-openapi` v0.139.0 -> v0.140.0
Fixes: #852, #850
## Documentation
- `v5preview/opensearchapi/README.md`: Partial Failure Errors
(Config.Errors, errmask, `OPENSEARCH_GO_ERROR_MASK`, typed errors,
`opensearchapi.Errors` helper, per-Resp helpers, helper functions,
operation constants) and Default Router Injection (truth table
for `OPENSEARCH_GO_ROUTER`, opt-out semantics).
- `v5preview/opensearchapi/MIGRATING.md`: v4 -> v5preview surface
delta (import path, `Indices -> Index` on multi-index Req types,
`DocumentID -> ID` on `IndexReq`, optional `Params` becomes
`*Params`, optional `bool` query params become `*bool`,
partial-failure type renames, errmask default flip, default
Router injection).
- `guides/error_handling.md`: Bulk/Search/Write partial-failure
examples as paired v4/v5preview blocks; per-Resp helper
subsection (`BulkItemFailures`, `SearchShardFailures`,
`WriteShardFailures`, `MultiSearchItemFailures`,
`PartialFailures(mask)`); error type reference covering v4 vs
v5preview internal-field-type divergence.
- `guides/bulk.md`: v4/v5preview field-name and `BulkResp.Items`
shape divergences with paired error-iteration examples for v4's
`[]map[string]BulkRespItem` and v5preview's `[]BulkItem`.
- `UPGRADING.md`: keeps version-history essentials for the >=5.0
partial-failure model and v5preview Router injection;
forward-links to the new package docs.
Ref: #816
Ref: opensearch-project/opensearch-api-specification/pull/11371 parent f591483 commit f95f305
328 files changed
Lines changed: 41870 additions & 8888 deletions
File tree
- .ci/opensearch
- .github/workflows
- cmd/osgen
- emit
- errwrap
- ir
- errmask
- guides
- internal
- build
- envvars
- opensearchapi
- testutil
- opensearchtransport
- testutil
- opensearchutil
- v5preview/opensearchapi
- plugins
- asynchronous_search
- flow_framework
- ism
- knn
- ltr
- ml
- notifications
- observability
- query
- rollups
- search_relevance
- security
- sm
- sql
- wlm
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
20 | | - | |
| 19 | + | |
| 20 | + | |
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
7 | 6 | | |
8 | 7 | | |
9 | 8 | | |
| |||
12 | 11 | | |
13 | 12 | | |
14 | 13 | | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
15 | 21 | | |
16 | 22 | | |
17 | 23 | | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
47 | 53 | | |
48 | | - | |
| 54 | + | |
49 | 55 | | |
50 | | - | |
| 56 | + | |
51 | 57 | | |
52 | 58 | | |
53 | 59 | | |
| |||
63 | 69 | | |
64 | 70 | | |
65 | 71 | | |
| 72 | + | |
66 | 73 | | |
67 | 74 | | |
68 | 75 | | |
| |||
108 | 115 | | |
109 | 116 | | |
110 | 117 | | |
| 118 | + | |
111 | 119 | | |
112 | 120 | | |
113 | 121 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
| 11 | + | |
10 | 12 | | |
11 | 13 | | |
12 | 14 | | |
| |||
89 | 91 | | |
90 | 92 | | |
91 | 93 | | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
92 | 115 | | |
93 | 116 | | |
94 | 117 | | |
| |||
198 | 221 | | |
199 | 222 | | |
200 | 223 | | |
| 224 | + | |
201 | 225 | | |
202 | 226 | | |
203 | 227 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
385 | 385 | | |
386 | 386 | | |
387 | 387 | | |
388 | | - | |
| 388 | + | |
389 | 389 | | |
390 | 390 | | |
391 | 391 | | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
392 | 409 | | |
393 | 410 | | |
394 | 411 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
501 | 501 | | |
502 | 502 | | |
503 | 503 | | |
504 | | - | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
505 | 514 | | |
506 | 515 | | |
507 | | - | |
| 516 | + | |
508 | 517 | | |
509 | 518 | | |
510 | 519 | | |
511 | 520 | | |
512 | 521 | | |
513 | | - | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
514 | 533 | | |
515 | 534 | | |
516 | 535 | | |
| |||
0 commit comments