Skip to content

Commit 03922a6

Browse files
sean-sean-
committed
Add x-error-responses partial-failure error mask
OpenSearch returns HTTP 200 for partial successes -- bulk item failures, search-shard failures, single-doc replica failures -- so callers must remember a second check after `err == nil`. v4 added an opt-in boolean (Config.ReturnQueryErrors) that converted ALL partial-failure shapes into typed Go errors. That single switch is too coarse: callers who want shard-level errors but tolerate bulk item failures (or vice versa) have no way to express it. Replace the boolean with internal/errmask.ErrorMask, a 15-bit field where each bit corresponds to one wrapper schema in the proposed x-error-responses OpenAPI extension (BulkItems, SearchShards, WriteShards, BroadcastShards, NodeFailures, BulkByScrollFailures, TaskFailures, MultiSearchItems, MultiDocItems, Snapshot{Create,Get}- ShardFailures, SimulateDocFailures, RankEvalFailures, IngestionShardFailures, PitNodeFailures). A set bit MASKS that category; the zero value reports every category. Callers express fine-grained policy in code (Config.Errors = errmask.BulkItems | errmask.SearchShards) or via OPENSEARCH_GO_ERROR_MASK using comma-separated +/- tokens (e.g. "+all,-bulk_items"). Lifecycle (matches OPENSEARCH_GO_ROUTER): v4 (this commit): default `errmask.All` -- preserves pre-bitfield behavior (no partial-failure errors). Config.ReturnQueryErrors=true is honored as a deprecated alias for `errmask.None`. v5: default flips to `errmask.None` (safe by default). v6: Config.Errors / OPENSEARCH_GO_ERROR_MASK removed; behavior is unconditionally `errmask.None`. The hand-written v4 opensearchapi/api_*.go call sites now read c.errors.Has(errmask.<Wrapper>) for each operation's wrapper category. A new hand-written v5preview/opensearchapi/errors.go ports the same typed-error surface (PartialBulkError, PartialSearchError, ShardFailureError, plus the IsPartialFailure / ToleratePartial- Failures / RequireSuccessRate helpers) using v5preview's BulkResponse- Item and ShardSearchFailure types. v5preview Config.Errors and the clientInit(rootClient, mask) signature are wired through both hand-written api.go and the generated clients_gen.go. Spec side: opensearch-openapi.yaml is patched with 15 `_common.errors___<Wrapper>` schemas under components.schemas and 115 operation entries get an x-error-responses annotation. This mirrors the upstream proposal in opensearch-api-specification (see issue-x-partial-failure-mode.md). Once that PR lands and we re-bundle from source, the local patch goes away cleanly. Generator side: cmd/osgen reads x-error-responses from the spec extension into ir.Operation.ErrorWrappers; cmd/osgen/errwrap supplies a hardcoded fallback for plugin operations the spec doesn't yet annotate. The dispatch fragment carries a data-driven `wrappers` map of {Template, Applies}: each wrapper has both an emission template and an Applies predicate that walks the response struct (including embeds via the type registry) to confirm the field path the template references actually exists. This keeps generated code compilable when spec annotations land before the underlying response schema models the relevant field -- v5preview's CreateResp and msearch's union response item are skipped today and will start emitting once those types acquire the missing fields. Ref: opensearch-project#816 Ref: opensearch-project/opensearch-api-specification/pull/1137 Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>
1 parent 0ab5947 commit 03922a6

49 files changed

Lines changed: 5924 additions & 380 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -89,16 +89,21 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
8989
- Document environment variables in `guides/routing.md`
9090
- Document read-after-write visibility guarantees with operation-aware routing in `guides/routing.md`
9191
- Add adaptive `max_concurrent_shard_requests` derived from cluster-wide AIMD congestion window ([#800](https://github.com/opensearch-project/opensearch-go/issues/800))
92-
- Add partial failure error types (`PartialBulkError`, `PartialSearchError`, `ShardFailureError`) that surface HTTP 200 partial failures as Go errors when `Config.ReturnQueryErrors` is enabled ([#816](https://github.com/opensearch-project/opensearch-go/issues/816))
92+
- Add partial failure error types (`PartialBulkError`, `PartialSearchError`, `ShardFailureError`, `MultiSearchItemError`) that surface HTTP 200 partial failures as typed Go errors, controlled by a per-category `errmask.ErrorMask` bitfield on `Config.Errors` ([#816](https://github.com/opensearch-project/opensearch-go/issues/816))
9393
- `PartialBulkError` returned from `Bulk` when `resp.Errors` is true, carries `FailedItems` and `SucceededCount`
9494
- `PartialSearchError` returned from `Search`, `MSearch`, `MSearchTemplate`, `SearchTemplate`, `Scroll.Get` when `_shards.failed > 0`
9595
- `ShardFailureError` returned from `Index`, `Document.Create`, `Document.Delete`, `Update` when replica shards fail
96-
- `PartialFailureError` marker interface with `IsPartial() bool` for type-switching across all partial failure types
96+
- `MultiSearchItemError` returned from `MSearch`/`MSearchTemplate` for per-sub-response Error envelopes
97+
- `MsearchErrors` / `MsearchTemplateErrors` per-op containers (Go 1.20+ multi-error contract via `Unwrap() []error`) when 2+ wrapper categories fire on the same response
98+
- `PartialFailureError` marker interface with `IsPartial() bool` for type-switching across all partial-failure types
99+
- Per-Resp helper methods (`BulkItemFailures`, `SearchShardFailures`, `WriteShardFailures`, `MultiSearchItemFailures`) plus `PartialFailures(mask)` aggregator for focused inspection at the call site
100+
- `opensearchapi.Errors(err) []error` package-level helper that flattens single- and multi-wrapper errors into a uniform slice for `switch` dispatch
97101
- Helper functions: `IsPartialFailure`, `ToleratePartialFailures`, `RequireSuccessRate` for threshold-based error tolerance
98102
- Operation constants: `OperationIndex`, `OperationCreate`, `OperationUpdate`, `OperationDelete`
99-
- `Config.ReturnQueryErrors` defaults to `false` in v4 (opt-in), will flip to `true` in v5
100-
- `OPENSEARCH_GO_PARTIAL_QUERY_ERRORS` environment variable overrides `Config.ReturnQueryErrors` at runtime
103+
- `Config.Errors *errmask.ErrorMask` replaces a single boolean: each bit suppresses one wrapper category. v4 defaults to `errmask.All` (mask everything, preserves pre-bitfield behavior); v5+ defaults to `errmask.Empty` (report everything)
104+
- `OPENSEARCH_GO_ERROR_MASK` environment variable overrides `Config.Errors` at runtime via comma-separated `+`/`-` tokens (lowercase snake_case wrapper names; unknown tokens silently dropped, debug-logged)
101105
- Both `(resp, error)` are non-nil on partial failure -- response is fully populated
106+
- `v5preview/opensearchapi` ports the same model with spec-driven types (regenerated from the OpenAPI `x-error-responses` extension on every `cmd/osgen` run)
102107
- Add `OperationClassifier` for zero-allocation HTTP method+path to `OperationID` mapping ([#816](https://github.com/opensearch-project/opensearch-go/issues/816))
103108
- Bit-packed `OperationID` (int64) encoding R/W flag, category, and minor operation
104109
- Masking helpers: `IsWrite`, `IsRead`, `Category`, `Minor`

‎UPGRADING.md‎

Lines changed: 123 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
- [Upgrading to >= 5.0.0](#upgrading-to->=-5.0.0)
2-
- [Partial failure errors (ReturnQueryErrors)](#partial-failure-errors-returnqueryerrors)
2+
- [Partial failure errors (Config.Errors)](#partial-failure-errors-configerrors)
33
- [Response.Body becomes a method](#responsebody-becomes-a-method)
44
- [StringError for unknown JSON responses](#stringerror-for-unknown-json-responses)
55
- [Upgrading to >= 4.7.0](#upgrading-to->=-4.7.0)
@@ -29,52 +29,148 @@
2929

3030
## Upgrading to >= 5.0.0
3131

32-
### Partial Failure Errors (ReturnQueryErrors)
32+
### Partial Failure Errors (Config.Errors)
3333

34-
Version 5.0.0 changes `Config.ReturnQueryErrors` to default to `true`. When enabled, API methods return typed errors for partial failures (HTTP 200 responses with embedded errors), so callers only need the standard `if err != nil` pattern. Both the response and the error are non-nil on partial failure -- the response is fully populated.
34+
Version 5.0.0 introduces typed partial-failure errors and a per-category bitmask that controls which categories surface as Go errors. OpenSearch returns HTTP 200 for many operations that partially succeed (bulk item failures, shard failures on search, replica failures on writes), so callers historically had to remember a second response inspection after every `if err != nil { ... }`. The new model turns those partial failures into typed errors callers can match on with `errors.As`.
3535

36-
**Why**: OpenSearch returns HTTP 200 for operations that partially succeed (bulk item failures, shard failures on search, replica failures on writes). Before this change, callers had to remember a second error check after every operation, which is easy to forget and non-idiomatic Go.
36+
#### Configuring the mask
3737

38-
**What changes in v5**: If you relied on `err == nil` meaning "no failures of any kind", your code is already correct -- you were ignoring partial failures. With `ReturnQueryErrors: true` (now the default), those partial failures surface as errors. Your `if err != nil` blocks will now catch them.
38+
`Config.Errors` is a `*errmask.ErrorMask` pointer. A set bit suppresses (masks) that category; an unset bit reports it. Three named values cover the common cases:
3939

40-
**If you need the old behavior**, set `ReturnQueryErrors: false`:
40+
| Value | Meaning |
41+
| ---------------- | ------------------------------------------------------------------- |
42+
| `nil` | Use the version's default (v4: `errmask.All`; v5+: `errmask.Empty`) |
43+
| `&errmask.Empty` | Mask nothing -- every category is reported as a typed error |
44+
| `&errmask.All` | Mask everything -- callers must inspect the response manually |
45+
46+
`errmask.None` and `errmask.Unknown` are aliases for `errmask.Empty`; all three equal 0. Composite masks (e.g. `errmask.SearchShards | errmask.MultiSearchItems`) suppress specific categories while leaving others reported.
47+
48+
The v4 default (`errmask.All`) preserves pre-bitfield behavior: partial failures are not surfaced as Go errors, so existing v4 code continues to work without modification. Opt in by setting `Config.Errors: &errmask.Empty` (or use `errmask.NewClient`-style helpers). The v5+ default flips to `errmask.Empty` so partial failures surface by default.
4149

4250
```go
51+
mask := errmask.Empty // report every category
4352
client, err := opensearchapi.NewClient(opensearchapi.Config{
44-
Client: opensearch.Config{Addresses: addrs},
45-
ReturnQueryErrors: false, // v4 behavior: partial failures don't return errors
53+
Client: opensearch.Config{Addresses: addrs},
54+
Errors: &mask,
4655
})
4756
```
4857

49-
**Handling partial failure errors**:
58+
#### Environment-variable override
59+
60+
`OPENSEARCH_GO_ERROR_MASK` accepts a comma-separated list of `+`/`-` tokens applied left-to-right on top of `Config.Errors`. Tokens are the lowercase snake_case wrapper-schema names from the OpenAPI `x-error-responses` extension (`bulk_items`, `search_shards`, `write_shards`, ...).
61+
62+
```sh
63+
# Mask everything except bulk-item errors (useful with v4: opt out of "mask everything" but suppress search-shard noise)
64+
export OPENSEARCH_GO_ERROR_MASK="+all,-bulk_items"
65+
66+
# Only mask search-shard failures; report every other category
67+
export OPENSEARCH_GO_ERROR_MASK="search_shards"
68+
69+
# Reset to "mask everything" (mimics the v4 default)
70+
export OPENSEARCH_GO_ERROR_MASK="all"
71+
72+
# Reset to "report everything" (the v5+ default)
73+
export OPENSEARCH_GO_ERROR_MASK="none"
74+
```
75+
76+
Unknown tokens are ignored (forward compatible: an older client tolerates new wrapper bits added by a newer release) and reported via the debug logger when `OPENSEARCH_GO_DEBUG` is enabled.
77+
78+
#### Handling typed errors
5079

51-
All partial failure errors implement the `PartialFailureError` interface and work with `errors.As`:
80+
Each operation returns a typed sub-error per detected wrapper category, and operations declaring multiple `x-error-responses` entries can fire more than one. The dispatch handler applies a runtime-collapse rule:
81+
82+
- 0 sub-errors fired: returns `nil`.
83+
- 1 sub-error fired: returns the bare sub-error (no wrapper allocated).
84+
- 2+ sub-errors fired: returns the per-op error type wrapping the slice.
85+
86+
`errors.As` against a known sub-error type works in **both** the single and multi cases (the per-op type implements `Unwrap() []error`):
5287

5388
```go
5489
resp, err := client.Bulk(ctx, opensearchapi.BulkReq{Body: body})
55-
if err != nil {
56-
var bulkErr *opensearchapi.PartialBulkError
57-
switch {
58-
case errors.As(err, &bulkErr):
59-
// resp is fully populated -- inspect individual items
60-
log.Printf("%d/%d items failed",
61-
len(bulkErr.FailedItems),
62-
bulkErr.SucceededCount+len(bulkErr.FailedItems))
90+
var bulkErr *opensearchapi.PartialBulkError
91+
if errors.As(err, &bulkErr) {
92+
log.Printf("%d/%d items failed",
93+
len(bulkErr.FailedItems),
94+
bulkErr.SucceededCount+len(bulkErr.FailedItems))
95+
}
96+
```
97+
98+
Callers wanting to enumerate every sub-error from a multi-wrapper op match on the per-op type:
99+
100+
```go
101+
resp, err := client.MSearch(ctx, req)
102+
var msErr *opensearchapi.MsearchErrors
103+
if errors.As(err, &msErr) {
104+
for _, sub := range msErr.Unwrap() {
105+
switch e := sub.(type) {
106+
case *opensearchapi.PartialSearchError: // shard aggregation
107+
case *opensearchapi.MultiSearchItemError: // per-sub-response Error
108+
}
109+
}
110+
}
111+
```
112+
113+
The `opensearchapi.Errors(err) []error` helper flattens the same shape uniformly across single- and multi-wrapper ops, so a single `switch` block handles both:
114+
115+
```go
116+
resp, err := client.MSearch(ctx, req)
117+
for _, sub := range opensearchapi.Errors(err) {
118+
switch e := sub.(type) {
119+
case *opensearchapi.PartialSearchError:
120+
// shard aggregation
121+
case *opensearchapi.MultiSearchItemError:
122+
// per-sub-response Error envelope
63123
default:
64-
return err // transport or HTTP error
124+
// transport / HTTP / decoding error
65125
}
66126
}
67127
```
68128

69-
**Error types**:
129+
A `nil` `err` returns `nil`; a non-partial err (transport, HTTP, decode) returns a single-element slice containing `err`. Adding new wrapper categories later is purely additive: a new `case` picks it up; the `default` keeps catching everything else.
130+
131+
#### Per-Resp helper methods
132+
133+
Every operation declaring `x-error-responses` exposes per-wrapper helper methods on its typed response, plus a `PartialFailures(mask)` aggregator. Use these when you want focused inspection at the call site without going through the dispatch error:
134+
135+
```go
136+
resp, _ := client.Bulk(ctx, req)
137+
if e := resp.BulkItemFailures(); e != nil {
138+
log.Printf("%d items failed", len(e.FailedItems))
139+
}
140+
141+
resp2, _ := client.MSearch(ctx, req)
142+
if e := resp2.SearchShardFailures(); e != nil { /* ... */ }
143+
if e := resp2.MultiSearchItemFailures(); e != nil { /* ... */ }
144+
```
145+
146+
The `r.PartialFailures(mask errmask.ErrorMask) []error` aggregator reports every wrapper category not suppressed by `mask` -- useful when reusing the dispatch's mask gating outside the dispatch path.
147+
148+
#### Error types in v4 `opensearchapi/`
149+
150+
| Error Type | Returned By | Key Fields |
151+
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
152+
| `*PartialBulkError` | `Bulk` | `FailedItems []BulkRespItem`, `SucceededCount int` |
153+
| `*PartialSearchError` | `Search`, `Scroll.Get`, `SearchTemplate` (single-bit); also via `*MsearchErrors` and `*MsearchTemplateErrors` for shard aggregation | `FailedShards int`, `TotalShards int`, `Failures []ResponseShardsFailure` |
154+
| `*ShardFailureError` | `Index`, `Document.Create`, `Document.Delete`, `Update` | `Operation string`, `FailedShards int`, `TotalShards int` |
155+
| `*MultiSearchItemError` | `MSearch`, `MSearchTemplate` (per-sub-response error inspection) | `Items []MultiSearchItemFailure`, `SucceededCount int` |
156+
| `*MsearchErrors` | `MSearch` when 2+ wrappers fire | `Unwrap() []error` (multi-error contract) |
157+
| `*MsearchTemplateErrors` | `MSearchTemplate` when 2+ wrappers fire | `Unwrap() []error` |
158+
159+
Per-op `*<Op>Errors` types are the Go 1.20+ multi-error containers; they implement `Unwrap() []error` so `errors.As` against any sub-error type still matches whether the response carried one sub-error or many.
160+
161+
#### Error types in v5preview `v5preview/opensearchapi/` (preview)
162+
163+
The v5preview surface ports the same model, but its error sub-types are spec-driven (regenerated from the OpenAPI spec on every `cmd/osgen` run):
164+
165+
| Field name | v4 (`opensearchapi`) | v5preview (`v5preview/opensearchapi`) |
166+
| ------------------------------- | ----------------------- | ------------------------------------------ |
167+
| Per-shard failure type | `ResponseShardsFailure` | `ShardSearchFailure` (spec-driven) |
168+
| Per-sub-response error envelope | inline `*DocumentError` | embedded `ErrorResponseBase` (spec-driven) |
169+
| Shard envelope type | `ResponseShards` | `ShardStatistics` (spec-driven) |
70170

71-
| Error Type | Returned By | Key Fields |
72-
|---|---|---|
73-
| `*PartialBulkError` | `Bulk` | `FailedItems []BulkRespItem`, `SucceededCount int` |
74-
| `*PartialSearchError` | `Search`, `MSearch`, `MSearchTemplate`, `SearchTemplate`, `Scroll.Get` | `FailedShards int`, `TotalShards int`, `Failures []ResponseShardsFailure` |
75-
| `*ShardFailureError` | `Index`, `Document.Create`, `Document.Delete`, `Update` | `Operation string`, `FailedShards int`, `TotalShards int` |
171+
The v5preview package additionally generates one per-op error type per operation declaring `x-error-responses` (e.g. `*v5preview/opensearchapi.MsearchErrors`, `*v5preview/opensearchapi.MsearchTemplateErrors`). Callers wanting v4-shaped field types should keep using the v4 `opensearchapi` package; v5preview is a preview surface that will become the default in v5.
76172

77-
**Helper functions** for common patterns:
173+
#### Helper functions
78174

79175
```go
80176
// Suppress all partial failures (best-effort operations)
@@ -87,7 +183,7 @@ err = opensearchapi.RequireSuccessRate(err, 0.99)
87183
if opensearchapi.IsPartialFailure(err) { ... }
88184
```
89185

90-
**Operation constants** for `ShardFailureError.Operation`:
186+
#### Operation constants for `ShardFailureError.Operation`
91187

92188
```go
93189
opensearchapi.OperationIndex // "index"

0 commit comments

Comments
 (0)