Skip to content

Commit 376b6df

Browse files
sean-sean-
committed
feat(opensearchapi): add partial failure errors for bulk, search, and write operations
OpenSearch returns HTTP 200 for partial failures (bulk item errors, shard failures), forcing callers to double-check responses after err == nil. This adds PartialBulkError, PartialSearchError, and ShardFailureError types so callers only need the standard Go if err != nil idiom. Both (resp, err) are non-nil on partial failure — the response is fully populated. Gated behind Config.ReturnQueryErrors (default false in v4, will flip to true in v5). Existing behavior is unchanged unless opted in. Adds helper functions IsPartialFailure, ToleratePartialFailures, and RequireSuccessRate for threshold-based error tolerance. Ref: opensearch-project#816 Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>
1 parent ac3eca1 commit 376b6df

17 files changed

Lines changed: 1133 additions & 19 deletions

‎CHANGELOG.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,16 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
8989
- Document environment variables in `guides/routing.md`
9090
- Document read-after-write visibility guarantees with operation-aware routing in `guides/routing.md`
9191
- Add adaptive `max_concurrent_shard_requests` derived from cluster-wide AIMD congestion window ([#800](https://github.com/opensearch-project/opensearch-go/issues/800))
92+
- Add partial failure error types (`PartialBulkError`, `PartialSearchError`, `ShardFailureError`) that surface HTTP 200 partial failures as Go errors when `Config.ReturnQueryErrors` is enabled ([#816](https://github.com/opensearch-project/opensearch-go/issues/816))
93+
- `PartialBulkError` returned from `Bulk` when `resp.Errors` is true, carries `FailedItems` and `SucceededCount`
94+
- `PartialSearchError` returned from `Search`, `MSearch`, `MSearchTemplate`, `SearchTemplate`, `Scroll.Get` when `_shards.failed > 0`
95+
- `ShardFailureError` returned from `Index`, `Document.Create`, `Document.Delete`, `Update` when replica shards fail
96+
- `PartialFailureError` marker interface with `IsPartial() bool` for type-switching across all partial failure types
97+
- Helper functions: `IsPartialFailure`, `ToleratePartialFailures`, `RequireSuccessRate` for threshold-based error tolerance
98+
- Operation constants: `OperationIndex`, `OperationCreate`, `OperationUpdate`, `OperationDelete`
99+
- `Config.ReturnQueryErrors` defaults to `false` in v4 (opt-in), will flip to `true` in v5
100+
- `OPENSEARCH_GO_PARTIAL_QUERY_ERRORS` environment variable overrides `Config.ReturnQueryErrors` at runtime
101+
- Both `(resp, error)` are non-nil on partial failure -- response is fully populated
92102
- Transport automatically sets `max_concurrent_shard_requests` query parameter on search requests routed through a coordinator node
93103
- Value derived from a cluster-wide aggregate of all polled nodes' search pool wait-time and completion deltas, clamped to `[floor, cap]` (default: 5–256)
94104
- Cluster-wide signal correctly models data-node fan-out capacity: single hot nodes are diluted by healthy peers, and MCSR only drops when aggregate cluster pressure rises

‎UPGRADING.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
- [Upgrading to >= 5.0.0](#upgrading-to->=-5.0.0)
2+
- [Partial failure errors (ReturnQueryErrors)](#partial-failure-errors-returnqueryerrors)
23
- [Response.Body becomes a method](#responsebody-becomes-a-method)
34
- [StringError for unknown JSON responses](#stringerror-for-unknown-json-responses)
45
- [Upgrading to >= 4.7.0](#upgrading-to->=-4.7.0)
@@ -28,6 +29,75 @@
2829

2930
## Upgrading to >= 5.0.0
3031

32+
### Partial Failure Errors (ReturnQueryErrors)
33+
34+
Version 5.0.0 changes `Config.ReturnQueryErrors` to default to `true`. When enabled, API methods return typed errors for partial failures (HTTP 200 responses with embedded errors), so callers only need the standard `if err != nil` pattern. Both the response and the error are non-nil on partial failure -- the response is fully populated.
35+
36+
**Why**: OpenSearch returns HTTP 200 for operations that partially succeed (bulk item failures, shard failures on search, replica failures on writes). Before this change, callers had to remember a second error check after every operation, which is easy to forget and non-idiomatic Go.
37+
38+
**What changes in v5**: If you relied on `err == nil` meaning "no failures of any kind", your code is already correct -- you were ignoring partial failures. With `ReturnQueryErrors: true` (now the default), those partial failures surface as errors. Your `if err != nil` blocks will now catch them.
39+
40+
**If you need the old behavior**, set `ReturnQueryErrors: false`:
41+
42+
```go
43+
client, err := opensearchapi.NewClient(opensearchapi.Config{
44+
Client: opensearch.Config{Addresses: addrs},
45+
ReturnQueryErrors: false, // v4 behavior: partial failures don't return errors
46+
})
47+
```
48+
49+
**Handling partial failure errors**:
50+
51+
All partial failure errors implement the `PartialFailureError` interface and work with `errors.As`:
52+
53+
```go
54+
resp, err := client.Bulk(ctx, opensearchapi.BulkReq{Body: body})
55+
if err != nil {
56+
var bulkErr *opensearchapi.PartialBulkError
57+
switch {
58+
case errors.As(err, &bulkErr):
59+
// resp is fully populated -- inspect individual items
60+
log.Printf("%d/%d items failed",
61+
len(bulkErr.FailedItems),
62+
bulkErr.SucceededCount+len(bulkErr.FailedItems))
63+
default:
64+
return err // transport or HTTP error
65+
}
66+
}
67+
```
68+
69+
**Error types**:
70+
71+
| Error Type | Returned By | Key Fields |
72+
|---|---|---|
73+
| `*PartialBulkError` | `Bulk` | `FailedItems []BulkRespItem`, `SucceededCount int` |
74+
| `*PartialSearchError` | `Search`, `MSearch`, `MSearchTemplate`, `SearchTemplate`, `Scroll.Get` | `FailedShards int`, `TotalShards int`, `Failures []ResponseShardsFailure` |
75+
| `*ShardFailureError` | `Index`, `Document.Create`, `Document.Delete`, `Update` | `Operation string`, `FailedShards int`, `TotalShards int` |
76+
77+
**Helper functions** for common patterns:
78+
79+
```go
80+
// Suppress all partial failures (best-effort operations)
81+
err = opensearchapi.ToleratePartialFailures(err)
82+
83+
// Fail only if success rate drops below threshold
84+
err = opensearchapi.RequireSuccessRate(err, 0.99)
85+
86+
// Test whether an error is a partial failure
87+
if opensearchapi.IsPartialFailure(err) { ... }
88+
```
89+
90+
**Operation constants** for `ShardFailureError.Operation`:
91+
92+
```go
93+
opensearchapi.OperationIndex // "index"
94+
opensearchapi.OperationCreate // "create"
95+
opensearchapi.OperationUpdate // "update"
96+
opensearchapi.OperationDelete // "delete"
97+
```
98+
99+
See [Error Handling and Partial Failures](guides/error_handling.md) for the full guide.
100+
31101
### StringError for Unknown JSON Responses
32102

33103
Version 5.0.0 returns `*opensearch.StringError` error type instead of `*fmt.wrapError` when response received from the server is an unknown JSON. For example, consider delete document API which returns an unknown JSON body when document is not found.

‎USER_GUIDE.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,10 @@ func example() error {
6161
// Optional: Enable intelligent request routing
6262
Router: router,
6363
},
64+
65+
// Optional: Surface partial failures (bulk item errors, shard failures)
66+
// as Go errors. See guides/error_handling.md for details.
67+
ReturnQueryErrors: true,
6468
},
6569
)
6670
if err != nil {
@@ -516,6 +520,7 @@ All `OPENSEARCH_GO_*` environment variables are evaluated once at client initial
516520
| `OPENSEARCH_GO_ACTIVE_LIST_CAP` | auto | Max active connections per pool |
517521
| `OPENSEARCH_GO_STANDBY_*` | (see guide) | Standby rotation and promotion tuning (3 variables) |
518522
| `OPENSEARCH_GO_OVERLOADED_*` | (see guide) | JVM heap and breaker thresholds (2 variables) |
523+
| `OPENSEARCH_GO_PARTIAL_QUERY_ERRORS`| `false` | Surface partial failures as Go errors (overrides `Config.ReturnQueryErrors`) |
519524

520525
## Guides by Topic
521526

‎guides/error_handling.md‎

Lines changed: 152 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,122 @@ This design maximizes availability but requires careful error checking in client
3232
| **Refresh** | 200 | `_shards.failed > 0` | Incomplete refresh |
3333
| **Cluster operations** | 200 | `_shards.failed > 0` | Incomplete stats/operations |
3434

35-
## Checking for Partial Failures
35+
## Automatic Partial Failure Errors (Recommended)
36+
37+
When `ReturnQueryErrors` is enabled, API methods return typed errors for partial failures alongside the fully populated response. This eliminates the need for manual double-checking and follows Go's `(result, error)` convention where both can be non-nil.
38+
39+
```go
40+
client, err := opensearchapi.NewClient(opensearchapi.Config{
41+
Client: opensearch.Config{Addresses: []string{"https://localhost:9200"}},
42+
ReturnQueryErrors: true,
43+
})
44+
```
45+
46+
> **Migration note**: `ReturnQueryErrors` defaults to `false` in v4 for backward compatibility. It will default to `true` in v5.
47+
>
48+
> You can also enable this via the `OPENSEARCH_GO_PARTIAL_QUERY_ERRORS=true` environment variable, which takes priority over the `Config` field. This is useful for toggling the behavior at deploy time without code changes.
49+
50+
### Bulk Operations
51+
52+
When `ReturnQueryErrors` is enabled, bulk operations return a `*PartialBulkError` when any items fail. The response is still fully populated -- callers can inspect both the error and the response.
53+
54+
```go
55+
resp, err := client.Bulk(ctx, opensearchapi.BulkReq{Body: body})
56+
if err != nil {
57+
var bulkErr *opensearchapi.PartialBulkError
58+
if errors.As(err, &bulkErr) {
59+
// resp is fully populated -- inspect individual items
60+
log.Printf("%d/%d items failed",
61+
len(bulkErr.FailedItems),
62+
bulkErr.SucceededCount+len(bulkErr.FailedItems))
63+
for _, item := range bulkErr.FailedItems {
64+
log.Printf(" %s %s/%s: %s",
65+
item.Error.Type, item.Index, item.DocumentID, item.Error.Reason)
66+
}
67+
} else {
68+
return err // transport or HTTP error
69+
}
70+
}
71+
```
72+
73+
### Search Operations
74+
75+
Search operations return a `*PartialSearchError` when shards fail. The response contains whatever hits came back from the successful shards.
76+
77+
```go
78+
resp, err := client.Search(ctx, &opensearchapi.SearchReq{
79+
Indices: []string{"events"},
80+
Body: body,
81+
})
82+
if err != nil {
83+
var shardErr *opensearchapi.PartialSearchError
84+
if errors.As(err, &shardErr) {
85+
log.Printf("%d/%d shards failed, got %d hits",
86+
shardErr.FailedShards, shardErr.TotalShards,
87+
len(resp.Hits.Hits))
88+
} else {
89+
return err
90+
}
91+
}
92+
```
93+
94+
Multi-search (`MSearch`, `MSearchTemplate`) and scroll (`Scroll.Get`) operations also return `PartialSearchError` when any sub-response has shard failures. The error aggregates failures across all sub-responses.
95+
96+
### Write Operations
97+
98+
Index, Create, Update, and Delete operations return a `*ShardFailureError` when the primary shard succeeds but replica shards fail. The `Operation` field identifies which write operation was performed.
99+
100+
```go
101+
resp, err := client.Index(ctx, opensearchapi.IndexReq{
102+
Index: "test",
103+
Body: strings.NewReader(`{"field": "value"}`),
104+
})
105+
if err != nil {
106+
var shardErr *opensearchapi.ShardFailureError
107+
if errors.As(err, &shardErr) {
108+
log.Printf("%s: %d/%d shards failed (primary succeeded)",
109+
shardErr.Operation, shardErr.FailedShards, shardErr.TotalShards)
110+
} else {
111+
return err
112+
}
113+
}
114+
```
115+
116+
### Helper Functions
117+
118+
Three helper functions simplify common patterns:
119+
120+
```go
121+
// Test whether an error is a partial failure (any type)
122+
if opensearchapi.IsPartialFailure(err) {
123+
log.Println("partial failure detected")
124+
}
125+
126+
// Suppress all partial failures -- useful for best-effort operations
127+
err = opensearchapi.ToleratePartialFailures(err)
128+
129+
// Threshold-based tolerance -- fail only if success rate drops below 99%
130+
err = opensearchapi.RequireSuccessRate(err, 0.99)
131+
if err != nil {
132+
log.Fatal(err) // only reached if <99% succeeded or non-partial error
133+
}
134+
```
135+
136+
### Error Type Reference
137+
138+
| Error Type | Returned By | Fields |
139+
| --------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------- |
140+
| `*PartialBulkError` | `Bulk` | `FailedItems []BulkRespItem`, `SucceededCount int` |
141+
| `*PartialSearchError` | `Search`, `MSearch`, `MSearchTemplate`, `SearchTemplate`, `Scroll.Get` | `FailedShards int`, `TotalShards int`, `Failures []ResponseShardsFailure` |
142+
| `*ShardFailureError` | `Index`, `Document.Create`, `Document.Delete`, `Update` | `Operation string`, `FailedShards int`, `TotalShards int` |
143+
144+
All three implement the `PartialFailureError` interface and work with `errors.As`.
145+
146+
---
147+
148+
## Manual Partial Failure Checking
149+
150+
When `ReturnQueryErrors` is `false` (the v4 default), callers must inspect response fields directly. The sections below document this pattern.
36151

37152
### 1. Bulk Operations
38153

@@ -188,9 +303,26 @@ func safeIndexOperation(client *opensearchapi.Client, ctx context.Context) error
188303

189304
## Best Practices
190305

191-
### 1. Always Check Partial Failure Indicators
306+
### 1. Enable ReturnQueryErrors
307+
308+
The simplest way to catch partial failures is to enable `ReturnQueryErrors` in the client config. This surfaces partial failures through the standard `error` return, so the idiomatic `if err != nil` catches everything:
309+
310+
```go
311+
client, err := opensearchapi.NewClient(opensearchapi.Config{
312+
Client: opensearch.Config{Addresses: addrs},
313+
ReturnQueryErrors: true,
314+
})
315+
316+
resp, err := client.Bulk(ctx, req)
317+
if err != nil {
318+
// Catches transport errors, HTTP errors, AND partial failures.
319+
return err
320+
}
321+
```
322+
323+
### 2. Always Check Partial Failure Indicators
192324

193-
**Never assume HTTP 2xx means complete success:**
325+
**When `ReturnQueryErrors` is disabled**, never assume HTTP 2xx means complete success:
194326

195327
```go
196328
// WRONG - Missing partial failure checks
@@ -210,7 +342,7 @@ if resp.Errors {
210342
}
211343
```
212344

213-
### 2. Define Your Error Tolerance
345+
### 3. Define Your Error Tolerance
214346

215347
Different applications have different requirements:
216348

@@ -232,7 +364,13 @@ if resp.Errors {
232364
}
233365
```
234366

235-
### 3. Implement Retry Logic for Failed Items
367+
With `ReturnQueryErrors` enabled, use `RequireSuccessRate` for the same effect:
368+
369+
```go
370+
err = opensearchapi.RequireSuccessRate(err, 0.50) // nil unless >50% failed
371+
```
372+
373+
### 4. Implement Retry Logic for Failed Items
236374

237375
```go
238376
func bulkWithRetry(client *opensearchapi.Client, ctx context.Context, items []string) error {
@@ -298,7 +436,7 @@ When implementing retry logic for bulk operations:
298436
- **Retry only the failed items**, not the entire batch. Items that succeeded in the original request do not need to be resubmitted (resubmitting may cause version conflicts or duplicate documents depending on whether document IDs are set).
299437
- **Set client-assigned `_id` values on bulk items.** After a timeout or ambiguous failure, the client can query for expected document IDs to determine which items were persisted, turning a blind retry into a targeted one. See [Bulk: Use client-assigned document IDs for recoverability](bulk.md#use-client-assigned-document-ids-for-recoverability).
300438

301-
### 4. Monitor Partial Failure Rates
439+
### 5. Monitor Partial Failure Rates
302440

303441
```go
304442
type OperationMetrics struct {
@@ -485,12 +623,13 @@ func isRetryableError(errType string) bool {
485623

486624
## Summary
487625

488-
1. **HTTP 2xx does not guarantee complete success.** Always check partial failure indicators.
489-
2. **Bulk operations**: Check the `resp.Errors` field and examine each item.
490-
3. **Search operations**: Check `resp.Shards.Failed` for incomplete results.
491-
4. **Write operations**: Check `resp.Shards.Failed` for replica failures.
492-
5. **Define error tolerance**: Determine what constitutes acceptable failure for the application.
493-
6. **Implement retry logic**: Retry transient failures with backoff.
494-
7. **Monitor failure rates**: Track and alert on partial failures.
626+
1. **Enable `ReturnQueryErrors: true`** for idiomatic `if err != nil` handling of partial failures.
627+
2. **HTTP 2xx does not guarantee complete success.** Without `ReturnQueryErrors`, always check partial failure indicators manually.
628+
3. **Bulk operations**: `PartialBulkError` (or manual `resp.Errors` check) for item-level failures.
629+
4. **Search operations**: `PartialSearchError` (or manual `resp.Shards.Failed` check) for incomplete results.
630+
5. **Write operations**: `ShardFailureError` (or manual `resp.Shards.Failed` check) for replica failures.
631+
6. **Define error tolerance**: Use `RequireSuccessRate` or custom logic to decide what constitutes acceptable failure.
632+
7. **Implement retry logic**: Retry transient failures with backoff.
633+
8. **Monitor failure rates**: Track and alert on partial failures.
495634

496635
Following these practices produces reliable applications that correctly handle OpenSearch's distributed partial-failure model.

‎opensearchapi/api_bulk.go‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,24 @@ func (c Client) Bulk(ctx context.Context, req BulkReq) (*BulkResp, error) {
2626
return &data, err
2727
}
2828

29+
if c.returnQueryErrors && data.Errors {
30+
var failed []BulkRespItem
31+
succeeded := 0
32+
for _, item := range data.Items {
33+
for _, v := range item {
34+
if v.Error != nil {
35+
failed = append(failed, v)
36+
} else {
37+
succeeded++
38+
}
39+
}
40+
}
41+
return &data, &PartialBulkError{
42+
FailedItems: failed,
43+
SucceededCount: succeeded,
44+
}
45+
}
46+
2947
return &data, nil
3048
}
3149

‎opensearchapi/api_document.go‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,14 @@ func (c documentClient) Create(ctx context.Context, req DocumentCreateReq) (*Doc
2626
return &data, err
2727
}
2828

29+
if c.apiClient.returnQueryErrors && data.Shards.Failed > 0 {
30+
return &data, &ShardFailureError{
31+
Operation: OperationCreate,
32+
FailedShards: data.Shards.Failed,
33+
TotalShards: data.Shards.Total,
34+
}
35+
}
36+
2937
return &data, nil
3038
}
3139

@@ -39,6 +47,14 @@ func (c documentClient) Delete(ctx context.Context, req DocumentDeleteReq) (*Doc
3947
return &data, err
4048
}
4149

50+
if c.apiClient.returnQueryErrors && data.Shards.Failed > 0 {
51+
return &data, &ShardFailureError{
52+
Operation: OperationDelete,
53+
FailedShards: data.Shards.Failed,
54+
TotalShards: data.Shards.Total,
55+
}
56+
}
57+
4258
return &data, nil
4359
}
4460

‎opensearchapi/api_index.go‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,14 @@ func (c Client) Index(ctx context.Context, req IndexReq) (*IndexResp, error) {
3030
return &data, err
3131
}
3232

33+
if c.returnQueryErrors && data.Shards.Failed > 0 {
34+
return &data, &ShardFailureError{
35+
Operation: OperationIndex,
36+
FailedShards: data.Shards.Failed,
37+
TotalShards: data.Shards.Total,
38+
}
39+
}
40+
3341
return &data, nil
3442
}
3543

0 commit comments

Comments
 (0)