Update metrics and error handling docs (#812)

* Add a doc explaining available metrics

Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>

* docs: drop per-Resp helpers from user docs, recommend for/switch

User-facing docs (v5preview/opensearchapi/README.md and
guides/error_handling.md) no longer document the per-Resp helper
methods (BulkItemFailures, SearchShardFailures, WriteShardFailures,
MultiSearchItemFailures, PartialFailures(mask)).

The Recommended pattern section presents two paths:

- Treat any server or API failure as a hard error -- the idiomatic
  `if err != nil { return err }` for operations where any failure is
  reason to stop.
- Inspect categories with a `for`/`switch` over
  opensearchapi.Errors(err) when partial error handling lets the
  application recover from known tolerated failure modes.

The antipattern discussion now covers errors.As, `Has`-style helpers,
and per-Resp helpers together: all three answer the narrow question
"did this category happen?" and silently miss categories added in a
future release. The type switch is the only category-aware pattern
recommended.

opensearchapi/partial_failure_methods.go file header reframes the
helpers as engine machinery for the dispatch and points at
guides/error_handling.md for the recommended call-site pattern. The
methods stay available without deprecation markers.

DEVELOPER_GUIDE.md and cmd/osgen/README.md describe the helpers as
engine machinery the dispatch consumes and redirect call-site authors
to opensearchapi.Errors(err) + for/switch.

Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>

* docs: round out v5-prep doc fixes

- guides/routing.md: add COORDINATOR and NULL rows to the policy
  override env-var table so it matches the 10 policy types in
  policy_override_env.go (USER_GUIDE.md already says "10 variables").
- guides/search.md: replace three Transport: &opensearchtransport.Client{Router: ...}
  blocks with Router: ... directly on opensearch.Config. The old form
  did not compile -- opensearch.Config.Transport is http.RoundTripper
  and *opensearchtransport.Client does not implement it; the canonical
  field for the router has been Config.Router since the routing system
  was introduced.

Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>

* docs: address PR #812 review feedback

- guides/metrics.md: replace nonexistent opensearchtransport.WithObserver()
  with the Observer field on opensearch.Config
- guides/metrics.md: hoist safeFloat above the for loop so the polling
  example is parseable Go
- guides/metrics.md: align aggregate tables (Request Counters, Pool State,
  Lifecycle, Health Checks) to Field (Go name) + JSON (tag) layout
- guides/metrics.md: soften mechanism claims on IsOverloaded, RTTBucket,
  EstLoad to describe field meaning rather than how they get set
- guides/error_handling.md, v5preview/opensearchapi/README.md: drop
  *PartialBulkError and *ShardFailureError from MSearch switch examples;
  fix the duplicate pre-existing example in the v5preview README
- guides/error_handling.md: qualify "folded into err" with the v4-vs-
  v5preview default-mask caveat
- guides/error_handling.md: nil-check ErrorCause.Reason deref in v5preview
  bulk example; switch the version-agnostic MSearch helper to log
  item.Status (portable across v4 and v5preview)
- v5preview/opensearchapi/README.md: fix NewMuxRouter() to single-value
  assignment (returns Router, not (Router, error))
- v5preview/opensearchapi/MIGRATING.md: drop the forward-compatible
  `replace` section -- the cited path has no go.mod and the directive
  cannot resolve
- UPGRADING.md: reword the partial-failure intro so the recommended
  type-switch pattern is the only advertised entry point (drops the
  in-tension `errors.As` mention)
- CHANGELOG.md: add (#812) backlink to the metrics-guide bullet

Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>

---------

Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>

Author: sean-

CHANGELOG.md

@@ -16,6 +16,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Add `OPENSEARCH_GO_SHARD_COST` environment variable and `WithShardCosts()` router option with `r:base`/`r:amplify`/`r:exponent` curve keys and static cost overrides
 - Add `ShardCostConfig` field to `Config` struct for programmatic shard cost override passthrough
 - Add `OPENSEARCH_GO_ROUTER` environment variable to enable the DefaultRouter without code changes; set to `true` to opt in (off by default in v4, on by default in v5, removed in v6) ([#815](https://github.com/opensearch-project/opensearch-go/pull/815))
+- Add client-side metrics guide covering Metrics API, ConnectionMetric, PolicySnapshot, and RouterSnapshot ([#812](https://github.com/opensearch-project/opensearch-go/pull/812))
 - Add `InsecureSkipVerify` config option to disable TLS certificate verification without constructing a custom `http.Transport`, preserving `DefaultTransport` connection pooling, HTTP/2, and timeout defaults ([#786](https://github.com/opensearch-project/opensearch-go/issues/786))
 - Add `DisableResponseBuffering` config option to skip eager `io.ReadAll` buffering of response bodies in `Perform()`, reducing per-request allocations and TTFB for proxy and streaming use cases ([#786](https://github.com/opensearch-project/opensearch-go/issues/786))
 - Add per-attempt `RequestTimeout` to bound individual HTTP round-trips, preventing indefinite hangs on stalled connections ([#786](https://github.com/opensearch-project/opensearch-go/issues/786))
@@ -98,10 +99,10 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
   - `MultiSearchItemError` returned from `MSearch`/`MSearchTemplate` for per-sub-response Error envelopes
   - `MSearchErrors` / `MSearchTemplateErrors` per-op containers (Go 1.20+ multi-error contract via `Unwrap() []error`) when 2+ wrapper categories fire on the same response
   - `PartialFailureError` marker interface with `IsPartial() bool` for type-switching across all partial-failure types
-  - Per-Resp helper methods (`BulkItemFailures`, `SearchShardFailures`, `WriteShardFailures`, `MultiSearchItemFailures`) plus `PartialFailures(mask)` aggregator for focused inspection at the call site
-  - `opensearchapi.Errors(err) []error` package-level helper that flattens single- and multi-wrapper errors into a uniform slice for `switch` dispatch
+  - `opensearchapi.Errors(err) []error` package-level helper that flattens single- and multi-wrapper errors into a uniform slice; recommended call-site pattern is a `for`/`switch` over the result (not `errors.As` against a specific type)
   - Helper functions: `IsPartialFailure`, `ToleratePartialFailures`, `RequireSuccessRate` for threshold-based error tolerance
   - Operation constants: `OperationIndex`, `OperationCreate`, `OperationUpdate`, `OperationDelete`
+  - Per-Resp helper methods (`BulkItemFailures`, `SearchShardFailures`, `WriteShardFailures`, `MultiSearchItemFailures`, `PartialFailures(mask)`) exist on the response types as engine machinery for the dispatch; new code should prefer a `for`/`switch` over `opensearchapi.Errors(err)` rather than the per-Resp helpers, for forward compatibility
   - `Config.Errors *errmask.ErrorMask` replaces a single boolean: each bit suppresses one wrapper category. v4 defaults to `errmask.All` (mask everything, preserves pre-bitfield behavior); v5+ defaults to `errmask.Empty` (report everything)
   - `OPENSEARCH_GO_ERROR_MASK` environment variable overrides `Config.Errors` at runtime via comma-separated `+`/`-` tokens (lowercase snake_case wrapper names; unknown tokens silently dropped, debug-logged)
   - Both `(resp, error)` are non-nil on partial failure -- response is fully populated

DEVELOPER_GUIDE.md

@@ -399,7 +399,7 @@ The `x-error-responses` extension on a spec operation declares the categories of
 
 - A typed Go error (e.g. `*PartialBulkError`, `*PartialSearchError`, `*ShardFailureError`, `*MultiSearchItemError`) decoded from the response body when that category fires.
 - A bit on `errmask.ErrorMask` (PascalCase, e.g. `errmask.BulkItems`) plus the corresponding env-var token (`bulk_items`) so callers can suppress or surface it via `Config.Errors` or `OPENSEARCH_GO_ERROR_MASK`.
-- A per-Resp helper method on the operation's typed response (e.g. `BulkResp.BulkItemFailures()`, `SearchResp.SearchShardFailures()`).
+- A per-Resp helper method on the operation's typed response (e.g. `BulkResp.BulkItemFailures()`, `SearchResp.SearchShardFailures()`). These exist as engine machinery for the dispatch and are not the recommended call-site pattern; user docs point callers at a `for`/`switch` over `opensearchapi.Errors(err)` instead.
 - A `PartialFailures(mask)` aggregator on the same Resp.
 
 Operations that declare two or more categories also get a per-op error container (e.g. `*MSearchErrors`) implementing `Unwrap() []error`, used when more than one category fires on a single response.

UPGRADING.md

@@ -34,7 +34,7 @@
 
 ### Partial Failure Errors (Config.Errors)
 
-Version 5.0.0 introduces typed partial-failure errors and a per-category bitmask that controls which categories surface as Go errors. OpenSearch returns HTTP 200 for many operations that partially succeed (bulk item failures, shard failures on search, replica failures on writes). The new model turns those partial failures into typed errors callers can match on with `errors.As`.
+Version 5.0.0 introduces typed partial-failure errors and a per-category bitmask that controls which categories surface as Go errors. OpenSearch returns HTTP 200 for many operations that partially succeed (bulk item failures, shard failures on search, replica failures on writes). The new model turns those partial failures into typed errors that callers can dispatch on; idiomatic partial error handling is shown below.
 
 **Default behavior change:**
 
@@ -52,9 +52,10 @@ A v4 caller upgrading to v5 who never set `Config.Errors` will start seeing part
 - Typed errors: `*PartialBulkError`, `*PartialSearchError`, `*ShardFailureError`, `*MultiSearchItemError`, `*MSearchErrors`, `*MSearchTemplateErrors`.
 - `opensearchapi.Errors(err) []error` to flatten single- and multi-wrapper error shapes into a uniform slice.
 - Helper functions: `IsPartialFailure(err)`, `ToleratePartialFailures(err)`, `RequireSuccessRate(err, threshold)`.
-- Per-Resp helper methods: `BulkItemFailures()`, `SearchShardFailures()`, `WriteShardFailures()`, `MultiSearchItemFailures()`, `PartialFailures(mask)`.
 - Operation constants: `OperationIndex`, `OperationCreate`, `OperationUpdate`, `OperationDelete`.
 
+The recommended call-site pattern is a `for`/`switch` over `opensearchapi.Errors(err)`, not `errors.As` against a specific type. Per-Resp helper methods (`BulkItemFailures()`, `SearchShardFailures()`, `WriteShardFailures()`, `MultiSearchItemFailures()`, `PartialFailures(mask)`) exist on the response types as engine machinery for the dispatch and remain available for focused inspection of a known category, but new code should use the type switch -- see [`guides/error_handling.md`](guides/error_handling.md#why-a-type-switch-not-errorsas-has-or-per-resp-helpers) for why.
+
 **Where to read more:**
 
 - [`v5preview/opensearchapi/README.md`](v5preview/opensearchapi/README.md) - full v5preview usage guide for these errors, including the type-switch pattern and the rationale for preferring it over `errors.As`/`Has`.

cmd/osgen/README.md

@@ -157,7 +157,7 @@ go run . api \
 3. Routes each operation to either the core `opensearchapi` package or a plugin package based on the operation group prefix.
 4. Renders Req structs (with path builder embedding, optional body, and header support), Params structs (with typed encode methods), and Resp stubs.
 5. Annotates generated code with availability (`x-version-added`), deprecation (`x-version-deprecated`, `x-deprecation-message`), and distribution exclusion metadata.
-6. Reads each operation's `x-error-responses` extension to emit typed partial-failure errors (`*PartialBulkError`, `*PartialSearchError`, `*ShardFailureError`, `*MultiSearchItemError`, ...), the corresponding `errmask` bits and env-var tokens, per-Resp helper methods (`BulkItemFailures()`, `SearchShardFailures()`, `WriteShardFailures()`, `MultiSearchItemFailures()`, `PartialFailures(mask)`), and -- for operations declaring two or more categories -- a per-op multi-error container implementing `Unwrap() []error`. See [`DEVELOPER_GUIDE.md` Partial-failure error generation](../../DEVELOPER_GUIDE.md#partial-failure-error-generation) for the full surface this produces, and [`v5preview/opensearchapi/README.md` Partial Failure Errors](../../v5preview/opensearchapi/README.md#partial-failure-errors) for the user-facing usage guide.
+6. Reads each operation's `x-error-responses` extension to emit typed partial-failure errors (`*PartialBulkError`, `*PartialSearchError`, `*ShardFailureError`, `*MultiSearchItemError`, ...), the corresponding `errmask` bits and env-var tokens, per-Resp helper methods (`BulkItemFailures()`, `SearchShardFailures()`, `WriteShardFailures()`, `MultiSearchItemFailures()`, `PartialFailures(mask)`) used internally by the dispatch, and -- for operations declaring two or more categories -- a per-op multi-error container implementing `Unwrap() []error`. The recommended call-site pattern in user code is a `for`/`switch` over `opensearchapi.Errors(err)`, not the per-Resp helpers; see [`DEVELOPER_GUIDE.md` Partial-failure error generation](../../DEVELOPER_GUIDE.md#partial-failure-error-generation) for the generated surface and [`v5preview/opensearchapi/README.md` Partial Failure Errors](../../v5preview/opensearchapi/README.md#partial-failure-errors) for the user-facing usage guide.
 
 ## Separate Module
 

guides/error_handling.md

@@ -86,14 +86,18 @@ for _, sub := range opensearchapi.Errors(err) {
             len(e.FailedItems),
             e.SucceededCount+len(e.FailedItems))
         for _, item := range e.FailedItems {
-            // BulkRespItem.ID and BulkRespItem.Error are pointers in v5preview.
+            // BulkRespItem.ID, BulkRespItem.Error, and ErrorCause.Reason are pointers in v5preview.
             id := ""
             if item.ID != nil {
                 id = *item.ID
             }
             if item.Error != nil {
+                reason := ""
+                if item.Error.Reason != nil {
+                    reason = *item.Error.Reason
+                }
                 log.Printf("  %s %s/%s: %s",
-                    item.Error.Type, item.Index, id, item.Error.Reason)
+                    item.Error.Type, item.Index, id, reason)
             }
         }
     default:
@@ -208,32 +212,41 @@ if err != nil {
 }
 ```
 
-### Per-Resp helper methods
+### Recommended pattern
+
+Two patterns cover every partial-failure use case. Pick the one that matches your operation's tolerance:
 
-Every operation that can return a partial failure exposes per-category helper methods on its typed response, plus a `PartialFailures(mask)` aggregator. Use these when you want focused inspection at the call site without going through the dispatch error. The helpers exist on both v4 `opensearchapi/` and v5preview `v5preview/opensearchapi/` Resp types and are nil-safe on a nil receiver.
+**Treat any server or API failure as a hard error** -- the simplest and most idiomatic Go path. Use this when the operation has no meaningful "partial success" -- any error is a reason to stop:
 
 ```go
-resp, _ := client.Bulk(ctx, req)
-if e := resp.BulkItemFailures(); e != nil {
-    log.Printf("%d items failed", len(e.FailedItems))
+resp, err := client.Bulk(ctx, req)
+if err != nil {
+    return err
 }
-
-resp2, _ := client.MSearch(ctx, req)
-if e := resp2.SearchShardFailures(); e != nil { /* ... */ }
-if e := resp2.MultiSearchItemFailures(); e != nil { /* ... */ }
-
-resp3, _ := client.Index(ctx, req)
-if e := resp3.WriteShardFailures(); e != nil { /* ... */ }
+// resp is fully populated; partial failures (if any) are folded into err
+// when the wrapper bits are unmasked (the v5preview default, or v4 with
+// Config.Errors: errmask.New()).
 ```
 
-`r.PartialFailures(mask errmask.ErrorMask) []error` returns every wrapper category that fired and was not suppressed by `mask`. Useful for recreating the dispatch's mask-gated behavior at the call site -- pass `errmask.Empty` to see every category, or any narrower mask to suppress specific ones:
+**Inspect categories with a `for`/`switch`** -- when partial error handling is appropriate. Partial error handling lets the client and its application recover from known failure modes they can tolerate (e.g. continue serving a search with a few failed shards, or retry only the bulk items the server rejected) instead of failing the whole operation. The `default` arm catches transport / HTTP / decode errors and any partial-failure category added in a future release:
 
 ```go
-for _, sub := range resp.PartialFailures(errmask.Empty) {
-    // same shape as opensearchapi.Errors(err)
+resp, err := client.MSearch(ctx, req)
+for _, sub := range opensearchapi.Errors(err) {
+    switch e := sub.(type) {
+    case *opensearchapi.PartialSearchError:
+        log.Printf("%d/%d shards failed", e.FailedShards, e.TotalShards)
+    case *opensearchapi.MultiSearchItemError:
+        log.Printf("%d sub-queries failed", len(e.Items))
+    default:
+        return err
+    }
 }
+// resp is fully populated; use it regardless of partial failure.
 ```
 
+`opensearchapi.Errors(err)` flattens every error shape into a uniform slice -- single sub-error, multi-wrapper container, transport error, or `nil` (returns `nil`). The switch is the only pattern this guide recommends for category-aware handling: it stays correct when the API adds new categories, and a missing `case` is reviewable / lint-able.
+
 ### Inspecting Multi-Wrapper Errors with `opensearchapi.Errors`
 
 Operations that can return more than one category of partial failure on the same response (today: `MSearch`, `MSearchTemplate`) sometimes do. The dispatch handler applies a runtime-collapse rule:
@@ -278,7 +291,7 @@ func handleMSearchError(err error) {
             metrics.ShardFailures.Add(int64(e.FailedShards))
         case *opensearchapi.MultiSearchItemError:
             for _, item := range e.Items {
-                log.Printf("sub-query %d failed: %s", item.Index, item.Error.Reason)
+                log.Printf("sub-query %d failed (status=%d)", item.Index, item.Status)
             }
         default:
             log.Printf("non-partial msearch error: %v", e)
@@ -289,11 +302,13 @@ func handleMSearchError(err error) {
 
 `opensearchapi.Errors(nil)` returns `nil`. A non-partial `err` (transport, HTTP, decode) returns a single-element slice containing `err`. Adding a new wrapper category later is purely additive: a new `case` in the switch picks it up; the `default` keeps catching everything else.
 
-### Why a type switch, not `errors.As` or `Has`-style helpers
+### Why a type switch, not `errors.As`, `Has`, or per-Resp helpers
+
+The set of partial-failure categories grows as the OpenSearch API evolves -- a future server or client release can add a category today's call sites have never seen. A type switch over `opensearchapi.Errors(err)` makes that growth visible: static analysis and code review can grep for the switch and flag missing cases, and the `default` arm keeps existing call sites safe in the meantime. `errors.As(err, &target)` and `Has`-style helpers (e.g. `multierror.Contains`, `errors.Has`) only answer "did _this_ category happen?" -- they cannot tell a call site that a _new_ category appeared and is being silently dropped, because the categories of interest are arguments rather than cases.
 
-The set of partial-failure categories grows as the OpenSearch API evolves -- a future server or client release can add a category today's call sites have never seen. A type switch over `opensearchapi.Errors(err)` makes that growth visible: static analysis and code review can grep for the switch and flag missing cases, and the `default` arm keeps existing call sites safe in the meantime. `errors.As(err, &target)` and HashiCorp-style helpers (`multierror.Contains`, `errors.Has`) only answer "did _this_ category happen?" -- they cannot tell a call site that a _new_ category appeared and is being silently dropped, because the categories of interest are arguments rather than cases.
+The per-Resp helper methods (`resp.BulkItemFailures()`, `resp.SearchShardFailures()`, `resp.WriteShardFailures()`, `resp.MultiSearchItemFailures()`) and the per-Resp `PartialFailures(mask)` aggregator suffer the same forward-compatibility problem: a call site only sees the categories whose helpers it explicitly invokes. They exist on the response types as engine machinery for the dispatch and remain available for focused inspection of a known category. New code should use the `for`/`switch` pattern shown above.
 
-Treat `As`/`Has` against the partial-failure error types as an antipattern: every call site that uses them becomes an audit liability the next time a category is added, because the omission is invisible to lint-time checks. The same reasoning applies to operations that today produce a single category -- preferring the type switch from day one means a future addition is purely additive rather than a silent behavior change.
+Treat `As`/`Has` and the per-Resp helpers against the partial-failure error types as an antipattern: every call site that uses them becomes an audit liability the next time a category is added, because the omission is invisible to lint-time checks. The same reasoning applies to operations that today produce a single category -- preferring the type switch from day one means a future addition is purely additive rather than a silent behavior change.
 
 ### Error Type Reference