Add generic Do[T] function for compile-time pointer enforcement (#809)

* Skip shard routing tests on OpenSearch < 2.2.0 with security

OpenSearch < 2.2.0 with the security plugin throws
java.io.OptionalDataException on shard-routed requests due to
non-thread-safe HashSet/HashMap in User serialization. Fixed in 2.2.0
by opensearch-project/security#1970 (50a94b47).

Widen the existing 2.1.0-only skip to cover all versions below 2.2.0.

Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>

* Add generic Do[T] function for compile-time pointer enforcement

Client.Do() accepts `any` for its dataPointer parameter, deferring
pointer validation to json.Unmarshal at runtime. Add a top-level
opensearch.Do[T]() function whose *T signature catches non-pointer
arguments at compile time. The generic wrapper is trivially inlined
by the compiler -- zero runtime overhead.

Mark Client.Do() with a Deprecated doc annotation to nudge callers
toward the safer alternative via staticcheck SA1019 and IDE tooling.
The method remains fully functional and will not be removed.

Convert the unexported `do` method in opensearchapi, plugins/security,
and plugins/ism to package-level generic functions, enforcing pointer
safety across all internal call sites.

Fixes: #808

Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>

* Improve transport test assertions

Replace magic status codes with http.StatusOK constants, use t.Context()
instead of manual context.WithCancel, and prefer require.Len/require.Positive
over less specific assertions.

Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>

* Use require.Error/require.NoError in ParseError test

Replace require.NotNil/require.Nil with the semantically correct error
assertion helpers.

Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>

* Add NoBody marker type and handle nil-body error responses

Introduce opensearch.NoBody as a marker type for Do[T] calls that expect
no response body, replacing the separate doRequest() function with a
unified dispatch path through the generic function.

Guard Do[T] against typed-nil pointers that would otherwise widen into a
non-nil any interface and reach json.Unmarshal. When dataPointer is nil
and the error response has no body, return a quoted-status error instead
of the misleading ErrUnexpectedEmptyBody.

Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>

---------

Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>

Author: sean-

CHANGELOG.md

@@ -7,6 +7,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 ### Added
 
 - Add `primary_terms_map` and `split_shards_metadata` fields to ClusterState index metadata for OpenSearch >=3.6.0 compatibility
+- Add generic `opensearch.Do[T]()` function for compile-time pointer enforcement on response types, preventing a class of bugs where non-pointer values are silently passed to `Client.Do()` and fail at runtime during JSON unmarshaling. Includes `opensearch.NoBody` marker type for calls that expect no response body, unifying all internal dispatch through a single generic path ([#809](https://github.com/opensearch-project/opensearch-go/pull/809))
 - Add `InsecureSkipVerify` config option to disable TLS certificate verification without constructing a custom `http.Transport`, preserving `DefaultTransport` connection pooling, HTTP/2, and timeout defaults ([#786](https://github.com/opensearch-project/opensearch-go/issues/786))
 - Add `DisableResponseBuffering` config option to skip eager `io.ReadAll` buffering of response bodies in `Perform()`, reducing per-request allocations and TTFB for proxy and streaming use cases ([#786](https://github.com/opensearch-project/opensearch-go/issues/786))
 - Add per-attempt `RequestTimeout` to bound individual HTTP round-trips, preventing indefinite hangs on stalled connections ([#786](https://github.com/opensearch-project/opensearch-go/issues/786))
@@ -153,13 +154,16 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
 ### Deprecated
 
+- Mark `Client.Do()` with a `Deprecated` doc annotation in favor of `opensearch.Do[T]()` for compile-time pointer safety; `Client.Do()` remains fully functional and will not be removed, but `staticcheck` SA1019 will nudge cross-package callers toward the safer generic alternative
+
 ### Removed
 
 ### Fixed
 
 - Fix bulk indexer HTML-escaping `_id` and `routing` values containing `<`, `>`, or `&` characters, causing OpenSearch to store escaped values (e.g., `\u003croot_account\u003e` stored instead of `<root_account>`), leading to duplicate documents, unreachable data on read-by-ID paths, and potential shard routing mismatches. Present since the `json.Marshal` migration in 2021 (commit `3da59092`). Replace `json.Marshal` with `json.NewEncoder` + `SetEscapeHTML(false)` in `opensearchutil.worker.writeMeta` and `opensearchutil.JSONReader`; replace per-worker `aux []byte` with `sync.Pool`-backed `*bytes.Buffer`; add table-driven test coverage for `writeMeta` edge cases and refactor remaining `TestBulkIndexer` subtests to table-driven `require`-based style ([#824](https://github.com/opensearch-project/opensearch-go/pull/824))
 - Fix pool replacement orphaning resurrection goroutines during node discovery, causing connections to become permanently dead with no active health checker ([#786](https://github.com/opensearch-project/opensearch-go/pull/786))
 - Extract `newMultiServerPoolFromClientWithLock` as single source of truth for Client-to-pool settings propagation ([#786](https://github.com/opensearch-project/opensearch-go/pull/786))
+- Skip shard routing integration tests on OpenSearch < 2.2.0 with security plugin due to server-side `OptionalDataException` from non-thread-safe User serialization (opensearch-project/security#1970)
 - Fix discovery pool wipe when all cluster nodes time out during `/_nodes/http` fan-out: parse `_nodes` metadata envelope and return `errDiscoveryEmpty` when `successful == 0`, preserving the existing connection pool for retry ([#821](https://github.com/opensearch-project/opensearch-go/pull/821))
 - Fix flaky `TestDefaultHealthCheck_RetryAfterMaxRetry`: replace wall-clock `time.Sleep` + `atomic.Int64` synchronization with context cancellation (`ctx.Done()`), and widen `maxRetryClusterHealth` to 5s so the baseline HTTP round-trip cannot race past the retry interval ([#787](https://github.com/opensearch-project/opensearch-go/pull/787))
 - Skip opensearchtransport integration tests on OpenSearch < 2.2.0 with security plugin due to server-side `OptionalDataException` from non-thread-safe User serialization (opensearch-project/security#1970)

USER_GUIDE.md

@@ -517,6 +517,7 @@ All `OPENSEARCH_GO_*` environment variables are evaluated once at client initial
 - [Advanced Index Actions](guides/advanced_index_actions.md)
 - [Index Templates](guides/index_template.md)
 - [Data Streams](guides/data_streams.md)
+- [Making Raw JSON REST Requests](guides/json.md)
 - [Request Routing](guides/routing.md)
 - [Cluster Health Checking](guides/cluster_health_checking.md)
 - [Node Discovery and Role Management](guides/node_discovery_and_roles.md)

error_test.go

@@ -318,11 +318,11 @@ func TestError(t *testing.T) {
 
 		// Parse the error
 		err := opensearch.ParseError(resp)
-		require.NotNil(t, err)
+		require.Error(t, err)
 
 		// Verify the body is still readable after ParseError
 		body, readErr := io.ReadAll(resp.Body)
-		require.Nil(t, readErr, "body should be readable after ParseError")
+		require.NoError(t, readErr, "body should be readable after ParseError")
 		require.NotEmpty(t, body, "body should not be empty after ParseError")
 
 		// Verify the body content matches the original

guides/json.md

@@ -1,5 +1,6 @@
 - [Making Raw JSON REST Requests](#making-raw-json-rest-requests)
   - [Setup](#setup)
+  - [Using Do for Typed Responses](#using-do-for-typed-responses)
   - [GET](#get)
   - [PUT](#put)
   - [POST](#post)
@@ -17,12 +18,14 @@ Let's create a client instance:
 package main
 
 import (
+	"context"
 	"fmt"
 	"io"
 	"net/http"
 	"os"
 	"strings"
 
+	"github.com/opensearch-project/opensearch-go/v4"
 	"github.com/opensearch-project/opensearch-go/v4/opensearchapi"
 )
 
@@ -40,6 +43,63 @@ func example() error {
 	}
 ```
 
+## Using Do for Typed Responses
+
+When you need to call an API that `opensearchapi` doesn't cover — plugin endpoints, newly released server APIs, or internal custom endpoints — use `opensearch.Do()` to execute a request and automatically unmarshal the JSON response into a struct.
+
+The `Client.Do()` method accepts `any` for its response parameter, which means passing a non-pointer compiles but fails at runtime during JSON unmarshaling. The generic `opensearch.Do[T]()` function catches this mistake at compile time. `Client.Do()` is marked with a `Deprecated` doc annotation to steer callers toward the safer alternative — it remains fully functional and will not be removed, but `staticcheck` SA1019 will flag cross-package usage as a nudge.
+
+First, define a request type that satisfies `opensearch.Request`:
+
+```go
+	// customReq wraps opensearch.BuildRequest to satisfy the opensearch.Request interface.
+	type customReq struct {
+		method string
+		path   string
+		body   io.Reader
+	}
+
+	func (r customReq) GetRequest() (*http.Request, error) {
+		return opensearch.BuildRequest(r.method, r.path, r.body, nil, nil)
+	}
+```
+
+Then use `opensearch.Do` to call the endpoint with a typed response:
+
+```go
+	type PluginStatusResp struct {
+		Status  string `json:"status"`
+		Version string `json:"version"`
+	}
+
+	ctx := context.Background()
+
+	// Preferred: opensearch.Do[T] enforces *T at compile time.
+	var pluginStatus PluginStatusResp
+	req := customReq{method: http.MethodGet, path: "/_plugins/my_plugin/status"}
+	resp, err := opensearch.Do(ctx, client.Client, req, &pluginStatus)
+	if err != nil {
+		return err
+	}
+	fmt.Printf("plugin status: %s (v%s), http: %d\n", pluginStatus.Status, pluginStatus.Version, resp.StatusCode)
+```
+
+If you pass a non-pointer value to `opensearch.Do`, the compiler rejects it:
+
+```go
+	// Compile error: cannot use pluginStatus (variable of type PluginStatusResp)
+	// as *PluginStatusResp value in argument to opensearch.Do
+	resp, err := opensearch.Do(ctx, client.Client, req, pluginStatus)
+```
+
+The three levels of the client API, from lowest to highest:
+
+| Level | Function                                                           | Response handling                                         | When to use                                                |
+| ----- | ------------------------------------------------------------------ | --------------------------------------------------------- | ---------------------------------------------------------- |
+| Low   | `client.Perform(req)`                                              | Raw `*http.Response`; caller reads and closes body        | Proxying, streaming, full control needed                   |
+| Mid   | `opensearch.Do(ctx, client, req, &resp)`                           | Automatic JSON unmarshal with compile-time pointer safety | Plugin APIs, unsupported endpoints, custom `Request` types |
+| High  | `client.Search(ctx, req)` / `client.Indices.Create(ctx, req)` etc. | Fully typed request and response                          | Standard OpenSearch APIs                                   |
+
 ## GET
 
 The following example returns the server version information via `GET /`.

opensearch.go

@@ -381,7 +381,12 @@ func (c *Client) Perform(req *http.Request) (*http.Response, error) {
 	return c.Transport.Perform(req)
 }
 
-// Do gets and performs the request. It also tries to parse the response into the dataPointer
+// Do gets and performs the request. It also tries to parse the response into the dataPointer.
+//
+// Deprecated: Use [Do] instead, which enforces that dataPointer is a pointer at compile time.
+// Client.Do accepts any, so passing a non-pointer compiles but fails at runtime during JSON
+// unmarshaling. The method remains fully functional and will not be removed; this annotation
+// exists to steer callers toward the safer generic alternative.
 func (c *Client) Do(ctx context.Context, req Request, dataPointer any) (*Response, error) {
 	httpReq, err := req.GetRequest()
 	if err != nil {
@@ -420,6 +425,24 @@ func (c *Client) Do(ctx context.Context, req Request, dataPointer any) (*Respons
 	return response, nil
 }
 
+// NoBody is a marker type for [Do] calls that expect no response body.
+// Pass (*NoBody)(nil) to skip JSON unmarshaling while retaining compile-time
+// pointer enforcement.
+type NoBody struct{}
+
+// Do is a generic version of [Client.Do] that enforces dataPointer as a pointer at compile time.
+// It delegates to [Client.Do] after the type system has guaranteed *T.
+//
+// A nil dataPointer is forwarded as untyped nil so that [Client.Do] skips
+// unmarshalling. This prevents a typed nil (e.g. (*MyResp)(nil)) from being
+// widened into a non-nil any interface that would reach [json.Unmarshal].
+func Do[T any](ctx context.Context, c *Client, req Request, dataPointer *T) (*Response, error) {
+	if dataPointer == nil {
+		return c.Do(ctx, req, nil)
+	}
+	return c.Do(ctx, req, dataPointer)
+}
+
 // Metrics returns the client metrics.
 func (c *Client) Metrics() (opensearchtransport.Metrics, error) {
 	if mt, ok := c.Transport.(opensearchtransport.Measurable); ok {

opensearch_internal_test.go

@@ -218,6 +218,35 @@ func TestClientInterfe(t *testing.T) {
 		assert.NotNil(t, resp)
 	})
 
+	t.Run("Generic Do()", func(t *testing.T) {
+		c, err := NewClient(Config{Transport: mockhttp.NewRoundTripFunc(t, defaultRoundTripFunc)})
+		require.NoError(t, err)
+
+		type versionInfo struct {
+			Number       string `json:"number"`
+			Distribution string `json:"distribution"`
+		}
+		type rootResp struct {
+			Version versionInfo `json:"version"`
+		}
+
+		var got rootResp
+		resp, err := Do(t.Context(), c, testReq{Path: "/"}, &got)
+		require.NoError(t, err)
+		require.NotNil(t, resp)
+		require.Equal(t, "1.0.0", got.Version.Number)
+		require.Equal(t, "opensearch", got.Version.Distribution)
+	})
+
+	t.Run("Generic Do() nil NoBody pointer", func(t *testing.T) {
+		c, err := NewClient(Config{Transport: mockhttp.NewRoundTripFunc(t, defaultRoundTripFunc)})
+		require.NoError(t, err)
+
+		resp, err := Do[NoBody](t.Context(), c, testReq{Path: "/"}, nil)
+		require.NoError(t, err)
+		require.NotNil(t, resp)
+	})
+
 	t.Run("Do() GetRequest error", func(t *testing.T) {
 		c, err := NewClient(Config{Transport: mockhttp.NewRoundTripFunc(t, defaultRoundTripFunc)})
 		require.NoError(t, err)

opensearchapi/api_aliases.go

@@ -20,7 +20,7 @@ func (c Client) Aliases(ctx context.Context, req AliasesReq) (*AliasesResp, erro
 		data AliasesResp
 		err  error
 	)
-	if data.response, err = c.do(ctx, req, &data); err != nil {
+	if data.response, err = do(ctx, &c, req, &data); err != nil {
 		return &data, err
 	}
 

opensearchapi/api_bulk.go

@@ -21,7 +21,7 @@ func (c Client) Bulk(ctx context.Context, req BulkReq) (*BulkResp, error) {
 		data BulkResp
 		err  error
 	)
-	if data.response, err = c.do(ctx, req, &data); err != nil {
+	if data.response, err = do(ctx, &c, req, &data); err != nil {
 		return &data, err
 	}