Skip to content

Commit e2d5dd3

Browse files
committed
feat(opensearchtransport): add OperationClassifier and bit-packed OperationID
Add a zero-allocation HTTP method+path classifier that reuses the existing routeTrie to map requests to structured OperationID values. Enables transparent metrics, tracing, and access-control middleware at the http.RoundTripper layer without per-operation wrapper code. OperationID is a bit-packed int64 encoding R/W flag, category, and minor operation. Masking helpers (IsWrite, Category, Minor) support efficient bitwise filtering. String() returns Prometheus-friendly labels. OperationClassifier is built from the canonical route table and is safe for concurrent use. Returns OpOther for unrecognized patterns. Adds OperationID field to trieLeaf/trieMatch, OpID() to the Route interface, and .Op() to RouteBuilder. All 124 routes are tagged. Ref: opensearch-project#816 Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com>
1 parent 0481faa commit e2d5dd3

11 files changed

Lines changed: 1876 additions & 162 deletions

CHANGELOG.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,12 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
9999
- `Config.ReturnQueryErrors` defaults to `false` in v4 (opt-in), will flip to `true` in v5
100100
- `OPENSEARCH_GO_PARTIAL_QUERY_ERRORS` environment variable overrides `Config.ReturnQueryErrors` at runtime
101101
- Both `(resp, error)` are non-nil on partial failure -- response is fully populated
102+
- Add `OperationClassifier` for zero-allocation HTTP method+path to `OperationID` mapping ([#816](https://github.com/opensearch-project/opensearch-go/issues/816))
103+
- Bit-packed `OperationID` (int64) encoding R/W flag, category, and minor operation
104+
- Masking helpers: `IsWrite`, `IsRead`, `Category`, `Minor`
105+
- `String()` returns Prometheus-friendly labels (e.g., `"search"`, `"bulk"`, `"doc_get"`)
106+
- Reuses existing `routeTrie` for O(path-segments) lookup, safe for concurrent use
107+
- Enables transparent metrics/tracing middleware at the `http.RoundTripper` layer
102108
- Transport automatically sets `max_concurrent_shard_requests` query parameter on search requests routed through a coordinator node
103109
- Value derived from a cluster-wide aggregate of all polled nodes' search pool wait-time and completion deltas, clamped to `[floor, cap]` (default: 5–256)
104110
- Cluster-wide signal correctly models data-node fan-out capacity: single hot nodes are diluted by healthy peers, and MCSR only drops when aggregate cluster pressure rises

USER_GUIDE.md

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -457,6 +457,47 @@ Key settings to verify on any custom transport:
457457

458458
`Clone()` copies all of these. If you must build from scratch (e.g. for a non-`*http.Transport` round tripper), set at minimum `ForceAttemptHTTP2: true` and `MaxIdleConnsPerHost` >= your expected concurrency.
459459

460+
## Operation Classifier
461+
462+
The `opensearchtransport.OperationClassifier` maps HTTP method+path pairs to structured `OperationID` values. This enables transparent metrics, tracing, or access-control middleware at the `http.RoundTripper` layer without per-operation wrapper code.
463+
464+
```go
465+
import "github.com/opensearch-project/opensearch-go/v4/opensearchtransport"
466+
467+
// Build once, reuse across requests. Safe for concurrent use.
468+
classifier := opensearchtransport.NewOperationClassifier()
469+
470+
// In an http.RoundTripper:
471+
func (t *MetricsTransport) RoundTrip(req *http.Request) (*http.Response, error) {
472+
op := t.classifier.Classify(req.Method, req.URL.Path)
473+
474+
start := time.Now()
475+
resp, err := t.next.RoundTrip(req)
476+
duration := time.Since(start)
477+
478+
status := 0
479+
if resp != nil {
480+
status = resp.StatusCode
481+
}
482+
t.histogram.WithLabelValues(op.String(), strconv.Itoa(status)).Observe(
483+
float64(duration.Milliseconds()),
484+
)
485+
return resp, err
486+
}
487+
```
488+
489+
`OperationID` is a bit-packed `int64` with masking helpers:
490+
491+
```go
492+
op := classifier.Classify("POST", "/my-index/_search")
493+
op.String() // "search"
494+
op.IsWrite() // false
495+
op.IsRead() // true
496+
op.Category() // CatSearch
497+
```
498+
499+
Returns `OpOther` for unrecognized patterns.
500+
460501
## Debugging
461502

462503
Set the `OPENSEARCH_GO_DEBUG` environment variable to enable debug logging for connection management, node discovery, and request routing. Debug output is written to stderr.

opensearchtransport/classify.go

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
// SPDX-License-Identifier: Apache-2.0
2+
//
3+
// The OpenSearch Contributors require contributions made to
4+
// this file be licensed under the Apache-2.0 license or a
5+
// compatible open source license.
6+
//
7+
// Modifications Copyright OpenSearch Contributors. See
8+
// GitHub history for details.
9+
10+
// Licensed to Elasticsearch B.V. under one or more contributor
11+
// license agreements. See the NOTICE file distributed with
12+
// this work for additional information regarding copyright
13+
// ownership. Elasticsearch B.V. licenses this file to you under
14+
// the Apache License, Version 2.0 (the "License"); you may
15+
// not use this file except in compliance with the License.
16+
// You may obtain a copy of the License at
17+
//
18+
// http://www.apache.org/licenses/LICENSE-2.0
19+
//
20+
// Unless required by applicable law or agreed to in writing,
21+
// software distributed under the License is distributed on an
22+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
23+
// KIND, either express or implied. See the License for the
24+
// specific language governing permissions and limitations
25+
// under the License.
26+
27+
package opensearchtransport
28+
29+
// OperationClassifier maps HTTP method+path pairs to [OperationID] values.
30+
// It reuses the same route trie used by [MuxPolicy], so classification is
31+
// zero-allocation for well-formed paths.
32+
//
33+
// Build once with [NewOperationClassifier] and reuse across requests.
34+
type OperationClassifier struct {
35+
trie routeTrie
36+
}
37+
38+
// NewOperationClassifier builds a classifier from the default route table.
39+
// The returned classifier is safe for concurrent use.
40+
func NewOperationClassifier() *OperationClassifier {
41+
c := &OperationClassifier{}
42+
43+
// Use a nil-safe null policy for all routes — we only need the
44+
// operationID from the leaf, not a working policy.
45+
p := NewNullPolicy()
46+
47+
routes := buildClassifierRoutes(p)
48+
for _, r := range routes {
49+
rm := r.(*RouteMux)
50+
method, path, err := splitMuxPattern(rm.Pattern)
51+
if err != nil {
52+
continue
53+
}
54+
c.trie.add([]string{method}, path, rm.policy, rm.attrs, rm.poolName, rm.operationID)
55+
}
56+
57+
return c
58+
}
59+
60+
// Classify returns the [OperationID] for the given HTTP method and path.
61+
// Returns [OpOther] for unrecognized method+path combinations.
62+
func (c *OperationClassifier) Classify(method, path string) OperationID {
63+
m, ok := c.trie.match(method, path)
64+
if !ok {
65+
return OpOther
66+
}
67+
return m.operationID
68+
}
69+
70+
// buildClassifierRoutes constructs the route table with OperationID tags
71+
// but using a single shared null policy. This avoids creating real role-based
72+
// policies (which need live connections) for pure classification use.
73+
func buildClassifierRoutes(p Policy) []Route {
74+
r := roleRoutes{
75+
ingestWrite: p,
76+
ingestMgmt: p,
77+
searchRead: p,
78+
getRead: p,
79+
dataWrite: p,
80+
dataRefresh: p,
81+
dataFlush: p,
82+
dataForceMerge: p,
83+
dataMgmt: p,
84+
searchMgmt: p,
85+
warmMgmt: p,
86+
}
87+
return buildRoleRoutes(r)
88+
}
Lines changed: 225 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,225 @@
1+
// SPDX-License-Identifier: Apache-2.0
2+
//
3+
// The OpenSearch Contributors require contributions made to
4+
// this file be licensed under the Apache-2.0 license or a
5+
// compatible open source license.
6+
7+
package opensearchtransport_test
8+
9+
import (
10+
"net/http"
11+
"testing"
12+
13+
"github.com/stretchr/testify/assert"
14+
"github.com/stretchr/testify/require"
15+
16+
"github.com/opensearch-project/opensearch-go/v4/opensearchtransport"
17+
)
18+
19+
func TestOperationClassifier(t *testing.T) {
20+
t.Parallel()
21+
c := opensearchtransport.NewOperationClassifier()
22+
23+
tests := []struct {
24+
name string
25+
method string
26+
path string
27+
want opensearchtransport.OperationID
28+
}{
29+
// Bulk
30+
{"bulk POST", http.MethodPost, "/_bulk", opensearchtransport.OpBulk},
31+
{"bulk PUT", http.MethodPut, "/_bulk", opensearchtransport.OpBulk},
32+
{"bulk with index", http.MethodPost, "/my-index/_bulk", opensearchtransport.OpBulk},
33+
{"bulk stream", http.MethodPost, "/_bulk/stream", opensearchtransport.OpBulkStream},
34+
{"reindex", http.MethodPost, "/_reindex", opensearchtransport.OpReindex},
35+
36+
// Search
37+
{"search GET", http.MethodGet, "/_search", opensearchtransport.OpSearch},
38+
{"search POST", http.MethodPost, "/_search", opensearchtransport.OpSearch},
39+
{"search with index", http.MethodPost, "/events/_search", opensearchtransport.OpSearch},
40+
{"msearch", http.MethodPost, "/_msearch", opensearchtransport.OpMSearch},
41+
{"count", http.MethodGet, "/_count", opensearchtransport.OpCount},
42+
{"count with index", http.MethodPost, "/events/_count", opensearchtransport.OpCount},
43+
{"delete_by_query", http.MethodPost, "/events/_delete_by_query", opensearchtransport.OpDeleteByQuery},
44+
{"update_by_query", http.MethodPost, "/events/_update_by_query", opensearchtransport.OpUpdateByQuery},
45+
{"validate", http.MethodGet, "/_validate/query", opensearchtransport.OpValidate},
46+
{"rank_eval", http.MethodPost, "/_rank_eval", opensearchtransport.OpRankEval},
47+
{"search shards", http.MethodGet, "/_search_shards", opensearchtransport.OpSearchShards},
48+
{"field caps", http.MethodGet, "/_field_caps", opensearchtransport.OpFieldCaps},
49+
50+
// Templates
51+
{"search template", http.MethodPost, "/_search/template", opensearchtransport.OpSearchTemplate},
52+
{"search template with index", http.MethodPost, "/events/_search/template", opensearchtransport.OpSearchTemplate},
53+
{"msearch template", http.MethodPost, "/_msearch/template", opensearchtransport.OpMSearchTmpl},
54+
55+
// Scroll
56+
{"scroll get", http.MethodGet, "/_search/scroll", opensearchtransport.OpScrollGet},
57+
{"scroll post", http.MethodPost, "/_search/scroll", opensearchtransport.OpScrollGet},
58+
{"scroll delete", http.MethodDelete, "/_search/scroll", opensearchtransport.OpScrollDelete},
59+
60+
// PIT
61+
{"pit create", http.MethodPost, "/events/_search/point_in_time", opensearchtransport.OpPITCreate},
62+
{"pit delete", http.MethodDelete, "/_search/point_in_time", opensearchtransport.OpPITDelete},
63+
{"pit list", http.MethodGet, "/_search/point_in_time/_all", opensearchtransport.OpPITList},
64+
65+
// Document ops
66+
{"doc get", http.MethodGet, "/events/_doc/123", opensearchtransport.OpDocGet},
67+
{"doc exists", http.MethodHead, "/events/_doc/123", opensearchtransport.OpDocExists},
68+
{"doc index PUT", http.MethodPut, "/events/_doc/123", opensearchtransport.OpDocIndex},
69+
{"doc index POST", http.MethodPost, "/events/_doc", opensearchtransport.OpDocIndex},
70+
{"doc create", http.MethodPut, "/events/_create/123", opensearchtransport.OpDocCreate},
71+
{"doc update", http.MethodPost, "/events/_update/123", opensearchtransport.OpDocUpdate},
72+
{"doc delete", http.MethodDelete, "/events/_doc/123", opensearchtransport.OpDocDelete},
73+
{"source get", http.MethodGet, "/events/_source/123", opensearchtransport.OpDocSourceGet},
74+
{"source exists", http.MethodHead, "/events/_source/123", opensearchtransport.OpDocSourceExist},
75+
{"mget", http.MethodPost, "/_mget", opensearchtransport.OpMGet},
76+
{"termvectors", http.MethodGet, "/events/_termvectors", opensearchtransport.OpTermVectors},
77+
{"mtermvectors", http.MethodPost, "/_mtermvectors", opensearchtransport.OpMTermVectors},
78+
{"explain", http.MethodPost, "/events/_explain/123", opensearchtransport.OpExplain},
79+
80+
// Ingest
81+
{"ingest get", http.MethodGet, "/_ingest/pipeline/my-pipe", opensearchtransport.OpIngestGet},
82+
{"ingest get all", http.MethodGet, "/_ingest/pipeline", opensearchtransport.OpIngestGet},
83+
{"ingest create", http.MethodPut, "/_ingest/pipeline/my-pipe", opensearchtransport.OpIngestCreate},
84+
{"ingest delete", http.MethodDelete, "/_ingest/pipeline/my-pipe", opensearchtransport.OpIngestDelete},
85+
{"ingest simulate", http.MethodPost, "/_ingest/pipeline/my-pipe/_simulate", opensearchtransport.OpIngestSimulate},
86+
87+
// Maintenance
88+
{"refresh", http.MethodPost, "/_refresh", opensearchtransport.OpRefresh},
89+
{"refresh index", http.MethodPost, "/events/_refresh", opensearchtransport.OpRefresh},
90+
{"flush", http.MethodPost, "/_flush", opensearchtransport.OpFlush},
91+
{"flush synced", http.MethodPost, "/_flush/synced", opensearchtransport.OpFlush},
92+
{"forcemerge", http.MethodPost, "/_forcemerge", opensearchtransport.OpForceMerge},
93+
{"segments", http.MethodGet, "/_segments", opensearchtransport.OpSegments},
94+
{"cache clear", http.MethodPost, "/_cache/clear", opensearchtransport.OpCacheClear},
95+
{"recovery", http.MethodGet, "/events/_recovery", opensearchtransport.OpRecovery},
96+
{"shard stores", http.MethodGet, "/events/_shard_stores", opensearchtransport.OpShardStores},
97+
{"stats", http.MethodGet, "/_stats", opensearchtransport.OpStats},
98+
{"stats with metric", http.MethodGet, "/_stats/indexing", opensearchtransport.OpStats},
99+
{"stats with index", http.MethodGet, "/events/_stats", opensearchtransport.OpStats},
100+
101+
// Rethrottle
102+
{"reindex rethrottle", http.MethodPost, "/_reindex/task1/_rethrottle", opensearchtransport.OpReindexRethrottle},
103+
{"ubq rethrottle", http.MethodPost, "/_update_by_query/task1/_rethrottle", opensearchtransport.OpUBQRethrottle},
104+
{"dbq rethrottle", http.MethodPost, "/_delete_by_query/task1/_rethrottle", opensearchtransport.OpDBQRethrottle},
105+
106+
// Cluster info
107+
{"root GET", http.MethodGet, "/", opensearchtransport.OpClusterInfo},
108+
{"root HEAD", http.MethodHead, "/", opensearchtransport.OpClusterInfo},
109+
110+
// Unknown
111+
{"unrecognized path", http.MethodGet, "/_unknown/endpoint", opensearchtransport.OpOther},
112+
{"unrecognized method", "DESTROY", "/_search", opensearchtransport.OpOther},
113+
{"empty path", http.MethodGet, "", opensearchtransport.OpOther},
114+
}
115+
116+
for _, tt := range tests {
117+
t.Run(tt.name, func(t *testing.T) {
118+
t.Parallel()
119+
got := c.Classify(tt.method, tt.path)
120+
assert.Equal(t, tt.want, got, "Classify(%q, %q) = %s, want %s",
121+
tt.method, tt.path, got, tt.want)
122+
})
123+
}
124+
}
125+
126+
func TestOperationClassifier_ConcurrentSafety(t *testing.T) {
127+
t.Parallel()
128+
c := opensearchtransport.NewOperationClassifier()
129+
130+
const goroutines = 50
131+
done := make(chan struct{})
132+
for range goroutines {
133+
go func() {
134+
defer func() { done <- struct{}{} }()
135+
for range 500 {
136+
c.Classify(http.MethodPost, "/events/_search")
137+
c.Classify(http.MethodPost, "/_bulk")
138+
c.Classify(http.MethodGet, "/events/_doc/123")
139+
}
140+
}()
141+
}
142+
for range goroutines {
143+
<-done
144+
}
145+
}
146+
147+
func TestOperationID_Masking(t *testing.T) {
148+
t.Parallel()
149+
150+
t.Run("IsWrite", func(t *testing.T) {
151+
t.Parallel()
152+
assert.False(t, opensearchtransport.OpSearch.IsWrite())
153+
assert.False(t, opensearchtransport.OpDocGet.IsWrite())
154+
assert.True(t, opensearchtransport.OpBulk.IsWrite())
155+
assert.True(t, opensearchtransport.OpDocIndex.IsWrite())
156+
assert.True(t, opensearchtransport.OpDocDelete.IsWrite())
157+
})
158+
159+
t.Run("IsRead", func(t *testing.T) {
160+
t.Parallel()
161+
assert.True(t, opensearchtransport.OpSearch.IsRead())
162+
assert.True(t, opensearchtransport.OpDocGet.IsRead())
163+
assert.False(t, opensearchtransport.OpBulk.IsRead())
164+
assert.False(t, opensearchtransport.OpDocIndex.IsRead())
165+
})
166+
167+
t.Run("Category", func(t *testing.T) {
168+
t.Parallel()
169+
assert.Equal(t, opensearchtransport.CatSearch, opensearchtransport.OpSearch.Category())
170+
assert.Equal(t, opensearchtransport.CatSearch, opensearchtransport.OpMSearch.Category())
171+
assert.Equal(t, opensearchtransport.CatSearch, opensearchtransport.OpCount.Category())
172+
assert.Equal(t, opensearchtransport.CatBulk, opensearchtransport.OpBulk.Category())
173+
assert.Equal(t, opensearchtransport.CatDocRead, opensearchtransport.OpDocGet.Category())
174+
assert.Equal(t, opensearchtransport.CatDocWrite, opensearchtransport.OpDocIndex.Category())
175+
})
176+
177+
t.Run("IsSearchFamily", func(t *testing.T) {
178+
t.Parallel()
179+
isSearchFamily := func(op opensearchtransport.OperationID) bool {
180+
return op.Category() == opensearchtransport.CatSearch
181+
}
182+
assert.True(t, isSearchFamily(opensearchtransport.OpSearch))
183+
assert.True(t, isSearchFamily(opensearchtransport.OpMSearch))
184+
assert.True(t, isSearchFamily(opensearchtransport.OpCount))
185+
assert.False(t, isSearchFamily(opensearchtransport.OpBulk))
186+
assert.False(t, isSearchFamily(opensearchtransport.OpDocGet))
187+
})
188+
}
189+
190+
func TestOperationID_String(t *testing.T) {
191+
t.Parallel()
192+
193+
tests := []struct {
194+
op opensearchtransport.OperationID
195+
want string
196+
}{
197+
{opensearchtransport.OpSearch, "search"},
198+
{opensearchtransport.OpMSearch, "msearch"},
199+
{opensearchtransport.OpCount, "count"},
200+
{opensearchtransport.OpBulk, "bulk"},
201+
{opensearchtransport.OpBulkStream, "bulk_stream"},
202+
{opensearchtransport.OpReindex, "reindex"},
203+
{opensearchtransport.OpDocGet, "doc_get"},
204+
{opensearchtransport.OpDocIndex, "doc_index"},
205+
{opensearchtransport.OpDocDelete, "doc_delete"},
206+
{opensearchtransport.OpDocCreate, "doc_create"},
207+
{opensearchtransport.OpDocUpdate, "doc_update"},
208+
{opensearchtransport.OpScrollGet, "scroll_get"},
209+
{opensearchtransport.OpScrollDelete, "scroll_delete"},
210+
{opensearchtransport.OpRefresh, "refresh"},
211+
{opensearchtransport.OpFlush, "flush"},
212+
{opensearchtransport.OpForceMerge, "forcemerge"},
213+
{opensearchtransport.OpStats, "stats"},
214+
{opensearchtransport.OpClusterInfo, "cluster_info"},
215+
{opensearchtransport.OpPing, "ping"},
216+
{opensearchtransport.OpOther, "other"},
217+
}
218+
219+
for _, tt := range tests {
220+
t.Run(tt.want, func(t *testing.T) {
221+
t.Parallel()
222+
require.Equal(t, tt.want, tt.op.String())
223+
})
224+
}
225+
}

0 commit comments

Comments
 (0)