Skip to content

Commit 9cb06c1

Browse files
ryanyuansean-
andauthored
feat: update osapifix, a version-hop migration tool for opensearch-go v3->v4 (#934)
* chore: baseline osupgrade-v4-to-v5 tool before generic refactor Snapshot the working v4->v5 migration tool (rewrite + vet passes, surface diffing, hand-authored type/call tables) as a diffable baseline. Subsequent commits generalize this into a multi-version 'osupgrade' with a transition registry, source auto-detection, and cross-hop composition. Excludes the built binary (now gitignored). Signed-off-by: Ryan Yuan <ryan.yuan@crowdstrike.com> * feat: add osupgrade, a multi-version opensearch-go migration tool osupgrade migrates a Go module across opensearch-go major versions. It is a generic frontend over a registry of per-adjacent-hop migration tables: it auto-detects the source major from the consumer's imports, defaults the target to the newest known version, and composes the tables across the src->dst chain. v4->v5 ships as the one fully worked hop; adding v3->v4 / v2->v3 is additive (a surface file, a hop file, and a registry entry) with no engine changes. Why a registry instead of one binary per transition: the rewrite engine, surface diffing, and CLI are all version-agnostic, so only the hand-authored hop DATA is migration-specific. A single tool that chains hops covers multi-version jumps without duplicating the machinery or accreting version flags. Two passes, because a major bump has two distinct kinds of change: - rewrite (syntactic, pre-compile): source-shaped code does not compile against the target, so a type-checking pass cannot load it. rewrite edits the AST from a composed, type-aware delta to get the module compiling against the into any-typed testify sinks and fail at run time; go/analysis analyzers catch these, and -fix rewrites the safe cases. Runs after build; targets one version. Design: - transitions.go: version-neutral types (Major, Hop, methodRegroup) plus the surfaces and hops registries. - hop_v4_to_v5.go: the worked hop (type renames, method regroups, removed helpers, semantic followups). - compose.go: resolve(src,dst) chains hops in [src,dst). The field delta is an endpoint diff of the source/target surfaces (intermediate versions never matter to fields); only type renames and call-site rules are folded across hops as ORDERED composition, with a cross-talk guard against a later hop reusing a source-version type name. - detect.go: source major read from .go import paths, robust to go.mod listing both majors mid-migration or already naming the target. - applydelta.go / internal/surface: the version-agnostic type-aware rewriter and surface model; engine consumes only composed inputs, no version-specific globals. Identifiers are From/To throughout. Usage (v4 -> v5): osupgrade rewrite -w ./... go get github.com/opensearch-project/opensearch-go/v5 && go build ./... osupgrade vet -fix ./... Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com> Signed-off-by: Ryan Yuan <ryan.yuan@crowdstrike.com> * feat: add osapifix v3->v4 hop Add the v3 -> v4 transition to the version-agnostic osapifix engine, chainable with the existing v4 -> v5 hop to migrate v3 -> v5 directly. Purely additive data work; no engine changes. The v3 -> v4 boundary is quiet: the generated opensearchapi Client and sub-clients are byte-identical, so the hop carries no type renames, field dispositions, method regroups, or removed helpers. Field changes are left to the fail-loud "unclassified" default rather than pre-enumerated. The one structural change is the error model: opensearchapi.{Error,Err, RootCause,StringError} moved to the root opensearch package and were redesigned (v3 Error{Err;Status} == v4 StructError; v4 Error is a new simpler {Err string}; Err gained CausedBy). rewriteTypeRef rewrites only the type name, never the package qualifier, so this cross-package move cannot be expressed as a TypeRename without emitting non-compiling code. It is reported as a SemanticFollowup instead, proven from the v3 and v4 error.go sources. Validated against real v3.1.0 consumers (GO-CE customerstats, config): a full chained v3 -> v4 -> v5 rewrite of customerstats rebuilds cleanly through the intermediate v4 and migrates every production file, with zero unclassified fields. That run also surfaced a pre-existing engine limitation now documented: files behind custom build tags are loaded under the default constraints and skipped without warning. Signed-off-by: Ryan Yuan <ryan.yuan@crowdstrike.com> --------- Signed-off-by: Ryan Yuan <ryan.yuan@crowdstrike.com> Signed-off-by: Sean Chittenden <sean.chittenden@crowdstrike.com> Co-authored-by: Sean Chittenden <sean.chittenden@crowdstrike.com>
1 parent 341a3f8 commit 9cb06c1

10 files changed

Lines changed: 21888 additions & 2 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
66

77
### Added
88

9-
- Add `cmd/osapifix`, a tool that migrates a Go module across opensearch-go major versions. `osapifix rewrite -w ./...` detects the source major from the module's imports and rewrites old-major API shapes (type renames, field dispositions, method regroups) into the target's; the pass is purely syntactic (go/parser + astutil + go/printer), so it runs before the code compiles against the target, and writes are sandboxed to the target module directory via `os.Root`. `osapifix vet -fix ./...` then runs go/analysis analyzers that catch runtime type-assertion hazards -- the target's precise `*int64`/`*string` types flowing into `any` sinks such as testify's `Equal`, which compile cleanly but panic at run time. Each adjacent version transition is a hand-authored `Hop` keyed against two committed API surfaces (`surface_vN.json`); a migration resolves to the ordered list of hops between source and target, and new hops (v3 -> v4, v2 -> v3) are added as data without engine changes. Ships the v4 -> v5 hop. See `cmd/osapifix/README.md` ([#933](https://github.com/opensearch-project/opensearch-go/pull/933))
9+
- Add `cmd/osapifix`, a tool that migrates a Go module across opensearch-go major versions. `osapifix rewrite -w ./...` detects the source major from the module's imports and rewrites old-major API shapes (type renames, field dispositions, method regroups) into the target's; the pass is purely syntactic (go/parser + astutil + go/printer), so it runs before the code compiles against the target, and writes are sandboxed to the target module directory via `os.Root`. `osapifix vet -fix ./...` then runs go/analysis analyzers that catch runtime type-assertion hazards -- the target's precise `*int64`/`*string` types flowing into `any` sinks such as testify's `Equal`, which compile cleanly but panic at run time. Each adjacent version transition is a hand-authored `Hop` keyed against two committed API surfaces (`surface_vN.json`); a migration resolves to the ordered list of hops between source and target, and new hops (v2 -> v3) are added as data without engine changes. Ships the v3 -> v4 and v4 -> v5 hops, chainable to migrate v3 -> v5 directly. See `cmd/osapifix/README.md` ([#933](https://github.com/opensearch-project/opensearch-go/pull/933))
1010
- Add `Close()` to `opensearch.Client` and `opensearchapi.Client` for explicit teardown of background goroutines (node discovery, health/stats pollers, DNS refresh) and idle connections, without type-asserting the transport. Cache implicitly-constructed default clients (`opensearch.NewDefaultClient`, `opensearchapi.NewDefaultClient`, and the client `opensearchutil.NewBulkIndexer` builds when none is supplied) in a process-wide, refcounted, idle-TTL cache keyed by config hash, so identical default clients share one transport instead of leaking one set of goroutines and its connection pool per construction. User-built `opensearch.NewClient`/`opensearchapi.NewClient` clients never enter the cache. `opensearchutil.NewBulkIndexer` now closes the client it implicitly creates when the indexer is closed. Tune the idle eviction window with `OPENSEARCH_GO_DEFAULT_CLIENT_TTL` (default `16m`; `0` = never evict; a negative value disables caching so every call builds a fresh client) ([#893](https://github.com/opensearch-project/opensearch-go/issues/893))
1111
- Add client-side DNS caching, enabled by default on the built-in transport. Resolved addresses are cached and re-resolved on an interval (default 60s, mirroring the TTL AWS publishes for managed OpenSearch Service endpoints). When the resolver becomes briefly unreachable, the last-known-good address continues to be served until the resolver recovers, so transient resolver outages (e.g. a node-local DNS blip producing `dial tcp: lookup ...: i/o timeout`) no longer fail requests for already-resolved hosts. Tune or disable via the `DNSCacheRefresh`, `DNSDialTimeout`, `DNSKeepAlive`, and `DNSTimeout` fields on `opensearch.Config` (or `OPENSEARCH_GO_DNS_CACHE_REFRESH`, `OPENSEARCH_GO_DNS_DIAL_TIMEOUT`, `OPENSEARCH_GO_DNS_KEEP_ALIVE`, `OPENSEARCH_GO_DNS_TIMEOUT`); each follows the 0 = default, <0 = disable, >0 = explicit convention. Caching is installed only when no custom `Transport` is supplied; a caller-provided `Transport` is never modified. A host that resolves to multiple addresses races up to three of them concurrently (random start offset per connection) and takes the first to connect, spreading load and tolerating a dead address. Refresh re-resolves cached hosts sequentially, so `DNSTimeout` (default 10s) bounds each lookup to keep one hung resolution from stalling a refresh tick. The refresh goroutine is bound to the client's root context, so it is reclaimed both when `Close` is called and when `New` returns an error after the context is created. Because Go's resolver does not expose record TTLs, the refresh interval is a re-resolution cadence, not a per-record TTL. Exposes `DNSLookups`, `DNSCacheMisses`, and `DNSLookupErrors` counters via `Transport.Metrics()`
1212
- `cmd/osgen`: guard `json.RawMessage` in generated request/response types behind a checked-in allowlist (`cmd/osgen/rawmessage_allowlist.txt`). Because a `json.RawMessage` is the symptom of a type the generator could not resolve, a generator bug can silently widen the raw-JSON surface of the public API; generation now fails (non-zero exit) when any `json.RawMessage` use is not listed, including nested forms such as `[]json.RawMessage`, `map[string]json.RawMessage`, and `[][]json.RawMessage` (the leaf is detected at any wrapper depth). Entries are keyed `GoTypeName/jsonFieldName` (whole-response raw bodies use `<Prefix>Resp/-`, and map/array responses whose element type is unresolved use `<Prefix>Resp/[entries]` and `<Prefix>Resp/[records]`). Add `-update-raw-message-allowlist` to regenerate the allowlist from current output (sorted and grouped for minimal diffs), and `-allow-unlisted-raw-message` to downgrade the check to a warning ([#890](https://github.com/opensearch-project/opensearch-go/pull/890))

UPGRADING.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,10 @@ Upgrading from the hand-written v4 `opensearchapi/` package to the code-generate
1515

1616
The [`osapifix`](cmd/osapifix/README.md) tool automates most of this delta (import bump, type/method/field renames, value-to-pointer adjustments); see the [Automated migration](opensearchapi/UPGRADING_V4_TO_V5.md#automated-migration) section.
1717

18+
## v3 to v4 `opensearchapi/` surface delta
19+
20+
The v4 `opensearchapi/` package keeps the v3 client and its sub-clients unchanged, so most call sites only need the new import path. The one change needing a human hand is the error model, which moved out of `opensearchapi` into the root `opensearch` package ([`UPGRADING_V4.md`](UPGRADING_V4.md) covers it). For the tool-assisted delta - the import bump, the error-model follow-ups, and the response/transport fields the tool reports rather than rewrites - see the deep-dive at [`opensearchapi/UPGRADING_V3_TO_V4.md`](opensearchapi/UPGRADING_V3_TO_V4.md) and its [Automated migration](opensearchapi/UPGRADING_V3_TO_V4.md#automated-migration) section.
21+
1822
## Related references
1923

2024
- [`COMPATIBILITY.md`](COMPATIBILITY.md) - client/server version support matrix.

UPGRADING_V4.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,8 @@ if errors.As(err, &opensearchError) {
119119
}
120120
```
121121

122+
The [`osapifix`](cmd/osapifix/README.md) tool automates the v3 -> v4 import bump and reports the error-model move (which it cannot rewrite mechanically) as a follow-up; see the deep-dive at [`opensearchapi/UPGRADING_V3_TO_V4.md`](opensearchapi/UPGRADING_V3_TO_V4.md).
123+
122124
### StringError for Unknown JSON Responses
123125

124126
Version 4.0.0 returns `*opensearch.StringError` error type instead of `*fmt.wrapError` when response received from the server is an unknown JSON. For example, consider delete document API which returns an unknown JSON body when document is not found.

cmd/osapifix/README.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
osapifix rewrite -w ./...
77
```
88

9-
Today it supports the v4 -> v5 hop. Additional hops (v3 -> v4, v2 -> v3) are added as data, without engine changes.
9+
Today it supports the v3 -> v4 and v4 -> v5 hops, chainable to migrate v3 -> v5 directly. Additional hops (v2 -> v3) are added as data, without engine changes.
1010

1111
## Install
1212

@@ -107,6 +107,7 @@ go test ./...
107107
| ----------------------------------------- | --------------------------------------------------------------------------------- |
108108
| `plan_test.go` | `planChain` and `DeriveDelta` field dispositions, via synthetic v7/v8/v9 surfaces |
109109
| `delta_test.go` | Drift guards over every hop's type renames and field dispositions |
110+
| `hop_v3_to_v4_test.go` | v3 -> v4 version-specific facts |
110111
| `hop_v4_to_v5_test.go` | v4 -> v5 version-specific facts |
111112
| `detect_test.go` | Source detection, version parsing, directory resolution |
112113
| `internal/surface/delta_internal_test.go` | Surface diffing internals |
@@ -115,3 +116,4 @@ go test ./...
115116

116117
- `vet` analyzers are v5-specific (`TypedAssertAnalyzer`) and target a single version; they do not chain across hops.
117118
- A module importing multiple majors migrates from the lowest; per-import-site source selection is not implemented.
119+
- Files behind custom build tags (`//go:build <tag>`) are loaded under the default build constraints, so they are not rewritten and are skipped without warning. Migrate those files by hand.

cmd/osapifix/hop_v3_to_v4.go

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
// SPDX-License-Identifier: Apache-2.0
2+
//
3+
// The OpenSearch Contributors require contributions made to
4+
// this file be licensed under the Apache-2.0 license or a
5+
// compatible open source license.
6+
7+
package main
8+
9+
// hop_v3_to_v4.go is the hand-authored v3 -> v4 migration data. It follows the
10+
// shape of hop_v4_to_v5.go (the fully worked example); see that file for the
11+
// rationale behind each field of Hop.
12+
//
13+
// The v3 -> v4 boundary is far quieter than v4 -> v5. The generated opensearchapi
14+
// Client and its sub-clients are byte-identical across the two versions, so there
15+
// are no method regroups and no *Req argument changes. The one real structural
16+
// change is the error model: the API error types moved out of the opensearchapi
17+
// package into the root opensearch package and were redesigned. That move cannot
18+
// be rewritten mechanically (see SemanticFollowups), so it is reported to the
19+
// operator rather than encoded as a rename. Everything else the surface diff
20+
// derives on its own (notably ~90 fields that became pointers), and any request/
21+
// response field that genuinely vanished is left to the fail-loud "unclassified"
22+
// default until a real consumer proves a ruling is needed.
23+
24+
const (
25+
v3api = "github.com/opensearch-project/opensearch-go/v3/opensearchapi"
26+
v3transport = "github.com/opensearch-project/opensearch-go/v3/opensearchtransport"
27+
)
28+
29+
// hopV3toV4 is the complete v3 -> v4 transition, registered in hops (see
30+
// transitions.go). Unlike hopV4toV5 most tables are empty: the surface diff plus
31+
// the fail-loud default cover the mechanical changes, and the sole hand ruling is
32+
// the error-model followup.
33+
//
34+
//nolint:gochecknoglobals // immutable data table; mirrors hopV4toV5
35+
var hopV3toV4 = Hop{
36+
From: 3,
37+
To: 4,
38+
39+
// TypeRenames: none. The four error types (Error, Err, RootCause, StringError)
40+
// change PACKAGE (opensearchapi -> root opensearch), not just name. The engine's
41+
// rewriteTypeRef rewrites only the type name, never the package qualifier, and
42+
// rewriteImports only version-bumps an import prefix - so a cross-package rename
43+
// would pass the drift guard yet emit non-compiling code. They are handled as
44+
// SemanticFollowups instead. Every other type keeps its name and package.
45+
TypeRenames: nil,
46+
47+
// FieldDispositions: none up front. The genuinely vanished fields
48+
// (opensearchapi.*Resp.Indices, which became an unexported field plus a
49+
// GetIndices() accessor; and opensearchtransport.Connection's dropped liveness
50+
// fields) are left to the fail-loud "unclassified" default, matching hopV4toV5's
51+
// discipline: a ruling is added only when a real consumer actually touches the
52+
// field, proven from source.
53+
FieldDispositions: nil,
54+
55+
// MethodRegroups: none. The opensearchapi Client and sub-clients are identical
56+
// v3 -> v4; no call site moves.
57+
MethodRegroups: nil,
58+
59+
// RemovedHelpers: none. No package-level opensearchapi helper was removed.
60+
RemovedHelpers: nil,
61+
62+
// SemanticFollowups: the error-model redesign, which cannot be rewritten
63+
// mechanically (cross-package move + shape change). Proven from the v3 and v4
64+
// error.go sources.
65+
SemanticFollowups: []string{
66+
"Error types moved from the opensearchapi package to the root opensearch package: " +
67+
"opensearchapi.{Error,Err,RootCause,StringError} are now opensearch.{Error,Err,RootCause,StringError}. " +
68+
"Update the imports and package qualifiers by hand - osapifix cannot rewrite a package qualifier.",
69+
"opensearchapi.Error changed shape: the v3 Error{Err Err; Status int} is now opensearch.StructError; " +
70+
"the v4 opensearch.Error is a different, simpler type ({Err string}). " +
71+
"Re-point type switches and assertions to opensearch.StructError where you decoded the detailed error.",
72+
"opensearch.Err gained an optional CausedBy *CausedBy field (nested causes); " +
73+
"existing field access is unaffected, but new nested-cause data is now available.",
74+
},
75+
}

cmd/osapifix/hop_v3_to_v4_test.go

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
// SPDX-License-Identifier: Apache-2.0
2+
//
3+
// The OpenSearch Contributors require contributions made to
4+
// this file be licensed under the Apache-2.0 license or a
5+
// compatible open source license.
6+
7+
package main
8+
9+
import (
10+
"strings"
11+
"testing"
12+
13+
"github.com/stretchr/testify/require"
14+
)
15+
16+
// hop_v3_to_v4_test.go pins the concrete v3->v4 facts the rewriter depends on.
17+
// The v3->v4 hop carries no type/field/method tables (the surface diff plus the
18+
// fail-loud default cover it), so these assertions pin the diff-derived behavior
19+
// instead: the pointer-wraps the diff produces, the error-model followup, the
20+
// deliberate fail-loud handling of the redesigned response fields, and the fact
21+
// that the hop chains cleanly onto v4->v5.
22+
23+
// planV3toV4 returns the single hop plan for v3->v4.
24+
func planV3toV4(t *testing.T) hopPlan {
25+
t.Helper()
26+
plans, err := planChain(3, 4)
27+
require.NoError(t, err)
28+
require.Len(t, plans, 1, "v3->v4 is a single hop")
29+
return plans[0]
30+
}
31+
32+
// TestHopV3toV4_KnownChanges verifies the v3->v4 delta carries the pointer-wraps
33+
// the diff derives and the error-model followup, and that no type/field tables
34+
// were hand-authored for this hop (all changes are diff-derived or fail-loud).
35+
func TestHopV3toV4_KnownChanges(t *testing.T) {
36+
p := planV3toV4(t)
37+
38+
// No hand-authored renames or dispositions: the diff plus the fail-loud
39+
// default carry this hop.
40+
require.Empty(t, p.renames, "v3->v4 declares no type renames")
41+
require.Empty(t, p.regroups, "v3->v4 declares no method regroups")
42+
43+
// A field that became a pointer in v4 is auto-detected as a pointerWrap by the
44+
// surface diff (no table entry needed): CatNodesItemResp.CPU int -> *int.
45+
assertChangeKind(t, p.delta.Structs[v3api+".CatNodesItemResp"].Changes, "CPU", "pointerWrap")
46+
47+
// The error-model move is reported as a semantic followup, never rewritten.
48+
require.True(t, containsSubstr(p.followups, "opensearchapi.{Error,Err,RootCause,StringError}"),
49+
"v3->v4 must report the error-package move as a followup")
50+
require.True(t, containsSubstr(p.followups, "opensearch.StructError"),
51+
"v3->v4 must flag the Error -> StructError shape change")
52+
}
53+
54+
// TestHopV3toV4_FailLoudForRedesignedFields asserts that the response fields
55+
// redesigned away in v4 (opensearchapi.*Resp.Indices, replaced by an unexported
56+
// field + GetIndices() accessor) are reported as "unclassified" rather than
57+
// silently dropped or wrongly renamed. This is deliberate: no disposition is
58+
// authored up front, so the fail-loud default surfaces the field only if a real
59+
// consumer actually reads it, at which point a proven ruling is added.
60+
func TestHopV3toV4_FailLoudForRedesignedFields(t *testing.T) {
61+
d := planV3toV4(t).delta
62+
63+
// Indices vanished on all six *Resp types; each must be unclassified.
64+
for _, typ := range []string{
65+
"AliasGetResp", "IndicesGetResp", "IndicesRecoveryResp",
66+
"MappingFieldResp", "MappingGetResp", "SettingsGetResp",
67+
} {
68+
assertChangeKind(t, d.Structs[v3api+"."+typ].Changes, "Indices", "unclassified")
69+
}
70+
71+
// opensearchtransport.Connection dropped its liveness fields; also fail-loud.
72+
conn := d.Structs[v3transport+".Connection"].Changes
73+
for _, f := range []string{"DeadSince", "Failures", "IsDead"} {
74+
assertChangeKind(t, conn, f, "unclassified")
75+
}
76+
}
77+
78+
// TestHopV3toV4_ChainsToV5 verifies the registered hop composes: a v3->v5 request
79+
// yields the two adjacent hops in order, applied serially by the driver.
80+
func TestHopV3toV4_ChainsToV5(t *testing.T) {
81+
plans, err := planChain(3, 5)
82+
require.NoError(t, err)
83+
require.Len(t, plans, 2, "v3->v5 chains v3->v4 then v4->v5")
84+
require.Equal(t, [2]Major{3, 4}, [2]Major{plans[0].from, plans[0].to})
85+
require.Equal(t, [2]Major{4, 5}, [2]Major{plans[1].from, plans[1].to})
86+
}
87+
88+
// containsSubstr reports whether any element of s contains sub.
89+
func containsSubstr(s []string, sub string) bool {
90+
for _, v := range s {
91+
if strings.Contains(v, sub) {
92+
return true
93+
}
94+
}
95+
return false
96+
}

cmd/osapifix/main.go

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,9 @@ import (
7878
// cmd/gensurface and re-embed when bumping the pinned opensearch-go versions.
7979
// Register each embedded surface in the surfaces map (transitions.go).
8080
//
81+
//go:embed surface_v3.json
82+
var surfaceV3JSON []byte
83+
8184
//go:embed surface_v4.json
8285
var surfaceV4JSON []byte
8386

0 commit comments

Comments
 (0)