Skip to content

Commit 041055e

Browse files
committed
Add Flush(context.Context) to BulkIndexer; replace channel with ring buffer
Signed-off-by: LeeFred3042U <109694901+LeeFred3042U@users.noreply.github.com> Fix golangci-lint v2 warnings (gosec and modernize) Signed-off-by: LeeFred3042U <109694901+LeeFred3042U@users.noreply.github.com> Add Flush(context.Context) to BulkIndexer; replace channel with ring buffer
1 parent ca3b71a commit 041055e

4 files changed

Lines changed: 481 additions & 96 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
66

77
### Added
88

9+
- Add `Flush(context.Context) error` to `BulkIndexer` interface: replaces the internal channel queue with a ring buffer and monotonic watermark so callers can drain all submitted items without closing the indexer. Fixes panic when `Add` is called after `Close` ([#336](https://github.com/opensearch-project/opensearch-go/issues/336))
910
- Add `Close()` to `opensearch.Client` and `opensearchapi.Client` for explicit teardown of background goroutines (node discovery, health/stats pollers, DNS refresh) and idle connections, without type-asserting the transport. Cache implicitly-constructed default clients (`opensearch.NewDefaultClient`, `opensearchapi.NewDefaultClient`, and the client `opensearchutil.NewBulkIndexer` builds when none is supplied) in a process-wide, refcounted, idle-TTL cache keyed by config hash, so identical default clients share one transport instead of leaking one set of goroutines and its connection pool per construction. User-built `opensearch.NewClient`/`opensearchapi.NewClient` clients never enter the cache. `opensearchutil.NewBulkIndexer` now closes the client it implicitly creates when the indexer is closed. Tune the idle eviction window with `OPENSEARCH_GO_DEFAULT_CLIENT_TTL` (default `16m`; `0` = never evict; a negative value disables caching so every call builds a fresh client) ([#893](https://github.com/opensearch-project/opensearch-go/issues/893))
1011
- Add client-side DNS caching, enabled by default on the built-in transport. Resolved addresses are cached and re-resolved on an interval (default 60s, mirroring the TTL AWS publishes for managed OpenSearch Service endpoints). When the resolver becomes briefly unreachable, the last-known-good address continues to be served until the resolver recovers, so transient resolver outages (e.g. a node-local DNS blip producing `dial tcp: lookup ...: i/o timeout`) no longer fail requests for already-resolved hosts. Tune or disable via the `DNSCacheRefresh`, `DNSDialTimeout`, `DNSKeepAlive`, and `DNSTimeout` fields on `opensearch.Config` (or `OPENSEARCH_GO_DNS_CACHE_REFRESH`, `OPENSEARCH_GO_DNS_DIAL_TIMEOUT`, `OPENSEARCH_GO_DNS_KEEP_ALIVE`, `OPENSEARCH_GO_DNS_TIMEOUT`); each follows the 0 = default, <0 = disable, >0 = explicit convention. Caching is installed only when no custom `Transport` is supplied; a caller-provided `Transport` is never modified. A host that resolves to multiple addresses races up to three of them concurrently (random start offset per connection) and takes the first to connect, spreading load and tolerating a dead address. Refresh re-resolves cached hosts sequentially, so `DNSTimeout` (default 10s) bounds each lookup to keep one hung resolution from stalling a refresh tick. The refresh goroutine is bound to the client's root context, so it is reclaimed both when `Close` is called and when `New` returns an error after the context is created. Because Go's resolver does not expose record TTLs, the refresh interval is a re-resolution cadence, not a per-record TTL. Exposes `DNSLookups`, `DNSCacheMisses`, and `DNSLookupErrors` counters via `Transport.Metrics()`
1112
- `cmd/osgen`: guard `json.RawMessage` in generated request/response types behind a checked-in allowlist (`cmd/osgen/rawmessage_allowlist.txt`). Because a `json.RawMessage` is the symptom of a type the generator could not resolve, a generator bug can silently widen the raw-JSON surface of the public API; generation now fails (non-zero exit) when any `json.RawMessage` use is not listed, including nested forms such as `[]json.RawMessage`, `map[string]json.RawMessage`, and `[][]json.RawMessage` (the leaf is detected at any wrapper depth). Entries are keyed `GoTypeName/jsonFieldName` (whole-response raw bodies use `<Prefix>Resp/-`, and map/array responses whose element type is unresolved use `<Prefix>Resp/[entries]` and `<Prefix>Resp/[records]`). Add `-update-raw-message-allowlist` to regenerate the allowlist from current output (sorted and grouped for minimal diffs), and `-allow-unlisted-raw-message` to downgrade the check to a warning ([#890](https://github.com/opensearch-project/opensearch-go/pull/890))

0 commit comments

Comments
 (0)