Skip to content

Add Flush(context.Context) to BulkIndexer; replace channel with ring … #930

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
LeeFred3042U wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add Flush(context.Context) to BulkIndexer; replace channel with ring … #930

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)

### Added

- Add `Flush(context.Context) error` to `BulkIndexer` interface: replaces the internal channel queue with a ring buffer and monotonic watermark so callers can drain all submitted items without closing the indexer. Fixes panic when `Add` is called after `Close` ([#336](https://github.com/opensearch-project/opensearch-go/issues/336))
- Add `Close()` to `opensearch.Client` and `opensearchapi.Client` for explicit teardown of background goroutines (node discovery, health/stats pollers, DNS refresh) and idle connections, without type-asserting the transport. Cache implicitly-constructed default clients (`opensearch.NewDefaultClient`, `opensearchapi.NewDefaultClient`, and the client `opensearchutil.NewBulkIndexer` builds when none is supplied) in a process-wide, refcounted, idle-TTL cache keyed by config hash, so identical default clients share one transport instead of leaking one set of goroutines and its connection pool per construction. User-built `opensearch.NewClient`/`opensearchapi.NewClient` clients never enter the cache. `opensearchutil.NewBulkIndexer` now closes the client it implicitly creates when the indexer is closed. Tune the idle eviction window with `OPENSEARCH_GO_DEFAULT_CLIENT_TTL` (default `16m`; `0` = never evict; a negative value disables caching so every call builds a fresh client) ([#893](https://github.com/opensearch-project/opensearch-go/issues/893))
- Add client-side DNS caching, enabled by default on the built-in transport. Resolved addresses are cached and re-resolved on an interval (default 60s, mirroring the TTL AWS publishes for managed OpenSearch Service endpoints). When the resolver becomes briefly unreachable, the last-known-good address continues to be served until the resolver recovers, so transient resolver outages (e.g. a node-local DNS blip producing `dial tcp: lookup ...: i/o timeout`) no longer fail requests for already-resolved hosts. Tune or disable via the `DNSCacheRefresh`, `DNSDialTimeout`, `DNSKeepAlive`, and `DNSTimeout` fields on `opensearch.Config` (or `OPENSEARCH_GO_DNS_CACHE_REFRESH`, `OPENSEARCH_GO_DNS_DIAL_TIMEOUT`, `OPENSEARCH_GO_DNS_KEEP_ALIVE`, `OPENSEARCH_GO_DNS_TIMEOUT`); each follows the 0 = default, <0 = disable, >0 = explicit convention. Caching is installed only when no custom `Transport` is supplied; a caller-provided `Transport` is never modified. A host that resolves to multiple addresses races up to three of them concurrently (random start offset per connection) and takes the first to connect, spreading load and tolerating a dead address. Refresh re-resolves cached hosts sequentially, so `DNSTimeout` (default 10s) bounds each lookup to keep one hung resolution from stalling a refresh tick. The refresh goroutine is bound to the client's root context, so it is reclaimed both when `Close` is called and when `New` returns an error after the context is created. Because Go's resolver does not expose record TTLs, the refresh interval is a re-resolution cadence, not a per-record TTL. Exposes `DNSLookups`, `DNSCacheMisses`, and `DNSLookupErrors` counters via `Transport.Metrics()`
- `cmd/osgen`: guard `json.RawMessage` in generated request/response types behind a checked-in allowlist (`cmd/osgen/rawmessage_allowlist.txt`). Because a `json.RawMessage` is the symptom of a type the generator could not resolve, a generator bug can silently widen the raw-JSON surface of the public API; generation now fails (non-zero exit) when any `json.RawMessage` use is not listed, including nested forms such as `[]json.RawMessage`, `map[string]json.RawMessage`, and `[][]json.RawMessage` (the leaf is detected at any wrapper depth). Entries are keyed `GoTypeName/jsonFieldName` (whole-response raw bodies use `<Prefix>Resp/-`, and map/array responses whose element type is unresolved use `<Prefix>Resp/[entries]` and `<Prefix>Resp/[records]`). Add `-update-raw-message-allowlist` to regenerate the allowlist from current output (sorted and grouped for minimal diffs), and `-allow-unlisted-raw-message` to downgrade the check to a warning ([#890](https://github.com/opensearch-project/opensearch-go/pull/890))
Expand Down
Loading
Loading