Skip to content

Add Flush(context.Context) to BulkIndexer; replace channel with ring … - #930

Open
LeeFred3042U wants to merge 1 commit into
opensearch-project:mainfrom
LeeFred3042U:flushFunction
Open

Add Flush(context.Context) to BulkIndexer; replace channel with ring …#930
LeeFred3042U wants to merge 1 commit into
opensearch-project:mainfrom
LeeFred3042U:flushFunction

Conversation

@LeeFred3042U

Copy link
Copy Markdown

Description

This PR implements a Flush(context.Context) error method on the BulkIndexer interface to allow callers to drain all submitted items without closing the indexer.

These were replaced:

  • The internal channel queue (bi.queue) was replaced with a lock-free ring buffer backed by three monotonic watermark counters (claimSeq, publishSeq, commitSeq).
  • Per-worker channel loops (worker.run) were removed in favor of a single, ordered sequencer goroutine.

Here's what changed:

  • Close() now sets an atomic closing flag and delegates directly to Flush(ctx) rather than closing a channel. This eliminates the "send on closed channel" panic if Add() is called after Close().
  • Add() now safely checks the closing flag and returns an error on a closed indexer, rather than panicking.
  • Context cancellation paths in Close() and Add() have been preserved and made fully deterministic.

here was what was added:

  • QueueSize field to BulkIndexerConfig (to size the underlying ring buffer, internally rounding up to the next power of two).
  • 6 new deterministic lifecycle tests in bulk_indexer_internal_test.go to verify concurrent flushing, reuse, and context cancellations.
  • A Flush release note entry to CHANGELOG.md.

Issues Resolved

Closes [#336]

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.35088% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.75%. Comparing base (ca3b71a) to head (1458208).
⚠️ Report is 49 commits behind head on main.

Files with missing lines Patch % Lines
opensearchutil/bulk_indexer.go 90.35% 7 Missing and 4 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #930      +/-   ##
==========================================
+ Coverage   48.70%   48.75%   +0.05%     
==========================================
  Files         637      637              
  Lines       61404    61459      +55     
==========================================
+ Hits        29904    29963      +59     
+ Misses      29749    29746       -3     
+ Partials     1751     1750       -1     
Flag Coverage Δ
integration 28.91% <61.40%> (+0.06%) ⬆️
unit 47.37% <90.35%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
opensearchutil/bulk_indexer.go 89.94% <90.35%> (+1.95%) ⬆️

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@LeeFred3042U
LeeFred3042U force-pushed the flushFunction branch 3 times, most recently from 041055e to eb37abb Compare July 8, 2026 23:09
…buffer

Signed-off-by: LeeFred3042U <109694901+LeeFred3042U@users.noreply.github.com>

Fix golangci-lint v2 warnings (gosec and modernize)

Signed-off-by: LeeFred3042U <109694901+LeeFred3042U@users.noreply.github.com>

Add Flush(context.Context) to BulkIndexer; replace channel with ring buffer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant