feat(sink): add Elasticsearch sink for document indexing#32
Closed
feat(sink): add Elasticsearch sink for document indexing#32
Conversation
- Adds Elasticsearch sink for indexing events as JSON documents - Feature-gated behind `sink-elasticsearch` flag - Uses `elasticsearch` crate (v8.19.0-alpha.1) with bulk indexing - Includes `ElasticsearchSinkConfigWithoutSecrets` for safe logging - Documents `routing` and `pipeline` payload extensions - Adds AnySink enum pattern for runtime sink dispatch - Integration tests use Elasticsearch testcontainer
…nly output - Make index field optional (can come from event.metadata["index"]) - Add resolve_index() for dynamic per-event index routing - Send only event.payload (no id, created_at, lsn, stream_id, metadata envelope) - Use DestinationError for publish failures, ConfigError for missing config - Remove verbose info! logging - Update tests for payload-only content and add metadata routing test
psteinroe
force-pushed
the
feat/sink-elasticsearch
branch
from
978a7c2 to
b35afc8
Compare
Use {"json_path": ..., "expression": ...} format instead of incorrect
{"key": ..., "source": ..., "value": ...} format.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <[email protected]>
Rely on HTTP status code for error detection. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Remove cloning in publish_events to improve performance. Events are now moved by value instead of borrowed and cloned. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
The borrowed index reference was keeping the event borrowed, preventing us from moving event.payload and event.id.id. Convert to String immediately to end the borrow. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ElasticsearchSinkthat indexes events as documentsevent.payloadas document body (with injectedidfield)Configuration
Routing
Index is resolved from:
indexfield in event metadata