Skip to content

Allow to specify an ingest pipeline to be used before writing documents #47227

@robertpaschedag

Description

@robertpaschedag

Component(s)

exporter/opensearch

Is your feature request related to a problem? Please describe.

An optional ingest pipeline in opensearch would allow post-processing of the received documents (by the collector) and before writing the documents into opensearch. Options could be:

  • adjust mapping of incoming documents
  • enrich documents
  • trigger other pipelines / notifications
  • ...

Describe the solution you'd like

The opensearchclient bulk indexer (https://github.com/opensearch-project/opensearch-go/blob/main/opensearchutil/bulk_indexer.go#L86) already supports a pipeline option to be passed. The opensearchexporter? just needs to allow such an pipeline option and pass it to the client.

Describe alternatives you've considered

No response

Additional context

https://docs.opensearch.org/latest/ingest-pipelines/

Tip

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions