[META] Pull-based Ingestion

[yupeng9]

Please describe the end goal of this project

This is a meta issue for all work related to the pull-based ingestion to ingest from streaming sources like Kafka and Kinesis directly.

Supporting References

[RFC] Streaming ingestion (pull based)

Issues

Indexing Core

• [Feature Request] Pull-based ingestion source APIs #16927 Pull-based ingestion source APIs
• [Feature Request] A new IngestionEngine that can pull data from streaming sources. #16929 A new IngestionEngine that can pull data from streaming sources
• [Feature Request] Metrics for pull-based ingestion #17077 Metrics for pull-based ingestion
• [Feature Request] Tests for multi-node recovery #17081 Tests for multi-node recover
• [Feature Request] FGAC support for pull-based ingestion #17083 FGAC support for pull-based ingestion
• [Feature Request] Error handling for pull-based ingestion #17085 Error handling for pull-based ingestion
• [Feature Request] Benchmark for pull-based ingestion #17086 Benchmark for pull-based ingestion
• [Feature Request] Ingestion management APIs for pull-based ingestion #17442 Ingestion management APIs for pull-based ingestion
• [Feature Request] Offset management for pull-based ingestion #17318 Offset management for pull-based ingestion

Plugin

• [Feature Request] Kafka plugin that implements the ingestion source API #16928 Kafka plugin that implements the ingestion source API
• [Pull-based Ingestion] Add Kinesis plugin support #17615 Kinesis plugin that implements the ingestion source API

TODO: will add more issues as this project moves forward

Related component

Indexing

[krisfreedain]

Catch All Triage - 1, 2, 3