This patch introduces new S3 source connectors for different formats:
- Text (CSV, JSON)
- Parquet
Features
- No size limit for single object / file
- Tunable batch size for latency and throughput
- Asynchronous file discovery
- Support both UTF-16 and UTF-8 encoding
- S3 Endpoints: AWS, NetApp ONTAP, StorageGRID
Contributors
- @niluka-insta made their contribution in #18
- @zheguang made their contribution in #20
Contributors
zheguang and niluka-insta