The backfill feature allows adding new fields to existing Milvus collections by reading collection data, joining it with new field data, and writing binlog files directly to S3.
Core Components
- MilvusBackfill.scala - Main orchestrator (src/main/scala/operations/backfill/)
- BackfillConfig.scala - Configuration (Milvus connection, S3 settings)
- SegmentPartitioner.scala - Custom Spark partitioner for segment-aware distribution
Data Flow
- Read metadata - From Milvus snapshot (preferred) or client API
- Load data - Read backfill Parquet (pk, new_fields...) + collection data (pk, segment_id, row_offset)
- Join - Sort merge join on primary key to match new data with original rows
- Partition by segment - Use SegmentPartitioner for 1:1 segment-to-Spark-partition mapping
- Write - Each executor writes binlog files to S3 via MilvusLoonWriter (Storage V2 FFI)
Key Features
- Snapshot mode - Zero-client offline backfill using MilvusSnapshotReader (Milvus 2.6+)
- Segment preservation - Maintains original segment structure with segment_id and row_offset
- NULL handling - Supports partial backfill (unmatched PKs get NULL values)
- Storage V2 - Direct FFI/Loon writer for efficient binlog generation
Output
Writes to: s3://{bucket}/{root_path}/insert_log/{collectionID}/{partitionID}/{segmentID}/new_field/
Returns Either[BackfillError, BackfillResult] with per-segment row counts and manifest paths.
The backfill feature allows adding new fields to existing Milvus collections by reading collection data, joining it with new field data, and writing binlog files directly to S3.
Core Components
Data Flow
Key Features
Output
Writes to: s3://{bucket}/{root_path}/insert_log/{collectionID}/{partitionID}/{segmentID}/new_field/
Returns Either[BackfillError, BackfillResult] with per-segment row counts and manifest paths.