Skip to content

feat: support milvus backfill with milvus snapshot #74

Description

@shaoting-huang

The backfill feature allows adding new fields to existing Milvus collections by reading collection data, joining it with new field data, and writing binlog files directly to S3.

Core Components

  • MilvusBackfill.scala - Main orchestrator (src/main/scala/operations/backfill/)
  • BackfillConfig.scala - Configuration (Milvus connection, S3 settings)
  • SegmentPartitioner.scala - Custom Spark partitioner for segment-aware distribution

Data Flow

  1. Read metadata - From Milvus snapshot (preferred) or client API
  2. Load data - Read backfill Parquet (pk, new_fields...) + collection data (pk, segment_id, row_offset)
  3. Join - Sort merge join on primary key to match new data with original rows
  4. Partition by segment - Use SegmentPartitioner for 1:1 segment-to-Spark-partition mapping
  5. Write - Each executor writes binlog files to S3 via MilvusLoonWriter (Storage V2 FFI)

Key Features

  • Snapshot mode - Zero-client offline backfill using MilvusSnapshotReader (Milvus 2.6+)
  • Segment preservation - Maintains original segment structure with segment_id and row_offset
  • NULL handling - Supports partial backfill (unmatched PKs get NULL values)
  • Storage V2 - Direct FFI/Loon writer for efficient binlog generation

Output

Writes to: s3://{bucket}/{root_path}/insert_log/{collectionID}/{partitionID}/{segmentID}/new_field/

Returns Either[BackfillError, BackfillResult] with per-segment row counts and manifest paths.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions