bug: service-mode recovery falls back to 127.0.0.1 for old WAL segment metadata without QuorumInfo

[FeilongHou]

Summary

After upgrading a Milvus cluster from 2.6 to 3.0/master with Woodpecker service-mode WAL, Milvus can hang during Flush because StreamingNode cannot recover old Woodpecker WAL segments. The Woodpecker client resolves the old segment quorum to 127.0.0.1, then repeatedly fails to read segment 0 with dial tcp 127.0.0.1:443: connect: connection refused.

This looks like a backward-compatibility issue for old segment metadata that does not contain the newer embedded QuorumInfo / service-mode quorum metadata.

Environment

• Milvus before upgrade: 2.6 image harbor.tencent-stage.zilliz.cn/milvusdb/milvus:2.6-20260707-e9ee9a47-amd64
• Milvus after upgrade: master/3.0 image harbor.tencent-stage.zilliz.cn/milvusdb/milvus:master-20260707-5617a46a-amd64
• Woodpecker runtime seen in logs: github.com/zilliztech/woodpecker@v0.1.33
• WAL backend: Woodpecker service mode
• Woodpecker service replicas: 4
• Configured service-mode seeds:

woodpecker:
  client:
    quorum:
      quorumBufferPools:
        - name: default
          seeds:
            - eric-json-milvus-woodpecker-0.eric-json-milvus-woodpecker-headless:18080
            - eric-json-milvus-woodpecker-1.eric-json-milvus-woodpecker-headless:18080
            - eric-json-milvus-woodpecker-2.eric-json-milvus-woodpecker-headless:18080
            - eric-json-milvus-woodpecker-3.eric-json-milvus-woodpecker-headless:18080
  storage:
    type: service

Reproduction Scenario

1. Deploy Milvus 2.6 with Woodpecker service-mode WAL.
2. Insert data into a collection.
3. Upgrade the same instance to Milvus 3.0/master, preserving etcd/Woodpecker metadata and data.
4. Run an operation that needs streaming WAL append/recovery, for example Milvus Flush.
5. Observe Proxy and StreamingNode logs.

In my case, the Python test command was:

python3 tests/python_client/json_shredding_insert_1m.py --skip-create --skip-insert

The script printed parallel insert skipped, then appeared stuck. The reason is that the script still calls client.flush() after skipping insert.

Actual Behavior

Proxy starts Flush, then fails to append the manual flush message to WAL because the pchannel assignment is not available:

flushTaskByStreamingService.Execute
assignment not found
append manual flush message to wal failed
Flush failed to WaitToFinish

StreamingNode recovery repeatedly fails while reading old Woodpecker WAL segment 0:

wp readNext msg exception
read entries error logName=by-dev-rootcoord-dml_* segmentId=0 entryId=0
read batch failed on node node=127.0.0.1
read batch failed on all quorum nodes totalNodes=1
rpc error: code = Unavailable desc = connection error:
  transport: Error while dialing: dial tcp 127.0.0.1:443: connect: connection refused

This prevents StreamingCoord from completing pchannel assignment. Proxy then keeps waiting/retrying for assignment, so user-facing Flush hangs or times out.

Expected Behavior

After upgrade, Woodpecker should be able to recover old WAL segments written by Milvus 2.6 in service mode.

For old segment metadata without embedded QuorumInfo, Woodpecker should not synthesize a localhost quorum in K8s/service mode. It should either:

• resolve the quorum from service-mode metadata/configured seeds,
• use a backward-compatible migration/fallback path,
• read sealed/eligible old segments through object storage if possible, or
• fail fast with an explicit incompatible metadata error instead of retrying 127.0.0.1:443.

Root Cause Analysis

The old Woodpecker segment metadata written before the upgrade appears to lack the newer quorum information. In etcd:

• old segment records such as woodpecker/logs/.../segments/0 do not contain the newer quorum field,
• newer segment records such as segments/3 include embedded QuorumInfo with real service-mode nodes like eric-json-milvus-woodpecker-*.headless:18080,
• woodpecker/quorums was empty in this environment.

The Woodpecker client code path then falls back to localhost when quorumId <= 0. In local module github.com/zilliztech/woodpecker@v0.1.29, woodpecker/segment/segment_handle.go has:

quorumId := s.segmentMetaCache.Load().Metadata.GetQuorumId()
if quorumId <= 0 {
    s.quorumInfo = &proto.QuorumInfo{
        Id: 0,
        Es: 1,
        Wq: 1,
        Aq: 1,
        Nodes: []string{
            "127.0.0.1",
        },
    }
    return s.quorumInfo, nil
}
return s.metadata.GetQuorumInfo(ctx, quorumId)

Runtime logs from v0.1.33 confirm the same behavior: reads are attempted against 127.0.0.1:443 instead of the configured Woodpecker service nodes.

There is also a direct-read path in Woodpecker, but it only applies when canDirectRead() is true, which requires the segment state to be Sealed. The observed failing segment is read through quorumReadBatch, so the service-mode quorum fallback is the active failure path.

Why This Is Not a Milvus StorageV3 / Loon Issue

The upgraded Milvus config also had common.storage.useLoonFFI=true, but the failure occurs before segment manifest/Loon read/write becomes relevant.

The failing chain is:

Milvus Flush
  -> append ManualFlush to streaming WAL
  -> pchannel assignment unavailable
  -> StreamingNode recovery blocked
  -> Woodpecker reads old WAL segment metadata
  -> quorum resolves to 127.0.0.1
  -> read batch fails forever/backoff loop

Logs did not show Loon/manifest read failures at the time of the hang. StorageV3 compaction/index logs were present and successful. The blocking error is in Woodpecker WAL recovery/quorum resolution.

Impact

This can make a Milvus 2.6 -> 3.0/master upgrade non-functional for clusters using Woodpecker service-mode WAL with preserved old metadata. User operations that need streaming WAL, including Flush, can hang or time out because pchannel assignment never completes.

Suggested Fix

The compatibility path for old segment metadata should not use 127.0.0.1 in service mode.

Possible directions:

1. Store or migrate a valid QuorumInfo for old segment metadata during upgrade/open.
2. When quorumId <= 0 and embedded quorum info is absent, resolve a service-mode quorum from the configured quorum buffer pool/seeds instead of localhost.
3. If localhost fallback is only valid for embedded/local mode, guard it by storage/deployment mode.
4. Add a regression test for service-mode old segment metadata without embedded QuorumInfo to ensure recovery does not dial 127.0.0.1.

[FeilongHou]

woodpecker_fix.zip Man, I ran into this exact same headache with the 3.0 upgrade. Turns out the old segment metadata is totally missing that new QuorumInfo field, so it defaults to localhost and dies. This script force-injects the missing quorum metadata into the old segments so they actually resolve.

I am not sure whether if your situation is the same as mine.

But, if you were using woodpecker in milvus 2.6, it was embed mode. When you upgrade, just keep the woodpecker embed setting. Do not turn on woodpecker service mode, i.e. woodpecker.enabled=false, streaming.woodpecker.embedded=true

[tinswzy]

Yes, that's correct. 2.6 does not support woodpecker in service mode. If the mode is changed from embedded to service during the upgrade from Milvus 2.6 to 3.0, it will trigger a panic. So keeping it in embedded mode is the right way to go for now.

For more details, you can check: milvus-io/milvus#50201