milvus cluster写入数据失败

[bozai111]

pulsar的日志信息

2025-03-24T08:44:21,900+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Compaction: entry log usage buckets[10% 20% 30% 40% 50% 60% 70% 80% 90% 100%] = [1, 0, 0, 0, 0, 0, 0, 0, 0, 0], compacted [1, 0, 0, 0, 0, 0, 0, 0, 0, 0]

2025-03-24T08:54:21,904+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Enter minor compaction, suspendMinor false

2025-03-24T08:54:21,904+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Do compaction to compact those files lower than 0.2

2025-03-24T08:54:21,904+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Compaction: entry log usage buckets before compaction [10% 20% 30% 40% 50% 60% 70% 80% 90% 100%] = [1, 0, 0, 0, 0, 0, 0, 0, 0, 0]

2025-03-24T08:54:21,904+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Compaction progress 0 / 1, current compaction entryLogId: 12

2025-03-24T08:54:22,270+0000 [GarbageCollectorThread-6-1] ERROR org.apache.bookkeeper.bookie.GarbageCollectorThread - Failed to compact entry log 12 due to unexpected error

java.lang.IllegalArgumentException: Keys and values must be >= 0

at org.apache.bookkeeper.util.collections.ConcurrentLongLongHashMap.checkBiggerEqualZero(ConcurrentLongLongHashMap.java:919) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]

at org.apache.bookkeeper.util.collections.ConcurrentLongLongHashMap.get(ConcurrentLongLongHashMap.java:227) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]

at org.apache.bookkeeper.util.collections.ConcurrentLongLongHashMap.containsKey(ConcurrentLongLongHashMap.java:233) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]

at org.apache.bookkeeper.bookie.EntryLogMetadata.containsLedger(EntryLogMetadata.java:64) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]

at org.apache.bookkeeper.bookie.EntryLogCompactor$CompactionScannerFactory$1.accept(EntryLogCompactor.java:90) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]

at org.apache.bookkeeper.bookie.DefaultEntryLogger.scanEntryLog(DefaultEntryLogger.java:1031) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]

at org.apache.bookkeeper.bookie.EntryLogCompactor.compact(EntryLogCompactor.java:61) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]

at org.apache.bookkeeper.bookie.GarbageCollectorThread.compactEntryLog(GarbageCollectorThread.java:732) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]

at org.apache.bookkeeper.bookie.GarbageCollectorThread.lambda$doCompactEntryLogs$9(GarbageCollectorThread.java:635) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]

at org.apache.bookkeeper.bookie.InMemoryEntryLogMetadataMap.forKey(InMemoryEntryLogMetadataMap.java:53) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]

at org.apache.bookkeeper.bookie.GarbageCollectorThread.doCompactEntryLogs(GarbageCollectorThread.java:622) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]

at org.apache.bookkeeper.bookie.GarbageCollectorThread.runWithFlags(GarbageCollectorThread.java:455) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]

at org.apache.bookkeeper.bookie.GarbageCollectorThread.run(GarbageCollectorThread.java:396) ~[org.apache.bookkeeper-bookkeeper-server-4.16.6.jar:4.16.6]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]

at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.113.Final.jar:4.1.113.Final]

at java.lang.Thread.run(Thread.java:840) ~[?:?]

2025-03-24T08:54:22,270+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Compaction: entry log usage buckets[10% 20% 30% 40% 50% 60% 70% 80% 90% 100%] = [1, 0, 0, 0, 0, 0, 0, 0, 0, 0], compacted [1, 0, 0, 0, 0, 0, 0, 0, 0, 0]

目前怀疑pulsar有异常，暂时看不出问题，如何排查？

[yhmo]

在相关的社区找线索，比如：
apache/bookkeeper#3445
apache/bookkeeper#2103

[xiaofan-luan]

I'm not an expert of pulsar, but it seems to be a pulsar bug.

To fix that:

1. restart bookeeper see if it works.
2. decommission the bookeeper with this problem. the error data should be rebuilt by the other two bookeeper.