DataDistribution: internal_error

[rourke750]

We are currently seeing this error with Data Distributor

trace.10.167.69.22.4592.1770482233.Nvuo9Y.0.2.xml:<Event Severity="40" ErrorKind="BugDetected" Time="1770490083.854772" DateTime="2026-02-07T18:48:03Z" Type="InternalError" ID="0000000000000000" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" FailedAssertion="value != endValue || endKey == maxWithPrefix.end" File="/home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/KeyRangeMap.actor.cpp" Line="297" ThreadID="7753631453163158563" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x55b671d 0x55b69e3 0x55b0be4 0x54dcd7a 0x4652b88 0x4652173 0x1d18d48 0x2ace81d 0x4912af8 0x4906446 0x4911d08 0x4911b19 0x4911318 0x490984b 0x24c24d8 0x24c1a4c 0x1c662b8 0x1c6600a 0x53850cd 0x53849a3 0x553bdd8 0x32b214c 0x7f3f76829d90" Machine="10.167.69.22:4592" LogGroup="default" Roles="DD" />
/mnt/logs/fdb/trace.10.167.69.22.4592.1770482233.Nvuo9Y.0.2.xml:<Event Severity="40" ErrorKind="Unset" Time="1770490083.854772" DateTime="2026-02-07T18:48:03Z" Type="SystemError" ID="0000000000000000" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" ThreadID="7753631453163158563" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x55b671d 0x55b69e3 0x55b0be4 0x54dd16c 0x54dcd8d 0x4652b88 0x4652173 0x1d18d48 0x2ace81d 0x4912af8 0x4906446 0x4911d08 0x4911b19 0x4911318 0x490984b 0x24c24d8 0x24c1a4c 0x1c662b8 0x1c6600a 0x53850cd 0x53849a3 0x553bdd8 0x32b214c 0x7f3f76829d90" Machine="10.167.69.22:4592" LogGroup="default" Roles="DD" />

Any idea on steps we can take to recover the data distributor? It just keeps crash looping and isn't able to do any work. We are fine with losing a small amount of data (the shard) if needed vs losing the cluster.
We are also running fdb 7.3.69

[jzhou77]

The assertion failure is at krmSetRangeCoalescing(). For DD, this function is called multiples times in MoveKeys related functions, either keyServer (\xff/keyServers/ key space) or serverKey (\xff/serverKeys/) prefixes, i.e., shard map metadata. In general, you will need to collect more data to understand what's going on. For shard map, fdbcli has a getlocations command to print shard map stored on commit proxies. For data within keyServer or serverKey prefixes, fdbcli has getrange command to query data. Unfortunately, the key value pairs returned are binary, so you'll need to learn to decode them (see relevant functions in fdbclient/SystemData.cpp).

Additionally, you probably need to enable more verbose logging on DD so that it's easier to narrow down the exact call site of krmSetRangeCoalescing() (based on adjacent trace events and backtrace).

Finally, it's better to collect the operation history (e.g., exclude, include) and failures (particularly storage servers) in the system.

The crash backtrace alone is not very helpful to pinpoint the exact call site of krmSetRangeCoalescing().

llvm-addr2line -e fdbserver.debug.x86_64 -p -C -f -i 0x55b671d 0x55b69e3 0x55b0be4 0x54dd16c 0x54dcd8d 0x4652b88 0x4652173 0x1d18d48 0x2ace81d 0x4912af8 0x4906446 0x4911d08 0x4911b19 0x4911318 0x490984b 0x24c24d8 0x24c1a4c 0x1c662b8 0x1c6600a 0x53850cd 0x53849a3 0x553bdd8 0x32b214c 0x7f3f76829d90
BaseTraceEvent::backtrace(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Trace.cpp:0
std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::__is_long[abi:v15006]() const at /usr/local/bin/../include/c++/v1/string:1499
 (inlined by) std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::~basic_string() at /usr/local/bin/../include/c++/v1/string:2333
 (inlined by) BaseTraceEvent::log() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Trace.cpp:1326
BaseTraceEvent::~BaseTraceEvent() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Trace.cpp:1369
Error::Error(int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Error.cpp:124
internal_error_impl(char const*, char const*, int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Error.cpp:74
(anonymous namespace)::KrmSetRangeCoalescing_ActorState<Transaction, (anonymous namespace)::KrmSetRangeCoalescing_Actor<Transaction>>::a_body1cont1(Void const&, int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/KeyRangeMap.actor.cpp:0
 (inlined by) (anonymous namespace)::KrmSetRangeCoalescing_ActorState<Transaction, (anonymous namespace)::KrmSetRangeCoalescing_Actor<Transaction>>::a_body1when1(Void const&, int) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbclient/KeyRangeMap.actor.g.cpp:1802
ActorCallback<(anonymous namespace)::KrmSetRangeCoalescing_Actor<Transaction>, 0, Void>::fire(Void const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:1439
SAV<Void>::finishSendAndDelPromiseRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:817
 (inlined by) void SAV<Void>::sendAndDelPromiseRef<Void>(Void&&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:811
QuorumCallback<Standalone<RangeResultRef>>::fire(Standalone<RangeResultRef> const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/genericactors.actor.h:1021
SAV<Standalone<RangeResultRef>>::finishSendAndDelPromiseRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:0
 (inlined by) (anonymous namespace)::GetRangeActorState<GetKeyValuesRequest, GetKeyValuesReply, Standalone<RangeResultRef>, (anonymous namespace)::GetRangeActor<GetKeyValuesRequest, GetKeyValuesReply, Standalone<RangeResultRef>>>::a_body1cont3loopBody1cont10(int) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbclient/NativeAPI.actor.g.cpp:21314
 (inlined by) (anonymous namespace)::GetRangeActorState<GetKeyValuesRequest, GetKeyValuesReply, Standalone<RangeResultRef>, (anonymous namespace)::GetRangeActor<GetKeyValuesRequest, GetKeyValuesReply, Standalone<RangeResultRef>>>::a_body1cont3loopBody1cont12cont2(int) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbclient/NativeAPI.actor.g.cpp:21512
ActorCallback<(anonymous namespace)::GetRangeActor<GetKeyValuesRequest, GetKeyValuesReply, Standalone<RangeResultRef>>, 2, GetKeyValuesReply>::fire(GetKeyValuesReply const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:1439
SAV<GetKeyValuesReply>::finishSendAndDelPromiseRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:817
 (inlined by) (anonymous namespace)::FmapActorState<Future<decltype(getReplyPromise(std::declval<GetKeyValuesRequest>()).getFuture().getValue())> (anonymous namespace)::loadBalance<StorageServerInterface, GetKeyValuesRequest, true>(DatabaseContext*, Reference<LocationInfo>, RequestStream<GetKeyValuesRequest, true> StorageServerInterface::*, GetKeyValuesRequest const&, TaskPriority, AtMostOnce, QueueModel*)::'lambda'(StorageServerInterface const&), GetKeyValuesReply, (anonymous namespace)::FmapActor<Future<decltype(getReplyPromise(std::declval<GetKeyValuesRequest>()).getFuture().getValue())> (anonymous namespace)::loadBalance<StorageServerInterface, GetKeyValuesRequest, true>(DatabaseContext*, Reference<LocationInfo>, RequestStream<GetKeyValuesRequest, true> StorageServerInterface::*, GetKeyValuesRequest const&, TaskPriority, AtMostOnce, QueueModel*)::'lambda'(StorageServerInterface const&), GetKeyValuesReply>>::a_body1cont1(GetKeyValuesReply const&, int) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/include/flow/genericactors.actor.g.h:22344
 (inlined by) (anonymous namespace)::FmapActorState<Future<decltype(getReplyPromise(std::declval<GetKeyValuesRequest>()).getFuture().getValue())> (anonymous namespace)::loadBalance<StorageServerInterface, GetKeyValuesRequest, true>(DatabaseContext*, Reference<LocationInfo>, RequestStream<GetKeyValuesRequest, true> StorageServerInterface::*, GetKeyValuesRequest const&, TaskPriority, AtMostOnce, QueueModel*)::'lambda'(StorageServerInterface const&), GetKeyValuesReply, (anonymous namespace)::FmapActor<Future<decltype(getReplyPromise(std::declval<GetKeyValuesRequest>()).getFuture().getValue())> (anonymous namespace)::loadBalance<StorageServerInterface, GetKeyValuesRequest, true>(DatabaseContext*, Reference<LocationInfo>, RequestStream<GetKeyValuesRequest, true> StorageServerInterface::*, GetKeyValuesRequest const&, TaskPriority, AtMostOnce, QueueModel*)::'lambda'(StorageServerInterface const&), GetKeyValuesReply>>::a_body1when1(GetKeyValuesReply const&, int) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/flow/include/flow/genericactors.actor.g.h:22363
ActorCallback<(anonymous namespace)::FmapActor<Future<decltype(getReplyPromise(std::declval<GetKeyValuesRequest>()).getFuture().getValue())> (anonymous namespace)::loadBalance<StorageServerInterface, GetKeyValuesRequest, true>(DatabaseContext*, Reference<LocationInfo>, RequestStream<GetKeyValuesRequest, true> StorageServerInterface::*, GetKeyValuesRequest const&, TaskPriority, AtMostOnce, QueueModel*)::'lambda'(StorageServerInterface const&), GetKeyValuesReply>, 0, GetKeyValuesReply>::fire(GetKeyValuesReply const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:1439
SAV<GetKeyValuesReply>::finishSendAndDelPromiseRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:817
 (inlined by) (anonymous namespace)::LoadBalanceActorState<StorageServerInterface, GetKeyValuesRequest, ReferencedInterface<StorageServerInterface>, true, (anonymous namespace)::LoadBalanceActor<StorageServerInterface, GetKeyValuesRequest, ReferencedInterface<StorageServerInterface>, true>>::a_body1loopBody1loopBody2when1(ErrorOr<GetKeyValuesReply> const&, int) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/include/fdbrpc/LoadBalance.actor.g.h:2188
ActorCallback<(anonymous namespace)::LoadBalanceActor<StorageServerInterface, GetKeyValuesRequest, ReferencedInterface<StorageServerInterface>, true>, 5, ErrorOr<GetKeyValuesReply>>::fire(ErrorOr<GetKeyValuesReply> const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:1439
SAV<ErrorOr<GetKeyValuesReply>>::finishSendAndDelPromiseRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:817
 (inlined by) (anonymous namespace)::WaitValueOrSignalActorState<GetKeyValuesReply, (anonymous namespace)::WaitValueOrSignalActor<GetKeyValuesReply>>::a_body1loopBody1when1(GetKeyValuesReply const&, int) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbrpc/include/fdbrpc/genericactors.actor.g.h:5947
ActorCallback<(anonymous namespace)::WaitValueOrSignalActor<GetKeyValuesReply>, 0, GetKeyValuesReply>::fire(GetKeyValuesReply const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:1439
SAV<GetKeyValuesReply>::finishSendAndDelPromiseRef() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:817
 (inlined by) void SAV<GetKeyValuesReply>::sendAndDelPromiseRef<GetKeyValuesReply&>(GetKeyValuesReply&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:811
std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, Error, EnsureTable<GetKeyValuesReply>>::index[abi:v15006]() const at /usr/local/bin/../include/c++/v1/variant:797
 (inlined by) std::__1::__variant_detail::__base<(std::__1::__variant_detail::_Trait)1, Error, EnsureTable<GetKeyValuesReply>>::valueless_by_exception[abi:v15006]() const at /usr/local/bin/../include/c++/v1/variant:792
 (inlined by) std::__1::__variant_detail::__dtor<std::__1::__variant_detail::__traits<Error, EnsureTable<GetKeyValuesReply>>, (std::__1::__variant_detail::_Trait)1>::__destroy[abi:v15006]() at /usr/local/bin/../include/c++/v1/variant:854
 (inlined by) std::__1::__variant_detail::__dtor<std::__1::__variant_detail::__traits<Error, EnsureTable<GetKeyValuesReply>>, (std::__1::__variant_detail::_Trait)1>::~__dtor() at /usr/local/bin/../include/c++/v1/variant:854
 (inlined by) std::__1::variant<Error, EnsureTable<GetKeyValuesReply>>::~variant() at /usr/local/bin/../include/c++/v1/variant:1375
 (inlined by) ErrorOr<EnsureTable<GetKeyValuesReply>>::~ErrorOr() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:137
 (inlined by) NetSAV<GetKeyValuesReply>::receive(ArenaObjectReader&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/include/fdbrpc/fdbrpc.h:122
(anonymous namespace)::DeliverActorState<(anonymous namespace)::DeliverActor>::a_body1cont1(int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbrpc/FlowTransport.actor.cpp:1184
ActorCallback<(anonymous namespace)::DeliverActor, 0, Void>::fire(Void const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:1439
void SAV<Void>::send<Void>(Void&&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:783
 (inlined by) void Promise<Void>::send<Void>(Void&&) const at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:1023
 (inlined by) N2::Net2::PromiseTask::operator()() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Net2.actor.cpp:267
 (inlined by) N2::Net2::run() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Net2.actor.cpp:1550
main at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/fdbserver.actor.cpp:0

[jzhou77]

FYI: Michael has made a patch #12682 to deal with this, if it's not too late.

[rourke750]

Hi, we can test this. We think we accidently corrupted data using exclude failed. We still had other issues with dd that caused us to try exclude failed since procs were getting stuck and not adding/removing data