[BUG] CN crash with SIGSEGV in publish_primary_compaction on shared-data cluster (3.5.10, aarch64)

[EdwardArchive]

Steps to reproduce the behavior (Required)

Environment:

• StarRocks Version: 3.5.10 RELEASE (build 9848c7f)
• Architecture: aarch64 (ARM64)
• OS: Ubuntu
• Cluster Mode: Shared-data (lake mode)
• Table Type: Primary Key table

Reproduction scenario:

---

Expected behavior (Required)

Compaction transactions for Primary Key tables should be published successfully without crashing the CN node.

---

Real behavior (Required)

CN node crashes with SIGSEGV (Segmentation Fault) during the publish phase of Primary Key table compaction.

Stack Trace:

3.5.10 RELEASE (build 9848c7f distro ubuntu arch aarch64)
query_id:00000000-0000-0000-0000-000000000000, fragment_instance:00000000-0000-0000-0000-000000000000, plan_node_id:-1
*** Aborted at 1765943709 (unix time) try "date -d @1765943709" if you are using GNU date ***
PC: @          0xf119ab0 google::protobuf::RepeatedPtrField<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::empty() const
*** SIGSEGV (@0x120) received by PID 27 (TID 0xffff1111fe80) LWP(410) from PID 288; stack trace: ***
    @     0xffffbdcb53dc (/usr/lib/aarch64-linux-gnu/libc.so.6+0x853db)
    @          0xed09718 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0xffffbf36b838 ([vdso]+0x837)
    @          0xf119ab0 google::protobuf::RepeatedPtrField<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::empty() const
    @          0x99600e8 starrocks::lake::UpdateManager::publish_primary_compaction(starrocks::TxnLogPB_OpCompaction const&, long, starrocks::TabletMetadataPB const&, starrocks::lake::Tablet const&, starrocks::DynamicCache<unsigned long, starrocks::lake::LakePrimaryIndex, std::mut
    @          0xb4e1e38 std::_Function_handler<starrocks::Status (), starrocks::lake::PrimaryKeyTxnLogApplier::apply(starrocks::TxnLogPB const&)::{lambda()#2}>::_M_invoke(std::_Any_data const&)
    @          0xb4e28f0 starrocks::lake::PrimaryKeyTxnLogApplier::check_and_recover(std::function<starrocks::Status ()> const&)
    @          0xb4e5248 starrocks::lake::PrimaryKeyTxnLogApplier::apply(starrocks::TxnLogPB const&)
    @          0xb4dd4c0 starrocks::lake::publish_version(starrocks::lake::TabletManager*, long, long, long, std::span<starrocks::TxnInfoPB const, 18446744073709551615ul>)
    @          0xb4cf218 starrocks::LakeServiceImpl::publish_version(google::protobuf::RpcController*, starrocks::PublishVersionRequest const*, starrocks::PublishVersionResponse*, google::protobuf::Closure*)::{lambda()#1}::operator()() const
    @          0xb6426e0 std::_Function_handler<void (), starrocks::ConcurrencyLimitedThreadPoolToken::submit(std::shared_ptr<starrocks::Runnable>, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda()#1}>::
    @          0xb649b94 starrocks::ThreadPool::dispatch_thread()
    @          0xb64017c starrocks::Thread::supervise_thread(void*)
    @     0xffffbdcb0398 (/usr/lib/aarch64-linux-gnu/libc.so.6+0x80397)
    @     0xffffbdd19e9c (/usr/lib/aarch64-linux-gnu/libc.so.6+0xe9e9b)

Memory statistics before crash:

I20251217 12:55:06.497229 281473749810816 daemon.cpp:139] Current memory statistics: 
process(1048808704), query_pool(348896), load(0), metadata(396326), compaction(0), 
schema_change(0), page_cache(91456), update(786560), chunk_allocator(0), passthrough(0), 
clone(0), consistency(0), datacache(496770243), jit(0)

---

StarRocks version (Required)

3.5.10 RELEASE (build 9848c7f distro ubuntu arch aarch64)

[EdwardArchive]

While analyzing the stack trace, I noticed something that might be related to this crash.

In be/src/storage/lake/update_manager.cpp

auto input_rowset = std::find_if(metadata.rowsets().begin(), metadata.rowsets().end(),
                                 [&](const RowsetMetadata& r) { return r.id() == max_rowset_id; });
uint32_t max_src_rssid = max_rowset_id + input_rowset->segments_size() - 1;

I'm not 100% sure, but it seems like there's no validation to check if input_rowset is end() before dereferencing it. If max_rowset_id doesn't exist in metadata.rowsets(), std::find_if would return end(), and dereferencing it could cause undefined behavior.

Could this be the reason for the SIGSEGV at 0x120?

[luohaha]

@EdwardArchive I don't think this is the reason, because max_rowset_id will exist in metadata.rowsets(). I will reproduce this issue in my env.

[EdwardArchive]

Thanks for the investigation @luohaha!

Here's more context about this issue:
The issue started when the FE Leader node was OOM killed. After that, CN nodes became unstable and started going down intermittently due to compaction operations. We tried disabling compaction, but the CN nodes continued to crash in a similar pattern.

We incidentally resolved the issue by deleting some transaction logs and partitions, though we haven't identified the root cause yet.

For now, we've increased the FE node memory as a precautionary measure.

I think this issue might be related to #56260

[luohaha]

@EdwardArchive Got it, thanks for your informations.

[luohaha]

@EdwardArchive Could you grep "input rowset not found" from Be logs, did these error logs show up at that time?

[EdwardArchive]

@luohaha
Hi Unfortunately, we don't have logs on that time. 😢

[luohaha]

While analyzing the stack trace, I noticed something that might be related to this crash.

In be/src/storage/lake/update_manager.cpp

auto input_rowset = std::find_if(metadata.rowsets().begin(), metadata.rowsets().end(),
                                 [&](const RowsetMetadata& r) { return r.id() == max_rowset_id; });
uint32_t max_src_rssid = max_rowset_id + input_rowset->segments_size() - 1;

I'm not 100% sure, but it seems like there's no validation to check if input_rowset is end() before dereferencing it. If max_rowset_id doesn't exist in metadata.rowsets(), std::find_if would return end(), and dereferencing it could cause undefined behavior.

Could this be the reason for the SIGSEGV at 0x120?

Concurrent compaction tasks within the same partition can trigger the anomaly you mentioned, which may lead to a crash. I’ve added a safeguard in this PR to handle this case. However, the root cause of why these concurrent compactions are occurring still requires further analysis.
#67154

[EdwardArchive]

@luohaha
Yes, I agree with your comment. The safeguard code you added looks reasonable and should help prevent crashes in this scenario.
If the same situation happens again, I’ll provide more detailed reports based on this conversation so we can continue the analysis of the root cause.

Based on this issue, my understanding of the timeline is as follows:

T1: FE1 (Leader) starts TX-1

• FE1 schedules compaction for partition P1
• runningCompactions = {P1: TX-1} (in-memory)
• TX-1 state: PREPARE
• CN starts executing compaction on rowsets [R1, R2]

T2: FE1 OOM / Crash

• runningCompactions is lost (in-memory state)
• CN continues executing TX-1 compaction

T3: FE2 becomes new Leader

• runningCompactions = {} (empty - lost with FE1)
• remainedActiveCompactionTxnWhenStart: TX-1 not included (only COMMITTED/VISIBLE txns are tracked)
• FE2 sees partition P1 as "available for compaction"

T4: FE2 schedules TX-2 on same partition P1

• FE2 queries metadata → R1, R2 still exist
• Same compaction selection logic → selects same rowsets [R1, R2]
• No protection! FE2 doesn't know TX-1 is running

T5: TX-1 Compaction completes

• CN finishes compaction: R1 + R2 → R3_tx1
• CN writes to S3:
  • rowset_R3_tx1 (new rowset file)
  • txn_log_100 (TxnLog)
• CN reports success to FE2
• FE2 changes TX-1 state: PREPARE → COMMITTED
• Note: TxnLog is only written here, not read

T6: TX-2 Compaction completes

• CN finishes compaction: R1 + R2 → R3_tx2
• CN writes to S3: rowset_R3_tx2, txn_log_200
• FE2 changes TX-2 state: PREPARE → COMMITTED

T7: TX-2 Publish (becomes VISIBLE first)

• PublishVersionDaemon picks TX-2
• CN reads txn_log_200, applies to metadata
• Metadata updated: V5 → V6
  • rowsets: [R1, R2] → rowsets: [R3_tx2]
  • R1, R2 removed from metadata
• update_primary_index_data_version(V6) called
• TX-2 state: COMMITTED → VISIBLE

T8: TX-1 Publish attempted → SIGSEGV!

• PublishVersionDaemon picks TX-1 (still COMMITTED)
• CN reads txn_log_100: input_rowsets=[R1, R2]
• cal_new_base_version() called:
  • Primary Index version = V6 (updated by TX-2)
  • V6 > base_version → loads metadata V6
• Metadata V6: rowsets = [R3_tx2] (R1, R2 already removed!)
• find_if(R1) returns end() → SIGSEGV!

T9: Continuous CN crashes

• FE retries TX-1 publish every 1 second
• Each CN that receives the request crashes
• CN1 → CN2 → CN3 → ... (rotation crash)

How do you think about this scenario?
BTW, Thanks for your work!

[luohaha]

T5: TX-1 Compaction completes

• CN finishes compaction: R1 + R2 → R3_tx1

• CN writes to S3:

  • rowset_R3_tx1 (new rowset file)
  • txn_log_100 (TxnLog)

• CN reports success to FE2

• FE2 changes TX-1 state: PREPARE → COMMITTED

• Note: TxnLog is only written here, not read

CN can't report back to FE2, because the whole compaction process is a RPC call. And when FE1 restart, this compaction rpc call will fail

[EdwardArchive]

Oh… thank you so much for sharing your thoughtful insight with me. It really gave me a new perspective and helped me see things more clearly.