edge and v3.1 validator fails on assertion: instruction.accounts.len() <= transaction_callee_map.len()

[levsha]

Edge and Beta canaries crashed on assertion failed.
The oldest commit crashed canaries run on is db91477 (it does not mean that the previous commit is ok - we haven't done bisect yet).

The crash trace:

[2025-11-17T14:04:02.189498849Z INFO  solana_core::replay_stage] scb1Z7du8NVSaHFXsafSjRdXr6xBjWR3iugikL739Y1 reset PoH to tick 24364090496 (within slot 380688913). My next leader slot is 380728732
[2025-11-17T14:04:02.189571849Z INFO  solana_poh::poh_recorder] reset poh from: AuTyA2uAQhJapaHEYc5vpUCZysdFFHQAhFoh62z6HzEh,24364090513,380688912 to: AuTyA2uAQhJapaHEYc5vpUCZysdFFHQAhFoh62z6HzEh,380688913
[2025-11-17T14:04:02.189741429Z INFO  solana_core::replay_stage] new fork:380688914 parent:380688913 root:380688882

thread 'solScHandleV04' panicked at /home/sol/src/agave/program-runtime/src/invoke_context.rs:451:9:
assertion failed: instruction.accounts.len() <= transaction_callee_map.len()
stack backtrace:
[2025-11-17T14:04:03.143476279Z INFO  solana_streamer::streamer] streamer send Repair hosts: count:269 [(64.130.53.232, SendStats { bytes: 16898, count: 14 }), (64.130.58.232, SendStats { bytes: 9656, count: 8 }), (185.172.191.7, SendStats { bytes: 8581, count: 8 }), (74.118.142.181, SendStats { bytes: 8449, count: 7 }), (198.13.
130.114, SendStats { bytes: 8449, count: 7 })]
   0:     0x5b7b21e35933 - std::backtrace_rs::backtrace::libunwind::trace::h2b8934d68a0c74cb
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
   1:     0x5b7b21e35933 - std::backtrace_rs::backtrace::trace_unsynchronized::h72d137991fa6eab8
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
   2:     0x5b7b21e35933 - std::sys::backtrace::_print_fmt::h3e831e32e6da920b
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/sys/backtrace.rs:66:9
   3:     0x5b7b21e35933 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h6d42cc84fc840290
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/sys/backtrace.rs:39:26
   4:     0x5b7b2008fa13 - core::fmt::rt::Argument::fmt::h7239af9e2199f2a8
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/fmt/rt.rs:177:76
   5:     0x5b7b2008fa13 - core::fmt::write::h5af61a909e3ec64d
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/fmt/mod.rs:1449:21
   6:     0x5b7b21e2fc23 - std::io::Write::write_fmt::h5a7b54aa6e4a315d
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/io/mod.rs:1890:15
   7:     0x5b7b21e35742 - std::sys::backtrace::BacktraceLock::print::h555579e7396c26ac
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/sys/backtrace.rs:42:9
   8:     0x5b7b21e36ba6 - std::panicking::default_hook::{{closure}}::h9128866118196224
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:298:22
   9:     0x5b7b21e36a40 - std::panicking::default_hook::h52e9e7314e0255f6
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:325:9
  10:     0x5b7b2119e615 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h6b218cc2bb4eafed
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/alloc/src/boxed.rs:1990:9
  11:     0x5b7b2119e615 - solana_metrics::metrics::set_panic_hook::{{closure}}::{{closure}}::h78e17191f1b01a26
                               at /home/sol/src/agave/metrics/src/metrics.rs:516:13
  12:     0x5b7b21e377a3 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h7c356b28a03897d7
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/alloc/src/boxed.rs:1990:9
  13:     0x5b7b21e377a3 - std::panicking::rust_panic_with_hook::h541791bcc774ef34
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:839:13
  14:     0x5b7b21e37446 - std::panicking::begin_panic_handler::{{closure}}::h6479a2f0137c7d19
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:697:13
  15:     0x5b7b21e35e69 - std::sys::backtrace::__rust_end_short_backtrace::ha04e7c0fc61ded91
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/sys/backtrace.rs:168:18
  16:     0x5b7b21e3710d - rust_begin_unwind
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:695:5
  17:     0x5b7b1fb59b70 - core::panicking::panic_fmt::h5764ee7030b7a73d
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/panicking.rs:75:14
  18:     0x5b7b1fb59bfc - core::panicking::panic::had768957450a0f86
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/panicking.rs:145:5
  19:     0x5b7b21cfe3eb - solana_program_runtime::invoke_context::InvokeContext::prepare_next_top_level_instruction::h9aac9d418ffcd5d4
                               at /home/sol/src/agave/program-runtime/src/invoke_context.rs:451:9
  20:     0x5b7b21ce3209 - solana_svm::message_processor::process_message::hc23270707bd17ede
                               at /home/sol/src/agave/svm/src/message_processor.rs:28:9
  21:     0x5b7b21cc5a6a - solana_svm::transaction_processor::TransactionBatchProcessor<FG>::execute_loaded_transaction::h728c2095d79b6197
                               at /home/sol/src/agave/svm/src/transaction_processor.rs:921:30
  22:     0x5b7b21cc5a6a - solana_svm::transaction_processor::TransactionBatchProcessor<FG>::load_and_execute_sanitized_transactions::h17114337e241c3d1
                               at /home/sol/src/agave/svm/src/transaction_processor.rs:507:39
  23:     0x5b7b21cd2543 - solana_runtime::bank::Bank::load_and_execute_transactions::h245c4067e1fca058
                               at /home/sol/src/agave/runtime/src/bank.rs:3283:32
  24:     0x5b7b21cd2543 - solana_runtime::bank::Bank::do_load_execute_and_commit_transactions_with_pre_commit_callback::h25eae3833313846b
                               at /home/sol/src/agave/runtime/src/bank.rs:3819:13
  25:     0x5b7b21ce0325 - solana_runtime::bank::Bank::load_execute_and_commit_transactions_with_pre_commit_callback::hda2cbe90ea833293
                               at /home/sol/src/agave/runtime/src/bank.rs:3794:9
  26:     0x5b7b21ce0325 - solana_ledger::blockstore_processor::execute_batch::hc29d3ae552d19d8e
                               at /home/sol/src/agave/ledger/src/blockstore_processor.rs:224:10
  27:     0x5b7b21ce68e6 - <solana_unified_scheduler_pool::DefaultTaskHandler as solana_unified_scheduler_pool::TaskHandler>::handle::h3973f2251b124a8c
                               at /home/sol/src/agave/unified-scheduler-pool/src/lib.rs:1161:19
  28:     0x5b7b20c2331e - solana_unified_scheduler_pool::ThreadManager<S,TH>::execute_task_with_handler::hecff83c027889622
                               at /home/sol/src/agave/unified-scheduler-pool/src/lib.rs:1663:9
  29:     0x5b7b20ae5dc8 - solana_unified_scheduler_pool::ThreadManager<S,TH>::start_threads::{{closure}}::{{closure}}::h88b5179f43e6fb29
                               at /home/sol/src/agave/unified-scheduler-pool/src/lib.rs:2375:21
  30:     0x5b7b20ae5dc8 - core::ops::function::FnOnce::call_once::h6c89d82cf774ee73
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/ops/function.rs:250:5
  31:     0x5b7b20ae5dc8 - std::sys::backtrace::__rust_begin_short_backtrace::hebe19831d7792781
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/sys/backtrace.rs:152:18
  32:     0x5b7b20c95b64 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h5858c9ca0d1aba5e
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/thread/mod.rs:559:17
  33:     0x5b7b20c95b64 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h52af259cb074468b
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/panic/unwind_safe.rs:272:9
  34:     0x5b7b20c95b64 - std::panicking::try::do_call::h6fe9b8935fc2ba2b
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:587:40
  35:     0x5b7b20c95b64 - std::panicking::try::h79275120bbaf04f4
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  36:     0x5b7b20c95b64 - std::panic::catch_unwind::h0616028e245bcdc5
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  37:     0x5b7b20c95b64 - std::thread::Builder::spawn_unchecked_::{{closure}}::h649b09685937f88b
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/thread/mod.rs:557:30
  38:     0x5b7b20c95b64 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h1380c3dbee85fa23
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/ops/function.rs:250:5
  39:     0x5b7b21e3c7ab - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h292d1663b0785fda
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/alloc/src/boxed.rs:1976:9
  40:     0x5b7b21e3c7ab - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h9d4af7b531852d72
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/alloc/src/boxed.rs:1976:9
  41:     0x5b7b21e3c7ab - std::sys::pal::unix::thread::Thread::new::thread_start::hcc5ed016d554f327
                               at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/sys/pal/unix/thread.rs:106:17
  42:     0x70997129caa4 - <unknown>
  43:     0x709971329c6c - <unknown>
  44:                0x0 - <unknown>
  [2025-11-17T14:04:04.468992842Z INFO  solana_streamer::streamer] streamer send Gossip hosts: count:1024 [(64.130.40.243, SendStats { bytes: 10410828, count: 11495 }), (212.83.42.95, SendStats { bytes: 6851394, count: 7613 }), (192.69.209.130, SendStats { bytes: 3091451, count: 3529 }), (86.105.224.133, SendStats { bytes: 1633780, count: 2972 }), (51.89.106.37, SendStats { bytes: 1633596, count: 2972 })]
[2025-11-17T14:04:04.616507098Z WARN  solana_metrics::metrics] submit error: error sending request for url (https://metrics.solana.com:8086/write?db=canaries-mb&u=canaries&p=quuD2ocUjA&precision=n)

[steviez]

The slot of interest is 380688914 and the bank hash that the cluster finalized:

[2025-11-17T14:04:02.337726571Z INFO  solana_runtime::bank]
bank frozen: 380688914
hash: BsfbincQeVumgW45xy8DfGWbQthTB9eYvrN7yn2CzsF1
signature_count: 1785
last_blockhash: FhuH8rxmqB1CoGqLMuHRWGnxbkV1kXSDcy6sgCTP8SDL
capitalization: 614240056132529952,
accounts_lt_hash checksum: Am7NPDGNzFMDJRzizC6aU85stJX1wD1vq2GRziNFVv9,
stats: BankHashStats { num_updated_accounts: 4370, num_removed_accounts: 68, num_lamports_stored: 1509775022172270, total_data_len: 17550214, num_executable_accounts: 1 }

[steviez]

Replaying the block with tip of master WITHOUT debug assertions produced the correct bank hash; retrying it with debug assertions enabled now

[steviez]

And confirmed that I got the same panic with ledger-tool:

thread 'main' panicked at /home/sol/src/agave/unified-scheduler-pool/src/lib.rs:2464:17:
assertion failed: instruction.accounts.len() <= transaction_callee_map.len() (From: Thread { id: ThreadId(248), name: Some("solScHandleV08"), .. })

Tracking down the tx now