Devnet Archive cannot load block 400187 3NLgcLnqqftWqurJPT3zahMNSQsHGPKZwP842saU1PuSgsHpUXKJ.json #18192
Description
Preliminary Checks
- This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/MinaProtocol/mina/issues
- This issue is not a question, feature request, RFC, or anything other than a bug report. Please post those things in GitHub Discussions: https://github.com/MinaProtocol/mina/discussions
Description
mina-archive -version
Commit 6929a7e
mina-archive fails to load devnet block 400187 3NLgcLnqqftWqurJPT3zahMNSQsHGPKZwP842saU1PuSgsHpUXKJ
Steps to Reproduce
- Restore latest backup
- Run missing blocks auditor
- Try to load missing block 400187 via
mina-archive-blocks --precomputed ...
Expected Result
Block will load to the archive db.
Actual Result
Block cannot be loaded, may crash mina-archive.
Daemon version
mina-archive -version
Commit 6929a7e
How frequently do you see this issue?
Rarely
What is the impact of this issue on your ability to run a node?
Low
Status
mina-archive-blocks --precomputed --archive-uri "$1" "$2"
Most attempts just "Error when adding block" - finally crashed the archive service:
Dec 03 19:46:03 nym3 mina-archive[820354]: {"timestamp":"2025-12-04 00:46:03.286111Z","level":"Info","source":{"module":"Archive_lib__Processor","location":>
Dec 03 19:46:13 nym3 mina-archive[820354]: (monitor.ml.Error
Dec 03 19:46:13 nym3 mina-archive[820354]: ("Async was unable to add a file descriptor to its table of open file descriptors"
Dec 03 19:46:13 nym3 mina-archive[820354]: (file_descr 8)
Dec 03 19:46:13 nym3 mina-archive[820354]: (error
Dec 03 19:46:13 nym3 mina-archive[820354]: "Attempt to register a file descriptor with Async that Async believes it is already managing.")
Dec 03 19:46:13 nym3 mina-archive[820354]: (backtrace
Dec 03 19:46:13 nym3 mina-archive[820354]: ("Raised by primitive operation at Base__Backtrace.get in file \"src/backtrace.ml\", line 10, characters 2-48"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_unix__Raw_scheduler.create_fd in file \"src/raw_scheduler.ml\", line 109, characters 66-84"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_unix__Fd.create in file \"src/fd.ml\" (inlined), line 40, characters 2-112"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Caqti_async.System_unix.Unix.wrap_fd in file \"caqti-async/lib/caqti_async.ml\", line 186, chara>
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Caqti_driver_postgresql.Connect_functor.Make_connection_base.reset.(fun) in file \"caqti-driver->
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Deferred0.bind.(fun) in file \"src/deferred0.ml\", line 54, characters 64-69"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Scheduler1.run_jobs in file \"src/scheduler1.ml\", line 335, characters 8-51"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Scheduler.run_cycle.run_jobs in file \"src/scheduler.ml\", line 173, characters 10>
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Scheduler.run_cycle in file \"src/scheduler.ml\", line 181, characters 2-12"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_unix__Raw_scheduler.have_lock_do_cycle in file \"src/raw_scheduler.ml\", line 631, charact>
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_unix__Raw_scheduler.be_the_scheduler.loop in file \"src/raw_scheduler.ml\", line 875, char>
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_unix__Raw_scheduler.be_the_scheduler in file \"src/raw_scheduler.ml\", line 879, character>
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_command.in_async.(fun) in file \"async_command/src/async_command.ml\", line 75, characters>
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Core_kernel__Command.For_unix.run.(fun) in file \"src/command.ml\", line 2453, characters 8-238"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Base__Exn.handle_uncaught_aux in file \"src/exn.ml\", line 111, characters 6-10"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Dune__exe__Archive in file \"src/app/archive/archive.ml\", line 15, characters 6-95"))
Dec 03 19:46:13 nym3 mina-archive[820354]: (scheduler
Dec 03 19:46:13 nym3 mina-archive[820354]: ((mutex
Dec 03 19:46:13 nym3 mina-archive[820354]: ((id_of_thread_holding_lock 0) (num_using_blocker 0) (blocker ())))
Dec 03 19:46:13 nym3 mina-archive[820354]: (is_running true) (have_called_go true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (fds_whose_watching_has_changed ())
Dec 03 19:46:13 nym3 mina-archive[820354]: (file_descr_watcher
Dec 03 19:46:13 nym3 mina-archive[820354]: (Epoll
Dec 03 19:46:13 nym3 mina-archive[820354]: ((timerfd 5)
Dec 03 19:46:13 nym3 mina-archive[820354]: (epoll
Dec 03 19:46:13 nym3 mina-archive[820354]: (In_use
Dec 03 19:46:13 nym3 mina-archive[820354]: ((epollfd 6)
Dec 03 19:46:13 nym3 mina-archive[820354]: (flags_by_fd
Dec 03 19:46:13 nym3 mina-archive[820354]: ((num_keys 65536) (alist ((7 (in)) (3 (in)) (5 (in et))))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (max_ready_events 256) (num_ready_events 0) (ready_events ()))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (handle_fd_read_ready <fun>) (handle_fd_write_ready <fun>))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (time_spent_waiting_for_io 642509882619)
Dec 03 19:46:13 nym3 mina-archive[820354]: (fd_by_descr
Dec 03 19:46:13 nym3 mina-archive[820354]: ((3
Dec 03 19:46:13 nym3 mina-archive[820354]: ((file_descr 3) (info interruptor_pipe_read) (kind Fifo)
Dec 03 19:46:13 nym3 mina-archive[820354]: (supports_nonblock true) (have_set_nonblock true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (state (Open Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching
Dec 03 19:46:13 nym3 mina-archive[820354]: ((read (Watch_repeatedly "<Pool.Pointer.t: 0x00000001>" Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (write Not_watching)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching_has_changed false) (num_active_syscalls 1)
Dec 03 19:46:13 nym3 mina-archive[820354]: (close_finished Empty)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (4
Dec 03 19:46:13 nym3 mina-archive[820354]: ((file_descr 4) (info interruptor_pipe_write) (kind Fifo)
Dec 03 19:46:13 nym3 mina-archive[820354]: (supports_nonblock true) (have_set_nonblock true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (state (Open Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching ((read Not_watching) (write Not_watching)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching_has_changed false) (num_active_syscalls 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (close_finished Empty)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (7
Dec 03 19:46:13 nym3 mina-archive[820354]: ((file_descr 7) (info ((type_ tcp) (listening_on 0.0.0.0:3086)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (kind (Socket Passive)) (supports_nonblock true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (have_set_nonblock true) (state (Open Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching ((read (Watch_once Empty)) (write Not_watching)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching_has_changed false) (num_active_syscalls 1)
Dec 03 19:46:13 nym3 mina-archive[820354]: (close_finished Empty)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (8
Dec 03 19:46:13 nym3 mina-archive[820354]: ((file_descr 8) (info "Caqti_async file descriptor")
Dec 03 19:46:13 nym3 mina-archive[820354]: (kind (Socket Active)) (supports_nonblock true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (have_set_nonblock false) (state (Open Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching ((read Not_watching) (write Not_watching)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching_has_changed false) (num_active_syscalls 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (close_finished Empty)))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (timerfd (5)) (timerfd_set_at (2025-12-03 19:46:13.091352576-05:00))
Dec 03 19:46:13 nym3 mina-archive[820354]: (scheduler_thread_id 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (interruptor
Dec 03 19:46:13 nym3 mina-archive[820354]: ((pipe
Dec 03 19:46:13 nym3 mina-archive[820354]: ((read
Dec 03 19:46:13 nym3 mina-archive[820354]: ((file_descr 3) (info interruptor_pipe_read) (kind Fifo)
Dec 03 19:46:13 nym3 mina-archive[820354]: (supports_nonblock true) (have_set_nonblock true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (state (Open Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching
Dec 03 19:46:13 nym3 mina-archive[820354]: ((read (Watch_repeatedly "<Pool.Pointer.t: 0x00000001>" Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (write Not_watching)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching_has_changed false) (num_active_syscalls 1)
Dec 03 19:46:13 nym3 mina-archive[820354]: (close_finished Empty)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (write
Dec 03 19:46:13 nym3 mina-archive[820354]: ((file_descr 4) (info interruptor_pipe_write) (kind Fifo)
Dec 03 19:46:13 nym3 mina-archive[820354]: (supports_nonblock true) (have_set_nonblock true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (state (Open Empty))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching ((read Not_watching) (write Not_watching)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (watching_has_changed false) (num_active_syscalls 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (close_finished Empty)))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (already_interrupted false) (clearbuffer <opaque>)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (signal_manager
Dec 03 19:46:13 nym3 mina-archive[820354]: ((handlers_by_signal ((-8 ((bag ()))))) (delivered <opaque>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (thread_safe_notify_signal_delivered <fun>)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (thread_pool
Dec 03 19:46:13 nym3 mina-archive[820354]: ((id 0) (state In_use)
Dec 03 19:46:13 nym3 mina-archive[820354]: (finished
Dec 03 19:46:13 nym3 mina-archive[820354]: ((value ()) (num_waiting 0) (mutex <opaque>) (full <opaque>)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (mutex
Dec 03 19:46:13 nym3 mina-archive[820354]: ((id_of_thread_holding_lock -1) (num_using_blocker 0) (blocker ())))
Dec 03 19:46:13 nym3 mina-archive[820354]: (default_priority 0) (max_num_threads 50) (cpu_affinity Inherit)
Dec 03 19:46:13 nym3 mina-archive[820354]: (num_threads 2) (thread_creation_failure_lockout 1s)
Dec 03 19:46:13 nym3 mina-archive[820354]: (last_thread_creation_failure ())
Dec 03 19:46:13 nym3 mina-archive[820354]: (thread_by_id
Dec 03 19:46:13 nym3 mina-archive[820354]: ((1
Dec 03 19:46:13 nym3 mina-archive[820354]: ((name shutdown) (thread_id (1)) (priority 0) (state Available)
Dec 03 19:46:13 nym3 mina-archive[820354]: (unfinished_work 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (work_queue
Dec 03 19:46:13 nym3 mina-archive[820354]: ((ev_q ()) (maxsize 1) (mutex <opaque>) (not_empty <opaque>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (not_full <opaque>) (finally <fun>)))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (2
Dec 03 19:46:13 nym3 systemd[695]: mina-archive.service: Main process exited, code=exited, status=1/FAILURE
Dec 03 19:46:13 nym3 mina-archive[820354]: ((name close) (thread_id (2)) (priority 0) (state Available)
Dec 03 19:46:13 nym3 mina-archive[820354]: (unfinished_work 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (work_queue
Dec 03 19:46:13 nym3 mina-archive[820354]: ((ev_q ()) (maxsize 1) (mutex <opaque>) (not_empty <opaque>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (not_full <opaque>) (finally <fun>)))))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (available_threads
Dec 03 19:46:13 nym3 mina-archive[820354]: (((name close) (thread_id (2)) (priority 0) (state Available)
Dec 03 19:46:13 nym3 mina-archive[820354]: (unfinished_work 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (work_queue
Dec 03 19:46:13 nym3 mina-archive[820354]: ((ev_q ()) (maxsize 1) (mutex <opaque>) (not_empty <opaque>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (not_full <opaque>) (finally <fun>))))
Dec 03 19:46:13 nym3 mina-archive[820354]: ((name shutdown) (thread_id (1)) (priority 0) (state Available)
Dec 03 19:46:13 nym3 mina-archive[820354]: (unfinished_work 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (work_queue
Dec 03 19:46:13 nym3 mina-archive[820354]: ((ev_q ()) (maxsize 1) (mutex <opaque>) (not_empty <opaque>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (not_full <opaque>) (finally <fun>))))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (work_queue ()) (unfinished_work 0) (num_work_completed 5)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (handle_thread_pool_stuck <fun>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (next_tsc_calibration 173303125748506281)
Dec 03 19:46:13 nym3 mina-archive[820354]: (kernel_scheduler
Dec 03 19:46:13 nym3 mina-archive[820354]: ((check_access ())
Dec 03 19:46:13 nym3 mina-archive[820354]: (job_pool
Dec 03 19:46:13 nym3 mina-archive[820354]: ((slots_per_tuple 3) (capacity 4) (length 2) (next_id 202)
Dec 03 19:46:13 nym3 mina-archive[820354]: (first_free (Free 9)) (dummy (<opaque>))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (normal_priority_jobs
Dec 03 19:46:13 nym3 mina-archive[820354]: ((num_jobs_run 112018) (jobs_left_this_cycle 467) (jobs <opaque>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (mask 3) (front 2) (length 0)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (low_priority_jobs
Dec 03 19:46:13 nym3 mina-archive[820354]: ((num_jobs_run 8) (jobs_left_this_cycle 500) (jobs <opaque>) (mask 0)
Dec 03 19:46:13 nym3 mina-archive[820354]: (front 0) (length 0)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (very_low_priority_workers ())
Dec 03 19:46:13 nym3 mina-archive[820354]: (main_execution_context
Dec 03 19:46:13 nym3 mina-archive[820354]: ((monitor
Dec 03 19:46:13 nym3 mina-archive[820354]: (((name main) (here ()) (id 1) (has_seen_error false)
Dec 03 19:46:13 nym3 mina-archive[820354]: (is_detached false))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (priority Normal) (local_storage ()) (tid 0) (backtrace_history ())))
Dec 03 19:46:13 nym3 mina-archive[820354]: (current_execution_context
Dec 03 19:46:13 nym3 mina-archive[820354]: ((monitor
Dec 03 19:46:13 nym3 mina-archive[820354]: (((name "") (here ()) (id 1620) (has_seen_error false)
Dec 03 19:46:13 nym3 mina-archive[820354]: (is_detached true))))
Dec 03 19:46:13 nym3 mina-archive[820354]: (priority Normal) (local_storage ()) (tid 0) (backtrace_history ())))
Dec 03 19:46:13 nym3 mina-archive[820354]: (uncaught_exn ()) (cycle_count 10880)
Dec 03 19:46:13 nym3 mina-archive[820354]: (cycle_start "2025-12-04 00:46:13.079433623Z") (in_cycle true)
Dec 03 19:46:13 nym3 mina-archive[820354]: (run_every_cycle_start (<fun>)) (run_every_cycle_end (<fun>))
Dec 03 19:46:13 nym3 systemd[695]: mina-archive.service: Failed with result 'exit-code'.
Dec 03 19:46:13 nym3 mina-archive[820354]: (last_cycle_time 97.002us) (last_cycle_num_jobs 5)
Dec 03 19:46:13 nym3 mina-archive[820354]: (total_cycle_time 2.267359177s)
Dec 03 19:46:13 nym3 mina-archive[820354]: (time_source (wall_clock (now 2025-12-03T19:46:13-0500)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (external_jobs
Dec 03 19:46:13 nym3 mina-archive[820354]: ((length 0) (front ((value None) (next <opaque>)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (back ((value None) (next <opaque>))) (unused_elts None)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (thread_safe_external_job_hook <fun>) (job_queued_hook ())
Dec 03 19:46:13 nym3 mina-archive[820354]: (event_added_hook ()) (yield <opaque>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (yield_until_no_jobs_remain <opaque>) (check_invariants false)
Dec 03 19:46:13 nym3 mina-archive[820354]: (max_num_jobs_per_priority_per_cycle 500) (record_backtraces false)
Dec 03 19:46:13 nym3 mina-archive[820354]: (on_start_of_cycle <fun>) (on_end_of_cycle <fun>)
Dec 03 19:46:13 nym3 mina-archive[820354]: (long_jobs_last_cycle ()) (cycle_started false)))
Dec 03 19:46:13 nym3 mina-archive[820354]: (have_lock_do_cycle ()) (max_inter_cycle_timeout 50ms)
Dec 03 19:46:13 nym3 mina-archive[820354]: (min_inter_cycle_timeout 0s) (initialized_at <opaque>))))
Dec 03 19:46:13 nym3 mina-archive[820354]: ("Raised at Base__Error.raise in file \"src/error.ml\" (inlined), line 8, characters 14-30"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Caqti_async.Fiber.finally.(fun) in file \"caqti-async/lib/caqti_async.ml\", line 44, characters 39>
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Deferred1.M.map.(fun) in file \"src/deferred1.ml\", line 17, characters 40-45"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5"
Dec 03 19:46:13 nym3 mina-archive[820354]: "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47"))
Dec 03 19:46:13 nym3 systemd[695]: mina-archive.service: Consumed 3.102s CPU time.Additional information
This was happening in my devnet node; I restored the latest archive backup from o1 and it is also missing the same block.