Error when releasing the schema

[ieQu1]

026-07-30T15:49:42.526367+00:00 [notice] Application: esockd. Exited: stopped. Type: permanent.
2026-07-30T15:49:42.528141+00:00 [notice] Application: emqx_conf. Exited: stopped. Type: temporary.
2026-07-30T15:49:42.528264+00:00 [warning] pid: <0.6980.0>, node: 'emqx@n3.local', reason: stop, '$kind': Stopping mria
2026-07-30T15:49:42.552927+00:00 [notice] Application: mria. Exited: stopped. Type: temporary.
2026-07-30T15:49:42.553662+00:00 [error] Error in process <0.6999.0> on node 'emqx@n3.local' with exit value:, {badarg,[{erlang,unlink,[undefined],[{error_info,#{module => erl_erts_errors}}]},{mnesia_controller,release_schema_commit_lock,0,[{file,"mnesia_controller.erl"},{line,338}]},{mnesia_dumper,update,3,[{file,"mnesia_dumper.erl"},{line,303}]},{mnesia_tm,do_commit,3,[{file,"mnesia_tm.erl"},{line,1859}]},{mnesia_tm,commit_participant,7,[{file,"mnesia_tm.erl"},{line,1789}]}]}
2026-07-30T15:49:42.557913+00:00 [notice] Application: mnesia. Exited: stopped. Type: temporary.
2026-07-30T15:49:42.592321+00:00 [error] Supervisor: {local,classy_membership_sup}. Context: child_terminated. Reason: shutdown. Offender: id=worker,pid=<0.7018.0>.
2026-07-30T15:49:42.592509+00:00 [notice] pid: <0.7020.0>, seed: 'emqx@n2.local', '$kind': Mria is restarting to join the cluster
2026-07-30T15:49:42.621859+00:00 [notice] pid: <0.7020.0>, seed: 'emqx@n2.local', result: ok, '$kind': Mria has joined the cluster
2026-07-30T15:49:42.621999+00:00 [notice] pid: <0.7057.0>, local: <<"z_tal6tS09fPNKK4G1eQOw">>, cluster: <<"SL9JOBNlbUAn5SnhWgP-0A">>, '$kind': classy_joined_cluster, intent: autocluster, joined_to_node: 'emqx@n2.local'
2026-07-30T15:49:42.629291+00:00 [notice] pid: <0.7066.0>, node: 'emqx@n4.local', site: <<"8PRSPJWMulqR0El8m2afHA">>, '$kind': classy_peer_connected
2026-07-30T15:49:42.629444+00:00 [notice] pid: <0.7070.0>, node: 'emqx@n9.local', site: <<"T94DE-s011qckpzRCvBu0g">>, '$kind': classy_peer_connected
2026-07-30T15:49:42.629517+00:00 [notice] pid: <0.7074.0>, node: 'emqx@n10.local', site: <<"n7_w69dmU42nsnJ8UtEbEw">>, '$kind': classy_peer_connected
2026-07-30T15:49:42.629586+00:00 [notice] pid: <0.7078.0>, node: 'emqx@n6.local', site: <<"nyOtEGYk66qIK4YTjehHEw">>, '$kind': classy_peer_connected
2026-07-30T15:49:42.629664+00:00 [notice] pid: <0.7082.0>, node: 'emqx@n5.local', site: <<"t8BEU53vulO2qYz_Rf9S7g">>, '$kind': classy_peer_connected
2026-07-30T15:49:42.629784+00:00 [notice] pid: <0.7087.0>, node: 'emqx@n1.local', site: <<"twMhEK9Ue_rtARkSDjaOyg">>, '$kind': classy_peer_connected
2026-07-30T15:49:42.629879+00:00 [notice] pid: <0.7092.0>, node: 'emqx@n2.local', site: <<"y0Pn2OZrjUg48P-ve418Eg">>, '$kind': classy_peer_connected
2026-07-30T15:49:42.630119+00:00 [notice] pid: <0.7096.0>, env: [{cluster_autoheal,true},{{callback,heal_partition},fun emqx_broker_heal:on_autoheal/1},{shard_transport,distr},{rlog_rpc_module,rpc},{bootstrap_batch_size,500},{db_backend,rlog},{{callback,lb_custom_info},fun emqx_machine:mria_lb_custom_info/0},{{callback,lb_custom_info_check},fun emqx_machine:mria_lb_custom_info_check/1},{node_role,core}], '$kind': Starting mria
2026-07-30T15:49:42.634111+00:00 [notice] pid: <0.7096.0>, '$kind': Starting mnesia
2026-07-30T15:49:42.635648+00:00 [notice] pid: <0.7096.0>, '$kind': Starting shards
2026-07-30T15:49:42.882167+00:00 [error] crasher: initial call: mria_schema:init/1, pid: <0.7117.0>, registered_name: [], error: {function_clause,[{mria_lib,ensure_ok,[{no_exists,{emqx_license_session_hwm,cstruct}}],[{file,"/media/aux/work/emqx/deps/mria/src/mria_lib.erl"},{line,197}]},{mria_schema,apply_schema_op,2,[{file,"/media/aux/work/emqx/deps/mria/src/mria_schema.erl"},{line,489}]},{lists,foldl_1,3,[{file,"lists.erl"},{line,2471}]},{mria_schema,init,1,[{file,"/media/aux/work/emqx/deps/mria/src/mria_schema.erl"},{line,250}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,2276}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,2236}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,333}]}]}, ancestors: [mria_sup,<0.7096.0>], message_queue_len: 0, messages: [], links: [<0.7115.0>,<0.7028.0>], dictionary: [], trap_exit: true, status: running, heap_size: 6772, stack_size: 29, reductions: 26639; neighbours:
2026-07-30T15:49:42.882370+00:00 [error] Supervisor: {local,mria_sup}. Context: start_error. Reason: {function_clause,[{mria_lib,ensure_ok,[{no_exists,{emqx_license_session_hwm,cstruct}}],[{file,"/media/aux/work/emqx/deps/mria/src/mria_lib.erl"},{line,197}]},{mria_schema,apply_schema_op,2,[{file,"/media/aux/work/emqx/deps/mria/src/mria_schema.erl"},{line,489}]},{lists,foldl_1,3,[{file,"lists.erl"},{line,2471}]},{mria_schema,init,1,[{file,"/media/aux/work/emqx/deps/mria/src/mria_schema.erl"},{line,250}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,2276}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,2236}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,333}]}]}. Offender: id=mria_schema,pid=undefined.
2026-07-30T15:49:42.882633+00:00 [error] crasher: initial call: application_master:init/3, pid: <0.7095.0>, registered_name: [], exit: {{{shutdown,{failed_to_start_child,mria_schema,{function_clause,[{mria_lib,ensure_ok,[{no_exists,{emqx_license_session_hwm,cstruct}}],[{file,"/media/aux/work/emqx/deps/mria/src/mria_lib.erl"},{line,197}]},{mria_schema,apply_schema_op,2,[{file,"/media/aux/work/emqx/deps/mria/src/mria_schema.erl"},{line,489}]},{lists,foldl_1,3,[{file,"lists.erl"},{line,2471}]},{mria_schema,init,1,[{file,"/media/aux/work/emqx/deps/mria/src/mria_schema.erl"},{line,250}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,2276}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,2236}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,333}]}]}}},{mria_app,start,[normal,[]]}},[{application_master,init,3,[{file,"application_master.erl"},{line,147}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,333}]}]}, ancestors: [application_controller,<0.10.0>], message_queue_len: 1, messages: [{'EXIT',<0.7096.0>,normal}], links: [<0.3953.0>], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 238; neighbours:
2026-07-30T15:49:42.882649+00:00 [notice] Application: mria. Exited: {{shutdown,{failed_to_start_child,mria_schema,{function_clause,[{mria_lib,ensure_ok,[{no_exists,{emqx_license_session_hwm,cstruct}}],[{file,"/media/aux/work/emqx/deps/mria/src/mria_lib.erl"},{line,197}]},{mria_schema,apply_schema_op,2,[{file,"/media/aux/work/emqx/deps/mria/src/mria_schema.erl"},{line,489}]},{lists,foldl_1,3,[{file,"lists.erl"},{line,2471}]},

[ieQu1]

2026-07-30T15:55:18.331148+00:00 [notice] Application: emqx_audit. Exited: stopped. Type: temporary.
2026-07-30T15:55:18.331596+00:00 [notice] tcp:default stopped on 0.0.0.0:1883
Listener tcp:default on 0.0.0.0:1883 stopped.
2026-07-30T15:55:18.332604+00:00 [notice] ssl:default stopped on 0.0.0.0:8883
Listener ssl:default on 0.0.0.0:8883 stopped.
Listener ws:default on 0.0.0.0:8083 stopped.
Listener wss:default on 0.0.0.0:8084 stopped.
2026-07-30T15:55:18.345517+00:00 [notice] msg: Shard allocation still in progress, pid: <0.5389.0>, reason: insufficient_sites_online, needed: 10, sites: [<<"1SUq0v_SwGo6wLtLvH5PYg">>], retry_in: 1000
2026-07-30T15:55:18.369445+00:00 [notice] msg: Shard allocation still in progress, pid: <0.5576.0>, reason: insufficient_sites_online, needed: 10, sites: [<<"1SUq0v_SwGo6wLtLvH5PYg">>], retry_in: 1000
2026-07-30T15:55:18.381324+00:00 [notice] msg: Shard allocation still in progress, pid: <0.5609.0>, reason: insufficient_sites_online, needed: 10, sites: [<<"1SUq0v_SwGo6wLtLvH5PYg">>], retry_in: 1000
2026-07-30T15:55:18.394559+00:00 [notice] msg: Shard allocation still in progress, pid: <0.5651.0>, reason: insufficient_sites_online, needed: 10, sites: [<<"1SUq0v_SwGo6wLtLvH5PYg">>], retry_in: 1000
2026-07-30T15:55:18.439219+00:00 [error] Supervisor: {local,emqx_persistent_session_ds_top_sup}. Context: shutdown_error. Reason: killed. Offender: id=bootstrapper,pid=<0.5334.0>.
2026-07-30T15:55:18.466549+00:00 [notice] Application: emqx. Exited: stopped. Type: permanent.
2026-07-30T15:55:18.468859+00:00 [notice] Application: emqx_durable_timer. Exited: stopped. Type: temporary.
2026-07-30T15:55:18.468945+00:00 [notice] Application: emqx_ds_backends. Exited: stopped. Type: temporary.
2026-07-30T15:55:18.481072+00:00 [notice] Application: emqx_ds_builtin_raft. Exited: stopped. Type: temporary.
2026-07-30T15:55:18.487112+00:00 [notice] Application: emqx_durable_storage. Exited: stopped. Type: temporary.
2026-07-30T15:55:18.489072+00:00 [notice] Application: cowboy. Exited: stopped. Type: temporary.
2026-07-30T15:55:18.489177+00:00 [notice] Application: emqx_utils. Exited: stopped. Type: temporary.
2026-07-30T15:55:18.489216+00:00 [notice] Application: emqx_http_lib. Exited: stopped. Type: temporary.
2026-07-30T15:55:18.491625+00:00 [notice] Application: emqx_bpapi. Exited: stopped. Type: temporary.
2026-07-30T15:55:18.493332+00:00 [notice] Application: ranch. Exited: stopped. Type: temporary.
2026-07-30T15:55:18.496260+00:00 [notice] Application: bcrypt. Exited: stopped. Type: temporary.
2026-07-30T15:55:18.496332+00:00 [notice] Application: emqx_gen_bridge. Exited: stopped. Type: temporary.
2026-07-30T15:55:18.498579+00:00 [notice] Application: esockd. Exited: stopped. Type: permanent.
2026-07-30T15:55:18.500672+00:00 [notice] Application: emqx_conf. Exited: stopped. Type: temporary.
2026-07-30T15:55:18.500834+00:00 [warning] pid: <0.6907.0>, node: 'emqx@n3.local', reason: stop, '$kind': Stopping mria
2026-07-30T15:55:18.533948+00:00 [notice] Application: mria. Exited: stopped. Type: temporary.
2026-07-30T15:55:18.540732+00:00 [notice] Application: mnesia. Exited: stopped. Type: temporary.
2026-07-30T15:55:18.566273+00:00 [error] Supervisor: {local,classy_membership_sup}. Context: child_terminated. Reason: shutdown. Offender: id=worker,pid=<0.6940.0>.
2026-07-30T15:55:18.566643+00:00 [notice] pid: <0.6942.0>, seed: 'emqx@n2.local', '$kind': Mria is restarting to join the cluster
2026-07-30T15:55:18.590259+00:00 [notice] pid: <0.6942.0>, seed: 'emqx@n2.local', result: ok, '$kind': Mria has joined the cluster
2026-07-30T15:55:18.590361+00:00 [notice] pid: <0.6974.0>, local: <<"1SUq0v_SwGo6wLtLvH5PYg">>, '$kind': classy_joined_cluster, cluster: <<"US1oi7Mpkj86RsSwh2rD_A">>, intent: autocluster, joined_to_node: 'emqx@n2.local'
2026-07-30T15:55:18.600493+00:00 [notice] pid: <0.7022.0>, env: [{{callback,lb_custom_info},fun emqx_machine:mria_lb_custom_info/0},{rlog_rpc_module,rpc},{bootstrap_batch_size,500},{{callback,lb_custom_info_check},fun emqx_machine:mria_lb_custom_info_check/1},{node_role,core},{{callback,heal_partition},fun emqx_broker_heal:on_autoheal/1},{shard_transport,distr},{cluster_autoheal,true},{db_backend,rlog}], '$kind': Starting mria
2026-07-30T15:55:18.605754+00:00 [notice] pid: <0.7022.0>, '$kind': Starting mnesia
2026-07-30T15:55:18.608448+00:00 [notice] pid: <0.7022.0>, '$kind': Starting shards
2026-07-30T15:55:23.598044+00:00 [critical] pid: <0.4177.0>, args: [<<"US1oi7Mpkj86RsSwh2rD_A">>,<<"1SUq0v_SwGo6wLtLvH5PYg">>,{join,#{node => 'emqx@n2.local',cluster => <<"vzutt_UYLJZuGMs9PG_mYQ">>,join_intent => autocluster}}], reason: {error,{exit,{noproc,{gen_server,call,[emqx_cluster_rpc,on_leave]}}}}, hook: fun emqx_cluster_rpc:on_kick_decided/3, hookpoint: on_kick_decided, '$kind': classy_hook_failure
2026-07-30T15:55:23.609244+00:00 [warning] mria_mnesia: still waiting for table(s): [mria_schema] 'emqx@n3.local'
2026-07-30T15:55:23.609597+00:00 [warning] Table mria_schema is waiting for one of the nodes: []
2026-07-30T15:55:28.610271+00:00 [warning] mria_mnesia: still waiting for table(s): [mria_schema] 'emqx@n3.local'
2026-07-30T15:55:28.610690+00:00 [warning] Table mria_schema is waiting for one of the nodes: []
2026-07-30T15:55:33.612235+00:00 [warning] mria_mnesia: still waiting for table(s): [mria_schema] 'emqx@n3.local'
2026-07-30T15:55:33.612705+00:00 [warning] Table mria_schema is waiting for one of the nodes: []
2026-07-30T15:55:38.614169+00:00 [warning] mria_mnesia: still waiting for table(s): [mria_schema] 'emqx@n3.local'
2026-07-30T15:55:38.614499+00:00 [warning] Table mria_schema is waiting for one of the nodes: []
2026-07-30T15:55:43.615220+00:00 [warning] mria_mnesia: still waiting for table(s): [mria_schema] 'emqx@n3.local'
2026-07-30T15:55:43.615707+00:00 [warning] Table mria_schema is waiting for one of the nodes: []
2026-07-30T15:55:48.600312+00:00 [critical] pid: <0.7000.0>, args: [stopped,single], reason: {error,{timeout,[{current_stacktrace,[{gen,do_call,4,[{file,"gen.erl"},{line,243}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,1297}]},{application_controller,call,2,[{file,"application_controller.erl"},{line,513}]},{application,enqueue_or_start_app,6,[{file,"application.erl"},{line,420}]},{application,enqueue_or_start,6,[{file,"application.erl"},{line,385}]},{application,ensure_all_started,3,[{file,"application.erl"},{line,360}]},{mria_app,on_run_level,2,[{file,"/media/aux/work/emqx/deps/mria/src/mria_app.erl"},{line,83}]},{classy_lib,safe_apply,3,[{file,"/media/aux/work/emqx/deps/classy/src/classy_lib.erl"},{line,100}]},{classy_lib,'-safe_apply_with_timeout/2-fun-0-',1,[{file,"/media/aux/work/emqx/deps/classy/src/classy_lib.erl"},{line,121}]}]}]}}, hook: fun mria_app:on_run_level/2, hookpoint: on_change_run_level, '$kind': classy_hook_failure
2026-07-30T15:55:48.600640+00:00 [warning] pid: <0.7620.0>, node: 'emqx@n3.local', reason: stop, '$kind': Stopping mria
2026-07-30T15:55:48.600876+00:00 [warning] mria_mnesia: still waiting for table(s): [mria_schema] 'emqx@n3.local'
2026-07-30T15:55:48.601247+00:00 [warning] Check is_schema_merged should get true but got timeout, Table mria_schema is waiting for one of the nodes: []
2026-07-30T15:55:48.601547+00:00 [error] crasher: initial call: mria_schema:init/1, pid: <0.7064.0>, registered_name: [], error: {{badmatch,{error,{node_not_running,'emqx@n3.local'}}},[{mria_schema,bootstrap,0,[{file,"/media/aux/work/emqx/deps/mria/src/mria_schema.erl"},{line,457}]},{mria_schema,init,1,[{file,"/media/aux/work/emqx/deps/mria/src/mria_schema.erl"},{line,245}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,2276}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,2236}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,333}]}]}, ancestors: [mria_sup,<0.7022.0>], message_queue_len: 0, messages: [], links: [<0.7062.0>], dictionary: [], trap_exit: true, status: running, heap_size: 2586, stack_size: 29, reductions: 71479; neighbours:
2026-07-30T15:55:48.601717+00:00 [error] Supervisor: {local,mria_sup}. Context: start_error. Reason: {{badmatch,{error,{node_not_running,'emqx@n3.local'}}},[{mria_schema,bootstrap,0,[{file,"/media/aux/work/emqx/deps/mria/src/mria_schema.erl"},{line,457}]},{mria_schema,init,1,[{file,"/media/aux/work/emqx/deps/mria/src/mria_schema.erl"},{line,245}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,2276}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,2236}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,333}]}]}. Offender: id=mria_schema,pid=undefined.
2026-07-30T15:55:48.601874+00:00 [error] crasher: initial call: application_master:init/3, pid: <0.7021.0>, registered_name: [], exit: {{{shutdown,{failed_to_start_child,mria_schema,{{badmatch,{error,{node_not_running,'emqx@n3.local'}}},[{mria_schema,bootstrap,0,[{file,"/media/aux/work/emqx/deps/mria/src/mria_schema.erl"},{line,457}]},{mria_schema,init,1,[{file,"/media/aux/work/emqx/deps/mria/src/mria_schema.erl"},{line,245}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,2276}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,2236}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,333}]}]}}},{mria_app,start,[normal,[]]}},[{application_master,init,3,[{file,"application_master.erl"},{line,147}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,333}]}]}, ancestors: [application_controller,<0.10.0>], message_queue_len: 1, messages: [{'EXIT',<0.7022.0>,normal}], links: [<0.3953.0>], dictionary: [], trap_exit: true, status: running, heap_size: 610, stack_size: 29, reductions: 223; neighbours:
2026-07-30T15:55:48.605424+00:00 [notice] Application: mnesia. Exited: stopped. Type: temporary.
2026-07-30T15:55:48.605479+00:00 [notice] Application: mria. Exited: {{shutdown,{failed_to_start_child,mria_schema,{{badmatch,{error,{node_not_running,'emqx@n3.local'}}},[{mria_schema,bootstrap,0,[{file,"/media/aux/work/emqx/deps/mria/src/mria_schema.erl"},{line,457}]},{mria_schema,init,1,[{file,"/media/aux/work/emqx/deps/mria/src/mria_schema.erl"},{line,245}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,2276}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,2236}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,333}]}]}}},{mria_app,start,[normal,[]]}}. Type: temporary.
lines 182-251