[Questions] RabbitMQ queue runs into an exception on Windows with anti-virus software enabled

[mister-mist]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.2.3

Erlang version used

27.3.x

Operating system (distribution) used

Windows Server

How is RabbitMQ deployed?

Windows installer

rabbitmq-diagnostics status output

NA

Logs from node 1 (with sensitive values edited out)

Details

[error] <0.3456856.0> ** Generic server <0.3456856.0> terminating
[error] <0.3456856.0> ** Last message in was {'$gen_cast',
[error] <0.3456856.0>                         {deliver,
[error] <0.3456856.0>                          {delivery,false,true,<0.3535883.0>,
[error] <0.3456856.0>                           {mc,mc_amqpl,
[error] <0.3456856.0>                            {content,60,none,
[error] <0.3456856.0>                             <<180,128,32,97,112,112,108,105,99,97,116,105,111,
[error] <0.3456856.0>                               110,47,118,110,100,46,109,97,115,115,116,114,97,
[error] <0.3456856.0>                               110,115,105,116,43,106,115,111,110,0,0,0,29,19,
[error] <0.3456856.0>                               120,45,100,111,116,110,101,116,45,112,117,98,45,
[error] <0.3456856.0>                               115,101,113,45,110,111,108,0,0,0,0,0,0,2,195,2,
[error] <0.3456856.0>                               36,98,48,50,57,48,48,48,48,45,57,50,55,49,45,53,
[error] <0.3456856.0>                               54,57,99,45,100,48,54,101,45,48,56,100,101,97,48,
[error] <0.3456856.0>                               52,53,53,97,55,57,36,98,48,50,57,48,48,48,48,45,
[error] <0.3456856.0>                               57,50,55,49,45,53,54,57,99,45,102,52,53,100,45,
[error] <0.3456856.0>                               48,56,100,101,97,48,52,53,53,98,49,57>>,
[error] <0.3456856.0>                             rabbit_framing_amqp_0_9_1,
[error] <0.3456856.0>                             [<<>>]},
[error] <0.3456856.0>                            #{id =>
[error] <0.3456856.0>                               <<239,201,59,47,174,189,28,142,11,224,118,117,50,
[error] <0.3456856.0>                                 116,105,131>>,
[error] <0.3456856.0>                              x => <<"vpl-job-routing-slip">>,
[error] <0.3456856.0>                              rk => [<<>>],
[error] <0.3456856.0>                              rts => 1776844887521}},
[error] <0.3456856.0>                           707,flow},
[error] <0.3456856.0>                          false}}
[error] <0.3456856.0> ** When Server state == {q,
[error] <0.3456856.0>                          {amqqueue,
[error] <0.3456856.0>                           {resource,<<"/">>,queue,<<"vpl-job-routing-slip">>},
[error] <0.3456856.0>                           true,false,none,
[error] <0.3456856.0>                           [{<<"x-queue-type">>,longstr,<<"classic">>}],
[error] <0.3456856.0>                           <0.3456856.0>,[],[],[],undefined,undefined,[],[],
[error] <0.3456856.0>                           crashed,0,[],<<"/">>,
[error] <0.3456856.0>                           #{user => <<"bob">>},
[error] <0.3456856.0>                           rabbit_classic_queue,#{}},
[error] <0.3456856.0>                          none,true,rabbit_priority_queue,
[error] <0.3456856.0>                          {passthrough,rabbit_variable_queue,
[error] <0.3456856.0>                           {vqstate,
[error] <0.3456856.0>                            {0,[],[]},
[error] <0.3456856.0>                            {0,[],[]},
[error] <0.3456856.0>                            {delta,undefined,0,0,undefined},
[error] <0.3456856.0>                            {0,[],[]},
[error] <0.3456856.0>                            {0,[],[]},
[error] <0.3456856.0>                            4504,4504,#{},#{},undefined,undefined,
[error] <0.3456856.0>                            {qi,
[error] <0.3456856.0>                             {resource,<<"/">>,queue,
[error] <0.3456856.0>                              <<"vpl-job-routing-slip">>},
[error] <0.3456856.0>                             <<"c:/Users/RABBITUSER/AppData/Roaming/RabbitMQ/db/rabbit@SERVER1-mnesia/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/2RPNPJQCGT5ZL48EVS6JN4OD3/">>,
[error] <0.3456856.0>                             #{},0,
[error] <0.3456856.0>                             #{4499 => ack,4500 => ack,4501 => ack},
[error] <0.3456856.0>                             #{},#{},#{},
[error] <0.3456856.0>                             #Fun<rabbit_variable_queue.2.62405665>,
[error] <0.3456856.0>                             #Fun<rabbit_variable_queue.3.62405665>},
[error] <0.3456856.0>                            {qs,
[error] <0.3456856.0>                             <<"c:/Users/RABBITUSER/AppData/Roaming/RabbitMQ/db/rabbit@SERVER1-mnesia/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/2RPNPJQCGT5ZL48EVS6JN4OD3/">>,
[error] <0.3456856.0>                             1,575088,#{},0,
[error] <0.3456856.0>                             #{4502 =>
[error] <0.3456856.0>                                {569235,2919,
[error] <0.3456856.0>                                 {mc,mc_amqpl,
[error] <0.3456856.0>                                  {content,60,none,
[error] <0.3456856.0>                                   <<180,128,32,97,112,112,108,105,99,97,116,
[error] <0.3456856.0>                                     105,111,110,47,118,110,100,46,109,97,115,
[error] <0.3456856.0>                                     115,116,114,97,110,115,105,116,43,106,115,
[error] <0.3456856.0>                                     111,110,0,0,0,29,19,120,45,100,111,116,110,
[error] <0.3456856.0>                                     101,116,45,112,117,98,45,115,101,113,45,
[error] <0.3456856.0>                                     110,111,108,0,0,0,0,0,0,2,193,2,36,98,48,
[error] <0.3456856.0>                                     50,57,48,48,48,48,45,57,50,55,49,45,53,54,
[error] <0.3456856.0>                                     57,99,45,100,101,98,53,45,48,56,100,101,97,
[error] <0.3456856.0>                                     48,52,53,53,57,55,54,36,98,48,50,57,48,48,
[error] <0.3456856.0>                                     48,48,45,57,50,55,49,45,53,54,57,99,45,97,
[error] <0.3456856.0>                                     99,55,98,45,48,56,100,101,97,48,52,53,53,
[error] <0.3456856.0>                                     97,97,52>>,
[error] <0.3456856.0>                                   rabbit_framing_amqp_0_9_1,
[error] <0.3456856.0>                                   [<<>>]},
[error] <0.3456856.0>                                  #{id =>
[error] <0.3456856.0>                                     <<86,28,79,44,241,7,78,156,228,41,77,90,
[error] <0.3456856.0>                                       165,93,106,251>>,
[error] <0.3456856.0>                                    x => <<"vpl-job-routing-slip">>,
[error] <0.3456856.0>                                    rk => [<<>>],
[error] <0.3456856.0>                                    rts => 1776844886969}}},
[error] <0.3456856.0>                               4503 =>
[error] <0.3456856.0>                                {572162,2918,
[error] <0.3456856.0>                                 {mc,mc_amqpl,
[error] <0.3456856.0>                                  {content,60,none,
[error] <0.3456856.0>                                   <<180,128,32,97,112,112,108,105,99,97,116,
[error] <0.3456856.0>                                     105,111,110,47,118,110,100,46,109,97,115,
[error] <0.3456856.0>                                     115,116,114,97,110,115,105,116,43,106,115,
[error] <0.3456856.0>                                     111,110,0,0,0,29,19,120,45,100,111,116,110,
[error] <0.3456856.0>                                     101,116,45,112,117,98,45,115,101,113,45,
[error] <0.3456856.0>                                     110,111,108,0,0,0,0,0,0,2,194,2,36,98,48,
[error] <0.3456856.0>                                     50,57,48,48,48,48,45,57,50,55,49,45,53,54,
[error] <0.3456856.0>                                     57,99,45,53,55,53,50,45,48,56,100,101,97,
[error] <0.3456856.0>                                     48,52,53,53,57,56,48,36,98,48,50,57,48,48,
[error] <0.3456856.0>                                     48,48,45,57,50,55,49,45,53,54,57,99,45,50,
[error] <0.3456856.0>                                     97,102,52,45,48,56,100,101,97,48,52,53,53,
[error] <0.3456856.0>                                     97,97,53>>,
[error] <0.3456856.0>                                   rabbit_framing_amqp_0_9_1,
[error] <0.3456856.0>                                   [<<>>]},
[error] <0.3456856.0>                                  #{id =>
[error] <0.3456856.0>                                     <<200,147,81,106,221,189,82,172,156,201,
[error] <0.3456856.0>                                       117,13,57,148,31,246>>,
[error] <0.3456856.0>                                    x => <<"vpl-job-routing-slip">>,
[error] <0.3456856.0>                                    rk => [<<>>],
[error] <0.3456856.0>                                    rts => 1776844886969}}}},
[error] <0.3456856.0>                             undefined,undefined},
[error] <0.3456856.0>                            {{client_msstate,<0.422.0>,
[error] <0.3456856.0>                              <<132,181,34,146,8,147,101,9,203,131,172,238,236,
[error] <0.3456856.0>                                141,60,123>>,
[error] <0.3456856.0>                              undefined,#Ref<0.2068921784.2808741891.138641>,
[error] <0.3456856.0>                              <<"c:/Users/RABBITUSER/AppData/Roaming/RabbitMQ/db/rabbit@SERVER1-mnesia/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent">>,
[error] <0.3456856.0>                              #Ref<0.2068921784.2808741891.138649>,
[error] <0.3456856.0>                              #Ref<0.2068921784.2808741891.138650>,
[error] <0.3456856.0>                              #Ref<0.2068921784.2808741891.138651>,
[error] <0.3456856.0>                              {4000,800}},
[error] <0.3456856.0>                             {client_msstate,<0.418.0>,
[error] <0.3456856.0>                              <<56,50,139,33,251,34,66,198,220,44,210,83,48,161,
[error] <0.3456856.0>                                238,40>>,
[error] <0.3456856.0>                              undefined,#Ref<0.2068921784.2808741889.144417>,
[error] <0.3456856.0>                              <<"c:/Users/RABBITUSER/AppData/Roaming/RabbitMQ/db/rabbit@SERVER1-mnesia/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient">>,
[error] <0.3456856.0>                              #Ref<0.2068921784.2808741889.144418>,
[error] <0.3456856.0>                              #Ref<0.2068921784.2808741889.144419>,
[error] <0.3456856.0>                              #Ref<0.2068921784.2808741889.144420>,
[error] <0.3456856.0>                              {4000,800}}},
[error] <0.3456856.0>                            true,0,4096,0,0,0,0,0,0,infinity,0,0,0,0,41,41,
[error] <0.3456856.0>                            {rates,1.374944114728699,1.374944114728699,
[error] <0.3456856.0>                             1.374944114728699,1.2858600581584827,
[error] <0.3456856.0>                             -5757233377858349},
[error] <0.3456856.0>                            #{},#{},#{},#{},44,41,11,4504,0,default,2,#{},
[error] <0.3456856.0>                            <<"/">>,false}},
[error] <0.3456856.0>                          {state,
[error] <0.3456856.0>                           {queue,
[error] <0.3456856.0>                            [{<0.3535793.0>,
[error] <0.3456856.0>                              {consumer,<<"amq.ctag-9eKX1bXy3mwatW1skCcwYg">>,
[error] <0.3456856.0>                               true,38,[],<<"bob">>}}],
[error] <0.3456856.0>                            [],1},
[error] <0.3456856.0>                           {active,-575746160107896,0.502659106762281}},
[error] <0.3456856.0>                          undefined,undefined,
[error] <0.3456856.0>                          {erlang,#Ref<0.2068921784.2846621697.20118>},
[error] <0.3456856.0>                          undefined,
[error] <0.3456856.0>                          {state,fine,5000,
[error] <0.3456856.0>                           #Ref<0.2068921784.2836398082.239426>},
[error] <0.3456856.0>                          #{},undefined,undefined,undefined,
[error] <0.3456856.0>                          {state,
[error] <0.3456856.0>                           #{<0.3535818.0> =>
[error] <0.3456856.0>                              #Ref<0.2068921784.2846359553.71469>,
[error] <0.3456856.0>                             <0.3535836.0> =>
[error] <0.3456856.0>                              #Ref<0.2068921784.2836398082.67636>,
[error] <0.3456856.0>                             <0.3535838.0> =>
[error] <0.3456856.0>                              #Ref<0.2068921784.2808610824.154888>,
[error] <0.3456856.0>                             <0.3535843.0> =>
[error] <0.3456856.0>                              #Ref<0.2068921784.2846359553.70852>,
[error] <0.3456856.0>                             <0.3535847.0> =>
[error] <0.3456856.0>                              #Ref<0.2068921784.2808872966.33327>,
[error] <0.3456856.0>                             <0.3535853.0> =>
[error] <0.3456856.0>                              #Ref<0.2068921784.2846359553.72541>,
[error] <0.3456856.0>                             <0.3535858.0> =>
[error] <0.3456856.0>                              #Ref<0.2068921784.2836398082.66676>,
[error] <0.3456856.0>                             <0.3535864.0> =>
[error] <0.3456856.0>                              #Ref<0.2068921784.2846359553.72648>,
[error] <0.3456856.0>                             <0.3535880.0> =>
[error] <0.3456856.0>                              #Ref<0.2068921784.2812018692.188973>,
[error] <0.3456856.0>                             <0.3535883.0> =>
[error] <0.3456856.0>                              #Ref<0.2068921784.2812018692.187652>,
[error] <0.3456856.0>                             <0.3535892.0> =>
[error] <0.3456856.0>                              #Ref<0.2068921784.2846097409.197033>,
[error] <0.3456856.0>                             <0.3535916.0> =>
[error] <0.3456856.0>                              #Ref<0.2068921784.2846359553.73171>,
[error] <0.3456856.0>                             <0.3535935.0> =>
[error] <0.3456856.0>                              #Ref<0.2068921784.2821718019.183693>,
[error] <0.3456856.0>                             <0.3535953.0> =>
[error] <0.3456856.0>                              #Ref<0.2068921784.2846359553.74968>},
[error] <0.3456856.0>                           delegate},
[error] <0.3456856.0>                          undefined,undefined,undefined,undefined,'drop-head',
[error] <0.3456856.0>                          1,0,running,false}
[error] <0.3456856.0> ** Reason for termination ==
[error] <0.3456856.0> ** {{badmatch,{error,eacces}},
[error] <0.3456856.0>     [{rabbit_classic_queue_index_v2,new_segment_file,3,
[error] <0.3456856.0>          [{file,"rabbit_classic_queue_index_v2.erl"},{line,611}]},
[error] <0.3456856.0>      {rabbit_classic_queue_index_v2,publish,8,
[error] <0.3456856.0>          [{file,"rabbit_classic_queue_index_v2.erl"},{line,600}]},
[error] <0.3456856.0>      {rabbit_variable_queue,maybe_write_index_to_disk,3,
[error] <0.3456856.0>          [{file,"rabbit_variable_queue.erl"},{line,1847}]},
[error] <0.3456856.0>      {rabbit_variable_queue,publish_delivered1,5,
[error] <0.3456856.0>          [{file,"rabbit_variable_queue.erl"},{line,1742}]},
[error] <0.3456856.0>      {rabbit_variable_queue,publish_delivered,4,
[error] <0.3456856.0>          [{file,"rabbit_variable_queue.erl"},{line,544}]},
[error] <0.3456856.0>      {rabbit_priority_queue,publish_delivered,4,
[error] <0.3456856.0>          [{file,"rabbit_priority_queue.erl"},{line,223}]},
[error] <0.3456856.0>      {rabbit_amqqueue_process,'-attempt_delivery/4-fun-0-',10,
[error] <0.3456856.0>          [{file,"rabbit_amqqueue_process.erl"},{line,733}]},
[error] <0.3456856.0>      {rabbit_queue_consumers,deliver_to_consumer,4,
[error] <0.3456856.0>          [{file,"rabbit_queue_consumers.erl"},{line,344}]}]}
[error] <0.3456856.0> 
[error] <0.3456856.0>   crasher:
[error] <0.3456856.0>     initial call: rabbit_amqqueue_process:init/1
[error] <0.3456856.0>     pid: <0.3456856.0>
[error] <0.3456856.0>     registered_name: []
[error] <0.3456856.0>     exception exit: {{badmatch,{error,eacces}},
[error] <0.3456856.0>                      [{rabbit_classic_queue_index_v2,new_segment_file,3,
[error] <0.3456856.0>                           [{file,"rabbit_classic_queue_index_v2.erl"},
[error] <0.3456856.0>                            {line,611}]},
[error] <0.3456856.0>                       {rabbit_classic_queue_index_v2,publish,8,
[error] <0.3456856.0>                           [{file,"rabbit_classic_queue_index_v2.erl"},
[error] <0.3456856.0>                            {line,600}]},
[error] <0.3456856.0>                       {rabbit_variable_queue,maybe_write_index_to_disk,3,
[error] <0.3456856.0>                           [{file,"rabbit_variable_queue.erl"},{line,1847}]},
[error] <0.3456856.0>                       {rabbit_variable_queue,publish_delivered1,5,
[error] <0.3456856.0>                           [{file,"rabbit_variable_queue.erl"},{line,1742}]},
[error] <0.3456856.0>                       {rabbit_variable_queue,publish_delivered,4,
[error] <0.3456856.0>                           [{file,"rabbit_variable_queue.erl"},{line,544}]},
[error] <0.3456856.0>                       {rabbit_priority_queue,publish_delivered,4,
[error] <0.3456856.0>                           [{file,"rabbit_priority_queue.erl"},{line,223}]},
[error] <0.3456856.0>                       {rabbit_amqqueue_process,'-attempt_delivery/4-fun-0-',
[error] <0.3456856.0>                           10,
[error] <0.3456856.0>                           [{file,"rabbit_amqqueue_process.erl"},{line,733}]},
[error] <0.3456856.0>                       {rabbit_queue_consumers,deliver_to_consumer,4,
[error] <0.3456856.0>                           [{file,"rabbit_queue_consumers.erl"},{line,344}]}]}
[error] <0.3456856.0>       in function  gen_server2:terminate/3 (gen_server2.erl, line 1174)
[error] <0.3456856.0>     ancestors: [<0.513.0>,<0.430.0>,<0.327.0>,<0.326.0>,
[error] <0.3456856.0>                   rabbit_vhost_sup_sup,rabbit_sup,<0.207.0>]
[error] <0.3456856.0>     message_queue_len: 2
[error] <0.3456856.0>     messages: [{'$gen_cast',
[error] <0.3456856.0>                       {deliver,
[error] <0.3456856.0>                           {delivery,false,true,<0.3535883.0>,
[error] <0.3456856.0>                               {mc,mc_amqpl,
[error] <0.3456856.0>                                   {content,60,none,
[error] <0.3456856.0>                                       <<180,128,32,97,112,112,108,105,99,97,
[error] <0.3456856.0>                                         116,105,111,110,47,118,110,100,46,109,
[error] <0.3456856.0>                                         97,115,115,116,114,97,110,115,105,116,
[error] <0.3456856.0>                                         43,106,115,111,110,0,0,0,29,19,120,45,
[error] <0.3456856.0>                                         100,111,116,110,101,116,45,112,117,98,
[error] <0.3456856.0>                                         45,115,101,113,45,110,111,108,0,0,0,0,
[error] <0.3456856.0>                                         0,0,2,196,2,36,98,48,50,57,48,48,48,48,
[error] <0.3456856.0>                                         45,57,50,55,49,45,53,54,57,99,45,53,52,
[error] <0.3456856.0>                                         57,51,45,48,56,100,101,97,48,52,53,53,
[error] <0.3456856.0>                                         97,56,50,36,98,48,50,57,48,48,48,48,45,
[error] <0.3456856.0>                                         57,50,55,49,45,53,54,57,99,45,52,49,51,
[error] <0.3456856.0>                                         97,45,48,56,100,101,97,48,52,53,53,98,
[error] <0.3456856.0>                                         49,101>>,
[error] <0.3456856.0>                                       rabbit_framing_amqp_0_9_1,
[error] <0.3456856.0>                                       [<<>>]},
[error] <0.3456856.0>                                   #{id =>
[error] <0.3456856.0>                                         <<123,125,129,169,222,32,235,82,231,
[error] <0.3456856.0>                                           180,244,164,213,192,157,39>>,
[error] <0.3456856.0>                                     x => <<"vpl-job-routing-slip">>,
[error] <0.3456856.0>                                     rk => [<<>>],
[error] <0.3456856.0>                                     rts => 1776844887521}},
[error] <0.3456856.0>                               708,flow},
[error] <0.3456856.0>                           false}},
[error] <0.3456856.0>                   {'$gen_cast',
[error] <0.3456856.0>                       {run_backing_queue,rabbit_variable_queue,
[error] <0.3456856.0>                           #Fun<rabbit_variable_queue.23.62405665>}}]
[error] <0.3456856.0>     links: [<0.513.0>]
[error] <0.3456856.0>     dictionary: [{credit_blocked,[]},
[error] <0.3456856.0>                   {{ch,<0.3535793.0>},
[error] <0.3456856.0>                    {cr,<0.3535793.0>,#Ref<0.2068921784.2808610823.197000>,
[error] <0.3456856.0>                        {0,[],[]},
[error] <0.3456856.0>                        1,
[error] <0.3456856.0>                        {queue,[],[],0},
[error] <0.3456856.0>                        {qstate,<0.3535809.0>,dormant,
[error] <0.3456856.0>                            {1,
[error] <0.3456856.0>                             {<<"amq.ctag-9eKX1bXy3mwatW1skCcwYg">>,
[error] <0.3456856.0>                              {credit,38,auto},
[error] <0.3456856.0>                              nil,nil}}},
[error] <0.3456856.0>                        37,#{}}},
[error] <0.3456856.0>                   {{credit_to,<0.3535880.0>},36},
[error] <0.3456856.0>                   {{credit_to,<0.3535916.0>},196},
[error] <0.3456856.0>                   {{credit_to,<0.3535843.0>},198},
[error] <0.3456856.0>                   {{credit_to,<0.3535935.0>},199},
[error] <0.3456856.0>                   {'$logger_metadata$',#{domain => [rabbitmq,queue]}},
[error] <0.3456856.0>                   {{credit_to,<0.3535892.0>},42},
[error] <0.3456856.0>                   {guid,{{942836513,4213326534,3693924947,815918632},1}},
[error] <0.3456856.0>                   {{credit_to,<0.3535864.0>},192},
[error] <0.3456856.0>                   {{credit_to,<0.3535883.0>},185},
[error] <0.3456856.0>                   {{credit_to,<0.3535953.0>},198},
[error] <0.3456856.0>                   {{credit_to,<0.3535853.0>},198},
[error] <0.3456856.0>                   {{credit_to,<0.3535818.0>},185},
[error] <0.3456856.0>                   {{credit_to,<0.3535847.0>},81},
[error] <0.3456856.0>                   {{credit_to,<0.3535858.0>},185},
[error] <0.3456856.0>                   {{credit_from,<0.422.0>},3905},
[error] <0.3456856.0>                   {rand_seed,
[error] <0.3456856.0>                       {#{max => 288230376151711743,type => exsplus,
[error] <0.3456856.0>                          next => #Fun<rand.5.40079776>,
[error] <0.3456856.0>                          jump => #Fun<rand.3.40079776>},
[error] <0.3456856.0>                        [159911850624263666|272430303914224767]}},
[error] <0.3456856.0>                   {{credit_to,<0.3535838.0>},25},
[error] <0.3456856.0>                   {process_name,
[error] <0.3456856.0>                       {rabbit_amqqueue_process,
[error] <0.3456856.0>                           {resource,<<"/">>,queue,
[error] <0.3456856.0>                               <<"vpl-job-routing-slip">>}}},
[error] <0.3456856.0>                   {{credit_to,<0.3535836.0>},192}]
[error] <0.3456856.0>     trap_exit: true
[error] <0.3456856.0>     status: running
[error] <0.3456856.0>     heap_size: 75113
[error] <0.3456856.0>     stack_size: 29
[error] <0.3456856.0>     reductions: 5674723
[error] <0.3456856.0>   neighbours:
[error] <0.3456856.0> 
[error] <0.513.0>     supervisor: {<0.513.0>,rabbit_amqqueue_sup}
[error] <0.513.0>     errorContext: child_terminated
[error] <0.513.0>     reason: {{badmatch,{error,eacces}},
[error] <0.513.0>              [{rabbit_classic_queue_index_v2,new_segment_file,3,
[error] <0.513.0>                   [{file,"rabbit_classic_queue_index_v2.erl"},{line,611}]},
[error] <0.513.0>               {rabbit_classic_queue_index_v2,publish,8,
[error] <0.513.0>                   [{file,"rabbit_classic_queue_index_v2.erl"},{line,600}]},
[error] <0.513.0>               {rabbit_variable_queue,maybe_write_index_to_disk,3,
[error] <0.513.0>                   [{file,"rabbit_variable_queue.erl"},{line,1847}]},
[error] <0.513.0>               {rabbit_variable_queue,publish_delivered1,5,
[error] <0.513.0>                   [{file,"rabbit_variable_queue.erl"},{line,1742}]},
[error] <0.513.0>               {rabbit_variable_queue,publish_delivered,4,
[error] <0.513.0>                   [{file,"rabbit_variable_queue.erl"},{line,544}]},
[error] <0.513.0>               {rabbit_priority_queue,publish_delivered,4,
[error] <0.513.0>                   [{file,"rabbit_priority_queue.erl"},{line,223}]},
[error] <0.513.0>               {rabbit_amqqueue_process,'-attempt_delivery/4-fun-0-',10,
[error] <0.513.0>                   [{file,"rabbit_amqqueue_process.erl"},{line,733}]},
[error] <0.513.0>               {rabbit_queue_consumers,deliver_to_consumer,4,
[error] <0.513.0>                   [{file,"rabbit_queue_consumers.erl"},{line,344}]}]}
[error] <0.513.0>     offender: [{pid,<0.3456856.0>},
[error] <0.513.0>                {id,rabbit_amqqueue},
[error] <0.513.0>                {mfargs,
[error] <0.513.0>                    {rabbit_amqqueue_process,start_link,
[error] <0.513.0>                        [{amqqueue,
[error] <0.513.0>                             {resource,<<"/">>,queue,
[error] <0.513.0>                                 <<"vpl-job-routing-slip">>},
[error] <0.513.0>                             true,false,none,
[error] <0.513.0>                             [{<<"x-queue-type">>,longstr,<<"classic">>}],
[error] <0.513.0>                             <0.514.0>,[],[],[],undefined,undefined,[],[],
[error] <0.513.0>                             stopped,0,[],<<"/">>,
[error] <0.513.0>                             #{user => <<"bob">>},
[error] <0.513.0>                             rabbit_classic_queue,#{}},
[error] <0.513.0>                         <0.512.0>]}},
[error] <0.513.0>                {restart_type,transient},
[error] <0.513.0>                {significant,true},
[error] <0.513.0>                {shutdown,600000},
[error] <0.513.0>                {child_type,worker}]
[error] <0.513.0> 
[error] <0.3567560.0> Restarting crashed queue 'vpl-job-routing-slip' in vhost '/'.
[error] <0.3567560.0> ** Generic server <0.3567560.0> terminating
[error] <0.3567560.0> ** Last message in was {'$gen_cast',init}
[error] <0.3567560.0> ** When Server state == {q,{amqqueue,
[error] <0.3567560.0>                                {resource,<<"/">>,queue,
[error] <0.3567560.0>                                    <<"vpl-job-routing-slip">>},
[error] <0.3567560.0>                                true,false,none,
[error] <0.3567560.0>                                [{<<"x-queue-type">>,longstr,<<"classic">>}],
[error] <0.3567560.0>                                <0.3567560.0>,[],[],[],undefined,undefined,[],
[error] <0.3567560.0>                                [],crashed,0,[],<<"/">>,
[error] <0.3567560.0>                                #{user => <<"bob">>},
[error] <0.3567560.0>                                rabbit_classic_queue,#{}},
[error] <0.3567560.0>                            none,false,undefined,backing_queue_state_truncated,
[error] <0.3567560.0>                            {state,
[error] <0.3567560.0>                                {queue,[],[],0},
[error] <0.3567560.0>                                {active,-575723332918092,1.0}},
[error] <0.3567560.0>                            undefined,undefined,undefined,undefined,
[error] <0.3567560.0>                            {state,fine,5000,undefined},
[error] <0.3567560.0>                            #{},undefined,undefined,undefined,
[error] <0.3567560.0>                            {state,#{},delegate},
[error] <0.3567560.0>                            undefined,undefined,undefined,undefined,
[error] <0.3567560.0>                            'drop-head',0,0,running,false}
[error] <0.3567560.0> ** Reason for termination ==
[error] <0.3567560.0> ** {{badmatch,{error,eacces}},
[error] <0.3567560.0>     [{rabbit_classic_queue_index_v2,recover_segments,5,
[error] <0.3567560.0>          [{file,"rabbit_classic_queue_index_v2.erl"},{line,328}]},
[error] <0.3567560.0>      {rabbit_classic_queue_index_v2,recover_segments,8,
[error] <0.3567560.0>          [{file,"rabbit_classic_queue_index_v2.erl"},{line,301}]},
[error] <0.3567560.0>      {rabbit_classic_queue_index_v2,recover,7,
[error] <0.3567560.0>          [{file,"rabbit_classic_queue_index_v2.erl"},{line,271}]},
[error] <0.3567560.0>      {rabbit_variable_queue,init,5,
[error] <0.3567560.0>          [{file,"rabbit_variable_queue.erl"},{line,459}]},
[error] <0.3567560.0>      {rabbit_priority_queue,init,3,
[error] <0.3567560.0>          [{file,"rabbit_priority_queue.erl"},{line,152}]},
[error] <0.3567560.0>      {rabbit_amqqueue_process,init_it2,3,
[error] <0.3567560.0>          [{file,"rabbit_amqqueue_process.erl"},{line,224}]},
[error] <0.3567560.0>      {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1056}]},
[error] <0.3567560.0>      {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,329}]}]}
[error] <0.3567560.0> 
[error] <0.3567560.0>   crasher:
[error] <0.3567560.0>     initial call: rabbit_amqqueue_process:init/1
[error] <0.3567560.0>     pid: <0.3567560.0>
[error] <0.3567560.0>     registered_name: []
[error] <0.3567560.0>     exception exit: {{badmatch,{error,eacces}},
[error] <0.3567560.0>                      [{rabbit_classic_queue_index_v2,recover_segments,5,
[error] <0.3567560.0>                           [{file,"rabbit_classic_queue_index_v2.erl"},
[error] <0.3567560.0>                            {line,328}]},
[error] <0.3567560.0>                       {rabbit_classic_queue_index_v2,recover_segments,8,
[error] <0.3567560.0>                           [{file,"rabbit_classic_queue_index_v2.erl"},
[error] <0.3567560.0>                            {line,301}]},
[error] <0.3567560.0>                       {rabbit_classic_queue_index_v2,recover,7,
[error] <0.3567560.0>                           [{file,"rabbit_classic_queue_index_v2.erl"},
[error] <0.3567560.0>                            {line,271}]},
[error] <0.3567560.0>                       {rabbit_variable_queue,init,5,
[error] <0.3567560.0>                           [{file,"rabbit_variable_queue.erl"},{line,459}]},
[error] <0.3567560.0>                       {rabbit_priority_queue,init,3,
[error] <0.3567560.0>                           [{file,"rabbit_priority_queue.erl"},{line,152}]},
[error] <0.3567560.0>                       {rabbit_amqqueue_process,init_it2,3,
[error] <0.3567560.0>                           [{file,"rabbit_amqqueue_process.erl"},{line,224}]},
[error] <0.3567560.0>                       {gen_server2,handle_msg,2,
[error] <0.3567560.0>                           [{file,"gen_server2.erl"},{line,1056}]},
[error] <0.3567560.0>                       {proc_lib,init_p_do_apply,3,
[error] <0.3567560.0>                           [{file,"proc_lib.erl"},{line,329}]}]}
[error] <0.3567560.0>       in function  gen_server2:terminate/3 (gen_server2.erl, line 1174)
[error] <0.3567560.0>     ancestors: [<0.513.0>,<0.430.0>,<0.327.0>,<0.326.0>,
[error] <0.3567560.0>                   rabbit_vhost_sup_sup,rabbit_sup,<0.207.0>]
[error] <0.3567560.0>     message_queue_len: 1
[error] <0.3567560.0>     messages: [{'$gen_cast',
[error] <0.3567560.0>                       {deliver,
[error] <0.3567560.0>                           {delivery,false,true,<0.3535847.0>,
[error] <0.3567560.0>                               {mc,mc_amqpl,
[error] <0.3567560.0>                                   {content,60,none,
[error] <0.3567560.0>                                       <<176,128,32,97,112,112,108,105,99,97,
[error] <0.3567560.0>                                         116,105,111,110,47,118,110,100,46,109,
[error] <0.3567560.0>                                         97,115,115,116,114,97,110,115,105,116,
[error] <0.3567560.0>                                         43,106,115,111,110,0,0,0,29,19,120,45,
[error] <0.3567560.0>                                         100,111,116,110,101,116,45,112,117,98,
[error] <0.3567560.0>                                         45,115,101,113,45,110,111,108,0,0,0,0,
[error] <0.3567560.0>                                         0,0,16,123,2,36,98,48,50,57,48,48,48,
[error] <0.3567560.0>                                         48,45,57,50,55,49,45,53,54,57,99,45,50,
[error] <0.3567560.0>                                         54,51,55,45,48,56,100,101,97,48,52,53,
[error] <0.3567560.0>                                         53,98,56,55>>,
[error] <0.3567560.0>                                       rabbit_framing_amqp_0_9_1,
[error] <0.3567560.0>                                       [<<>>]},
[error] <0.3567560.0>                                   #{id =>
[error] <0.3567560.0>                                         <<126,86,227,29,127,184,61,59,203,228,
[error] <0.3567560.0>                                           242,160,73,43,104,43>>,
[error] <0.3567560.0>                                     x =>
[error] <0.3567560.0>                                         <<"XYZ.VPL.Contracts.Pipelines:JobProcessingStarted">>,
[error] <0.3567560.0>                                     rk => [<<>>],
[error] <0.3567560.0>                                     rts => 1776844888196}},
[error] <0.3567560.0>                               4219,flow},
[error] <0.3567560.0>                           false}}]
[error] <0.3567560.0>     links: [<0.513.0>]
[error] <0.3567560.0>     dictionary: [{'$logger_metadata$',#{domain => [rabbitmq,queue]}},
[error] <0.3567560.0>                   {rand_seed,
[error] <0.3567560.0>                       {#{max => 288230376151711743,type => exsplus,
[error] <0.3567560.0>                          next => #Fun<rand.5.40079776>,
[error] <0.3567560.0>                          jump => #Fun<rand.3.40079776>},
[error] <0.3567560.0>                        [165056089596967982|267316131352707923]}},
[error] <0.3567560.0>                   {guid,{{1290392614,2134180762,2553636000,86269042},1}},
[error] <0.3567560.0>                   {process_name,
[error] <0.3567560.0>                       {rabbit_amqqueue_process,
[error] <0.3567560.0>                           {resource,<<"/">>,queue,
[error] <0.3567560.0>                               <<"vpl-job-routing-slip">>}}}]
[error] <0.3567560.0>     trap_exit: true
[error] <0.3567560.0>     status: running
[error] <0.3567560.0>     heap_size: 10958
[error] <0.3567560.0>     stack_size: 29
[error] <0.3567560.0>     reductions: 34631
[error] <0.3567560.0>   neighbours:
[error] <0.3567560.0> 
[error] <0.513.0>     supervisor: {<0.513.0>,rabbit_amqqueue_sup}
[error] <0.513.0>     errorContext: child_terminated
[error] <0.513.0>     reason: {{badmatch,{error,eacces}},
[error] <0.513.0>              [{rabbit_classic_queue_index_v2,recover_segments,5,
[error] <0.513.0>                   [{file,"rabbit_classic_queue_index_v2.erl"},{line,328}]},
[error] <0.513.0>               {rabbit_classic_queue_index_v2,recover_segments,8,
[error] <0.513.0>                   [{file,"rabbit_classic_queue_index_v2.erl"},{line,301}]},
[error] <0.513.0>               {rabbit_classic_queue_index_v2,recover,7,
[error] <0.513.0>                   [{file,"rabbit_classic_queue_index_v2.erl"},{line,271}]},
[error] <0.513.0>               {rabbit_variable_queue,init,5,
[error] <0.513.0>                   [{file,"rabbit_variable_queue.erl"},{line,459}]},
[error] <0.513.0>               {rabbit_priority_queue,init,3,
[error] <0.513.0>                   [{file,"rabbit_priority_queue.erl"},{line,152}]},
[error] <0.513.0>               {rabbit_amqqueue_process,init_it2,3,
[error] <0.513.0>                   [{file,"rabbit_amqqueue_process.erl"},{line,224}]},
[error] <0.513.0>               {gen_server2,handle_msg,2,
[error] <0.513.0>                   [{file,"gen_server2.erl"},{line,1056}]},
[error] <0.513.0>               {proc_lib,init_p_do_apply,3,
[error] <0.513.0>                   [{file,"proc_lib.erl"},{line,329}]}]}
[error] <0.513.0>     offender: [{pid,<0.3567560.0>},
[error] <0.513.0>                {id,rabbit_amqqueue},
[error] <0.513.0>                {mfargs,
[error] <0.513.0>                    {rabbit_amqqueue_process,start_link,
[error] <0.513.0>                        [{amqqueue,
[error] <0.513.0>                             {resource,<<"/">>,queue,
[error] <0.513.0>                                 <<"vpl-job-routing-slip">>},
[error] <0.513.0>                             true,false,none,
[error] <0.513.0>                             [{<<"x-queue-type">>,longstr,<<"classic">>}],
[error] <0.513.0>                             <0.514.0>,[],[],[],undefined,undefined,[],[],
[error] <0.513.0>                             stopped,0,[],<<"/">>,
[error] <0.513.0>                             #{user => <<"bob">>},
[error] <0.513.0>                             rabbit_classic_queue,#{}},
[error] <0.513.0>                         <0.512.0>]}},
[error] <0.513.0>                {restart_type,transient},
[error] <0.513.0>                {significant,true},
[error] <0.513.0>                {shutdown,600000},
[error] <0.513.0>                {child_type,worker}]
[error] <0.513.0> 
[error] <0.3567622.0> Restarting crashed queue 'vpl-job-routing-slip' in vhost '/'.
[error] <0.3567622.0> ** Generic server <0.3567622.0> terminating
[error] <0.3567622.0> ** Last message in was {'$gen_cast',init}
[error] <0.3567622.0> ** When Server state == {q,{amqqueue,
[error] <0.3567622.0>                                {resource,<<"/">>,queue,
[error] <0.3567622.0>                                    <<"vpl-job-routing-slip">>},
[error] <0.3567622.0>                                true,false,none,
[error] <0.3567622.0>                                [{<<"x-queue-type">>,longstr,<<"classic">>}],
[error] <0.3567622.0>                                <0.3567622.0>,[],[],[],undefined,undefined,[],
[error] <0.3567622.0>                                [],live,0,[],<<"/">>,
[error] <0.3567622.0>                                #{user => <<"bob">>},
[error] <0.3567622.0>                                rabbit_classic_queue,#{}},
[error] <0.3567622.0>                            none,false,undefined,backing_queue_state_truncated,
[error] <0.3567622.0>                            {state,
[error] <0.3567622.0>                                {queue,[],[],0},
[error] <0.3567622.0>                                {active,-575723332417258,1.0}},
[error] <0.3567622.0>                            undefined,undefined,undefined,undefined,
[error] <0.3567622.0>                            {state,fine,5000,undefined},
[error] <0.3567622.0>                            #{},undefined,undefined,undefined,
[error] <0.3567622.0>                            {state,#{},delegate},
[error] <0.3567622.0>                            undefined,undefined,undefined,undefined,
[error] <0.3567622.0>                            'drop-head',0,0,running,false}
[error] <0.3567622.0> ** Reason for termination ==
[error] <0.3567622.0> ** {{badmatch,{error,eacces}},
[error] <0.3567622.0>     [{rabbit_classic_queue_index_v2,recover_segments,5,
[error] <0.3567622.0>          [{file,"rabbit_classic_queue_index_v2.erl"},{line,328}]},
[error] <0.3567622.0>      {rabbit_classic_queue_index_v2,recover_segments,8,
[error] <0.3567622.0>          [{file,"rabbit_classic_queue_index_v2.erl"},{line,301}]},
[error] <0.3567622.0>      {rabbit_classic_queue_index_v2,recover,7,
[error] <0.3567622.0>          [{file,"rabbit_classic_queue_index_v2.erl"},{line,271}]},
[error] <0.3567622.0>      {rabbit_variable_queue,init,5,
[error] <0.3567622.0>          [{file,"rabbit_variable_queue.erl"},{line,459}]},
[error] <0.3567622.0>      {rabbit_priority_queue,init,3,
[error] <0.3567622.0>          [{file,"rabbit_priority_queue.erl"},{line,152}]},
[error] <0.3567622.0>      {rabbit_amqqueue_process,init_it2,3,
[error] <0.3567622.0>          [{file,"rabbit_amqqueue_process.erl"},{line,224}]},
[error] <0.3567622.0>      {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1056}]},
[error] <0.3567622.0>      {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,329}]}]}
[error] <0.3567622.0> 
[error] <0.3567622.0>   crasher:
[error] <0.3567622.0>     initial call: rabbit_amqqueue_process:init/1
[error] <0.3567622.0>     pid: <0.3567622.0>
[error] <0.3567622.0>     registered_name: []
[error] <0.3567622.0>     exception exit: {{badmatch,{error,eacces}},
[error] <0.3567622.0>                      [{rabbit_classic_queue_index_v2,recover_segments,5,
[error] <0.3567622.0>                           [{file,"rabbit_classic_queue_index_v2.erl"},
[error] <0.3567622.0>                            {line,328}]},
[error] <0.3567622.0>                       {rabbit_classic_queue_index_v2,recover_segments,8,
[error] <0.3567622.0>                           [{file,"rabbit_classic_queue_index_v2.erl"},
[error] <0.3567622.0>                            {line,301}]},
[error] <0.3567622.0>                       {rabbit_classic_queue_index_v2,recover,7,
[error] <0.3567622.0>                           [{file,"rabbit_classic_queue_index_v2.erl"},
[error] <0.3567622.0>                            {line,271}]},
[error] <0.3567622.0>                       {rabbit_variable_queue,init,5,
[error] <0.3567622.0>                           [{file,"rabbit_variable_queue.erl"},{line,459}]},
[error] <0.3567622.0>                       {rabbit_priority_queue,init,3,
[error] <0.3567622.0>                           [{file,"rabbit_priority_queue.erl"},{line,152}]},
[error] <0.3567622.0>                       {rabbit_amqqueue_process,init_it2,3,
[error] <0.3567622.0>                           [{file,"rabbit_amqqueue_process.erl"},{line,224}]},
[error] <0.3567622.0>                       {gen_server2,handle_msg,2,
[error] <0.3567622.0>                           [{file,"gen_server2.erl"},{line,1056}]},
[error] <0.3567622.0>                       {proc_lib,init_p_do_apply,3,
[error] <0.3567622.0>                           [{file,"proc_lib.erl"},{line,329}]}]}
[error] <0.3567622.0>       in function  gen_server2:terminate/3 (gen_server2.erl, line 1174)
[error] <0.3567622.0>     ancestors: [<0.513.0>,<0.430.0>,<0.327.0>,<0.326.0>,
[error] <0.3567622.0>                   rabbit_vhost_sup_sup,rabbit_sup,<0.207.0>]
[error] <0.3567622.0>     message_queue_len: 3
[error] <0.3567622.0>     messages: [{'$gen_cast',
[error] <0.3567622.0>                       {deliver,
[error] <0.3567622.0>                           {delivery,false,true,<0.3535847.0>,
[error] <0.3567622.0>                               {mc,mc_amqpl,
[error] <0.3567622.0>                                   {content,60,none,
[error] <0.3567622.0>                                       <<176,128,32,97,112,112,108,105,99,97,
[error] <0.3567622.0>                                         116,105,111,110,47,118,110,100,46,109,
[error] <0.3567622.0>                                         97,115,115,116,114,97,110,115,105,116,
[error] <0.3567622.0>                                         43,106,115,111,110,0,0,0,29,19,120,45,
[error] <0.3567622.0>                                         100,111,116,110,101,116,45,112,117,98,
[error] <0.3567622.0>                                         45,115,101,113,45,110,111,108,0,0,0,0,
[error] <0.3567622.0>                                         0,0,16,126,2,36,98,48,50,57,48,48,48,
[error] <0.3567622.0>                                         48,45,57,50,55,49,45,53,54,57,99,45,55,
[error] <0.3567622.0>                                         99,98,98,45,48,56,100,101,97,48,52,53,
[error] <0.3567622.0>                                         53,98,99,102>>,
[error] <0.3567622.0>                                       rabbit_framing_amqp_0_9_1,
[error] <0.3567622.0>                                       [<<>>]},
[error] <0.3567622.0>                                   #{id =>
[error] <0.3567622.0>                                         <<89,185,129,154,219,118,27,17,146,93,
[error] <0.3567622.0>                                           115,58,57,169,9,46>>,
[error] <0.3567622.0>                                     x =>
[error] <0.3567622.0>                                         <<"XYZ.VPL.Contracts.Pipelines:JobProcessingStarted">>,
[error] <0.3567622.0>                                     rk => [<<>>],
[error] <0.3567622.0>                                     rts => 1776844888680}},
[error] <0.3567622.0>                               4222,flow},
[error] <0.3567622.0>                           false}},
[error] <0.3567622.0>                   {'$gen_cast',
[error] <0.3567622.0>                       {deliver,
[error] <0.3567622.0>                           {delivery,false,true,<0.3535847.0>,
[error] <0.3567622.0>                               {mc,mc_amqpl,
[error] <0.3567622.0>                                   {content,60,none,
[error] <0.3567622.0>                                       <<176,128,32,97,112,112,108,105,99,97,
[error] <0.3567622.0>                                         116,105,111,110,47,118,110,100,46,109,
[error] <0.3567622.0>                                         97,115,115,116,114,97,110,115,105,116,
[error] <0.3567622.0>                                         43,106,115,111,110,0,0,0,29,19,120,45,
[error] <0.3567622.0>                                         100,111,116,110,101,116,45,112,117,98,
[error] <0.3567622.0>                                         45,115,101,113,45,110,111,108,0,0,0,0,
[error] <0.3567622.0>                                         0,0,16,127,2,36,98,48,50,57,48,48,48,
[error] <0.3567622.0>                                         48,45,57,50,55,49,45,53,54,57,99,45,97,
[error] <0.3567622.0>                                         98,49,49,45,48,56,100,101,97,48,52,53,
[error] <0.3567622.0>                                         53,98,100,48>>,
[error] <0.3567622.0>                                       rabbit_framing_amqp_0_9_1,
[error] <0.3567622.0>                                       [<<>>]},
[error] <0.3567622.0>                                   #{id =>
[error] <0.3567622.0>                                         <<102,165,83,120,47,142,59,83,132,122,
[error] <0.3567622.0>                                           65,71,189,211,72,105>>,
[error] <0.3567622.0>                                     x =>
[error] <0.3567622.0>                                         <<"XYZ.VPL.Contracts.Pipelines:JobProcessingStarted">>,
[error] <0.3567622.0>                                     rk => [<<>>],
[error] <0.3567622.0>                                     rts => 1776844888680}},
[error] <0.3567622.0>                               4223,flow},
[error] <0.3567622.0>                           false}},
[error] <0.3567622.0>                   {'$gen_cast',
[error] <0.3567622.0>                       {deliver,
[error] <0.3567622.0>                           {delivery,false,true,<0.3535847.0>,
[error] <0.3567622.0>                               {mc,mc_amqpl,
[error] <0.3567622.0>                                   {content,60,none,
[error] <0.3567622.0>                                       <<176,128,32,97,112,112,108,105,99,97,
[error] <0.3567622.0>                                         116,105,111,110,47,118,110,100,46,109,
[error] <0.3567622.0>                                         97,115,115,116,114,97,110,115,105,116,
[error] <0.3567622.0>                                         43,106,115,111,110,0,0,0,29,19,120,45,
[error] <0.3567622.0>                                         100,111,116,110,101,116,45,112,117,98,
[error] <0.3567622.0>                                         45,115,101,113,45,110,111,108,0,0,0,0,
[error] <0.3567622.0>                                         0,0,16,128,2,36,98,48,50,57,48,48,48,
[error] <0.3567622.0>                                         48,45,57,50,55,49,45,53,54,57,99,45,51,
[error] <0.3567622.0>                                         53,49,48,45,48,56,100,101,97,48,52,53,
[error] <0.3567622.0>                                         53,98,101,98>>,
[error] <0.3567622.0>                                       rabbit_framing_amqp_0_9_1,
[error] <0.3567622.0>                                       [<<>]},
[error] <0.3567622.0>                                   #{id =>
[error] <0.3567622.0>                                         <<185,23,213,49,18,227,175,37,43,74,
[error] <0.3567622.0>                                           166,11,150,153,238,98>>,
[error] <0.3567622.0>                                     x =>
[error] <0.3567622.0>                                         <<"XYZ.VPL.Contracts.Pipelines:JobProcessingStarted">>,
[error] <0.3567622.0>                                     rk => [<<>>],
[error] <0.3567622.0>                                     rts => 1776844888893}},
[error] <0.3567622.0>                               4224,flow},
[error] <0.3567622.0>                           false}}]
[error] <0.3567622.0>     links: [<0.513.0>]
[error] <0.3567622.0>     dictionary: [{'$logger_metadata$',#{domain => [rabbitmq,queue]}},
[error] <0.3567622.0>                   {rand_seed,
[error] <0.3567622.0>                       {#{max => 288230376151711743,type => exsplus,
[error] <0.3567622.0>                          next => #Fun<rand.5.40079776>,
[error] <0.3567622.0>                          jump => #Fun<rand.3.40079776>},
[error] <0.3567622.0>                        [214491311071608366|1605824199608501]}},
[error] <0.3567622.0>                   {guid,{{826360828,2098099157,2657016660,2469477058},1}},
[error] <0.3567622.0>                   {process_name,
[error] <0.3567622.0>                       {rabbit_amqqueue_process,
[error] <0.3567622.0>                           {resource,<<"/">>,queue,
[error] <0.3567622.0>                               <<"vpl-job-routing-slip">>}}}]
[error] <0.3567622.0>     trap_exit: true
[error] <0.3567622.0>     status: running
[error] <0.3567622.0>     heap_size: 17731
[error] <0.3567622.0>     stack_size: 29
[error] <0.3567622.0>     reductions: 34434
[error] <0.3567622.0>   neighbours:
[error] <0.3567622.0> 
[error] <0.513.0>     supervisor: {<0.513.0>,rabbit_amqqueue_sup}
[error] <0.513.0>     errorContext: child_terminated
[error] <0.513.0>     reason: {{badmatch,{error,eacces}},
[error] <0.513.0>              [{rabbit_classic_queue_index_v2,recover_segments,5,
[error] <0.513.0>                   [{file,"rabbit_classic_queue_index_v2.erl"},{line,328}]},
[error] <0.513.0>               {rabbit_classic_queue_index_v2,recover_segments,8,
[error] <0.513.0>                   [{file,"rabbit_classic_queue_index_v2.erl"},{line,301}]},
[error] <0.513.0>               {rabbit_classic_queue_index_v2,recover,7,
[error] <0.513.0>                   [{file,"rabbit_classic_queue_index_v2.erl"},{line,271}]},
[error] <0.513.0>               {rabbit_variable_queue,init,5,
[error] <0.513.0>                   [{file,"rabbit_variable_queue.erl"},{line,459}]},
[error] <0.513.0>               {rabbit_priority_queue,init,3,
[error] <0.513.0>                   [{file,"rabbit_priority_queue.erl"},{line,152}]},
[error] <0.513.0>               {rabbit_amqqueue_process,init_it2,3,
[error] <0.513.0>                   [{file,"rabbit_amqqueue_process.erl"},{line,224}]},
[error] <0.513.0>               {gen_server2,handle_msg,2,
[error] <0.513.0>                   [{file,"gen_server2.erl"},{line,1056}]},
[error] <0.513.0>               {proc_lib,init_p_do_apply,3,
[error] <0.513.0>                   [{file,"proc_lib.erl"},{line,329}]}]}
[error] <0.513.0>     offender: [{pid,<0.3567622.0>},
[error] <0.513.0>                {id,rabbit_amqqueue},
[error] <0.513.0>                {mfargs,
[error] <0.513.0>                    {rabbit_amqqueue_process,start_link,
[error] <0.513.0>                        [{amqqueue,
[error] <0.513.0>                             {resource,<<"/">>,queue,
[error] <0.513.0>                                 <<"vpl-job-routing-slip">>},
[error] <0.513.0>                             true,false,none,
[error] <0.513.0>                             [{<<"x-queue-type">>,longstr,<<"classic">>}],
[error] <0.513.0>                             <0.514.0>,[],[],[],undefined,undefined,[],[],
[error] <0.513.0>                             stopped,0,[],<<"/">>,
[error] <0.513.0>                             #{user => <<"bob">>},
[error] <0.513.0>                             rabbit_classic_queue,#{}},
[error] <0.513.0>                         <0.512.0>]}},
[error] <0.513.0>                {restart_type,transient},
[error] <0.513.0>                {significant,true},
[error] <0.513.0>                {shutdown,600000},
[error] <0.513.0>                {child_type,worker}]
[error] <0.513.0> 
[error] <0.3567617.0> Restarting crashed queue 'vpl-job-routing-slip' in vhost '/'.

Logs from node 2 (if applicable, with sensitive values edited out)

NA

Logs from node 3 (if applicable, with sensitive values edited out)

NA

rabbitmq.conf

Default

Steps to deploy RabbitMQ cluster

NA

Steps to reproduce the behavior in question

NA

advanced.config

NA

Application code

NA

Kubernetes deployment file

NA

What problem are you trying to solve?

Could someone review the attached log file and decide if it may be solvable by applying the same methods used in #15134.
The work done in #15134 significantly improved the overall stability of the system.

[michaelklishin]

@lhoguin we should investigate this, it could be that more places need filesystem operation retries similar to #14231, #15136, #14108, #14178

[lhoguin]

new_segment_file and recover_segments already use the open_eventually function, including on 4.2. Whatever happened it failed despite our best effort there. I guess we could try to increase how many times open_eventually tries by default.

[michaelklishin]

We currently do up to 3 retries with 10 ms wait intervals. We can try bumping to 10.

[lhoguin]

@mister-mist Do you have an anti-virus or other file scanner running on these machines? Alternatively are the rabbit files shared to network or similar, or is there software that does something with the rabbit files?

[MirahImage]

From the description, it does seem there's an antivirus enabled in the environment. Antivirus on Windows almost always leads to problems with RabbitMQ.

[mister-mist]

• Microsoft Defender is enabled on the machine
• The recommended folders have been added to the exclusion list in Microsoft Defender
• The rabbit files are not shared to the network, and to our knowledge no other software are using the files.

[michaelklishin]

@mister-mist we don't have much to add: disable all anti-virus software on the host that runs RabbitMQ nodes. Or run nodes in a container.

We will not troubleshoot Microsoft Defender's behavior on your host for you.