Broker log stops updating

[pshields2]

Describe the bug
The broker container is outputting to stdout and that is put into the kubernetes log?
I am stress testing the broker and generating a lot of messages in the log. At some point the log stops updating and I have to restart the broker pod to get logging back. The broker otherwise does not seam to be impacted by this. We do not have max log set so it should log until the node runs out of space. Is there some configuration parameters for logging? Rotating or wrapping the log?

Tip

Vote this issue reacting with 👍 or 👎

[brusdev]

Since 1.0.23 activemq-artemis-broker-kubernetes-image only prints the log to the console, see https://github.com/arkmq-org/activemq-artemis-broker-kubernetes-image/blob/main/modules/activemq-artemis-launch/added/log4j2.properties.
This seems to be a Kubernetes config issue. How many bytes does the log contain when it stops updating?

[pshields2]

I captured the terminal output to a file to check the size of the first log -f run.

ls -l ~/Downloads/Terminal_Saved_Output.txt
-rw-r--r--@ 1 pshields  staff  9031290 Oct  1 12:19 /Users/pshields/Downloads/Terminal_Saved_Output.txt

I did some testing and it looks like the log is wrapping?

The size of the log is not consistent either.

Would this behavior be controlled in Kubernetes?

What I did to test.

1. Start logging with kubectl logs -f mutt-pod-ss-0 . log output to terminal eventually stops.
2. When it does ctl^c and then quickly repeat logging cmd but direct output to file. Broker startup message are not present in file.
3. restart tailing log output to terminal. repeat step 2 when output stops.

Examine log output saved to files for incrementing time stamps. Time stamps increase through progression of captured log files.

# wc broker0-930-25.log
  22248   93769 3605586 broker0-930-25.log
 # wc broker0-930-25-2.log
   7494   31427 1208269 broker0-930-25-2.log
 # wc broker0-930-25-3.log
  54206  222081 8524732 broker0-930-25-3.log
  # wc broker0-930-25-4.log
  51536  196618 7519089 broker0-930-25-4.log
 # wc broker0-930-25-5.log
  44142  163389 6243361 broker0-930-25-5.log

 # ls -l broker0-930-25*.log
-rw-r--r-- 1 root root 1208269 Sep 30 19:42 broker0-930-25-2.log
-rw-r--r-- 1 root root 8524732 Sep 30 19:52 broker0-930-25-3.log
-rw-r--r-- 1 root root 7519089 Sep 30 20:36 broker0-930-25-4.log
-rw-r--r-- 1 root root 6243361 Sep 30 20:58 broker0-930-25-5.log
-rw-r--r-- 1 root root 3605586 Sep 30 19:38 broker0-930-25.log

 # tail -20 broker0-930-25.log | grep ^2025
2025-09-30 19:38:25,392 ERROR [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error processing control packet: MqttSubscribeMessage[fixedHeader=MqttFixedHeader[messageType=SUBSCRIBE, isDup=false, qosLevel=AT_LEAST_ONCE, isRetain=false, remainingLength=82], variableHeader=MqttMessageIdAndPropertiesVariableHeader[messageId=1, properties=io.netty.handler.codec.mqtt.MqttProperties@3ebd3662], payload=MqttSubscribePayload[MqttTopicSubscription[topicFilter=trusted/+/dvs/dropped_messages/x3000c0s28b4n0:77, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]], MqttTopicSubscription[topicFilter=trusted/+/dvs/server/state, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]]

 # head -20 broker0-930-25-2.log | grep ^2025
2025-09-30 19:41:38,229 ERROR [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error processing control packet: MqttSubscribeMessage[fixedHeader=MqttFixedHeader[messageType=SUBSCRIBE, isDup=false, qosLevel=AT_LEAST_ONCE, isRetain=false, remainingLength=82], variableHeader=MqttMessageIdAndPropertiesVariableHeader[messageId=1, properties=io.netty.handler.codec.mqtt.MqttProperties@3ebd3662], payload=MqttSubscribePayload[MqttTopicSubscription[topicFilter=trusted/+/dvs/dropped_messages/x3000c0s28b3n0:17, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]], MqttTopicSubscription[topicFilter=trusted/+/dvs/server/state, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]]
2025-09-30 19:41:38,241 ERROR [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error processing control packet: MqttSubscribeMessage[fixedHeader=MqttFixedHeader[messageType=SUBSCRIBE, isDup=false, qosLevel=AT_LEAST_ONCE, isRetain=false, remainingLength=82], variableHeader=MqttMessageIdAndPropertiesVariableHeader[messageId=1, properties=io.netty.handler.codec.mqtt.MqttProperties@3ebd3662], payload=MqttSubscribePayload[MqttTopicSubscription[topicFilter=trusted/+/dvs/dropped_messages/x3000c0s28b3n0:22, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]], MqttTopicSubscription[topicFilter=trusted/+/dvs/server/state, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]]

 # tail -20 broker0-930-25-2.log | grep ^2025
2025-09-30 19:42:17,589 ERROR [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error processing control packet: MqttSubscribeMessage[fixedHeader=MqttFixedHeader[messageType=SUBSCRIBE, isDup=false, qosLevel=AT_LEAST_ONCE, isRetain=false, remainingLength=83], variableHeader=MqttMessageIdAndPropertiesVariableHeader[messageId=1, properties=io.netty.handler.codec.mqtt.MqttProperties@3ebd3662], payload=MqttSubscribePayload[MqttTopicSubscription[topicFilter=trusted/+/dvs/dropped_messages/x3000c0s26b3n0:172, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]], MqttTopicSubscription[topicFilter=trusted/+/dvs/server/state, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]]

 # head -20 broker0-930-25-3.log | grep ^2025
2025-09-30 19:47:02,257 ERROR [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error processing control packet: MqttSubscribeMessage[fixedHeader=MqttFixedHeader[messageType=SUBSCRIBE, isDup=false, qosLevel=AT_LEAST_ONCE, isRetain=false, remainingLength=82], variableHeader=MqttMessageIdAndPropertiesVariableHeader[messageId=1, properties=io.netty.handler.codec.mqtt.MqttProperties@3ebd3662], payload=MqttSubscribePayload[MqttTopicSubscription[topicFilter=trusted/+/dvs/dropped_messages/x3000c0s28b3n0:93, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]], MqttTopicSubscription[topicFilter=trusted/+/dvs/server/state, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]]
2025-09-30 19:47:02,942 ERROR [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error processing control packet: MqttSubscribeMessage[fixedHeader=MqttFixedHeader[messageType=SUBSCRIBE, isDup=false, qosLevel=AT_LEAST_ONCE, isRetain=false, remainingLength=83], variableHeader=MqttMessageIdAndPropertiesVariableHeader[messageId=1, properties=io.netty.handler.codec.mqtt.MqttProperties@3ebd3662], payload=MqttSubscribePayload[MqttTopicSubscription[topicFilter=trusted/+/dvs/dropped_messages/x3000c0s26b4n0:286, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]], MqttTopicSubscription[topicFilter=trusted/+/dvs/server/state, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]]

 # tail -20 broker0-930-25-3.log | grep ^2025
2025-09-30 19:52:22,367 ERROR [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error processing control packet: MqttSubscribeMessage[fixedHeader=MqttFixedHeader[messageType=SUBSCRIBE, isDup=false, qosLevel=AT_LEAST_ONCE, isRetain=false, remainingLength=83], variableHeader=MqttMessageIdAndPropertiesVariableHeader[messageId=1, properties=io.netty.handler.codec.mqtt.MqttProperties@3ebd3662], payload=MqttSubscribePayload[MqttTopicSubscription[topicFilter=trusted/+/dvs/dropped_messages/x3000c0s19b1n0:216, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]], MqttTopicSubscription[topicFilter=trusted/+/dvs/server/state, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]]

 # head -20 broker0-930-25-4.log | grep ^2025
2025-09-30 20:31:28,378 ERROR [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error processing control packet: MqttSubscribeMessage[fixedHeader=MqttFixedHeader[messageType=SUBSCRIBE, isDup=false, qosLevel=AT_LEAST_ONCE, isRetain=false, remainingLength=82], variableHeader=MqttMessageIdAndPropertiesVariableHeader[messageId=1, properties=io.netty.handler.codec.mqtt.MqttProperties@3ebd3662], payload=MqttSubscribePayload[MqttTopicSubscription[topicFilter=trusted/+/dvs/dropped_messages/x3000c0s19b2n0:43, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]], MqttTopicSubscription[topicFilter=trusted/+/dvs/server/state, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]]
2025-09-30 20:31:28,379 ERROR [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error processing control packet: MqttSubscribeMessage[fixedHeader=MqttFixedHeader[messageType=SUBSCRIBE, isDup=false, qosLevel=AT_LEAST_ONCE, isRetain=false, remainingLength=83], variableHeader=MqttMessageIdAndPropertiesVariableHeader[messageId=1, properties=io.netty.handler.codec.mqtt.MqttProperties@3ebd3662], payload=MqttSubscribePayload[MqttTopicSubscription[topicFilter=trusted/+/dvs/dropped_messages/x3000c0s19b2n0:268, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]], MqttTopicSubscription[topicFilter=trusted/+/dvs/server/state, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]]
# tail -20 broker0-930-25-4.log | grep ^2025
2025-09-30 20:36:35,741 ERROR [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834000: Error removing subscription.

 # head -20 broker0-930-25-5.log | grep ^2025
2025-09-30 20:54:11,652 ERROR [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error processing control packet: MqttSubscribeMessage[fixedHeader=MqttFixedHeader[messageType=SUBSCRIBE, isDup=false, qosLevel=AT_LEAST_ONCE, isRetain=false, remainingLength=82], variableHeader=MqttMessageIdAndPropertiesVariableHeader[messageId=1, properties=io.netty.handler.codec.mqtt.MqttProperties@3ebd3662], payload=MqttSubscribePayload[MqttTopicSubscription[topicFilter=trusted/+/dvs/dropped_messages/x3000c0s28b3n0:45, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]], MqttTopicSubscription[topicFilter=trusted/+/dvs/server/state, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]]
2025-09-30 20:54:11,685 ERROR [org.apache.activemq.artemis.core.protocol.mqtt] AMQ834002: Error processing control packet: MqttSubscribeMessage[fixedHeader=MqttFixedHeader[messageType=SUBSCRIBE, isDup=false, qosLevel=AT_LEAST_ONCE, isRetain=false, remainingLength=82], variableHeader=MqttMessageIdAndPropertiesVariableHeader[messageId=1, properties=io.netty.handler.codec.mqtt.MqttProperties@3ebd3662], payload=MqttSubscribePayload[MqttTopicSubscription[topicFilter=trusted/+/dvs/dropped_messages/x3000c0s28b4n0:41, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]], MqttTopicSubscription[topicFilter=trusted/+/dvs/server/state, option=SubscriptionOption[qos=AT_MOST_ONCE, noLocal=false, retainAsPublished=false, retainHandling=SEND_AT_SUBSCRIBE]]]]

[brusdev]

It seems you are hitting the default kubelet log-rotation configuration settings, containerLogMaxSize (default 10Mi) and containerLogMaxFiles (default 5), for further details see https://kubernetes.io/docs/concepts/cluster-administration/logging/#log-rotation