Status of ActiveMQ Artemis is false when using shared storage (readiness/liveness issue

[subhidh-agarwal]

I’m running ActiveMQ Artemis on Kubernetes using the AMQ Broker Operator with a shared store HA policy.

When scaling to 2 pods, one pod becomes primary and the second pod waits indefinitely to acquire the shared store lock (expected behavior). However, the waiting pod never becomes Ready, which causes issues with Kubernetes readiness/liveness handling.

apiVersion: broker.amq.io/v1beta1
kind: ActiveMQArtemis
metadata:
  name: artemis-broker
spec:
  env:
  - name: FQ_HOST_NAME
    value: '0.0.0.0'
  - name: JAVA_ARGS_APPEND
    value: '-Dwebconfig.bindings.artemis.uri=http://0.0.0.0:8161 -Xms8G -Xmx8G -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -XX:ReservedCodeCacheSize=1024m -XX:NewRatio=2 -Xss256k -XX:MaxDirectMemorySize=1536m -XX:+ExitOnOutOfMemoryError -XX:+UseContainerSupport'
  deploymentPlan:
    size: 1
    persistenceEnabled: false
    clustered: true
    livenessProbe:
      exec:
        command:
        - test
        - -f
        - /home/jboss/amq-broker/lock/cli.lock
      initialDelaySeconds: 5
      timeoutSeconds: 5
      periodSeconds: 5
      successThreshold: 1
      failureThreshold: 2
    extraVolumes:
    - name: shared-data
      persistentVolumeClaim:
        claimName: artemis-broker-artemis-broker-ss-0
    extraVolumeMounts:
    - name: shared-data
      mountPath: /opt/amq-broker/data
  brokerProperties:
  - "HAPolicyConfiguration=SHARED_STORE_PRIMARY"
  - "journalDirectory=/opt/amq-broker/data/journal"
  - "pagingDirectory=/opt/amq-broker/data/paging"
  - "bindingsDirectory=/opt/amq-broker/data/bindings"
  - "largeMessagesDirectory=/opt/amq-broker/data/largemessages"
  acceptors:
  - name: artemis
    port: 61616
    protocols: all

Observed behavior

• When 2 pods are running:
  • Pod A acquires the shared store lock and becomes active
  • Pod B waits indefinitely to acquire the lock
• Pod B remains in 0/1 Ready
• Kubernetes Services will not route traffic to Pod B (even though it is expected to be a standby)

What I’ve tried

• Modified the liveness probe so the standby pod does not restart
• However, readiness still fails, and I need readiness to reflect:
  • Ready if broker is active
  • Not ready if broker is standby

Question

Is there a recommended readiness/liveness probe configuration for shared store HA deployments?

Specifically:

• Is there a way for the standby broker to report a meaningful readiness state?
• Is the operator expected to manage this automatically?
• Should readiness be disabled entirely for standby pods in shared-store HA?

Any guidance or examples would be appreciated.

[gtully]

Have you seen https://github.com/arkmq-org/activemq-artemis-operator/blob/main/controllers/activemqartemis_rwm_pvc_ha_test.go

two independent artemis crs that share a rwm volume. one service that routes to the active/ready artemis.

when the leader/follower are in the same stateful set (size 2) they are no longer independent. It needs to be two "peer" crs.