Mattermost-k8s crashed if installed in cluster mode (two units) without the enterprise license (in Juju 3.6-rc2)

[taurus-forever]

Bug Description

Hi,

Data team is using Mattermost charm to test PgBouncer K8s charm stability.
The pgbouncer CI test did install Mattermost in the clustered way (two units) and experiencing random crashes::

App                       Version                         Status  Scale  Charm                     Channel        Rev  Address         Exposed  Message
mattermost-k8s            .../mattermost:v8.1.3-20.04...  error       2  mattermost-k8s            latest/stable   27  10.152.183.83   no       crash loop backoff: back-off 5m0s restarting failed container=mattermost-k8s pod=mattermost-k8s-69995677fb-7w27h_test...
...

Unit                         Workload  Agent  Address       Ports     Message
mattermost-k8s/0*            error     idle   10.1.163.213  8065/TCP  crash loop backoff: back-off 5m0s restarting failed container=mattermost-k8s pod=mattermost-k8s-69995677fb-7w27h_test...
mattermost-k8s/1             error     idle   10.1.163.212  8065/TCP  crash loop backoff: back-off 5m0s restarting failed container=mattermost-k8s pod=mattermost-k8s-69995677fb-98tsj_test...
...

We read here, the extra license is required, maybe it is a reason?
For now we are going to test Mattermost in a single unit mode
(it looks like the issue is not reproducible if Mattermost is installed in a single unit mode).

To Reproduce

juju deploy mattermost-k8s -n 2
juju deploy pgbouncer-k8s --trust -n 2
juju deploy postgresql-k8s  --trust -n 3
juju deploy self-signed-certificates

juju relate self-signed-certificates postgresql-k8s
juju relate self-signed-certificates pgbouncer-k8s
juju relate pgbouncer-k8s postgresql-k8s
juju relate mattermost-k8s pgbouncer-k8s:db

Environment

Name              Version      Rev    Tracking            Publisher      Notes
juju              3.6-rc2      28954  3.6/candidate       canonical✓     -
microk8s          v1.31.2      7371   1.31-strict/stable  canonical✓     -

Relevant log output

The juju debug-log has no related logs.
The entire juju status with relations:


Model   Controller  Cloud/Region        Version  SLA          Timestamp
testpg  microk8s    microk8s/localhost  3.6-rc2  unsupported  12:38:27+01:00

App                       Version                         Status  Scale  Charm                     Channel        Rev  Address         Exposed  Message
mattermost-k8s            .../mattermost:v8.1.3-20.04...  error       2  mattermost-k8s            latest/stable   27  10.152.183.239  no       crash loop backoff: back-off 5m0s restarting failed container=mattermost-k8s pod=mattermost-k8s-dc4c9dcd8-6fwv7_testp...
pgbouncer-k8s             1.21.0                          active      2  pgbouncer-k8s             1/edge         345  10.152.183.184  no       
postgresql-k8s            14.13                           active      3  postgresql-k8s            14/edge        459  10.152.183.100  no       
self-signed-certificates                                  active      1  self-signed-certificates  latest/stable  155  10.152.183.56   no       

Unit                         Workload  Agent  Address     Ports     Message
mattermost-k8s/2*            error     idle   10.1.61.35  8065/TCP  crash loop backoff: back-off 5m0s restarting failed container=mattermost-k8s pod=mattermost-k8s-dc4c9dcd8-6fwv7_testp...
mattermost-k8s/3             error     idle   10.1.61.3   8065/TCP  crash loop backoff: back-off 5m0s restarting failed container=mattermost-k8s pod=mattermost-k8s-dc4c9dcd8-mgp4n_testp...
pgbouncer-k8s/0*             active    idle   10.1.61.24            
pgbouncer-k8s/1              active    idle   10.1.61.4             
postgresql-k8s/0*            active    idle   10.1.61.20            Primary
postgresql-k8s/1             active    idle   10.1.61.28            
postgresql-k8s/2             active    idle   10.1.61.1             
self-signed-certificates/0*  active    idle   10.1.61.21            

Integration provider                   Requirer                        Interface          Type     Message
pgbouncer-k8s:db                       mattermost-k8s:db               pgsql              regular  
pgbouncer-k8s:pgb-peers                pgbouncer-k8s:pgb-peers         pgb_peers          peer     
pgbouncer-k8s:upgrade                  pgbouncer-k8s:upgrade           upgrade            peer     
postgresql-k8s:database                pgbouncer-k8s:backend-database  postgresql_client  regular  
postgresql-k8s:database-peers          postgresql-k8s:database-peers   postgresql_peers   peer     
postgresql-k8s:restart                 postgresql-k8s:restart          rolling_op         peer     
postgresql-k8s:upgrade                 postgresql-k8s:upgrade          upgrade            peer     
self-signed-certificates:certificates  pgbouncer-k8s:certificates      tls-certificates   regular  
self-signed-certificates:certificates  postgresql-k8s:certificates     tls-certificates   regular  

Additional context

Note: two units deployment used to work well in Juju 2.9.51 and Juju 3.4.6 but crashes in Juju 3.6-rc2.

[javierdelapuente]

@swetha1654 can you have a look at this issue? Thanks!

[swetha1654]

sure! will get back as soon as possible

[swetha1654]

@taurus-forever update: I tried to reproduce this issue in my local environment
I was able to install microk8s from the 1.31-strict/stable track. But when i tried to install juju 3.6-rc2 it threw the following error-

ubuntu@my-juju-vm:~$ sudo snap install juju --channel 3.6/candidate
juju (3.6/stable) 3.6.0 from Canonical✓ installed
Channel 3.6/candidate for juju is closed; temporarily forwarding to 3.6/stable.
ubuntu@my-juju-vm:~$ juju --version
Since Juju 3 is being run for the first time, it has downloaded the latest public cloud information.
3.6.0-genericlinux-amd64

I proceeded with 3.6 stable and ran the steps to reproduce the issue but i was unable to reproduce it. All the charms were in idle and active state:

Every 2.0s: juju status --relations --color                                      my-juju-vm: Fri Nov 29 12:43:56 2024

Model     Controller     Cloud/Region        Version  SLA          Timestamp
my-model  my-controller  microk8s/localhost  3.6.0    unsupported  12:43:57+05:30

App                       Version                         Status  Scale  Charm                     Channel        Rev
  Address         Exposed  Message
mattermost-k8s            .../mattermost:v8.1.3-20.04...  active      2  mattermost-k8s            latest/stable   27
  10.152.183.81   no
pgbouncer-k8s             1.21.0                          active      2  pgbouncer-k8s             1/stable       269
  10.152.183.24   no
postgresql-k8s            14.12                           active      3  postgresql-k8s            14/stable      444
  10.152.183.244  no
self-signed-certificates                                  active      1  self-signed-certificates  latest/stable  155
  10.152.183.230  no

Unit                         Workload  Agent  Address      Ports     Message
mattermost-k8s/0*            active    idle   10.1.32.153  8065/TCP
mattermost-k8s/1             active    idle   10.1.32.152  8065/TCP
pgbouncer-k8s/0*             active    idle   10.1.32.143
pgbouncer-k8s/1              active    idle   10.1.32.144
postgresql-k8s/0*            active    idle   10.1.32.149            Primary
postgresql-k8s/1             active    idle   10.1.32.151
postgresql-k8s/2             active    idle   10.1.32.150
self-signed-certificates/0*  active    idle   10.1.32.148

Integration provider                   Requirer                        Interface          Type     Message
pgbouncer-k8s:db                       mattermost-k8s:db               pgsql              regular
pgbouncer-k8s:pgb-peers                pgbouncer-k8s:pgb-peers         pgb_peers          peer
pgbouncer-k8s:upgrade                  pgbouncer-k8s:upgrade           upgrade            peer
postgresql-k8s:database                pgbouncer-k8s:backend-database  postgresql_client  regular
postgresql-k8s:database-peers          postgresql-k8s:database-peers   postgresql_peers   peer
postgresql-k8s:restart                 postgresql-k8s:restart          rolling_op         peer
postgresql-k8s:upgrade                 postgresql-k8s:upgrade          upgrade            peer
self-signed-certificates:certificates  pgbouncer-k8s:certificates      tls-certificates   regular

This might probably be because this issue does not occur in 3.6 stable version of juju or because its not reproduce. Could you update the juju version and see if this issue still persists? or if you have the debug-logs from that run it would be helpful to see why it went into crashloopbackoff

Thanks!

[swetha1654]

closing this issue as unable to reproduce