Skip to content

Override systemd-journald with ForwardToSyslog=no to avoid disk exhaustion issues #914

Description

@kot0dama

Bug Description

The k8s snap defaults to trace log level, while the Ubuntu system defaults to ForwardToSyslog=yes.
This is a high of disk exhaustion risk on any k8s charm deployment.

I believe the charm should enforce ForwardToSyslog=no until at least one of these bugs is resolved with a remediation:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2147513
canonical/k8sd#31

Thank you!

To Reproduce

Deploy the charm and enable many workloads, watch syslog fill with kube api server trace level logs up to GBs instead of usual KBs.

Environment

Juju: 3.6.9

App                         Version  Status  Scale  Charm                       Channel        Rev  Exposed  Message
cinder-csi                  v1.27.1  active      1  cinder-csi                  1.33/stable     38  no       Versions: cinder-csi-driver=v1.27.1
easyrsa                     v3.0.9   active      1  easyrsa                     1.33/stable     74  no       Certificate Authority connected.
etcd                        3.4.37   active      3  etcd                        1.33/stable    788  no       Healthy with 3 known peers
grafana-agent                        active     10  grafana-agent               1/stable       489  no       tracing: off
k8s-control                 1.32.6   active      3  k8s                         1.32/stable    987  yes      Ready
k8s-worker                  1.32.6   active      3  k8s-worker                  1.32/stable    987  yes      Ready
landscape-client                     active     10  landscape-client            latest/edge     72  no       Client registered!
openstack-cloud-controller  v1.30.0  active      1  openstack-cloud-controller  1.33/stable     40  no       Versions: openstack-cloud-controller-manager=v1.30.0
openstack-integrator        yoga     active      1  openstack-integrator        1.33/stable    115  no       Ready
ubuntu-advantage                     active     10  ubuntu-advantage            latest/stable  143  no       Attached (esm-apps,esm-infra,landscape,livepatch)

Relevant log output

Confidential, please ask if required.

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions