NodeFilesystemAlmostOutOfSpace - Filesystem on tmpfs running out of space

[ThomasUllrich666]

We installed the COS Plugin on our Satellite Openshift Worker:
Version 2.2
Revision 12

We receive this error message: Filesystem on tmpfs at has only 0.00% available space left.
path: mountpoint=/var/lib/ibmc-s3fs/3b4b9705ecf886490489df3d603a71fc168da0241ae57bbeba8edc59a84db65e

we discovered that the mount point indeed didn't have sufficient space left:

sh-4.4# df |grep /var/lib/ibmc-s3fs/
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/ce3254fda8e07cf7dd00c6092e843cf22312c81215796b2cbc302f32471445c7
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/230bc0a1464aae1f4117781cd54ded7b698420d17299bc2db69e314109f8ab5c
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/eb4d47c212812236399cf2454d879ee2cff76f866a91408e49c2edef6c9f4db2
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/c749ffa58939d30041fc0158d5652ff50a009e2d6fdff7aa809a777c1e819159
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/ec0ad791fdf18886496f313aaa7a5fa4c5080f60745365d610f90881687e3249
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/157a6f07baf8d326ce6127d24b438d0793a87a466b79e2833d0d01f78f30cf42
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/9e2c1dbfefbb65a04879bc0653db526cdce441646016ad426bf5e2ba0695af3a
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/d90ed51b14efdaeeb8a97688b940532f2333b57c4f1e2544d30db3f4cc0d236b
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/03962ef66ea5070bdf2bea8169b48540cd870a4138c4ea3a3c92ac93f4c96c0a
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/dc21d8d8211688ebe2466bc8f0da7274ff451de73f5a00ae0afb517f5858bc39
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/7c4d297faa151e8f4940dbd2f1d68034634eecafe1337fab7cfc7e027658fe21
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/d46e06b8ef86b179921390731063dfeb5d274d751c9570094fb7de86118b8346
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/016f4835abe7c1cdd47938099617fb7b13438dec648cfc7157815bb42629fe1c
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/bd32d16aab229743e02b0e5c17b6fe053954186373fa73152f11e2c9ac45d068
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/2099a995c40095092bf5da124be69baed336228ad3e241fc02fb0dd893eb768d
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/6507df254918a0b2d49fa0b055de61afaf503bbdad27a6f407ed983fe9c5ca5d
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/8aca7df34b6dee49dffcccd37d7644fa945b2b69e5d8e4a20adf17c411d0eb48
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/f3e9aa1bec73bc1f9ba95ae51ddea19cda6f8ea321f7f8cc9d505c1c6f1296f7
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/e36e01444b3c644f05c257498e8b2e2fb747c7ae3307292b36d73147c2ca716f
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/6cf531fd800ace5037de39619966fd4f98ccdec24c499e436ee298ee460a0f0c
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/6e62de6783745b9d2facc7e79bedb094ba4255fb9b82dfe8bba8a6bf4c8137f7
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/b33ac99c610143470091167aa483aaacc1713af9500a7046f4bcd90419ef89fc
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/5fa5e148f70172f040667c0757bc84e2719d8284855aa669a76c65ca928307e4
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/2320a47f4b0f35e4827c7096384ff9cb5430980ad0a60f4f949451acd7331481
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/50aaeafb085a04504b778909272bb16ff19d306f4cd426ac5b46a03fed88d8da
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/3c90cdc19470f2906c0128df143765c31992c55c653d58ee1909b792e5899c82
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/4f73e161114e58a92e97cf3f9fe600677db7f5dfde2dc38183fbdde3939499aa
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/e5454b273fb9d3acb44016aeab2a90f6237b781e0c103a7a675ae5608b070ca2
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/b78edf44febf364870e64277ae7611a18082108a835ecd36930bd0ff5b667c27
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/6401d1f6e609d041478e7f09ccef1acb89f7ca7b761e50123e1a4d4ab37c9893
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/33ba840df4704fcee56cdcbf841404d94feecdd58c1875499873d15eb6fee2fe
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/8405792b2e8e25bf1804c0942b08394549cb9831fc4f1c814497f506162af16f
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/688f80195b081169ec32f30a407dc906692d85ffb7de939c37087415b5e01c9b
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/55b722035afa0940b9f88bff328125a02d6c094191168930281cbc405701ff71
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/059db21227a80bc2c8aeeb2b36b0a09b6fd638cf817286dad7679c0f57f37833
tmpfs                          4        4                 0 100% /var/lib/ibmc-s3fs/c112bcb65c838745d8a8661e62ef0b9028b550171963b2acf234d7a9dd801f3a

An example of our PVC:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  annotations:
    ibm.io/object-path: static-content
    pv.kubernetes.io/bind-completed: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: ibm.io/ibmc-s3fs
    ibm.io/stat-cache-expire-seconds: '0'
    ibm.io/auto-create-bucket: 'false'
    ibm.io/secret-namespace: ru-aokbw-idbroker
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"ibm.io/auto-cache":"false","ibm.io/auto-create-bucket":"false","ibm.io/auto-delete-bucket":"false","ibm.io/bucket":"sek-idp-ru-aokbw-hsm-config","ibm.io/object-path":"static-content","ibm.io/secret-name":"sekidp-cos-api-key","ibm.io/secret-namespace":"ru-aokbw-idbroker","ibm.io/stat-cache-expire-seconds":"0","volume.beta.kubernetes.io/storage-provisioner":"ibm.io/ibmc-s3fs","volume.kubernetes.io/storage-provisioner":"ibm.io/ibmc-s3fs"},"labels":{"app.kubernetes.io/component":"persistent-storage","app.kubernetes.io/instance":"idbroker-entrygw-ru-aokbw-nue","app.kubernetes.io/name":"entrygw-content-pv-claim","app.kubernetes.io/part-of":"entrygw"},"name":"entrygw-content-pv-claim","namespace":"ru-aokbw-idbroker"},"spec":{"accessModes":["ReadWriteMany"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"ibmc-s3fs-cos","volumeMode":"Filesystem"}}
    ibm.io/auto-cache: 'false'
    ibm.io/secret-name: sekidp-cos-api-key
    ibm.io/bucket: sek-idp-ru-aokbw-hsm-config
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.kubernetes.io/storage-provisioner: ibm.io/ibmc-s3fs
    ibm.io/auto-delete-bucket: 'false'
  resourceVersion: '131639451'
  name: entrygw-content-pv-claim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  volumeName: pvc-92c87042-6bc1-4398-84d6-505573e54dbb
  storageClassName: ibmc-s3fs-cos
  volumeMode: Filesystem
status:
  phase: Bound
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 1Gi

Any idea what's wrong?