/plugins-defualt/ and GRAYLOG_HOME

[scottix]

Hi I am trying to upgrade from Graylog 4.2 to 4.3 and I am getting a weird start error.

graylog_1        | find: ‘/plugins-default/’: No such file or directory

Looking through the code it seems there was an env variable added called GRAYLOG_HOME that is not being initialized it seems.

I couldn't find any documentation stating whether we had to set that or not.

Any help would be appreciated.

[kroepke]

Need to verify this is still the case with 5.0

[scottix]

I am using docker-compose like this

  graylog:
    image: graylog/graylog:4.2
    volumes:
      - ./graylog/config:/usr/share/graylog/data/config
      - graylog_journal:/usr/share/graylog/data/journal
    restart: always
    environment:
      - GRAYLOG_HTTP_EXTERNAL_URI=https://hidden/
    depends_on:
      - mongo
      - elasticsearch
    ports:
      # Web interface
      - 9000:9000
      # syslog TCP
      - 1514:1514
      # syslog UDP
      - 1514:1514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp

4.2 runs fine, but if I switch to 4.3, that's when I get the issue.

[ccaballero646]

As a workaround, you can set the value for that variable in your docker-compose file like this:

graylog:
  image: graylog/graylog:5.0
  # other relevant configurations
  environment:
    - GRAYLOG_HOME: /usr/share/graylog
    # other relevant environment variables

[Kraktorist]

For some reason this doesn't work well on docker-1.13.1

The variable GRAYLOG_HOME is exported from /etc/profile.d/graylog.sh

graylog-docker/docker/oss/Dockerfile

Line 52 in 74912b5

ARG GRAYLOG_HOME=/usr/share/graylog

graylog-docker/docker/oss/Dockerfile

Line 68 in 74912b5

echo "export GRAYLOG_HOME=${GRAYLOG_HOME}" >> /etc/profile.d/graylog.sh && \

which is executed by the following code in the entrypoint

graylog-docker/docker-entrypoint.sh

Line 9 in 74912b5

source /etc/profile

This is how it works on docker-1.13.1

[vagrant@vagrant-01 ~]$ sudo docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-209.git7d71120.el7.centos.x86_64
 Go version:      go1.10.3
 Git commit:      7d71120/1.13.1
 Built:           Wed Mar  2 15:25:43 2022
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-209.git7d71120.el7.centos.x86_64
 Go version:      go1.10.3
 Git commit:      7d71120/1.13.1
 Built:           Wed Mar  2 15:25:43 2022
 OS/Arch:         linux/amd64
 Experimental:    false

[vagrant@vagrant-01 ~]$ sudo docker run -it --entrypoint=bash graylog/graylog:4.3.9
graylog@d97caf9def6e:~$ bash -x /docker-entrypoint.sh 
+ set -e
+ __GRAYLOG_SERVER_JAVA_OPTS=
+ source /etc/profile
++ '[' '' ']'
++ '[' -d /etc/profile.d ']'
++ for i in /etc/profile.d/*.sh
++ '[' -r /etc/profile.d/01-locale-fix.sh ']'
++ for i in /etc/profile.d/*.sh
++ '[' -r /etc/profile.d/graylog.sh ']'
++ unset i
+ [[ -z '' ]]
+ /opt/java/openjdk/bin/java -XX:+PrintFlagsFinal
+ grep -q UseParNewGC
+ /opt/java/openjdk/bin/java -XX:+PrintFlagsFinal
+ grep -q UseConcMarkSweepGC
+ [[ ! -z '' ]]
++ sed -r 's/^(GRAYLOG_[^=]*)__FILE=.*/\1/g'
++ env
++ grep '^GRAYLOG_[^=]\+__FILE=.\+'
+ [[ -e /tmp/graylog.pid ]]
+ [[ ! -z '' ]]
+ [[ ! -z '' ]]
+ export GRAYLOG_PLUGIN_DIR=/plugins-merged
+ GRAYLOG_PLUGIN_DIR=/plugins-merged
+ rm -f '/plugins-merged/*'
+ find /plugins-default/ -type f -exec cp '{}' /plugins-merged/ ';'
find: ‘/plugins-default/’: No such file or directory

and this is how it works on docker-ce: 20.10.23:

[vagrant@vagrant-02 ~]$ sudo docker version
Client: Docker Engine - Community
 Version:           20.10.23
 API version:       1.41
 Go version:        go1.18.10
 Git commit:        7155243
 Built:             Thu Jan 19 17:36:21 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.23
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.10
  Git commit:       6051f14
  Built:            Thu Jan 19 17:34:26 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.16
  GitCommit:        31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
[vagrant@vagrant-02 ~]$ sudo docker run -it --entrypoint=bash graylog/graylog:4.3.9
graylog@8a1aceb42bbe:~$ bash -x /docker-entrypoint.sh 
+ set -e
+ __GRAYLOG_SERVER_JAVA_OPTS=
+ source /etc/profile
++ '[' '' ']'
++ '[' -d /etc/profile.d ']'
++ for i in /etc/profile.d/*.sh
++ '[' -r /etc/profile.d/01-locale-fix.sh ']'
++ . /etc/profile.d/01-locale-fix.sh
++++ /usr/bin/locale-check C.UTF-8
+++ eval
++ for i in /etc/profile.d/*.sh
++ '[' -r /etc/profile.d/graylog.sh ']'
++ . /etc/profile.d/graylog.sh
+++ export BUILD_DATE=2022-11-02T16:01:28Z
+++ BUILD_DATE=2022-11-02T16:01:28Z
+++ export GRAYLOG_VERSION=4.3.9
+++ GRAYLOG_VERSION=4.3.9
+++ export 'GRAYLOG_SERVER_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true -Djdk.tls.acknowledgeCloseNotify=true -XX:+UnlockExperimentalVMOptions -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow'
+++ GRAYLOG_SERVER_JAVA_OPTS='-Dlog4j2.formatMsgNoLookups=true -Djdk.tls.acknowledgeCloseNotify=true -XX:+UnlockExperimentalVMOptions -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow'
+++ export GRAYLOG_HOME=/usr/share/graylog
+++ GRAYLOG_HOME=/usr/share/graylog
+++ export GRAYLOG_USER=graylog
+++ GRAYLOG_USER=graylog
+++ export GRAYLOG_GROUP=graylog
+++ GRAYLOG_GROUP=graylog
+++ export GRAYLOG_UID=1100
+++ GRAYLOG_UID=1100
+++ export GRAYLOG_GID=1100
+++ GRAYLOG_GID=1100
+++ export PATH=/usr/share/graylog/bin:/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+++ PATH=/usr/share/graylog/bin:/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ unset i
+ [[ -z '' ]]
+ /opt/java/openjdk/bin/java -XX:+PrintFlagsFinal
+ grep -q UseParNewGC
+ GRAYLOG_SERVER_JAVA_OPTS='-Dlog4j2.formatMsgNoLookups=true -Djdk.tls.acknowledgeCloseNotify=true -XX:+UnlockExperimentalVMOptions -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -XX:+UseParNewGC'
+ export GRAYLOG_SERVER_JAVA_OPTS
+ /opt/java/openjdk/bin/java -XX:+PrintFlagsFinal
+ grep -q UseConcMarkSweepGC
+ GRAYLOG_SERVER_JAVA_OPTS='-Dlog4j2.formatMsgNoLookups=true -Djdk.tls.acknowledgeCloseNotify=true -XX:+UnlockExperimentalVMOptions -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled'
+ export GRAYLOG_SERVER_JAVA_OPTS
+ [[ ! -z '' ]]
++ env
++ grep '^GRAYLOG_[^=]\+__FILE=.\+'
++ sed -r 's/^(GRAYLOG_[^=]*)__FILE=.*/\1/g'
+ [[ -e /tmp/graylog.pid ]]
+ [[ ! -z '' ]]
+ [[ ! -z '' ]]

On docker-1.13.1 the scripts from /etc/profile.d/ directory cannot be exported and therefore GRAYLOG_HOME variable is empty

[smithandre]

I could go from v4.2 to v4.3.3 without issues. v4.3.4 threw a premision denied error on a /data driectory, and later on (I think 4.3.10 onwards) gives me the above find: ‘/plugins-default/’: No such file or directory error as well.

Not yet been able to check against v5.0, as I had to get to 4.3 in order to take MoggoDB up to version 5.

[dparmar]

It seems this happens in Graylog 5.0.6 too.

graylog@9f10cfcdecd9:~$ bash -x /docker-entrypoint.sh 
+ set -e
+ __GRAYLOG_SERVER_JAVA_OPTS=
+ source /etc/profile
++ '[' '' ']'
++ '[' -d /etc/profile.d ']'
++ for i in /etc/profile.d/*.sh
++ '[' -r /etc/profile.d/01-locale-fix.sh ']'
++ for i in /etc/profile.d/*.sh
++ '[' -r /etc/profile.d/graylog.sh ']'
++ unset i
+ [[ ! -z '' ]]
++ env
++ grep '^GRAYLOG_[^=]\+__FILE=.\+'
++ sed -r 's/^(GRAYLOG_[^=]*)__FILE=.*/\1/g'
+ [[ -e /tmp/graylog.pid ]]
+ [[ ! -z '' ]]
+ [[ ! -z '' ]]
+ export GRAYLOG_PLUGIN_DIR=/plugins-merged
+ GRAYLOG_PLUGIN_DIR=/plugins-merged
+ rm -f '/plugins-merged/*'
+ find /plugins-default/ -type f -exec cp '{}' /plugins-merged/ ';'
find: ‘/plugins-default/’: No such file or directory`

Docker version on Ubuntu:

$ docker version
Client: Docker Engine - Community
 Version:           19.03.6
 API version:       1.40
 Go version:        go1.12.16
 Git commit:        369ce74a3c
 Built:             Thu Feb 13 01:27:49 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.16
  Git commit:       369ce74a3c
  Built:            Thu Feb 13 01:26:23 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[dparmar]

The workaround of explicitly setting GRAYLOG_HOME was useful but then it gives insufficient memory error.

graylog@6e4e56af28ce:~$ export GRAYLOG_HOME=/usr/share/graylog 
graylog@6e4e56af28ce:~$ bash -x /docker-entrypoint.sh 
+ set -e
+ __GRAYLOG_SERVER_JAVA_OPTS=
+ source /etc/profile
++ '[' '' ']'
++ '[' -d /etc/profile.d ']'
++ for i in /etc/profile.d/*.sh
++ '[' -r /etc/profile.d/01-locale-fix.sh ']'
++ for i in /etc/profile.d/*.sh
++ '[' -r /etc/profile.d/graylog.sh ']'
++ unset i
+ [[ ! -z '' ]]
++ env
++ grep '^GRAYLOG_[^=]\+__FILE=.\+'
++ sed -r 's/^(GRAYLOG_[^=]*)__FILE=.*/\1/g'
+ [[ -e /tmp/graylog.pid ]]
+ [[ ! -z '' ]]
+ [[ ! -z '' ]]
+ export GRAYLOG_PLUGIN_DIR=/usr/share/graylog/plugins-merged
+ GRAYLOG_PLUGIN_DIR=/usr/share/graylog/plugins-merged
+ rm -f '/usr/share/graylog/plugins-merged/*'
+ find /usr/share/graylog/plugins-default/ -type f -exec cp '{}' /usr/share/graylog/plugins-merged/ ';'
+ find /usr/share/graylog/plugin '!' -readable -prune -o -type f -a -readable -exec cp '{}' /usr/share/graylog/plugins-merged/ ';'
+ run
+ setup
+ for d in journal log plugin config contentpacks
+ dir=/usr/share/graylog/data/journal
+ [[ -d /usr/share/graylog/data/journal ]]
++ stat --format=%U:%G /usr/share/graylog/data/journal
+ [[ graylog:graylog != \g\r\a\y\l\o\g\:\g\r\a\y\l\o\g ]]
+ for d in journal log plugin config contentpacks
+ dir=/usr/share/graylog/data/log
+ [[ -d /usr/share/graylog/data/log ]]
++ stat --format=%U:%G /usr/share/graylog/data/log
+ [[ graylog:graylog != \g\r\a\y\l\o\g\:\g\r\a\y\l\o\g ]]
+ for d in journal log plugin config contentpacks
+ dir=/usr/share/graylog/data/plugin
+ [[ -d /usr/share/graylog/data/plugin ]]
++ stat --format=%U:%G /usr/share/graylog/data/plugin
+ [[ graylog:graylog != \g\r\a\y\l\o\g\:\g\r\a\y\l\o\g ]]
+ for d in journal log plugin config contentpacks
+ dir=/usr/share/graylog/data/config
+ [[ -d /usr/share/graylog/data/config ]]
++ stat --format=%U:%G /usr/share/graylog/data/config
+ [[ graylog:graylog != \g\r\a\y\l\o\g\:\g\r\a\y\l\o\g ]]
+ for d in journal log plugin config contentpacks
+ dir=/usr/share/graylog/data/contentpacks
+ [[ -d /usr/share/graylog/data/contentpacks ]]
+ mkdir -p /usr/share/graylog/data/contentpacks
++ stat --format=%U:%G /usr/share/graylog/data/contentpacks
+ [[ graylog:graylog != \g\r\a\y\l\o\g\:\g\r\a\y\l\o\g ]]
+ '[' 0 = 0 ']'
+ graylog server
+ exec /opt/java/openjdk/bin/java -jar -Dlog4j.configurationFile=/usr/share/graylog/data/config/log4j2.xml -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker /usr/share/graylog/graylog.jar server -f /usr/share/graylog/data/config/graylog.conf
[0.006s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# An error report file with more information is saved as:
# /usr/share/graylog/hs_err_pid31.log