CrashLoopBackOff for nexus-iq pod after installing nexus-iq chart

[grigoryevandrey]

I've installed repository manager and it works fine, but nexus-iq-server giving me troubles. I am not sure how to debug it since i can not check logs of nexus-iq-server pod because of ./start.sh: line 2: /var/log/nexus-iq-server/stderr.log: Permission denied error.

Both repository manager and nexus-iq are in one nexus namespace.
RM storing data in one pv, and nexus-iq storing logs and error logs in different pvs.

I am not sure if this is an issue, or i am simply missing something in my configs.

Nexus IQ Config:

---
# Default values for iqserver.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
statefulset:
  # This is not supported
  enabled: false
replicaCount: 1
deploymentStrategy: Recreate
image:
  # Sonatype Official Public Image
  repository: sonatype/nexus-iq-server
  tag: 1.142.0
  pullPolicy: IfNotPresent
imagePullSecrets:
# for image registries that require login, specify the name of the existing
# kubernetes secret
#   - name: <pull-secret-name>

iq:
  name: nxiq
  hostname: iq-server.mydomain.com
  applicationPort: 8070
  adminPort: 8071
  # base 64 encoded license file with no line breaks
  licenseSecret: ''
  # add this line with this file path and the `licenseSecret` above to autoconfigure licensing
  # licenseFile: /etc/nexus-iq-license/license_lic
  extraLabels:
  # add the following two lines to mount a secrets volume within the container at the specified location
  # secretName: secret-jks
  # secretMountName: /etc/secret-volume
  env:
    - name: JAVA_OPTS
      value: '-Djava.util.prefs.userRoot=$(SONATYPE_WORK)/javaprefs'
  # In conjunction with 'secretName' and 'secretMountName' above, this is an example of how to inject required password
  # secrets into the runtime environment, and how to modify the startup of the server to utilize custom Java SSL stores.
  #    - name: TRUSTSTORE_PASSWORD
  #        valueFrom:
  #          secretKeyRef:
  #            name: secret-jks
  #            key: truststorePassword
  #    - name: KEYSTORE_PASSWORD
  #      valueFrom:
  #        secretKeyRef:
  #          name: secret-jks
  #          key: keystorePassword
  #    - name: JAVA_OPTS
  #      value: "-Djavax.net.ssl.keyStoreType=jks -Djavax.net.ssl.keyStore=/etc/secret-volume/keystore.jks -Djavax.net.ssl.keyStorePassword=$(KEYSTORE_PASSWORD) -Djavax.net.ssl.trustStoreType=jks -Djavax.net.ssl.trustStore=/etc/secret-volume/truststore.jks -Djavax.net.ssl.trustStorePassword=$(TRUSTSTORE_PASSWORD) -Djava.util.prefs.userRoot=${SONATYPE_WORK}/javaprefs"

  # Configures the liveness probe for IQ pod
  livenessProbe:
    initialDelaySeconds: 10
    periodSeconds: 10
    failureThreshold: 3
    timeoutSeconds: 2
    successThreshold: 1
  # Configures the readiness probe for IQ pod
  readinessProbe:
    initialDelaySeconds: 30
    periodSeconds: 30
    failureThreshold: 10
    timeoutSeconds: 2
    successThreshold: 1

nameOverride: ''
fullnameOverride: ''

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name:

podSecurityContext: {}
#  fsGroup: 2000

securityContext: {}
#  capabilities:
#    drop:
#    - ALL
#  readOnlyRootFilesystem: true
#  runAsNonRoot: true
#  runAsUser: 1000

service:
  type: 'ClusterIP'
  port: 80

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: '0'
    kubernetes.io/ingress.allow-http: 'false'
    cert-manager.io/cluster-issuer: letsencrypt-issuer
  hostUI: iq-server.mydomain.com
  hostUIPath: /
  hostAdmin: admin-iq-server.mydomain.com
  hostAdminPath: /
  tls:
    - secretName: nxiq-tls
      hosts:
        - iq-server.mydomain.com
        - admin-iq-server.mydomain.com

resources:
# We usually recommend not to specify default resources and to leave this as a
# conscious choice for the user. This also increases chances charts run on
# environments with little resources, such as Minikube. If you do want to
# specify resources, uncomment the lines below. These values ensure minimum
# resources for production.
#  requests:
#    cpu: 8
#    memory: 8Gi
# Optionally limit the resources to the recommended minimum requirements.
#  limits:
#    cpu: 8
#    memory: 8Gi

nodeSelector: {}

tolerations: []

affinity: {}

persistence:
  enabled: true
  accessMode: ReadWriteOnce
  storageSize: 1Gi

  # # If defined, storageClass: <storageClass>
  # # If set to "-", storageClass: "", which disables dynamic provisioning
  # # If undefined (the default) or set to null, no storageClass spec is
  # #   set, choosing the default provisioner.  (gp2 on AWS, standard on
  # #   GKE, AWS & OpenStack)
  # storageClass: "-"

  # # annotations are only applied to persistentVolumeClaim
  # annotations:
  #  "helm.sh/resource-policy": keep

  # # For existing PVCs
  # existingClaim: nexus-data-disk
  # existingLogClaim: nexus-log-disk

  # # persistentVolume settings below here

  # # Uncomment these to create a persistentVolume for data and/or logs volumes
  # pvName: nexus-data-disk
  # logpvName: nexus-log-disk

  # # Uncomment this for gcePersistentDisk
  # gcePersistentDisk:
  # # Uncomment the next two lines if you want to use an existing GCE PersistentDisk for IQ Data.
  #   pdName: nexus-data-disk
  #   fsType: ext4
  # # Uncomment the next two lines if you want to use an existing GCE PersistentDisk for IQ Logs.
  #   logpdName: nexus-log-disk
  #   logfsType: ext4

  # #Uncomment the following section for using an existing AWS EBS Volume for data and/or logs
  # awsElasticBlockStore:
  #   # Uncomment the next two lines if you want to use an existing AWS EBS Volume for IQ Data.
  #   volumeID: vol-xxxxxxxxxxxxxxxxx
  #   fsType: ext4
  #   # Uncomment the next two lines if you want to use an existing AWS EBS Volume for IQ Data.
  #   logvolumeID: vol-xxxxxxxxxxxxxxxxx
  #   logfsType: ext4

  # # Uncomment the following block if you want to use ANY CSI Storage Driver
  # # The YAML under 'csi:' or 'logCSI:' are taken as you choose to write them in order to support any CSI
  # # The following is the csi block for the Data PV
  # csi:
  #   driver: ebs.csi.aws.com
  #   volumeHandle: vol-xxxxxxxxxxxxxxxxx
  # # The following is the csi block for the Log PV
  # logCSI:
  #   driver: ebs.csi.aws.com
  #   volumeHandle: vol-xxxxxxxxxxxxxxxxx

  # # Uncomment the following block if you need PV affinity
  # # The nodeSelectorTerms block is taken as you choose to write it in whole
  # affinity:
  #   nodeSelectorTerms:
  #     - matchExpressions:
  #           - key: topology.ebs.csi.aws.com/zone
  #             operator: In
  #             values:
  #               - us-east-2a

# configYaml is the full text of the config.yml file that will be passed to IQ Server
configYaml:
  baseUrl: http://iq-server.mydomain.com
  sonatypeWork: /sonatype-work
  licenseFile: /etc/nexus-iq-license/license_lic
  initialAdminPassword: admin123
  features:
    enableUnauthenticatedPages: true
  server:
    applicationConnectors:
      - type: http
        port: 8070
    adminConnectors:
      - type: http
        port: 8071
    # HTTP request log settings.
    requestLog:
      appenders:
        # All appenders set to console
        - type: file
          currentLogFilename: /var/log/nexus-iq-server/request.log
          # Do not display log statements below this threshold to stdout.
          # threshold: INFO
          logFormat: '%clientHost %l %user [%date] "%requestURL" %statusCode %bytesSent %elapsedTime "%header{User-Agent}"'
          archivedLogFilenamePattern: /var/log/nexus-iq-server/request-%d.log.gz
          archivedFileCount: 50

  createSampleData: true

  logging:
    # The default level of all loggers. Can be OFF, ERROR, WARN, INFO, DEBUG, TRACE, or ALL.
    level: DEBUG

    # Logger-specific settings.
    loggers:
      'com.sonatype.insight.scan': INFO
      'eu.medsea.mimeutil.MimeUtil2': INFO
      'org.apache.http': INFO
      'org.apache.http.wire': ERROR
      'org.eclipse.birt.report.engine.layout.pdf.font.FontConfigReader': WARN
      'org.eclipse.jetty': INFO
      'org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter': INFO # WARNING: This reveals credentials at DEBUG level
      'com.sonatype.insight.audit':
        appenders:
          # All appenders set to console
          - type: file
            currentLogFilename: /var/log/nexus-iq-server/audit.log
            # Do not display log statements below this threshold to stdout.
            # threshold: INFO
            archivedLogFilenamePattern: /var/log/nexus-iq-server/audit-%d.log.gz
            archivedFileCount: 50

      'com.sonatype.insight.policy.violation':
        appenders:
          - type: console

            # Do not display log statements below this threshold to stdout.
            threshold: INFO

    appenders:
      # Settings for logging to stdout.
      - type: console

        # Do not display log statements below this threshold to stdout.
        threshold: INFO

        logFormat: "%d{'yyyy-MM-dd HH:mm:ss,SSSZ'} %level [%thread] %X{username} %logger - %msg%n"

      - type: file

        # Do not display log statements below this threshold to stdout.
        threshold: ALL

        logFormat: "%d{'yyyy-MM-dd HH:mm:ss,SSSZ'} %level [%thread] %X{username} %logger - %msg%n"
        # The file to which current statements will be logged.
        currentLogFilename: /var/log/nexus-iq-server/clm-server.log
        archivedLogFilenamePattern: /var/log/nexus-iq-server/clm-server-%d.log.gz
        archivedFileCount: 50

Nexus Repository Manager Config:

---
statefulset:
  # This is not supported
  enabled: false
deploymentStrategy: Recreate
image:
  # Sonatype Official Public Image
  repository: sonatype/nexus3
  tag: 3.41.0
  pullPolicy: IfNotPresent
imagePullSecrets:
# for image registries that require login, specify the name of the existing
# kubernetes secret
#   - name: <pull-secret-name>

nexus:
  docker:
    enabled: true
    registries:
      - port: 5000
        host: docker-registry.mydomain.com
        secretName: docker-registry-tls
  env:
    # minimum recommended memory settings for a small, person instance from
    # https://help.sonatype.com/repomanager3/product-information/system-requirements
    - name: INSTALL4J_ADD_VM_PARAMS
      value: |-
        -Xms2048m -Xmx2048m
        -XX:ActiveProcessorCount=4
        -XX:MaxDirectMemorySize=2703M
        -XX:+UnlockExperimentalVMOptions
        -XX:+UseCGroupMemoryLimitForHeap
        -Djava.util.prefs.userRoot=/nexus-data/javaprefs
    - name: NEXUS_SECURITY_RANDOMPASSWORD
      value: 'true'
  properties:
    override: false
    data:
      nexus.scripts.allowCreation: true
      # See this article for ldap configuratioon options https://support.sonatype.com/hc/en-us/articles/216597138-Setting-Advanced-LDAP-Connection-Properties-in-Nexus-Repository-Manager
      # nexus.ldap.env.java.naming.security.authentication: simple
  # nodeSelector:
  #   cloud.google.com/gke-nodepool: default-pool
  resources:
    # minimum recommended memory settings for a small, person instance from
    # https://help.sonatype.com/repomanager3/product-information/system-requirements
    # requests:
    #   cpu: 4
    #   memory: 8Gi
    # limits:
    #   cpu: 4
    #   memory: 8Gi

  # The ports should only be changed if the nexus image uses a different port
  nexusPort: 8081

  # Default the pods UID and GID to match the nexus3 container.
  # Customize or remove these values from the securityContext as appropriate for
  # your deployment environment.
  securityContext:
    runAsUser: 200
    runAsGroup: 200
    fsGroup: 200
  podAnnotations: {}
  livenessProbe:
    initialDelaySeconds: 30
    periodSeconds: 30
    failureThreshold: 6
    timeoutSeconds: 10
    path: /
  readinessProbe:
    initialDelaySeconds: 30
    periodSeconds: 30
    failureThreshold: 6
    timeoutSeconds: 10
    path: /
  # hostAliases allows the modification of the hosts file inside a container
  hostAliases: []
  # - ip: "192.168.1.10"
  #   hostnames:
  #   - "example.com"
  #   - "www.example.com"

nameOverride: ''
fullnameOverride: ''

deployment:
  # # Add annotations in deployment to enhance deployment configurations
  annotations: {}
  # # Add init containers. e.g. to be used to give specific permissions for nexus-data.
  # # Add your own init container or uncomment and modify the given example.
  initContainers:
  #   - name: fmp-volume-permission
  #   image: busybox
  #   imagePullPolicy: IfNotPresent
  #   command: ['chown','-R', '200', '/nexus-data']
  #   volumeMounts:
  #     - name: nexus-data
  #       mountPath: /nexus-data
  # Uncomment and modify this to run a command after starting the nexus container.
  postStart:
    command: # '["/bin/sh", "-c", "ls"]'
  preStart:
    command: # '["/bin/rm", "-f", "/path/to/lockfile"]'
  terminationGracePeriodSeconds: 120
  additionalContainers:
  additionalVolumes:
  additionalVolumeMounts:

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: '0'
    kubernetes.io/ingress.allow-http: 'false'
    cert-manager.io/cluster-issuer: letsencrypt-issuer
  hostPath: /
  hostRepo: storage.mydomain.com
  tls:
    - secretName: nexus-tls
      hosts:
        - storage.mydomain.com

service:
  name: nexus3
  enabled: true
  labels: {}
  annotations: {}
  type: ClusterIP

route:
  enabled: false
  name: docker
  portName: docker
  labels:
  annotations:
  # path: /docker

nexusProxyRoute:
  enabled: false
  labels:
  annotations:
  # path: /nexus

persistence:
  enabled: true
  accessMode: ReadWriteOnce
  ## If defined, storageClass: <storageClass>
  ## If set to "-", storageClass: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClass spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  # existingClaim:
  annotations:
    'helm.sh/resource-policy': keep
  # storageClass: "-"
  storageSize: 8Gi
  # If PersistentDisk already exists you can create a PV for it by including the 2 following keypairs.
  # pdName: nexus-data-disk
  # fsType: ext4

tolerations: []

# Enable configmap and add data in configmap
config:
  enabled: false
  mountPath: /sonatype-nexus-conf
  data: []

# # To use an additional secret, set enable to true and add data
secret:
  enabled: false
  mountPath: /etc/secret-volume
  readOnly: true
  data: []

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ''

[gss-jmorris]

Having the exact same issue here--not able to see anything missing in our configuration either. PVC/PV mounted properly, and we are running Nexus IQ in its own dedicated namespace

[bigspotteddog]

To review the Nexus IQ Server logs, connect to the pod via bash shell. The logs are in /var/log/nexus-iq-server.

Here are the commands I used to shell into a pod:

$ kubectl get pods --all-namespaces
$ kubectl --namespace <namespace> exec -it <pod> -- bash

[rudolphjacksonm]

We can't view the logs as the container comes up and immediately crashes. I think the filesystem permission error is a hard exit

[gss-jmorris]

Right so I edited the startup command for the container so I could go in and manually try starting it to see what happens. The file permissions are set as follows:

$ ls -la /var/log
total 292
drwxr-xr-x 1 root root     44 Jul 27 16:36 .
drwxr-xr-x 1 root root     30 May  3 08:55 ..
-rw-rw-r-- 1 root utmp 292292 Jul 27 16:36 lastlog
drwxr-xr-x 3 root root   4096 Aug 30 09:52 nexus-iq-server

$ whoami
nexus

The nexus user doesn't have permissions to this directory, and therefore can't write logs as the error message states

[gss-jmorris]

Found the fix for this @grigoryevandrey, you need to set the fsGroup under podSecurityContext so the volumes are writeable. I've raised a PR to change the default values file to reflect this, as the chart will not work out-of-the-box without this uncommented.

https://github.com/sonatype/helm3-charts/pull/226/files

[jflinchbaugh]

This PR has merged and should apply a similar change as #226. Please let us know if it works better in your security environment. #227