Modsecurity failing with "Database is not open. Use: SecGeoLookupDb directive"

[chlynott]

We have been running ingress-nginx with modsecurity rules in place for over two years. Recently the we have been unable to block geolocations and are getting this error in the debug logs

Database is not open. Use: SecGeoLookupDb directive.

I have the GeoLite2-Country.mmdb mounted using an azure file back PV:

# Create an Azure File StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: geoip-country-database-sc
  namespace: nginx-ingress-controller
provisioner: kubernetes.io/azure-file
mountOptions:
  - dir_mode=0775
  - file_mode=0775
  - uid=0
  - gid=0
  - mfsymlinks
  - cache=strict
  - actimeo=30
parameters:
  skuName: Standard_ZRS
allowVolumeExpansion: true

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: geoip-country-database-pvc
  namespace: nginx-ingress-controller
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: geoip-country-database-sc
  resources:
    requests:
      storage: 1Gi

This is my ingress-nginx-controller ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: nginx-ingress-controller
  annotations:
    meta.helm.sh/release-name: ingress-nginx
    meta.helm.sh/release-namespace: nginx-ingress-controller
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.9.3
    helm.sh/chart: ingress-nginx-4.8.2
data:
  add-headers: nginx-ingress-controller/ingress-nginx-custom-add-headers
  allow-snippet-annotations: "true"
  strict-validate-path-type: "true"
  enable-modsecurity: 'true'
  modsecurity-snippet: |
    Include /etc/nginx/modsecurity/modsecurity.conf
  proxy-cookie-path: |
    $uri "/; HTTPOnly; Secure; SameSite=strict"

extra values:

controller:
  extraVolumeMounts:
    - name: geoip-country-database
      mountPath: "/etc/nginx/geoip/GeoLite2-Country.mmdb"
      subPath: GeoLite2-Country.mmdb
    - name: modsecurity
      mountPath: "/etc/nginx/modsecurity/modsecurity.conf"
      subPath: modsecurity.conf
  extraVolumes:
    - name: geoip-country-database
      persistentVolumeClaim:
            claimName: geoip-country-database-pvc
    - name: modsecurity
      configMap:
        name: "modsecurity-conf"

Modsecurity Configuration:

SecRuleEngine On
SecRequestBodyAccess On
SecResponseBodyAccess On
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogParts ABIFHJKZ
SecAuditLogType Concurrent
SecAuditLog /var/log/modsec_audit.log
SecDebugLog /tmp/modsec_debug.log
SecCookieFormat 0
SecStatusEngine On
SecAuditLogStorageDir /var/log/audit/
SecRule REMOTE_ADDR "@ipMatch 10.180.0.0/16" "phase:1,id:200000001,nolog,allow"
SecGeoLookup /etc/nginx/geoip/GeoLite2-Country.mmdb
SecRule REMOTE_ADDR "@geoLookup" "phase:1,chain,id:22,drop,msg:'Non-allowed country IP address'"
SecRule GEO:COUNTRY_CODE "!@pm US CA BS MX TC TT VC MF LC KN BL AG AI AW BB VG KY CU DM DO GD GP HT JM MQ MS CR GB AU NZ"

ingress-nginx helm chart version 4.8.3
AKS version 1.30.0
ingress-nginx App Version 1.9.4

We have this configuration running in four Kubernetes clusters. Up until recently it was working just fine, then we started getting the above error in the logs.

I have tried restarting the ingress-nginx-controller deployments, and that helped in our development clusters, for a little while, then we started seeing the error message again, and I was able to reach all sites from a blocked country.

I have tried upgrading the helm chart from 4.8.2 to 4.8.3 in all clusters, which oddly worked for one replica in one of our production clusters (there are two replicas running in each cluster) but not any of the others.
This also fixed one of or dev clusters for a few days, but it has started giving the above error message again.

I expect the configuration to be able to find the database and use it when I set SecGeoLookupDb to the correct file path, everytime.
Not sure what else to try to resolve this, and find it odd that it sometimes works, and other times does not.

[longwuyuan]

/remove-kind bug
/kind support
/triage needs-information

I think we can work together and possibility locate root cause.
To rule out Read/Write delay or error, caused by the DB being a mount, is there any possibility that you can change the design to have the DB downloaded to the filesystem of the controller. Download from a well-known mirror or pre-download and publish from a adjacent pod.

[chlynott]

Thank you @longwuyuan. I have created a job to pull the download to the controller filesystem from the MaxMind permalink. I will be testing for a few days in our development environment to ensure it continues to work, as we have seen it stop working randomly.

[chlynott]

Tested in another one of our dev clusters - this is the Job and PVC updated; I am creating these before the nginx is deployed and updating the modsecurity.conf to point to the correct file location. We are still getting the error in the debug logs

kind: Job
metadata:
  name: geoip-job
  namespace: nginx-ingress-controller
spec:
  ttlSecondsAfterFinished: 3600
  template:
    metadata:
      name: geoip-job
    spec:
      containers:
        - name: geoip-download
          image: alpine
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -x
              apk add --no-cache bash curl
              echo "Downloading GeoLite2 Country database..."
              curl -O -J -L -u ACCOUNT_ID:LICENSE_KEY "https://download.maxmind.com/geoip/databases/GeoLite2-Country/download?suffix=tar.gz"
              echo "Download complete. Listing files:"
              ls -lah
              echo "Extracting files..."
              tar -xzvf GeoLite2-Country_*.tar.gz
              mv GeoLite2-Country_*/GeoLite2-Country.mmdb /mnt/share
              echo "Extraction complete. Listing files in /mnt/share:"
              ls -lah /mnt/share
              sleep 3600
          volumeMounts:
            - name: geoip-country-database
              mountPath: /mnt/share
      restartPolicy: Never
      volumes:
        - name: geoip-country-database
          persistentVolumeClaim:
            claimName: geoip-country-database-pvc
---
 apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: geoip-country-database-pvc
  namespace: nginx-ingress-controller
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: azurefile-premium
  resources:
    requests:
      storage: 1Gi

It appears as though it started then stopped working, then started working again? an example of the debug log

[174066786327.544008] [/s/1000-4/api/b/] [4] Starting phase REQUEST_HEADERS.  (SecRules 1)
[174066786327.544008] [/s/1000-4/api/b/] [4] (Rule: 200000001) Executing operator "IpMatch" with param "10.180.0.0/16" against REMOTE_ADDR.
[174066786327.544008] [/s/1000-4/api/b/] [4] Rule returned 0.
[174066786327.544008] [/s/1000-4/api/b/] [4] (Rule: 22) Executing operator "GeoLookup against REMOTE_ADDR.
[174066786327.544008] [/s/1000-4/api/b/] [4] Database is not open. Use: SecGeoLookupDb directive.
[174066786327.544008] [/s/1000-4/api/b/] [4] Rule returned 0.
[174066786327.544008] [/s/1000-4/api/b/] [4] Starting phase REQUEST_BODY. (SecRules 2)
[174066786327.544008] [/s/1000-4/api/b/] [4] Starting phase RESPONSE_HEADERS. (SecRules 3)
[174066786327.544008] [/s/1000-4/api/b/] [4] Starting phase RESPONSE_BODY. (SecRules 4)
[174066786327.544008] [/s/1000-4/api/b/] [4] Starting phase LOGGING. (SecRules 5)
[174066786596.483884] [] [4] Initializing transaction
[174066786596.483884] [] [4] Transaction context created.
[174066786596.483884] [] [4] Starting phase CONNECTION. (SecRules 0)
[174066786596.483884] [] [4] Starting phase URI. (SecRules 0 + 1/2)
[174066786596.483884] [/] [4] Starting phase REQUEST_HEADERS.  (SecRules 1)
[174066786596.483884] [/] [4] (Rule: 200000001) Executing operator "IpMatch" with param "10.180.0.0/16" against REMOTE_ADDR.
[174066786596.483884] [/] [4] Rule returned 1.
[174066786596.483884] [/] [4] Running (disruptive)     action: allow.
[174066786596.483884] [/] [4] Dropping the evaluation of upcoming rules in favor of an `allow' action of type: FromNowOn
[174066786596.483884] [/] [4] Starting phase REQUEST_BODY. (SecRules 2)
[174066786596.483884] [/] [4] Starting phase RESPONSE_HEADERS. (SecRules 3)
[174066786596.483884] [/] [4] Starting phase RESPONSE_BODY. (SecRules 4)
[174066786596.483884] [/] [4] Starting phase LOGGING. (SecRules 5)
[174066786596.483884] [/] [5] Saving this request as part of the audit logs.

I was able to switch to Germany using a VPN and was able to reach the applications website

[longwuyuan]

My suggestion was different from what you tried.
My suggestion is that you create a new workload. Even just using plain simple image like nginx:alpine. Or anything else of your choice that can listen on port 80.

Then download the db inside that image and serve the image from webroot.
Then use the usual controller config but specify this image as the URL to download the db from, in the controller config.

[chlynott]

Tested following the comment here #11435 (comment) This worked for about five days before seeing the same log message in the debug.log.
Also attempted to run a nginx image 'registry.k8s.io/ingress-nginx/controller:v1.9.3', download everything directly and use docker commit to make our own image. This attempt worked for about a day. Not sure what else to test at this point
Our maxmind mirror pod and service example:

apiVersion: v1
kind: Pod
metadata:
  name: maxmindmirror
  namespace: nginx-ingress-controller
  labels:
    app: maxmindmirror
spec:
  containers:
  - name: maxmindmirror
    image: nginx:alpine
    ports:
      - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: maxmindmirror
  namespace: nginx-ingress-controller
spec:
  selector:
    app: maxmindmirror
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

The nginx values we are passing to the helm chart:

controller:
  config:
    use-geoip2: true
  extraArgs:
    maxmind-mirror: "http://maxmindmirror"
  service:
    externalTrafficPolicy: Local

[longwuyuan]

• That maxmindmirror image should be built by you using nginx:alpine as base and adding the db file to it in Dockerfile of maxmindmirror image. Not create pod using nginx:alpine and then attempt to download inside its pod. I have tested this and it works perfectly consistently.
• What is the error message after 5 days ?
• If you use any custom image fro controller, then we can not support
• Use a recent supported release of the controller please. 1.9 is old and not supported