ServiceMonitor cannot access service port

[r2ronoha]

I'm trying to set up a service monitor on the authentication service, and it does not seem to be able to access the port.

Checks:

• app is up and running properly
• port is reachable from the cluster and other apps
• I've deployed a prometheus test container on the deployment + service, and pointed the servicemonitor to it, and it works as expected

Is it possible to monitor the port?
If so, is there any additional configuration required to do so?

• Standard config

apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik-forward-auth
  namespace: ingress-traefik
  labels:
    app: traefik-forward-auth
    chart: traefik-forward-auth-0.0.1
    release: wandera
    heritage: Tiller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik-forward-auth
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: traefik-forward-auth
    spec:
      terminationGracePeriodSeconds: 60
      containers:          
      - image: mesosphere/traefik-forward-auth:3.1.0
        name: traefik-forward-auth
        ports:
        - containerPort: 4181
          protocol: TCP
        env:
        - name: LOG_LEVEL
          value: debug
        - name: DOMAIN
          value: my-domain
        - name: INSECURE_COOKIE
          value: "false"
        - name: URL_PATH
          value: /oauth2/callback
        - name: SECRET
          valueFrom:
            secretKeyRef:
              name: traefik-forward-auth
              key: secret
        - name: PROVIDER_URI
          value: https://my-okta-domain/oauth2/default
        - name: CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: traefik-forward-auth
              key: client-id
        - name: CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: traefik-forward-auth
              key: client-secret
        - name: SCOPE
          value: email profile openid
---
apiVersion: v1
kind: Service
metadata:
  name: traefik-forward-auth
  namespace: ingress-traefik
  labels:
    app: traefik-forward-auth
    chart: traefik-forward-auth-0.0.1
    release: wandera
    heritage: Tiller
spec:
  type: ClusterIP
  selector:
    app: traefik-forward-auth
  ports:
  - name: auth-http
    port: 4181
    targetPort: 4181
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: traefik-forward-auth
  namespace: ingress-traefik
  labels:
    app: traefik-forward-auth
    chart: traefik-forward-auth-0.0.1
    heritage: Tiller
    release: wandera
spec:
  jobLabel: app
  selector:
    matchLabels:
      app: traefik-forward-auth
      chart: traefik-forward-auth-0.0.1
  namespaceSelector:
    matchNames:
    - ingress-traefik
  endpoints:
  - port: auth-http
    interval: 30s

• Config with additional container/service

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik-forward-auth
  namespace: ingress-traefik
  labels:
    app: traefik-forward-auth
    chart: traefik-forward-auth-0.0.1
    release: wandera
    heritage: Tiller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik-forward-auth
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: traefik-forward-auth
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: prom-test
        image: supergiantkir/prometheus-test-app
        ports:
        - name: prom-test
          containerPort: 8081
        resources:
          limits:
            cpu: 100m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
          
      - image: mesosphere/traefik-forward-auth:3.1.0
        name: traefik-forward-auth
        ports:
        - containerPort: 4181
          protocol: TCP
        env:
        - name: LOG_LEVEL
          value: debug
        - name: DOMAIN
          value: my-domain
        - name: INSECURE_COOKIE
          value: "false"
        - name: URL_PATH
          value: /oauth2/callback
        - name: SECRET
          valueFrom:
            secretKeyRef:
              name: traefik-forward-auth
              key: secret
        - name: PROVIDER_URI
          value: https://my-okta-domain/oauth2/default
        - name: CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: traefik-forward-auth
              key: client-id
        - name: CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: traefik-forward-auth
              key: client-secret
        - name: SCOPE
          value: email profile openid

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: traefik-forward-auth
  namespace: ingress-traefik
  labels:
    app: traefik-forward-auth
    chart: traefik-forward-auth-0.0.1
    heritage: Tiller
    release: wandera
spec:
  jobLabel: app
  selector:
    matchLabels:
      app: traefik-forward-auth
      chart: traefik-forward-auth-0.0.1
  namespaceSelector:
    matchNames:
    - ingress-traefik
  endpoints:
  - port: prom-test
    interval: 30s

---
apiVersion: v1
kind: Service
metadata:
  name: traefik-forward-auth
  namespace: ingress-traefik
  labels:
    app: traefik-forward-auth
    chart: traefik-forward-auth-0.0.1
    release: wandera
    heritage: Tiller
spec:
  type: ClusterIP
  selector:
    app: traefik-forward-auth
  ports:
  - name: auth-http
    port: 4181
    targetPort: 4181
---
apiVersion: v1
kind: Service
metadata:
  name: traefik-forward-auth-monitoring
  namespace: ingress-traefik
  labels:
    app: traefik-forward-auth
    chart: traefik-forward-auth-0.0.1
    release: wandera
    heritage: Tiller
spec:
  selector:
    app: traefik-forward-auth
  ports:
  - name: prom-test
    port: 8081
    targetPort: 8081