[🐛 Bug]: Event Bus container unable to use telemetry Java Agents

[quickjp2]

What happened?

Looking to use Azure Application Insights for tracing within my installation of Selenium Grid (fully distributed), instead of using Jaeger. I followed the docs for installing the agent along side the Selenium jar (see AppInsights Agent Docs), and added an initContainer to each of the Selenium Grid component deployments. All of the other components pick-up the AppInsights Java Agent and check-in with Azure, except for the Event Bus. As well, no browser nodes are able to check-in with the Grid.

However, when I remove the initContainer from just the Event Bus deployment, and remove the JAVA_TOOL_OPTIONS and APPLICATIONINSIGHTS_CONNECTION_STRING env vars from just there, the Event Bus does not throw the errors, and browser nodes are able to register with the grid.

Expectations:

• Event Bus uses the Java agent to self-register within Azure AppInsights application map
• Nodes are able register

As part of my troubleshooting, I tried using a different Java Agent, and the Event Bus container outputs the same errors (see logs).

Link to Azure AppInsights Agent Docs
Link to Open-Telemetry Agent Docs

Command used to start Selenium Grid with Docker (or Kubernetes)

# Source: selenium-grid/templates/event-bus-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: selenium-grid-selenium-event-bus-config
  namespace: selenium-grid-demo-ns
  labels:
    app.kubernetes.io/managed-by: helm
    app.kubernetes.io/instance: selenium-grid
    app.kubernetes.io/version: 4.29.0-20250303
    app.kubernetes.io/component: selenium-grid-4.29.0-20250303
    helm.sh/chart: selenium-grid-0.40.1
data:
  SE_EVENT_BUS_HOST: "selenium-grid-selenium-event-bus"
  SE_EVENT_BUS_PUBLISH_PORT: "4442"
  SE_EVENT_BUS_SUBSCRIBE_PORT: "4443"
  SE_JAVA_OPTS: "-XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -XX:MaxRAMPercentage=100 -Dos.name=linux"
  start-selenium-grid-eventbus.sh: |-
      #!/usr/bin/env bash

      # set -e: exit asap if a command exits with a non-zero status
      set -e

      echo "Starting Selenium Grid EventBus..."

      function append_se_opts() {
        local option="${1}"
        local value="${2:-""}"
        local log_message="${3:-true}"
        if [[ "${SE_OPTS}" != *"${option}"* ]]; then
          if [ "${log_message}" = "true" ]; then
            echo "Appending Selenium option: ${option} ${value}"
          else
            echo "Appending Selenium option: ${option} $(mask ${value})"
          fi
          SE_OPTS="${SE_OPTS} ${option}"
          if [ ! -z "${value}" ]; then
            SE_OPTS="${SE_OPTS} ${value}"
          fi
        else
          echo "Selenium option: ${option} already set in env variable SE_OPTS. Ignore new option: ${option} ${value}"
        fi
      }

      if [ ! -z "$SE_EVENT_BUS_HOST" ]; then
        echo "Using SE_EVENT_BUS_HOST: ${SE_EVENT_BUS_HOST}"
        HOST_CONFIG="--host ${SE_EVENT_BUS_HOST}"
      fi

      if [ ! -z "$SE_EVENT_BUS_PORT" ]; then
        echo "Using SE_EVENT_BUS_PORT: ${SE_EVENT_BUS_PORT}"
        PORT_CONFIG="--port ${SE_EVENT_BUS_PORT}"
      fi

      if [ ! -z "$SE_OPTS" ]; then
        echo "Appending Selenium options: ${SE_OPTS}"
      fi

      if [ ! -z "$SE_LOG_LEVEL" ]; then
        append_se_opts "--log-level" "${SE_LOG_LEVEL}"
      fi

      if [ ! -z "$SE_HTTP_LOGS" ]; then
        append_se_opts "--http-logs" "${SE_HTTP_LOGS}"
      fi

      if [ ! -z "$SE_STRUCTURED_LOGS" ]; then
        append_se_opts "--structured-logs" "${SE_STRUCTURED_LOGS}"
      fi

      if [ ! -z "$SE_EXTERNAL_URL" ]; then
        append_se_opts "--external-url" "${SE_EXTERNAL_URL}"
      fi

      if [ "${SE_ENABLE_TLS}" = "true" ]; then
        # Configure truststore for the server
        if [ ! -z "$SE_JAVA_SSL_TRUST_STORE" ]; then
          echo "Appending Java options: -Djavax.net.ssl.trustStore=${SE_JAVA_SSL_TRUST_STORE}"
          SE_JAVA_OPTS="$SE_JAVA_OPTS -Djavax.net.ssl.trustStore=${SE_JAVA_SSL_TRUST_STORE}"
        fi
        if [ -f "${SE_JAVA_SSL_TRUST_STORE_PASSWORD}" ]; then
          echo "Getting Truststore password from ${SE_JAVA_SSL_TRUST_STORE_PASSWORD} to set Java options: -Djavax.net.ssl.trustStorePassword"
          SE_JAVA_SSL_TRUST_STORE_PASSWORD="$(cat ${SE_JAVA_SSL_TRUST_STORE_PASSWORD})"
        fi
        if [ ! -z "${SE_JAVA_SSL_TRUST_STORE_PASSWORD}" ]; then
          echo "Appending Java options: -Djavax.net.ssl.trustStorePassword=$(mask ${SE_JAVA_SSL_TRUST_STORE_PASSWORD})"
          SE_JAVA_OPTS="$SE_JAVA_OPTS -Djavax.net.ssl.trustStorePassword=${SE_JAVA_SSL_TRUST_STORE_PASSWORD}"
        fi
        echo "Appending Java options: -Djdk.internal.httpclient.disableHostnameVerification=${SE_JAVA_DISABLE_HOSTNAME_VERIFICATION}"
        SE_JAVA_OPTS="$SE_JAVA_OPTS -Djdk.internal.httpclient.disableHostnameVerification=${SE_JAVA_DISABLE_HOSTNAME_VERIFICATION}"
        # Configure certificate and private key for component communication
        if [ ! -z "$SE_HTTPS_CERTIFICATE" ]; then
          append_se_opts "--https-certificate" "${SE_HTTPS_CERTIFICATE}"
        fi
        if [ ! -z "$SE_HTTPS_PRIVATE_KEY" ]; then
          append_se_opts "--https-private-key" "${SE_HTTPS_PRIVATE_KEY}"
        fi
      fi

      EXTRA_LIBS=""

      if [ "${SE_ENABLE_TRACING}" = "true" ] && [ -n "${SE_OTEL_EXPORTER_ENDPOINT}" ]; then
        EXTERNAL_JARS=$(</external_jars/.classpath.txt)
        [ -n "$EXTRA_LIBS" ] && [ -n "${EXTERNAL_JARS}" ] && EXTRA_LIBS=${EXTRA_LIBS}:
        EXTRA_LIBS="--ext "${EXTRA_LIBS}${EXTERNAL_JARS}
        echo "Tracing is enabled"
        echo "Classpath will be enriched with these external jars : " ${EXTRA_LIBS}
        if [ -n "$SE_OTEL_SERVICE_NAME" ]; then
          SE_OTEL_JVM_ARGS="$SE_OTEL_JVM_ARGS -Dotel.resource.attributes=service.name=${SE_OTEL_SERVICE_NAME}"
        fi
        if [ -n "$SE_OTEL_TRACES_EXPORTER" ]; then
          SE_OTEL_JVM_ARGS="$SE_OTEL_JVM_ARGS -Dotel.traces.exporter=${SE_OTEL_TRACES_EXPORTER}"
        fi
        if [ -n "$SE_OTEL_EXPORTER_ENDPOINT" ]; then
          SE_OTEL_JVM_ARGS="$SE_OTEL_JVM_ARGS -Dotel.exporter.otlp.endpoint=$(envsubst < <(echo ${SE_OTEL_EXPORTER_ENDPOINT}))"
        fi
        if [ -n "$SE_OTEL_JAVA_GLOBAL_AUTOCONFIGURE_ENABLED" ]; then
          SE_OTEL_JVM_ARGS="$SE_OTEL_JVM_ARGS -Dotel.java.global-autoconfigure.enabled=${SE_OTEL_JAVA_GLOBAL_AUTOCONFIGURE_ENABLED}"
        fi
        if [ -n "$SE_OTEL_JVM_ARGS" ]; then
          SE_JAVA_OPTS="$SE_JAVA_OPTS ${SE_OTEL_JVM_ARGS}"
        fi
      else
        if [ -n "${APPLICATIONINSIGHTS_CONNECTION_STRING}" ]; then
          echo "Using Azure App Insights"
          APPLICATIONINSIGHTS_ROLE_NAME="${SE_OTEL_SERVICE_NAME}"
          if [ -n "$SE_OTEL_JAVA_GLOBAL_AUTOCONFIGURE_ENABLED" ]; then
            SE_OTEL_JVM_ARGS="$SE_OTEL_JVM_ARGS -Dotel.java.global-autoconfigure.enabled=${SE_OTEL_JAVA_GLOBAL_AUTOCONFIGURE_ENABLED}"
          fi
          if [ -n "$SE_OTEL_TRACES_EXPORTER" ]; then
            SE_OTEL_JVM_ARGS="$SE_OTEL_JVM_ARGS -Dotel.traces.exporter=${SE_OTEL_TRACES_EXPORTER}"
          fi
          if [ -n "$SE_OTEL_JVM_ARGS" ]; then
            SE_JAVA_OPTS="$SE_JAVA_OPTS ${SE_OTEL_JVM_ARGS}"
          fi
        else
          append_se_opts "--tracing" "false"
          SE_JAVA_OPTS="$SE_JAVA_OPTS -Dwebdriver.remote.enableTracing=false"
          echo "Tracing is disabled"
        fi
      fi

      if [ -n "${SE_JAVA_HTTPCLIENT_VERSION}" ]; then
        SE_JAVA_OPTS="$SE_JAVA_OPTS -Dwebdriver.httpclient.version=${SE_JAVA_HTTPCLIENT_VERSION}"
      fi

      if [ -n "${SE_JAVA_OPTS_DEFAULT}" ]; then
        SE_JAVA_OPTS="${SE_JAVA_OPTS_DEFAULT} $SE_JAVA_OPTS"
      fi

      if [ -n "${JAVA_OPTS:-$SE_JAVA_OPTS}" ]; then
        echo "Using JAVA_OPTS: ${JAVA_OPTS:-$SE_JAVA_OPTS}"
      fi

      function handle_heap_dump() {
        /opt/bin/handle_heap_dump.sh /opt/selenium/logs
      }
      if [ "${SE_JAVA_HEAP_DUMP}" = "true" ]; then
        trap handle_heap_dump ERR SIGTERM SIGINT
      else
        trap handle_heap_dump ERR
      fi

      java ${JAVA_OPTS:-$SE_JAVA_OPTS} \
        -jar /opt/selenium/selenium-server.jar \
        ${EXTRA_LIBS} event-bus \
        --bind-host ${SE_BIND_HOST} \
        ${HOST_CONFIG} \
        ${PORT_CONFIG} \
        ${SE_OPTS}
---
# Source: selenium-grid/templates/event-bus-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: selenium-grid-selenium-event-bus
  namespace: selenium-grid-demo-ns
  labels: &event_bus_labels
    app: selenium-grid-selenium-event-bus
    app.kubernetes.io/name: selenium-grid-selenium-event-bus
    app.kubernetes.io/managed-by: helm
    app.kubernetes.io/instance: selenium-grid
    app.kubernetes.io/version: 4.29.0-20250303
    app.kubernetes.io/component: selenium-grid-4.29.0-20250303
    helm.sh/chart: selenium-grid-0.40.1
spec:
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: selenium-grid-selenium-event-bus
      app.kubernetes.io/instance: selenium-grid
  template:
    metadata:
      labels: *event_bus_labels
      annotations:
        checksum/logging-configmap: 340341e72ca26257adeb74743bec46d3a16f7a94cbe37ac71703d23bf1a4144e
        checksum/server-configmap: bd08b4ce23320321f4ef8e3903ff63e449101c0e0c8bdd76957ff7974b30d37e
        checksum/secrets: 8068a715b5d90514c7f111b339be79a8d9915ad9391c853e2ba8afa1d58a8df6
    spec:
      serviceAccountName: selenium-grid-selenium-serviceaccount
      serviceAccount: selenium-grid-selenium-serviceaccount
      initContainers:
      - name: app-insights-clone
        image: selenium/hub:4.29.0-20250303
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh","-c"]
        args: ["wget -P /tmp/jars https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar"]
        # args: ["wget -P /tmp/jars https://github.com/microsoft/ApplicationInsights-Java/releases/download/3.7.1/applicationinsights-agent-3.7.1.jar"]
        volumeMounts:
        - mountPath: /tmp/jars
          name: appinsights-jar
      containers:
        - name: selenium-grid-selenium-event-bus
          image: selenium/event-bus:4.29.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5557
              protocol: TCP
            - containerPort: 4442
              protocol: TCP
            - containerPort: 4443
              protocol: TCP
          env:
            - name: KUBERNETES_NODE_HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: APPLICATIONINSIGHTS_ROLE_NAME
              value: 'selenium-grid-selenium-event-bus'
            - name: SE_OTEL_SERVICE_NAME
              value: 'selenium-grid-selenium-event-bus'
            - name: SE_EVENT_BUS_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: SE_EVENT_BUS_PORT
              value: "5557"
            - name: SE_LOG_LEVEL
              value: "FINEST"
            - name: APPLICATIONINSIGHTS_CONNECTION_STRING
              value: ""
            - name: JAVA_TOOL_OPTIONS
              value: "-javaagent:/tmp/jars/opentelemetry-javaagent.jar"
          envFrom:
            - configMapRef:
                name: selenium-grid-selenium-logging-config
            - configMapRef:
                name: selenium-grid-selenium-server-config
            - secretRef:
                name: selenium-grid-selenium-secrets
            - secretRef:
                name: selenium-grid-selenium-basic-auth-secrets
          volumeMounts:
            - mountPath: /opt/bin/start-selenium-grid-eventbus.sh
              name: selenium-grid-selenium-event-bus-config
              subPath: start-selenium-grid-eventbus.sh
            - mountPath: /tmp/jars
              name: appinsights-jar
          resources:
            limits:
              cpu: "1"
              memory: 2Gi
            requests:
              cpu: "0.5"
              memory: 512Mi
      volumes:
        - name: "selenium-grid-selenium-event-bus-config"
          configMap:
            name: selenium-grid-selenium-event-bus-config
            defaultMode: 493
        - emptyDir: {}
          name: appinsights-jar

Relevant log output

# Opentelemetry-Agent Logs
15:02:23.308 DEBUG [PlatformDependent0.<clinit>] - direct buffer constructor: unavailable
java.lang.UnsupportedOperationException: Reflective setAccessible(true) disabled
        at io.netty.util.internal.ReflectionUtil.trySetAccessible(ReflectionUtil.java:31)
        at io.netty.util.internal.PlatformDependent0$5.run(PlatformDependent0.java:332)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
        at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:325)
        at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:334)
        at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:90)
        at io.netty.channel.nio.NioEventLoop.<clinit>(NioEventLoop.java:84)
        at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:182)
        at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:38)
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:60)
        at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:52)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:97)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:92)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:73)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:53)
        at org.openqa.selenium.netty.server.NettyServer.<init>(NettyServer.java:102)
        at org.openqa.selenium.netty.server.NettyServer.<init>(NettyServer.java:68)
        at org.openqa.selenium.grid.commands.EventBusCommand$1.<init>(EventBusCommand.java:158)
        at org.openqa.selenium.grid.commands.EventBusCommand.asServer(EventBusCommand.java:133)
        at org.openqa.selenium.grid.commands.EventBusCommand.execute(EventBusCommand.java:186)
        at org.openqa.selenium.grid.TemplateGridCommand.lambda$configure$4(TemplateGridCommand.java:122)
        at org.openqa.selenium.grid.Main.launch(Main.java:83)
        at org.openqa.selenium.grid.Main.go(Main.java:56)
        at org.openqa.selenium.grid.Main.main(Main.java:41)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.openqa.selenium.grid.Bootstrap.runMain(Bootstrap.java:77)
        at org.openqa.selenium.grid.Bootstrap.main(Bootstrap.java:70)
{"class": "PlatformDependent0","log-level": "DEBUG","log-message": "direct buffer constructor: unavailable","log-name": "io.netty.util.internal.PlatformDependent0","log-time-local": "2025-03-18T15:02:23.308Z","log-time-utc": "2025-03-18T15:02:23.308Z","method": "\u003cclinit>"}
15:02:23.316 DEBUG [PlatformDependent0.<clinit>] - java.nio.Bits.unaligned: available, true
{"class": "PlatformDependent0","log-level": "DEBUG","log-message": "java.nio.Bits.unaligned: available, true","log-name": "io.netty.util.internal.PlatformDependent0","log-time-local": "2025-03-18T15:02:23.316Z","log-time-utc": "2025-03-18T15:02:23.316Z","method": "\u003cclinit>"}
15:02:23.320 DEBUG [PlatformDependent0.<clinit>] - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable
java.lang.IllegalAccessException: class io.netty.util.internal.PlatformDependent0$7 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @1757cd72
        at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:394)
        at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:714)
        at java.base/java.lang.reflect.Method.invoke(Method.java:571)
        at io.netty.util.internal.PlatformDependent0$7.run(PlatformDependent0.java:468)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
        at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:459)
        at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:334)
        at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:90)
        at io.netty.channel.nio.NioEventLoop.<clinit>(NioEventLoop.java:84)
        at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:182)
        at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:38)
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:60)
        at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:52)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:97)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:92)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:73)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:53)
        at org.openqa.selenium.netty.server.NettyServer.<init>(NettyServer.java:102)
        at org.openqa.selenium.netty.server.NettyServer.<init>(NettyServer.java:68)
        at org.openqa.selenium.grid.commands.EventBusCommand$1.<init>(EventBusCommand.java:158)
        at org.openqa.selenium.grid.commands.EventBusCommand.asServer(EventBusCommand.java:133)
        at org.openqa.selenium.grid.commands.EventBusCommand.execute(EventBusCommand.java:186)
        at org.openqa.selenium.grid.TemplateGridCommand.lambda$configure$4(TemplateGridCommand.java:122)
        at org.openqa.selenium.grid.Main.launch(Main.java:83)
        at org.openqa.selenium.grid.Main.go(Main.java:56)
        at org.openqa.selenium.grid.Main.main(Main.java:41)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.openqa.selenium.grid.Bootstrap.runMain(Bootstrap.java:77)
        at org.openqa.selenium.grid.Bootstrap.main(Bootstrap.java:70)
{"class": "PlatformDependent0","log-level": "DEBUG","log-message": "jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable","log-name": "io.netty.util.internal.PlatformDependent0","log-time-local": "2025-03-18T15:02:23.32Z","log-time-utc": "2025-03-18T15:02:23.32Z","method": "\u003cclinit>"}

# Azure ApplicationInsights Agent Logs
22:32:36.926 DEBUG [PlatformDependent0.<clinit>] - direct buffer constructor: unavailable
java.lang.UnsupportedOperationException: Reflective setAccessible(true) disabled
        at io.netty.util.internal.ReflectionUtil.trySetAccessible(ReflectionUtil.java:31)
        at io.netty.util.internal.PlatformDependent0$5.run(PlatformDependent0.java:332)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
        at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:325)
        at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:334)
        at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:90)
        at io.netty.channel.nio.NioEventLoop.<clinit>(NioEventLoop.java:84)
        at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:182)
        at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:38)
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:60)
        at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:52)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:97)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:92)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:73)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:53)
        at org.openqa.selenium.netty.server.NettyServer.<init>(NettyServer.java:102)
        at org.openqa.selenium.netty.server.NettyServer.<init>(NettyServer.java:68)
        at org.openqa.selenium.grid.commands.EventBusCommand$1.<init>(EventBusCommand.java:158)
        at org.openqa.selenium.grid.commands.EventBusCommand.asServer(EventBusCommand.java:133)
        at org.openqa.selenium.grid.commands.EventBusCommand.execute(EventBusCommand.java:186)
        at org.openqa.selenium.grid.TemplateGridCommand.lambda$configure$4(TemplateGridCommand.java:122)
        at org.openqa.selenium.grid.Main.launch(Main.java:83)
        at org.openqa.selenium.grid.Main.go(Main.java:56)
        at org.openqa.selenium.grid.Main.main(Main.java:41)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.openqa.selenium.grid.Bootstrap.runMain(Bootstrap.java:77)
        at org.openqa.selenium.grid.Bootstrap.main(Bootstrap.java:70)
{"class": "PlatformDependent0","log-level": "DEBUG","log-message": "direct buffer constructor: unavailable","log-name": "io.netty.util.internal.PlatformDependent0","log-time-local": "2025-03-17T22:32:36.926Z","log-time-utc": "2025-03-17T22:32:36.926Z","method": "\u003cclinit>"}
22:32:36.937 DEBUG [PlatformDependent0.<clinit>] - java.nio.Bits.unaligned: available, true
{"class": "PlatformDependent0","log-level": "DEBUG","log-message": "java.nio.Bits.unaligned: available, true","log-name": "io.netty.util.internal.PlatformDependent0","log-time-local": "2025-03-17T22:32:36.937Z","log-time-utc": "2025-03-17T22:32:36.937Z","method": "\u003cclinit>"}
22:32:36.938 DEBUG [PlatformDependent0.<clinit>] - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable
java.lang.IllegalAccessException: class io.netty.util.internal.PlatformDependent0$7 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @43bd930a
        at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:394)
        at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:714)
        at java.base/java.lang.reflect.Method.invoke(Method.java:571)
        at io.netty.util.internal.PlatformDependent0$7.run(PlatformDependent0.java:468)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
        at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:459)
        at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:334)
        at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:90)
        at io.netty.channel.nio.NioEventLoop.<clinit>(NioEventLoop.java:84)
        at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:182)
        at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:38)
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:60)
        at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:52)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:97)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:92)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:73)
        at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:53)
        at org.openqa.selenium.netty.server.NettyServer.<init>(NettyServer.java:102)
        at org.openqa.selenium.netty.server.NettyServer.<init>(NettyServer.java:68)
        at org.openqa.selenium.grid.commands.EventBusCommand$1.<init>(EventBusCommand.java:158)
        at org.openqa.selenium.grid.commands.EventBusCommand.asServer(EventBusCommand.java:133)
        at org.openqa.selenium.grid.commands.EventBusCommand.execute(EventBusCommand.java:186)
        at org.openqa.selenium.grid.TemplateGridCommand.lambda$configure$4(TemplateGridCommand.java:122)
        at org.openqa.selenium.grid.Main.launch(Main.java:83)
        at org.openqa.selenium.grid.Main.go(Main.java:56)
        at org.openqa.selenium.grid.Main.main(Main.java:41)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.openqa.selenium.grid.Bootstrap.runMain(Bootstrap.java:77)
        at org.openqa.selenium.grid.Bootstrap.main(Bootstrap.java:70)
{"class": "PlatformDependent0","log-level": "DEBUG","log-message": "jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable","log-name": "io.netty.util.internal.PlatformDependent0","log-time-local": "2025-03-17T22:32:36.938Z","log-time-utc": "2025-03-17T22:32:36.938Z","method": "\u003cclinit>"}

Operating System

Kubernetes (AKS)

Docker Selenium version (image tag)

4.29.0-20250303

Selenium Grid chart version (chart version)

0.40.1

[github-actions]

@quickjp2, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[VietND96]

I saw you download extra jars

    args: ["wget -P /tmp/jars https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar"]

Can you include those classpath in part of launch java -jar /opt/selenium/selenium-server.jar --ext ...

Which is current handle in this step, with download opentelemetry jars

      EXTRA_LIBS=""

      if [ "${SE_ENABLE_TRACING}" = "true" ] && [ -n "${SE_OTEL_EXPORTER_ENDPOINT}" ]; then
        EXTERNAL_JARS=$(</external_jars/.classpath.txt)
        [ -n "$EXTRA_LIBS" ] && [ -n "${EXTERNAL_JARS}" ] && EXTRA_LIBS=${EXTRA_LIBS}:
        EXTRA_LIBS="--ext "${EXTRA_LIBS}${EXTERNAL_JARS}
        echo "Tracing is enabled"
        echo "Classpath will be enriched with these external jars : " ${EXTRA_LIBS}

[VietND96]

By default, when enable tracing, we downloaded those jars to /external_jars and define classpath as below, so you can do the same, and define your ext classpath

Tracing is enabled
Classpath will be enriched with these external jars : --ext /external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-exporter-otlp/1.46.0/opentelemetry-exporter-otlp-1.46.0.jar:/external_jars/https/repo1.maven.org/maven2/io/grpc/grpc-netty/1.70.0/grpc-netty-1.70.0.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-codec-http/4.1.118.Final/netty-codec-http-4.1.118.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-handler/4.1.118.Final/netty-handler-4.1.118.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-common/4.1.118.Final/netty-common-4.1.118.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-sdk-trace/1.46.0/opentelemetry-sdk-trace-1.46.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-sdk-metrics/1.46.0/opentelemetry-sdk-metrics-1.46.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-sdk-logs/1.46.0/opentelemetry-sdk-logs-1.46.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-exporter-otlp-common/1.46.0/opentelemetry-exporter-otlp-common-1.46.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-exporter-sender-okhttp/1.46.0/opentelemetry-exporter-sender-okhttp-1.46.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-sdk-extension-autoconfigure-spi/1.46.0/opentelemetry-sdk-extension-autoconfigure-spi-1.46.0.jar:/external_jars/https/repo1.maven.org/maven2/io/grpc/grpc-api/1.70.0/grpc-api-1.70.0.jar:/external_jars/https/repo1.maven.org/maven2/org/codehaus/mojo/animal-sniffer-annotations/1.24/animal-sniffer-annotations-1.24.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-codec-http2/4.1.110.Final/netty-codec-http2-4.1.110.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/grpc/grpc-core/1.70.0/grpc-core-1.70.0.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-handler-proxy/4.1.110.Final/netty-handler-proxy-4.1.110.Final.jar:/external_jars/https/repo1.maven.org/maven2/com/google/guava/guava/33.3.1-android/guava-33.3.1-android.jar:/external_jars/https/repo1.maven.org/maven2/com/google/errorprone/error_prone_annotations/2.30.0/error_prone_annotations-2.30.0.jar:/external_jars/https/repo1.maven.org/maven2/io/perfmark/perfmark-api/0.27.0/perfmark-api-0.27.0.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-transport-native-unix-common/4.1.118.Final/netty-transport-native-unix-common-4.1.118.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/grpc/grpc-util/1.70.0/grpc-util-1.70.0.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-buffer/4.1.118.Final/netty-buffer-4.1.118.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-transport/4.1.118.Final/netty-transport-4.1.118.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-codec/4.1.118.Final/netty-codec-4.1.118.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-resolver/4.1.118.Final/netty-resolver-4.1.118.Final.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-api/1.46.0/opentelemetry-api-1.46.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-sdk-common/1.46.0/opentelemetry-sdk-common-1.46.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-api-incubator/1.46.0-alpha/opentelemetry-api-incubator-1.46.0-alpha.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-exporter-common/1.46.0/opentelemetry-exporter-common-1.46.0.jar:/external_jars/https/repo1.maven.org/maven2/com/squareup/okhttp3/okhttp/4.12.0/okhttp-4.12.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-sdk/1.46.0/opentelemetry-sdk-1.46.0.jar:/external_jars/https/repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar:/external_jars/https/repo1.maven.org/maven2/com/google/code/gson/gson/2.11.0/gson-2.11.0.jar:/external_jars/https/repo1.maven.org/maven2/com/google/android/annotations/4.1.1.4/annotations-4.1.1.4.jar:/external_jars/https/repo1.maven.org/maven2/io/grpc/grpc-context/1.70.0/grpc-context-1.70.0.jar:/external_jars/https/repo1.maven.org/maven2/io/netty/netty-codec-socks/4.1.110.Final/netty-codec-socks-4.1.110.Final.jar:/external_jars/https/repo1.maven.org/maven2/com/google/guava/failureaccess/1.0.2/failureaccess-1.0.2.jar:/external_jars/https/repo1.maven.org/maven2/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/external_jars/https/repo1.maven.org/maven2/org/checkerframework/checker-qual/3.43.0/checker-qual-3.43.0.jar:/external_jars/https/repo1.maven.org/maven2/com/google/j2objc/j2objc-annotations/3.0.0/j2objc-annotations-3.0.0.jar:/external_jars/https/repo1.maven.org/maven2/io/opentelemetry/opentelemetry-context/1.46.0/opentelemetry-context-1.46.0.jar:/external_jars/https/repo1.maven.org/maven2/com/squareup/okio/okio/3.6.0/okio-3.6.0.jar:/external_jars/https/repo1.maven.org/maven2/org/jetbrains/kotlin/kotlin-stdlib-jdk8/1.9.10/kotlin-stdlib-jdk8-1.9.10.jar:/external_jars/https/repo1.maven.org/maven2/com/squareup/okio/okio-jvm/3.6.0/okio-jvm-3.6.0.jar:/external_jars/https/repo1.maven.org/maven2/org/jetbrains/kotlin/kotlin-stdlib/1.9.10/kotlin-stdlib-1.9.10.jar:/external_jars/https/repo1.maven.org/maven2/org/jetbrains/kotlin/kotlin-stdlib-jdk7/1.9.10/kotlin-stdlib-jdk7-1.9.10.jar:/external_jars/https/repo1.maven.org/maven2/org/jetbrains/kotlin/kotlin-stdlib-common/1.9.10/kotlin-stdlib-common-1.9.10.jar:/external_jars/https/repo1.maven.org/maven2/org/jetbrains/annotations/13.0/annotations-13.0.jar

[VietND96]

Moreover, in container, we also have util cs to fetch jars and construct classpath, it usage and location here

java -jar /tmp/cs fetch --classpath --cache /external_jars \
        io.opentelemetry:opentelemetry-exporter-otlp:${OPENTELEMETRY_VERSION} \
        io.grpc:grpc-netty:${GRPC_VERSION} \
        io.netty:netty-codec-http:${NETTY_VERSION} \
        io.netty:netty-handler:${NETTY_VERSION} \
        io.netty:netty-common:${NETTY_VERSION} \
        > /external_jars/.classpath.txt \

[VietND96]

Also, currently we are using SE_OTEL_TRACES_EXPORTER="otlp" (Open Telemetry) it isn't specific to Jaeger. Does Azure Application Insights support an endpoint, that Grid can export traces to, the endpoint specify via SE_OTEL_EXPORTER_ENDPOINT

[VietND96]

Can you look into this? https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/azuremonitorexporter/README.md

[quickjp2]

Hi @VietND96,

Thanks for the quick responses!

The custom jar I left in was what I used for verifying it wasn't just the Application Insights agent having issues. While I included the app insights fetch command as a comment, here's that line again:
args: ["wget -P /tmp/jars https://github.com/microsoft/ApplicationInsights-Java/releases/download/3.7.1/applicationinsights-agent-3.7.1.jar"]

I'll add the EXTRA_LIBS logic and try that out within the hour, and report back!

As well, I'll look at the docs for Open-Telemetry, since that does appear to add more info than what I saw in the MS docs I linked in the ticket.

[quickjp2]

I updated the start script to get the EXTRA_LIBS added in. Below is the the updated copy of both that configMap and the Event Bus deployment. I still get the same errors.

In all cases, I've had the connection string App Insights says to use added as an environment var from a config map (see here). I've declared the variable blank in the below deployment as a placeholder, as a way to indicate that it is being set.

The MS docs for adding in the appInsights jar is to use the -javaagent flag (see here). That being said, I'll be trying adding the jar to the class path, as you indicated earlier, right after I post this update.

# Source: selenium-grid/templates/event-bus-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: selenium-grid-selenium-event-bus-config
  namespace: selenium-grid-demo-ns
  labels:
    app.kubernetes.io/managed-by: helm
    app.kubernetes.io/instance: selenium-grid
    app.kubernetes.io/version: 4.29.0-20250303
    app.kubernetes.io/component: selenium-grid-4.29.0-20250303
    helm.sh/chart: selenium-grid-0.40.1
data:
  SE_EVENT_BUS_HOST: "selenium-grid-selenium-event-bus"
  SE_EVENT_BUS_PUBLISH_PORT: "4442"
  SE_EVENT_BUS_SUBSCRIBE_PORT: "4443"
  SE_JAVA_OPTS: "-XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -XX:MaxRAMPercentage=100 -Dos.name=linux"
  start-selenium-grid-eventbus.sh: |-
      #!/usr/bin/env bash

      # set -e: exit asap if a command exits with a non-zero status
      set -e

      echo "Starting Selenium Grid EventBus..."

      function append_se_opts() {
        local option="${1}"
        local value="${2:-""}"
        local log_message="${3:-true}"
        if [[ "${SE_OPTS}" != *"${option}"* ]]; then
          if [ "${log_message}" = "true" ]; then
            echo "Appending Selenium option: ${option} ${value}"
          else
            echo "Appending Selenium option: ${option} $(mask ${value})"
          fi
          SE_OPTS="${SE_OPTS} ${option}"
          if [ ! -z "${value}" ]; then
            SE_OPTS="${SE_OPTS} ${value}"
          fi
        else
          echo "Selenium option: ${option} already set in env variable SE_OPTS. Ignore new option: ${option} ${value}"
        fi
      }

      if [ ! -z "$SE_EVENT_BUS_HOST" ]; then
        echo "Using SE_EVENT_BUS_HOST: ${SE_EVENT_BUS_HOST}"
        HOST_CONFIG="--host ${SE_EVENT_BUS_HOST}"
      fi

      if [ ! -z "$SE_EVENT_BUS_PORT" ]; then
        echo "Using SE_EVENT_BUS_PORT: ${SE_EVENT_BUS_PORT}"
        PORT_CONFIG="--port ${SE_EVENT_BUS_PORT}"
      fi

      if [ ! -z "$SE_OPTS" ]; then
        echo "Appending Selenium options: ${SE_OPTS}"
      fi

      if [ ! -z "$SE_LOG_LEVEL" ]; then
        append_se_opts "--log-level" "${SE_LOG_LEVEL}"
      fi

      if [ ! -z "$SE_HTTP_LOGS" ]; then
        append_se_opts "--http-logs" "${SE_HTTP_LOGS}"
      fi

      if [ ! -z "$SE_STRUCTURED_LOGS" ]; then
        append_se_opts "--structured-logs" "${SE_STRUCTURED_LOGS}"
      fi

      if [ ! -z "$SE_EXTERNAL_URL" ]; then
        append_se_opts "--external-url" "${SE_EXTERNAL_URL}"
      fi

      if [ "${SE_ENABLE_TLS}" = "true" ]; then
        # Configure truststore for the server
        if [ ! -z "$SE_JAVA_SSL_TRUST_STORE" ]; then
          echo "Appending Java options: -Djavax.net.ssl.trustStore=${SE_JAVA_SSL_TRUST_STORE}"
          SE_JAVA_OPTS="$SE_JAVA_OPTS -Djavax.net.ssl.trustStore=${SE_JAVA_SSL_TRUST_STORE}"
        fi
        if [ -f "${SE_JAVA_SSL_TRUST_STORE_PASSWORD}" ]; then
          echo "Getting Truststore password from ${SE_JAVA_SSL_TRUST_STORE_PASSWORD} to set Java options: -Djavax.net.ssl.trustStorePassword"
          SE_JAVA_SSL_TRUST_STORE_PASSWORD="$(cat ${SE_JAVA_SSL_TRUST_STORE_PASSWORD})"
        fi
        if [ ! -z "${SE_JAVA_SSL_TRUST_STORE_PASSWORD}" ]; then
          echo "Appending Java options: -Djavax.net.ssl.trustStorePassword=$(mask ${SE_JAVA_SSL_TRUST_STORE_PASSWORD})"
          SE_JAVA_OPTS="$SE_JAVA_OPTS -Djavax.net.ssl.trustStorePassword=${SE_JAVA_SSL_TRUST_STORE_PASSWORD}"
        fi
        echo "Appending Java options: -Djdk.internal.httpclient.disableHostnameVerification=${SE_JAVA_DISABLE_HOSTNAME_VERIFICATION}"
        SE_JAVA_OPTS="$SE_JAVA_OPTS -Djdk.internal.httpclient.disableHostnameVerification=${SE_JAVA_DISABLE_HOSTNAME_VERIFICATION}"
        # Configure certificate and private key for component communication
        if [ ! -z "$SE_HTTPS_CERTIFICATE" ]; then
          append_se_opts "--https-certificate" "${SE_HTTPS_CERTIFICATE}"
        fi
        if [ ! -z "$SE_HTTPS_PRIVATE_KEY" ]; then
          append_se_opts "--https-private-key" "${SE_HTTPS_PRIVATE_KEY}"
        fi
      fi

      EXTRA_LIBS=""

      if [ "${SE_ENABLE_TRACING}" = "true" ] && { [ -n "${APPLICATIONINSIGHTS_CONNECTION_STRING}" ] || [ -n "${SE_OTEL_EXPORTER_ENDPOINT}" ]; }; then
        EXTERNAL_JARS=$(</external_jars/.classpath.txt)
        [ -n "$EXTRA_LIBS" ] && [ -n "${EXTERNAL_JARS}" ] && EXTRA_LIBS=${EXTRA_LIBS}:
        EXTRA_LIBS="--ext "${EXTRA_LIBS}${EXTERNAL_JARS}
        echo "Tracing is enabled"
        echo "Classpath will be enriched with these external jars : " ${EXTRA_LIBS}
        if [ -n "$SE_OTEL_SERVICE_NAME" ]; then
          SE_OTEL_JVM_ARGS="$SE_OTEL_JVM_ARGS -Dotel.resource.attributes=service.name=${SE_OTEL_SERVICE_NAME}"
        fi
        if [ -n "$SE_OTEL_TRACES_EXPORTER" ]; then
          SE_OTEL_JVM_ARGS="$SE_OTEL_JVM_ARGS -Dotel.traces.exporter=${SE_OTEL_TRACES_EXPORTER}"
        fi
        if [ -n "$SE_OTEL_EXPORTER_ENDPOINT" ]; then
          SE_OTEL_JVM_ARGS="$SE_OTEL_JVM_ARGS -Dotel.exporter.otlp.endpoint=$(envsubst < <(echo ${SE_OTEL_EXPORTER_ENDPOINT}))"
        fi
        if [ -n "${APPLICATIONINSIGHTS_CONNECTION_STRING}" ]; then
          echo "Using Azure App Insights"
          APPLICATIONINSIGHTS_ROLE_NAME="${SE_OTEL_SERVICE_NAME}"
        fi
        if [ -n "$SE_OTEL_JAVA_GLOBAL_AUTOCONFIGURE_ENABLED" ]; then
          SE_OTEL_JVM_ARGS="$SE_OTEL_JVM_ARGS -Dotel.java.global-autoconfigure.enabled=${SE_OTEL_JAVA_GLOBAL_AUTOCONFIGURE_ENABLED}"
        fi
        if [ -n "$SE_OTEL_JVM_ARGS" ]; then
          SE_JAVA_OPTS="$SE_JAVA_OPTS ${SE_OTEL_JVM_ARGS}"
        fi
      else
        append_se_opts "--tracing" "false"
        SE_JAVA_OPTS="$SE_JAVA_OPTS -Dwebdriver.remote.enableTracing=false"
        echo "Tracing is disabled"
      fi

      if [ -n "${SE_JAVA_HTTPCLIENT_VERSION}" ]; then
        SE_JAVA_OPTS="$SE_JAVA_OPTS -Dwebdriver.httpclient.version=${SE_JAVA_HTTPCLIENT_VERSION}"
      fi

      if [ -n "${SE_JAVA_OPTS_DEFAULT}" ]; then
        SE_JAVA_OPTS="${SE_JAVA_OPTS_DEFAULT} $SE_JAVA_OPTS"
      fi

      if [ -n "${JAVA_OPTS:-$SE_JAVA_OPTS}" ]; then
        echo "Using JAVA_OPTS: ${JAVA_OPTS:-$SE_JAVA_OPTS}"
      fi

      function handle_heap_dump() {
        /opt/bin/handle_heap_dump.sh /opt/selenium/logs
      }
      if [ "${SE_JAVA_HEAP_DUMP}" = "true" ]; then
        trap handle_heap_dump ERR SIGTERM SIGINT
      else
        trap handle_heap_dump ERR
      fi

      java ${JAVA_OPTS:-$SE_JAVA_OPTS} \
        -jar /opt/selenium/selenium-server.jar \
        ${EXTRA_LIBS} event-bus \
        --bind-host ${SE_BIND_HOST} \
        ${HOST_CONFIG} \
        ${PORT_CONFIG} \
        ${SE_OPTS}
---
# Source: selenium-grid/templates/event-bus-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: selenium-grid-selenium-event-bus
  namespace: selenium-grid-demo-ns
  labels: &event_bus_labels
    app: selenium-grid-selenium-event-bus
    app.kubernetes.io/name: selenium-grid-selenium-event-bus
    app.kubernetes.io/managed-by: helm
    app.kubernetes.io/instance: selenium-grid
    app.kubernetes.io/version: 4.29.0-20250303
    app.kubernetes.io/component: selenium-grid-4.29.0-20250303
    helm.sh/chart: selenium-grid-0.40.1
spec:
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: selenium-grid-selenium-event-bus
      app.kubernetes.io/instance: selenium-grid
  template:
    metadata:
      labels: *event_bus_labels
      annotations:
        checksum/logging-configmap: 340341e72ca26257adeb74743bec46d3a16f7a94cbe37ac71703d23bf1a4144e
        checksum/server-configmap: bd08b4ce23320321f4ef8e3903ff63e449101c0e0c8bdd76957ff7974b30d37e
        checksum/secrets: 8068a715b5d90514c7f111b339be79a8d9915ad9391c853e2ba8afa1d58a8df6
    spec:
      serviceAccountName: selenium-grid-selenium-serviceaccount
      serviceAccount: selenium-grid-selenium-serviceaccount
      initContainers:
      - name: app-insights-clone
        image: selenium/hub:4.29.0-20250303
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh","-c"]
        args: ["wget -P /tmp/jars https://github.com/microsoft/ApplicationInsights-Java/releases/download/3.7.1/applicationinsights-agent-3.7.1.jar"]
        volumeMounts:
        - mountPath: /tmp/jars
          name: appinsights-jar
      containers:
        - name: selenium-grid-selenium-event-bus
          image: selenium/event-bus:4.29.0-20250303
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5557
              protocol: TCP
            - containerPort: 4442
              protocol: TCP
            - containerPort: 4443
              protocol: TCP
          env:
            - name: KUBERNETES_NODE_HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: APPLICATIONINSIGHTS_ROLE_NAME
              value: 'selenium-grid-selenium-event-bus'
            - name: SE_OTEL_SERVICE_NAME
              value: 'selenium-grid-selenium-event-bus'
            - name: SE_EVENT_BUS_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: SE_EVENT_BUS_PORT
              value: "5557"
            - name: SE_LOG_LEVEL
              value: "FINEST"
            - name: APPLICATIONINSIGHTS_CONNECTION_STRING
              value: ""
            - name: JAVA_TOOL_OPTIONS
              value: "-javaagent:/tmp/jars/applicationinsights-agent-3.7.1.jar"
          envFrom:
            - configMapRef:
                name: selenium-grid-selenium-logging-config
            - configMapRef:
                name: selenium-grid-selenium-server-config
            - secretRef:
                name: selenium-grid-selenium-secrets
            - secretRef:
                name: selenium-grid-selenium-basic-auth-secrets
          volumeMounts:
            - mountPath: /opt/bin/start-selenium-grid-eventbus.sh
              name: selenium-grid-selenium-event-bus-config
              subPath: start-selenium-grid-eventbus.sh
            - mountPath: /tmp/jars
              name: appinsights-jar
          resources:
            limits:
              cpu: "1"
              memory: 2Gi
            requests:
              cpu: "0.5"
              memory: 512Mi
      volumes:
        - name: "selenium-grid-selenium-event-bus-config"
          configMap:
            name: selenium-grid-selenium-event-bus-config
            defaultMode: 493
        - emptyDir: {}
          name: appinsights-jar

[quickjp2]

Can you look into this? https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/azuremonitorexporter/README.md

So this went over my head the first few times; apologies for that. Combined with the message right before this quoted one, I'm assuming that I can update the SE_OTEL_TRACES_EXPORTER var to use the azuremonitor exporter. I'm looking through documentation to confirm how I would do that.

[quickjp2]

Update:
Looks like the azuremonitorexporter document link is for if I were developing a custom collector, which I'm not in this case.

so, currently we are using SE_OTEL_TRACES_EXPORTER="otlp" (Open Telemetry) it isn't specific to Jaeger. Does Azure Application Insights support an endpoint, that Grid can export traces to, the endpoint specify via SE_OTEL_EXPORTER_ENDPOINT

From what I've read in Microsoft's documentation, it can accept otlp formatted traces. It does not call out a specific endpoint to use though.

I did attempt to add the jar to the classpath, but that presented different errors. I'm hesitant to put more time into that route of troubleshooting because that isn't needed for every other Grid component to work.

Since I'm using K8s, I figured this would be the best first stop for asking my question, but if there's a more fitting location to ask, please let me know! 😄

And thank you for all your help and ideas so far @VietND96!