gRPC with TLS and authentication - missing selected ALPN property

[martinmorch-rsyd]

Is your question related to a specific component?

victoriametrics/victoria-traces:v0.7.1
opentelemetry-collector-contrib:0.145.0

Describe the question in detail

First off, apologies in advance, I'm somewhat new to VictoriaTraces and Opentelemetry Collector.

When trying to set up Opentelemetry Collector exporters towards VictoriaTraces gRPC with TLS, the collector throws this error:

2026-02-18T06:28:28.844Z info internal/retry_sender.go:133 Exporting failed. Will retry the request after interval. {"resource": {"service.instance.id": "4720fee1-4b03-4e35-8529-695f5dea8093", "service.name": "otelcol-contrib", "service.version": "0.145.0"}, "otelcol.component.id": "otlp/vt_grpc_0", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property. If you upgraded from a grpc-go version earlier than 1.67, your TLS connections may have stopped working due to ALPN enforcement. For more details, see: https://github.com/grpc/grpc-go/issues/434\"", "interval": "7.111375977s"}

When I test the gRPC port on VT with:
echo QUIT | openssl s_client -connect 10.162.131.63:20192 -alpn h2

I get:
"No ALPN negotiated"

Through searching, the error indicates that the gRPC service collector is trying to communicate with, doesn't support gRPC HTTP/2, where the collector requires H2 ALPN support.

Is this correct with regards to VictoriaTraces, or am I doing something horribly incorrect?

Thanks in advance!

Key concepts docs

• VictoriaTraces key concepts - https://docs.victoriametrics.com/victoriatraces/keyconcepts/
• LogsQL - https://docs.victoriametrics.com/victorialogs/logsql/

[jiekun]

Hi. The gRPC endpoints in VictoriaTraces was built on native HTTP2 server directly, without using external packages like gRPC-Go. It may miss some features (that we dont want to include, because OTLP doesnt use them), but the TLS support is not the case.

I assume it's a bug and will investigate later.

And before we figure out something, it's recommended to use OTLP/HTTP instead of OTLP/gRPC.

[jiekun]

Could you provide the OTLP exporter config and the command-line flags of VictoriaTraces? I'd like to check how you config the TLS on both side first. Thanks!

[martinmorch-rsyd]

@jiekun Thanks for taking the time!

VictoriaTraces parameters:

args = [
  "-storageDataPath=/victoria-traces-data",
  "-httpAuth.username=${VT_HTTPAUTH_USERNAME}",
  "-httpAuth.password=${VT_HTTPAUTH_PASSWORD}",
  "-metrics.exposeMetadata=true",
  "-tls=true",
  "-tlsMinVersion=TLS13",
  "-tlsCertFile=${VT_TLS_CERT_FILE}",
  "-tlsKeyFile=${VT_TLS_KEY_FILE}",
  "-otlpGRPCListenAddr=0.0.0.0:4317",
  "-otlpGRPC.tls=true",
  "-otlpGRPC.tlsMinVersion=TLS13",
  "-otlpGRPC.tlsCertFile=${VT_TLS_CERT_FILE}",
  "-otlpGRPC.tlsKeyFile=${VT_TLS_KEY_FILE}",
  "-memory.allowedPercent=${local.victoria_traces.memory_allowed_percent}",
  "-retentionPeriod=${local.victoria_traces.retention_period}",
  "-http.shutdownDelay=${local.victoria_traces.http_shutdown_delay}",
]

Opentelemetry Collector config:

extensions:
  file_storage:
    create_directory: true
    directory: /otelcol/storage
  basicauth/vt:
    client_auth:
      username: "<user>"
      password: "<pass>"
  headers_setter:
    additional_auth: basicauth/vt
    headers:
      - action: insert
        key: "AccountID"
        from_context: "AccountID"
        default_value: "0"
      - action: insert
        key: "ProjectID"
        from_context: "ProjectID"
        default_value: "0"

receivers:
  otlp:
    protocols:
      http:
        endpoint: "0.0.0.0:4318"
        include_metadata: true
      grpc:
        endpoint: "0.0.0.0:4317"
        include_metadata: true

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  batch:
    timeout: 5s
    send_batch_size: 8192
    metadata_keys:
      - AccountID
      - ProjectID

exporters:
  otlp_grpc/vt_grpc_0:
    endpoint: "<ip>:20192"
    tls:
      ca_file: "/secrets/victoria-traces-tls-ca.crt"
      server_name_override: staging-traces.rsyd.net
    auth:
      authenticator: headers_setter
    sending_queue:
      enabled: true
      storage: file_storage
      sizer: bytes
      queue_size: 1073741824
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 0
  otlp_grpc/vt_grpc_1:
    endpoint: "<ip>:26511"
    tls:
      ca_file: "/secrets/victoria-traces-tls-ca.crt"
      server_name_override: staging-traces.rsyd.net
    auth:
      authenticator: headers_setter
    sending_queue:
      enabled: true
      storage: file_storage
      sizer: bytes
      queue_size: 1073741824
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 0

service:
  extensions: [file_storage, basicauth/vt, headers_setter]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters:
        - otlp_grpc/vt_grpc_0
        - otlp_grpc/vt_grpc_1

I'm actually not sure if basic auth is even "a thing" when doing gRPC, but alas, above is the relevant configs.