Document how to use OTLP exporters with self-signed certificates

[xue20xi]

Description

The goal is to use the OTLP Log gRPC Exporter in golang application to send application logs to the opentelemetry collector with TLS enabled on the server side.

Environment

• OS: WSL(Ubuntu)
• Architecture: x86_64
• Go Version: go1.23.8
• opentelemetry-go version: v1.35.0

Steps To Reproduce

1. Export the CA certificate of the opentelemetry collector
   export OTEL_EXPORTER_OTLP_CERTIFICATE=xxx
2. For the opentelemetry collector endpoint, using the default https://localhost:4317
3. Follow the example in the README, add the last 5 lines to emit the log

package main

import (
	"context"

	"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc"
	otellog "go.opentelemetry.io/otel/log"
	"go.opentelemetry.io/otel/log/global"
	"go.opentelemetry.io/otel/log/logtest"
	"go.opentelemetry.io/otel/sdk/log"
)

func main() {
	ctx := context.Background()
	exp, err := otlploggrpc.New(ctx)
	if err != nil {
		panic(err)
	}

	processor := log.NewBatchProcessor(exp)
	provider := log.NewLoggerProvider(log.WithProcessor(processor))
	defer func() {
		if err := provider.Shutdown(ctx); err != nil {
			panic(err)
		}
	}()

	global.SetLoggerProvider(provider)

	// From here, the provider can be used by instrumentation to collect
	// telemetry.
	mylogger := provider.Logger("mylogger")
	rf := logtest.RecordFactory{}
	rf.Body = otellog.StringValue("hello")
	r1 := rf.NewRecord()
	mylogger.Emit(ctx, r1)
}

4. Below error popup

panic: context deadline exceeded: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"

goroutine 1 [running]:

5. It seems the OTEL_EXPORTER_OTLP_CERTIFICATE variable is not working as described in the README.

OTEL_EXPORTER_OTLP_CERTIFICATE, OTEL_EXPORTER_OTLP_LOGS_CERTIFICATE (default: none) - the filepath to the trusted certificate to use when verifying a server's TLS credentials. 

6. I have tried export the SSL_CERT_FILE variable to the same value as OTEL_EXPORTER_OTLP_CERTIFICATE and the same code works fine.

Expected behavior

Expect the OTLP Log gRPC Exporter can work when specify the environment variable OTEL_EXPORTER_OTLP_CERTIFICATE with no errors like specifying the SSL_CERT_FILE environment variable

[xue20xi]

The call chain is as below:

otlploggrpc.New(ctx) -> otlploggrpc.newClient(cfg config) -> otlploggrpc.newGRPCDialOptions(cfg config)

Inside the newGRPCDialOptions(cfg config) function
As the cfg.gRPCCredentials.Value is nil, and we're using the telemetry collector with TLS enabled, so the default else logic is used.

func newGRPCDialOptions(cfg config) []grpc.DialOption {
        ...
	// Prioritize GRPCCredentials over Insecure (passing both is an error).
	if cfg.gRPCCredentials.Value != nil {
		dialOpts = append(dialOpts, grpc.WithTransportCredentials(cfg.gRPCCredentials.Value))
	} else if cfg.insecure.Value {
		dialOpts = append(dialOpts, grpc.WithTransportCredentials(insecure.NewCredentials()))
	} else {
		// Default to using the host's root CA.
		dialOpts = append(dialOpts, grpc.WithTransportCredentials(
			credentials.NewTLS(nil),
		))
	}
       ...

opentelemetry-go/exporters/otlp/otlplog/otlploggrpc/client.go

Line 94 in 1f9051c

dialOpts = append(dialOpts, grpc.WithTransportCredentials(

[xue20xi]

The proposed solution is to generate the gRPCCredentials information in otlploggrpc.newConfig() method when populate the config values.

opentelemetry-go/exporters/otlp/otlplog/otlploggrpc/config.go

Line 122 in 1f9051c

c.headers = c.headers.Resolve(

[pellared]

It seems the OTEL_EXPORTER_OTLP_CERTIFICATE variable is not working as described in the README.

From https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc#pkg-overview:

OTEL_EXPORTER_OTLP_CERTIFICATE, OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE (default: none) - the filepath to the trusted certificate to use when verifying a server's TLS credentials.

The certificate is self-signed therefore it is not trusted.

This follows the OTel Specification:

• https://github.com/open-telemetry/opentelemetry-specification/blob/2b9ef9613f3bc4b135f0c1393be641f3a6245643/specification/protocol/exporter.md?plain=1#L41

I think you simply need to use https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc#WithTLSCredentials and use it together with https://pkg.go.dev/google.golang.org/grpc/credentials#NewTLS.

After quick Googling you can take a look at e.g. https://medium.com/@mertkimyonsen/securing-grpc-connection-with-ssl-tls-certificate-using-go-db3852fe89dd

[xue20xi]

Hi @pellared , thanks very much for your reply.

From the document, we can use OTEL_EXPORTER_OTLP_CERTIFICATE to specify the trusted CA certificate.

OTEL_EXPORTER_OTLP_CERTIFICATE, OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE (default: none) - the filepath to the trusted certificate to use when verifying a server's TLS credentials.

If it should not work by specifying this environment variable only, maybe we can consider to change the README and say that this should work when together with this method https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc#WithTLSCredentials

And by the way, when we specify SSL_CERT_FILE using the same CA file, it works fine. To keep the same behavior, I think we'd better let this OTEL_EXPORTER_OTLP_CERTIFICATE environment variable works in the same way as SSL_CERT_FILE instead of have to work together with WithTLSCredentials() method

[pellared]

If it should not work by specifying this environment variable only

The documentation is correct. The a self-signed certificate is (by default) not trusted.

when we specify SSL_CERT_FILE using the same CA file, it works fine.

It is worth mentioning that SSL_CERT_FILE is not working on all operating systems. See https://pkg.go.dev/crypto/x509#SystemCertPool:

On Unix systems other than macOS the environment variables SSL_CERT_FILE and SSL_CERT_DIR can be used to override the system default locations for the SSL certificate file and SSL certificate files directory, respectively. The latter can be a colon-separated list.

I am OK improving the documentation to describe how to setup OTLP that use self-signed certificates.
Maybe the we could add testable examples for WithTLSCredentials and WithTLSClientConfig to show how it can be done?
SSL_CERT_FILE and SSL_CERT_DIR for Unix (but not macOS) systems could also be documented or adding some reference to https://pkg.go.dev/crypto/x509#SystemCertPool.

I change the issue title to reflect this.
@xue20xi, do you want to work on it?

[xue20xi]

@pellared , yes, I will update the doc with one example to address this specific scenario.