Distributor connects to ingester via plan http despite configured TLS

[bobrik]

Describe the bug

You can see the code here:

pyroscope/pkg/clientpool/ingester_client_pool.go

Line 65 in 39e0e7c

IngesterServiceClient: ingesterv1connect.NewIngesterServiceClient(httpClient, "http://"+inst.Addr, f.options...),

It is using plain HTTP despite me having TLS configured throughout.

In the logs:

Oct 31 03:59:30 143dm6 pyroscope[4186086]: http: TLS handshake error from [2400:...2844]:48136: tls: first record does not look like a TLS handshake

Oct 31 03:59:30 143dm6 pyroscope[4186086]: ts=2025-10-31T03:59:30.77202765Z caller=distributor.go:396 component=distributor tenant=anonymous user=anonymous level=warn msg="profile rejected" service_name=grafana.alloy.ebpf profile_type=process_cpu matched_usage_groups=[] detected_language=go profile_time=2025-10-31T03:59:27.807Z ingestion_delay=2.963s decompressed_size=21244 sample_count=149 err="unavailable: write tcp [2400:...2844]:48136->[2400:...2844]:4040: write: connection reset by peer"

Oct 31 03:59:30 143dm6 pyroscope[4186086]: ts=2025-10-31T03:59:30.207856516Z caller=distributor.go:396 component=distributor tenant=anonymous user=anonymous level=warn msg="profile rejected" service_name=grafana.alloy.ebpf profile_type=process_cpu matched_usage_groups=[] detected_language=unknown profile_time=2025-10-31T03:59:27.812Z ingestion_delay=2.394s decompressed_size=420 sample_count=1 err="unavailable: unexpected EOF"

I can see plaintext queries in tcpdump:

To Reproduce

Steps to reproduce the behavior:

1. Configure TLS for all components in a single node deployment.
2. Send some data to the node via HTTPS.
3. Observe complaints about ingestion.

Expected behavior

I expect TLS to work.

Environment

• Infrastructure: bare metal, single node
• Deployment tool: manual

[saurabh-m-w]

Hi, any update or workaround on this issue? We are also trying to use pyroscope with TLS enabled