Error with Alloy forwarding: pushing IngestInput-pprof failed invalid_argument: profile with labels ... has duplicate label name: 'service_name'

[cbismuth]

From Go code, when pushing directly to Pyroscope, everything is fine, when pushing to Alloy for Pyroscope forwarding, we have this error below:

Logs:

pyroscope   | ts=2025-03-14T14:33:26.586451063Z caller=ingest_handler.go:56 level=error msg="pyroscope ingest" err="pushing IngestInput-pprof failed invalid_argument: profile with labels '{__delta__=\"false\", __name__=\"process_cpu\", pyroscope_spy=\"gospy\", service_name=\"etf1.dice\", service_name=\"etf1.dice\"}' has duplicate label name: 'service_name'" orgID=anonymous
alloy       | ts=2025-03-14T14:33:26.587831766Z level=error msg="Failed to append profile" component_path=/ component_id=pyroscope.receive_http.default appendable=0 err="pyroscope write error: status 422"

Go code:

_, err := pyroscope.Start(pyroscope.Config{
	ApplicationName: ApplicationName,
	ServerAddress:   os.Getenv("PYROSCOPE_ENDPOINT"),
	ProfileTypes: []pyroscope.ProfileType{
		pyroscope.ProfileCPU,
		pyroscope.ProfileInuseObjects,
		pyroscope.ProfileAllocObjects,
		pyroscope.ProfileInuseSpace,
		pyroscope.ProfileAllocSpace,
		pyroscope.ProfileGoroutines,
		pyroscope.ProfileMutexCount,
		pyroscope.ProfileMutexDuration,
		pyroscope.ProfileBlockCount,
		pyroscope.ProfileBlockDuration,
	},
	Logger: pyroscope.StandardLogger,
})
if err != nil {
	fmt.Println(fmt.Errorf("failed to setup pyroscope: %w", err).Error())
	return
}

config.alloy:

pyroscope.receive_http "default" {
   http {
       listen_address = "0.0.0.0"
       listen_port = 9999
   }
   forward_to = [pyroscope.write.backend.receiver]
}

pyroscope.write "backend" {
    endpoint {
        url = "http://pyroscope:4100"
    }
}

[korniltsev]

Likely fixed in grafana/alloy#2918

[kppullin]

We've run into the same issue with other pyroscope SDKs.

Building alloy from the latest code in main, including the referenced possible fix grafana/alloy#2918, does not resolve the issue.

It looks to stem from grafana/alloy#2675 which added the service_name label. If the service_name label is removed prior to inclusion in the query string in internal/component/pyroscope/write/write.go, everything works again. Best I can tell, service_name is not duplicated elsewhere in the query string, so my guess is there's something happening on the server end between the payload and query string.

[korniltsev]

This should have been fixed in grafana/alloy#2918

Any chance you can double check with a devel image in your test environment?
https://hub.docker.com/layers/grafana/alloy-dev/v1.8.0-devel-8625747/images/sha256-671fc64299db64926af8f9b621ad5ae3936b402c4220a6c77859979b427ac2fa

[simonswine]

Sorry about this, the bug is in fact in Pyroscope, we will have a fix released soon see the details in my message here: grafana/pyroscope#4017 (comment)