/querier.v1.QuerierService/Diff api failing over http

[ndatar18]

Describe the bug

Running into 404 errors when trying to access the profile diff api over HTTP https://grafana.com/docs/pyroscope/latest/reference-server-api/#querierv1querierservicediff

Has the http endpoint changed from the docs?

All of the other endpoints on these docs work.

Expected behavior

Expected to receive returned data similar to other APIs on this page, instead got 404 error.

Environment

Fails even with just a simple curl command.

Additional Context

[aleks-p]

Hi @ndatar18 , thank you for reporting this issue.

I gave it a quick try and didn't get a 404. What I did was

docker run -p 4040:4040 grafana/pyroscope:1.17.0

and then a simplified curl, based on the one in the docs you linked

curl \
  -H "Content-Type: application/json" \
  -d '{
      "left": {
        "end": '$(date +%s)000',
        "labelSelector": "{}",
        "profileTypeID": "process_cpu:cpu:nanoseconds:cpu:nanoseconds",
        "start": '$(expr $(date +%s) - 3600 )000'
      },
      "right": {
        "end": '$(date +%s)000',
        "labelSelector": "{}",
        "profileTypeID": "process_cpu:cpu:nanoseconds:cpu:nanoseconds",
        "start": '$(expr $(date +%s) - 3600 )000'
      }
    }' \
  http://localhost:4040/querier.v1.QuerierService/Diff

This produced a result with data in my test.

Can you share some more details on how you are running Pyroscope, and the query you are testing with?

[ndatar18]

Hi @ndatar18 , thank you for reporting this issue.

I gave it a quick try and didn't get a 404. What I did was

docker run -p 4040:4040 grafana/pyroscope:1.17.0

and then a simplified curl, based on the one in the docs you linked

curl \
  -H "Content-Type: application/json" \
  -d '{
      "left": {
        "end": '$(date +%s)000',
        "labelSelector": "{}",
        "profileTypeID": "process_cpu:cpu:nanoseconds:cpu:nanoseconds",
        "start": '$(expr $(date +%s) - 3600 )000'
      },
      "right": {
        "end": '$(date +%s)000',
        "labelSelector": "{}",
        "profileTypeID": "process_cpu:cpu:nanoseconds:cpu:nanoseconds",
        "start": '$(expr $(date +%s) - 3600 )000'
      }
    }' \
  http://localhost:4040/querier.v1.QuerierService/Diff

This produced a result with data in my test.

Can you share some more details on how you are running Pyroscope, and the query you are testing with?

Thanks @aleks-p ! I am using Grafana Cloud managed pyroscope and trying to access via the datasource proxy. I was trying this command. It works when I use any of the other endpoints (LabelNames, ProfileTypes, SelectMergeStacktraces, etc.) but still running into 404s with the Diff endpoint.

curl \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
      "left": {
        "end": '$(expr $(date +%s) - 900 )000',
        "labelSelector": "{}",
        "profileTypeID": "wall:cpu:nanoseconds:wall:nanoseconds",
        "start": '$(expr $(date +%s) - 1800 )000'
      },
      "right": {
        "end": '$(date +%s)000',
        "labelSelector": "{}",
        "profileTypeID": "wall:cpu:nanoseconds:wall:nanoseconds",
        "start": '$(expr $(date +%s) - 900 )000'
      }
    }' \
  https://<grafana-url>/api/datasources/proxy/uid/<datasource-uid>/querier.v1.QuerierService/Diff

[aleks-p]

Thank you, that explains things. We don't seem to expose this particular endpoint in Grafana Cloud, probably because the frontend application there (Profiles Drilldown) uses a different endpoint (/pyroscope/render-diff).

You can use that endpoint for now, the equivalent of what you were doing would be something like this GET request:

curl \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  https://<grafana-url>/api/datasources/proxy/uid/<datasource-uid>/pyroscope/render-diff?leftQuery=wall%3Acpu%3Ananoseconds%3Awall%3Ananoseconds%7B%7D&leftFrom=now-15m&leftUntil=now&rightQuery=wall%3Acpu%3Ananoseconds%3Awall%3Ananoseconds%7B%7D&rightFrom=now-15m&rightUntil=now'

In the meantime I'll look into making /querier.v1.QuerierService/Diff available in Grafana Cloud Profiles.

[ndatar18]

I see, thanks for the help Aleksandar! And please let me know when you are able to expose /querier.v1.QuerierService/Diff in Grafana Cloud :)