After setting up the Prometheus data source, using /resources/api/v1/label/service/values?match[]=service_traffic does not retrieve the service information.

[sakeri-l]

I set the selector of the telemetry node to Prometheus in SkyWalking, and also added a Prometheus node in Grafana.

When I use the template at https://demo.skywalking.apache.org:3000, I can't get the service information. The request to /resources/api/v1/label/service/values?match[]=service_traffic returns an empty result. I want to know what is wrong with my configuration and what I should check. Thank you very much.

[wu-sheng]

What do you mean? I think you mixed two things. Prometheus for so11y is to collect metrics of OAP.
The query for service traffic is about getting OAP observed app services.
What do you actually expect?

[sakeri-l]

I want to use Grafana to display the application metrics collected by SkyWalking on a dashboard, similar to https://demo.skywalking.apache.org:3000, but I am unable to get the service information in Grafana.

I followed the General Service panel configuration at https://demo.skywalking.apache.org:3000/ but couldn't retrieve my service information.

I am not sure if I understand correctly. I followed the configuration at https://skywalking.apache.org/docs/main/latest/en/setup/backend/ui-grafana/. If my configuration is wrong, please let me know. Thank you very much.

1、my skywalking application.yml
telemetry:
selector: ${SW_TELEMETRY:prometheus}
none:
prometheus:
host: ${SW_TELEMETRY_PROMETHEUS_HOST:0.0.0.0}
port: ${SW_TELEMETRY_PROMETHEUS_PORT:9083}
sslEnabled: ${SW_TELEMETRY_PROMETHEUS_SSL_ENABLED:false}
sslKeyPath: ${SW_TELEMETRY_PROMETHEUS_SSL_KEY_PATH:""}
sslCertChainPath: ${SW_TELEMETRY_PROMETHEUS_SSL_CERT_CHAIN_PATH:""}

2、Then I added the SkyWalking data source to my Prometheus. This is my prometheus.yml
scrape_configs:
- job_name: 'skywalking-local'
metrics_path: '/metrics'
static_configs:
- targets: ['42.194.190.145:9083']

3、I added Prometheus and SkyWalking data sources in Grafana.

4、I imported the template from https://demo.skywalking.apache.org:3000, but found that I couldn't get the service. I can collect the metrics of my service on SkyWalking, but I'm not sure where the problem lies. Thank you very much for your guidance.

[wu-sheng]

Where do you learn this? I am totally confused about what you are doing.
OAP itself would pretend itself as a Prometheus server if you want to query via PromQL. You don't need Prometheus and OTEL at all.

telemetry module is feeding oap's performance, not oap collected metrics to Prometheus. It has nothing related to that. As well as your OTEL collector setup.

This doc(https://skywalking.apache.org/docs/main/latest/en/setup/backend/ui-grafana/) says nothing about your step 1 and 2. I don't know where you get that.

[wu-sheng]

You should point Prometheus dashboard to OAP's promQL point.

#This module is for PromQL API.
promql:
  selector: ${SW_PROMQL:default}
  default:
    # For HTTP server
    restHost: ${SW_PROMQL_REST_HOST:0.0.0.0}
    restPort: ${SW_PROMQL_REST_PORT:9090}
    restContextPath: ${SW_PROMQL_REST_CONTEXT_PATH:/}
    restIdleTimeOut: ${SW_PROMQL_REST_IDLE_TIMEOUT:30000}
    restAcceptQueueSize: ${SW_PROMQL_REST_QUEUE_SIZE:0}
    # The below config is for the API buildInfo, set the value to mock the build info.
    buildInfoVersion: ${SW_PROMQL_BUILD_INFO_VERSION:"2.45.0"}
    buildInfoRevision: ${SW_PROMQL_BUILD_INFO_REVISION:""}
    buildInfoBranch: ${SW_PROMQL_BUILD_INFO_BRANCH:""}
    buildInfoBuildUser: ${SW_PROMQL_BUILD_INFO_BUILD_USER:""}
    buildInfoBuildDate: ${SW_PROMQL_BUILD_INFO_BUILD_DATE:""}
    buildInfoGoVersion: ${SW_PROMQL_BUILD_INFO_GO_VERSION:""}

[sakeri-l]

Thank you very much for your guidance. It was my own misunderstanding. When I accessed PromQL, the page returned Status: 404. I thought the default PromQL service was unavailable. It was my own mistake. Thank you very much for your help.