how to enable Self-Observability in version-9.3.0?

[xiongshiyan]

I can enable Self-Observability through doing some configs like below in config/application.yml when I use Skywalking version 8

telemetry:
  selector: ${SW_TELEMETRY:prometheus}
  none:
  prometheus:
    host: ${SW_TELEMETRY_PROMETHEUS_HOST:0.0.0.0}
    port: ${SW_TELEMETRY_PROMETHEUS_PORT:1234}

prometheus-fetcher:
  selector: ${SW_PROMETHEUS_FETCHER:default}
  default:
    active: ${SW_PROMETHEUS_FETCHER_ACTIVE:true}

but I can't find the option of 'prometheus-fetcher' in config/application.yml when I use Skywalking version 9, how can I enable Self-Observability ?

thanks !!

[wu-sheng]

Where does the doc say anything about prometheus-fetcher? Don't read a 3rd party blog and ask for the community.
Every version has its document. Read the official docs, and post the link of the doc if you have any question about this.

[hnxydq]

As the doc v9.3.0 describe below.
For the step 2 "Set up OpenTelemetry to scrape the metrics from OAP telemetry", i found the test case( https://github.com/apache/skywalking/blob/7ad04062b20ecf2bdb0afc4ba8c6ce353e5ae4d8/test/e2e-v2/cases/so11y/otel-collector-config.yaml ), but which directory should we put the yaml? any demo for more details ?

[wu-sheng]

Showcase demos most key features, https://skywalking.apache.org/docs/skywalking-showcase/next/readme/

[hnxydq]

According to the source code, I found the way to enable Self-Observability in version-9.3.0 when we just use in local windows for test as follows:
1.set up prometheus telemetry as describe in doc v9.3.0

telemetry:
  selector: ${SW_TELEMETRY:prometheus}
  none:
  prometheus:
    host: ${SW_TELEMETRY_PROMETHEUS_HOST:0.0.0.0}
    port: ${SW_TELEMETRY_PROMETHEUS_PORT:1234}
    sslEnabled: ${SW_TELEMETRY_PROMETHEUS_SSL_ENABLED:false}
    sslKeyPath: ${SW_TELEMETRY_PROMETHEUS_SSL_KEY_PATH:""}
    sslCertChainPath: ${SW_TELEMETRY_PROMETHEUS_SSL_CERT_CHAIN_PATH:""}

the default host and port can be found in file:

public class PrometheusConfig extends ModuleConfig {
    private String host = "0.0.0.0";
    private int port = 1234;
    private boolean sslEnabled = false;
    private String sslKeyPath;
    private String sslCertChainPath;
}

2.download opentelemetry-collector according to your operation system:
opentelemetry-collector-releases

3.run the collector with yaml config file (otel-collector-config.yaml):

receivers:
  prometheus:
    config:
     scrape_configs:
       - job_name: 'skywalking-so11y'
         scrape_interval: 5s
         static_configs:
           - targets: ['localhost:1234']
             labels:
               host_name: http://localhost:1234
               service: oap-server
processors:
  batch:

exporters:
  ####################################################################################
  # If you want to use otlp exporter please ensure that your OAP version is >= 9.2.0 #
  # or you can use oc exporter, The config format should be:                         #
  # opencensus:                                                                      #
  #   endpoint: "oap:11800"                                                          #
  #   tls:                                                                           #
  #      insecure: true                                                              #
  ####################################################################################
  otlp:
    endpoint: localhost:11800
    tls:
      insecure: true
  logging:
    logLevel: debug

service:
  pipelines:
    metrics:
      receivers:
      - prometheus
      processors:
      - batch
      exporters:
      #########################################
      # oc exporter config format:            #
      # - opencensus                          #
      #########################################
      - otlp
      - logging

run with command:

otelcol-contrib --config=file:D:/deploy/skywalking/otelcol-contrib_0.69.0_windows_amd64/otel-collector-config.yaml

4. restart the oapService.bat then refresh the UI.

[JeffryFeng]

For the step 2 "Set up OpenTelemetry to scrape the metrics from OAP telemetry"
---- there is no explanation : how to setup ???
---- who can give more infos.

[JeffryFeng]

Since I deployed skywalking as POD in K8S, cannot see anything in GUI .
who can give steps : how to enable self-observability in version-9.3.0
Great thanks

[XiamenGeek]

Official showcase is deployed on K8S，I have done on virtual machine. You can refer this steps:

1. Enable prometheus telemetry on your OAP servers. Edit application.yml file, and restart OAP:

telemetry:
selector: ${SW_TELEMETRY:prometheus}
none:
prometheus:
host: ${SW_TELEMETRY_PROMETHEUS_HOST:0.0.0.0}
port: ${SW_TELEMETRY_PROMETHEUS_PORT:1234}
sslEnabled: ${SW_TELEMETRY_PROMETHEUS_SSL_ENABLED:false}
sslKeyPath: ${SW_TELEMETRY_PROMETHEUS_SSL_KEY_PATH:""}
sslCertChainPath: ${SW_TELEMETRY_PROMETHEUS_SSL_CERT_CHAIN_PATH:""}

2. Download otel collector package
   https://github.com/open-telemetry/opentelemetry-collector-releases
3. untar the package.
4. Copy the configration from here and save it to file otel-collector-config.yaml, replace it with your OAP address .
   https://github.com/apache/skywalking/blob/7ad04062b20ecf2bdb0afc4ba8c6ce353e5ae4d8/test/e2e-v2/cases/so11y/otel-collector-config.yaml
5. run it
   ./otelcol --config=conf/otel-collector-config.yaml

[JeffryFeng]

thanks for your answer. Now I can see the data in Self-Observability GUI , based on your steps.
But there are 2 problems:

1. in file otel-collector-config.yaml, expoters--->logging ---->loglevel is deprecated and not used, Now use verbosity .
   If still use loglevel，there will be error when run command ( ./otelcol --config=otel-collector-config.yaml ).
   So in my ENV, I changed to verbosity: normal.
   reference:
   https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/loggingexporter/README.md

2. if skywalking Pod restart, POD ip will be changed,too.
   then the IP in otel-collector-config.yaml will be useless.

[Herman19890]

i flow the steps ;but when i run ：otelcol-contrib --config=/skywalking/otel-collector-config.yaml；it becomes some worn log shows :

    {"kind": "exporter", "data_type": "metrics", "name": "logging"}

2023-08-27T11:32:53.176+0800 warn internal/transaction.go:111 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1693107173174, "target_labels": "{name="up", host_name="centos7u9", instance="oap:1534", job="skywalking-so11y", service="oap-server"}"}
2023-08-27T11:32:53.273+0800 info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "resource metrics": 1, "metrics": 5, "data points": 5}

my env is blow:

apache-skywalking-apm-9.5.0
mysql Ver 8.0.33
otelcol-contrib_0.83.0_linux_amd64.rpm