Update setting up Prometheus docs

[DevSusu]

psschwei: originally opened in serving and moved here. updates needed for docs can be found in this comment: #5217 (comment)

/area monitoring

What version of Knative?

1.7.1 (installed with https://github.com/knative/serving/releases/download/knative-v1.7.1/serving-core.yaml)

Expected Behavior

• install knative-serving
• port-fowarded one of them to debug on prometheus metrics
• pods report metrics described in https://knative.dev/docs/serving/observability/metrics/serving-metrics/

Actual Behavior

kubectl port-forward service/activator-service -n knative-serving 9090:9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
Handling connection for 9090
E0914 16:42:51.809345 3007207 portforward.go:400] an error occurred forwarding 9090 -> 9090: error forwarding port 9090 to pod 4f48d8d6708f2f6cc464957b6718d2ec89088951c6430acfb0fb04857e595a94, uid : exit status 1: 2022/09/14 07:42:51 socat[13441] E connect(5, AF=2 127.0.0.1:9090, 16): Connection refused

I had to specify the metrics.backend-destination and metrics.request-metrics-backend-destination to get this working.

Steps to Reproduce the Problem

• follow instructions on https://knative.dev/docs/install/yaml-install/serving/install-serving-with-yaml/#prerequisites
• port-forward one of the components with 9090 port exposed
• curl localhost:9090 (same error with curl localhost:9090/metrics)

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-observability
  namespace: knative-serving
  labels:
    app.kubernetes.io/name: knative-serving
    app.kubernetes.io/component: observability
    app.kubernetes.io/version: "1.7.1"
  annotations:
    knative.dev/example-checksum: "fed4756e"
data: {}

[DevSusu]

related issue: knative/eventing#6509

[DevSusu]

resolved by editing configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-observability
  namespace: knative-serving
  labels:
    app.kubernetes.io/name: knative-serving
    app.kubernetes.io/component: observability
    app.kubernetes.io/version: "1.7.1"
  annotations:
    knative.dev/example-checksum: "fed4756e"
data:
  metrics.backend-destination: prometheus
  metrics.request-metrics-backend-destination: prometheus

[DevSusu]

I think we need to add below block to setting up prometheus part.

https://knative.dev/docs/serving/observability/metrics/collecting-metrics/#setting-up-prometheus

kubectl patch --namespace knative-serving configmap/config-observability \
  --type merge \
  --patch '{"data":{"metrics.backend-destination":"promethuus","metrics.request-metrics-backend-destination":"prometheus"}}'

kubectl patch --namespace knative-serving configmap/config-observability \
  --type merge \
  --patch '{"data":{"metrics.backend-destination":"promethuus","metrics.request-metrics-backend-destination":"prometheus"}}'

[knative-prow]

@DevSusu: The label(s) area/monitoring cannot be applied, because the repository doesn't have them.

In response to this:

/area monitoring

What version of Knative?

1.7.1 (installed with https://github.com/knative/serving/releases/download/knative-v1.7.1/serving-core.yaml)

Expected Behavior

• install knative-serving
• port-fowarded one of them to debug on prometheus metrics
• pods report metrics described in https://knative.dev/docs/serving/observability/metrics/serving-metrics/

Actual Behavior

kubectl port-forward service/activator-service -n knative-serving 9090:9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
Handling connection for 9090
E0914 16:42:51.809345 3007207 portforward.go:400] an error occurred forwarding 9090 -> 9090: error forwarding port 9090 to pod 4f48d8d6708f2f6cc464957b6718d2ec89088951c6430acfb0fb04857e595a94, uid : exit status 1: 2022/09/14 07:42:51 socat[13441] E connect(5, AF=2 127.0.0.1:9090, 16): Connection refused

I had to specify the metrics.backend-destination and metrics.request-metrics-backend-destination to get this working.

Steps to Reproduce the Problem

• follow instructions on https://knative.dev/docs/install/yaml-install/serving/install-serving-with-yaml/#prerequisites
• port-forward one of the components with 9090 port exposed
• curl localhost:9090 (same error with curl localhost:9090/metrics)

apiVersion: v1
kind: ConfigMap
metadata:
 name: config-observability
 namespace: knative-serving
 labels:
   app.kubernetes.io/name: knative-serving
   app.kubernetes.io/component: observability
   app.kubernetes.io/version: "1.7.1"
 annotations:
   knative.dev/example-checksum: "fed4756e"
data: {}

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.