feat: add Loki-based admin usage dashboard with Perses datasource by tgitelman · Pull Request #995 · opendatahub-io/models-as-a-service

tgitelman · 2026-06-14T14:37:42Z

Summary
Add admin usage dashboard (usage-admin-loki-dashboard) that queries Loki structured logs via LogQL, showing token consumption, request counts, rate limiting, success rate, and active users
Add Loki datasource infrastructure: PersesDatasource CR, TLS CA ConfigMap, RBAC ClusterRoleBinding, and SA token Secret
Wire into existing observability/dashboards kustomization alongside the Prometheus-based dashboards
Details
The dashboard provides an admin-level view of MaaS API usage by querying structured metadata from Envoy access logs stored in Loki (via OTel Collector). It complements the existing Prometheus-based dashboards with per-request granularity including user_id, tokens_total, response_code, and model.

Dashboard panels:

Total Tokens, Total Requests, Total Rate Limited, Success Rate, Active Users (stat panels)
Token consumption over time (time series chart with "View by" model/subscription selector)
Usage breakdown table (model × subscription with token and request counts)
Loki infra (deployment/components/observability/loki/):

loki-datasource.yaml — PersesDatasource pointing to LokiStack gateway with SA token + TLS
loki-ca.yaml — ConfigMap with service CA injection for TLS
loki-rbac.yaml — ClusterRoleBinding granting perses-sa the cluster-logging-application-view role
loki-secret.yaml — SA token secret for LokiStack gateway authentication
Prerequisites
LokiStack deployed and receiving structured logs from the Envoy OTel access logger
Cluster Observability Operator (COO) with Perses enabled
OTel Collector forwarding Envoy access logs to Loki via OTLP
Test plan

Deploy with kubectl apply -k deployment/components/observability/observability/dashboards/

Verify dashboard appears in OpenShift Console under Observe > Dashboards

Confirm all panels render data (Total Tokens, Requests, Rate Limited, Success Rate, Active Users)

Verify "View by" selector switches token chart grouping between model and subscription

Confirm usage breakdown table shows model × subscription detail

openshift-ci · 2026-06-14T14:37:48Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tgitelman
Once this PR has been reviewed and has the lgtm label, please assign jland-redhat for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

coderabbitai · 2026-06-14T14:37:49Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 2a469a53-501e-4909-a83c-1c42ce6e31a4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

ahadas · 2026-06-14T14:50:34Z

+    app.kubernetes.io/managed-by: maas-observability
+    app.kubernetes.io/component: perses
+  annotations:
+    service.beta.openshift.io/inject-cabundle: "true"


did you check if we can enable unencrypted communication from within the cluster? I saw we have unencrypted communication with the Prometheus instance of ODH - I think it makes sense here as well, if possible

LokiStack gateway in openshift-logging mode enforces HTTPS exclusively (PR #6288). There's no HTTP port to connect to. This matches the Prometheus/Thanos Querier pattern — the ODH operator's own cluster-prometheus-datasource also uses HTTPS + service CA + bearer token to Thanos Querier (same pattern we follow here for Loki). The CA ConfigMap + TLS config are required.

This matches the Prometheus/Thanos Querier pattern — the ODH operator's own cluster-prometheus-datasource also uses HTTPS + service CA + bearer token to Thanos Querier (same pattern we follow here for Loki)

the Prometheus instance I referred to is not the cluster-prometheus-datasource but the one that is specific to ODH

ahadas · 2026-06-14T14:51:37Z

+        proxy:
+          kind: HTTPProxy
+          spec:
+            url: https://lokistack-sample-gateway-http.logging4.svc.cluster.local:8080/api/logs/v1/application


this should not rely on the namespace being logging4

ahadas · 2026-06-14T14:52:20Z

@@ -0,0 +1,18 @@
+# Grant Perses SA read access to application logs via LokiStack gateway.


this probably needs to move to the opendatahub-operator instead

ahadas · 2026-06-14T14:52:54Z

+    app.kubernetes.io/managed-by: maas-observability
+    app.kubernetes.io/component: perses
+  annotations:
+    kubernetes.io/service-account.name: loki-query-proxy


I guess we don't need this here, right?

ahadas · 2026-06-14T14:53:24Z

@@ -0,0 +1,36 @@
+# Admin Loki datasource — direct to LokiStack, no user filtering.


the datasource needs to be placed in observability/dashboards folder, next to the dashboard

ahadas · 2026-06-14T14:54:05Z

+          kind: StaticListVariable
+          spec:
+            values:
+              - "dev-subscription-a"


version 1.5 of cluster observability operator is expected later this week, we need to test their fix for dynamic values in those lists

…io#2 — move datasource, fix LokiStack URL - Move loki-datasource.yaml to observability/dashboards/ next to its dashboard (follows Prometheus datasource co-location convention) - Change hardcoded LokiStack URL from lokistack-sample/logging4 to logging-loki/openshift-logging (standard namespace) - Update loki/ and dashboards/ kustomization references accordingly

Add admin usage dashboard querying Loki structured logs via LogQL, with supporting Loki datasource infrastructure (CA, RBAC, SA token secret).

rhods-ci-bot · 2026-06-14T16:53:04Z

@tgitelman: The following test has Failed:

OCI Artifact Browser URL

View in Artifact Browser

Inspecting Test Artifacts Manually

To inspect your test artifacts manually, follow these steps:

Install ORAS (see the ORAS installation guide).
Download artifacts with the following commands:

mkdir -p oras-artifacts
cd oras-artifacts
oras pull quay.io/opendatahub/odh-ci-artifacts:maas-group-test-758wt

…io#2 — move datasource, fix LokiStack URL - Move loki-datasource.yaml to observability/dashboards/ next to its dashboard (follows Prometheus datasource co-location convention) - Change hardcoded LokiStack URL from lokistack-sample/logging4 to logging-loki/openshift-logging (standard namespace) - Update loki/ and dashboards/ kustomization references accordingly

openshift-ci Bot added the do-not-merge/work-in-progress label Jun 14, 2026

ahadas reviewed Jun 14, 2026

View reviewed changes

feat: add Loki-based admin usage dashboard with Perses datasource

2d63a11

Add admin usage dashboard querying Loki structured logs via LogQL, with supporting Loki datasource infrastructure (CA, RBAC, SA token secret).

tgitelman force-pushed the feature-loki-admin-dashboard branch from 17576cd to 2d63a11 Compare June 14, 2026 15:55

Merge branch 'main' into feature-loki-admin-dashboard

d14a700

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add Loki-based admin usage dashboard with Perses datasource#995

feat: add Loki-based admin usage dashboard with Perses datasource#995
tgitelman wants to merge 2 commits into
opendatahub-io:mainfrom
tgitelman:feature-loki-admin-dashboard

tgitelman commented Jun 14, 2026

Uh oh!

openshift-ci Bot commented Jun 14, 2026

Uh oh!

coderabbitai Bot commented Jun 14, 2026 •

edited

Loading

Review skipped

Uh oh!

ahadas Jun 14, 2026

Uh oh!

tgitelman Jun 14, 2026

Uh oh!

ahadas Jun 14, 2026

Uh oh!

ahadas Jun 14, 2026

Uh oh!

ahadas Jun 14, 2026

Uh oh!

ahadas Jun 14, 2026

Uh oh!

ahadas Jun 14, 2026

Uh oh!

ahadas Jun 14, 2026

Uh oh!

rhods-ci-bot commented Jun 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1,18 @@
		# Grant Perses SA read access to application logs via LokiStack gateway.

		@@ -0,0 +1,36 @@
		# Admin Loki datasource — direct to LokiStack, no user filtering.

Conversation

tgitelman commented Jun 14, 2026

Uh oh!

openshift-ci Bot commented Jun 14, 2026

Uh oh!

coderabbitai Bot commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rhods-ci-bot commented Jun 14, 2026

OCI Artifact Browser URL

Inspecting Test Artifacts Manually

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coderabbitai Bot commented Jun 14, 2026 •

edited

Loading