Skip to content

feat: add Loki query proxy for user-scoped log isolation#999

Draft
tgitelman wants to merge 1 commit into
opendatahub-io:mainfrom
tgitelman:feature/loki-user-proxy
Draft

feat: add Loki query proxy for user-scoped log isolation#999
tgitelman wants to merge 1 commit into
opendatahub-io:mainfrom
tgitelman:feature/loki-user-proxy

Conversation

@tgitelman

Copy link
Copy Markdown
Contributor

Summary

  • Add a Go-based Loki query proxy that injects user_id filtering into LogQL queries based on the caller's Kubernetes identity (TokenReview API)
  • Bridges per-user Loki log isolation until Perses/COO adds native user-scoped querying
  • Runs on stock ubi9/go-toolset:1.25 — no custom image build required

Design

The proxy sits between Perses and the LokiStack gateway. When a non-admin user queries Loki:

  1. Extracts the user's bearer token from the request
  2. Validates it via the Kubernetes TokenReview API to resolve username + groups
  3. Injects | user_id="<username>" into every LogQL stream selector
  4. Forwards the modified query to LokiStack using the proxy's own SA token (with cluster-logging-application-view RBAC)

Admins (system:cluster-admins, system:masters) bypass filtering and see all logs.

Files (5 files, 646 lines)

File Purpose
proxy-source-configmap.yaml Go source code (main, auth, rewriter, config) mounted as ConfigMap
deployment-user.yaml Deployment on stock UBI9 Go image, hardened security context
rbac.yaml ServiceAccount + ClusterRoleBindings (logging-view, tokenreview)
service.yaml ClusterIP service on port 8080
kustomization.yaml Namespace replacement for ClusterRoleBinding subjects

Security

  • All tokens validated server-side via TokenReview (never parsed locally)
  • TLS 1.2+ for all upstream connections (LokiStack, K8s API)
  • runAsNonRoot, drop ALL capabilities, RuntimeDefault seccomp profile
  • Least-privilege RBAC: only cluster-logging-application-view + tokenreviews create
  • Non-admin users restricted to query endpoints only (/loki/api/v1/query, /query_range)
  • Username values escaped to prevent LogQL injection

Bridging solution

This proxy is a temporary bridge until Perses adds native user-scoped Loki filtering (tracked in COO 1.5+). Known POC limitations:

  • go run on startup (~60-90s cold start)
  • Full response buffering (io.ReadAll)
  • CA certificates loaded once at startup (no runtime rotation)

Prerequisites

  • LokiStack deployed in openshift-logging with structured logs flowing from OTel Collector
  • Admin dashboard PR merged (provides loki/ infrastructure: CA, RBAC, secret)

Test plan

  • Deploy with kustomize build deployment/components/observability/loki-proxy | kubectl apply -f -
  • Verify proxy pod starts and passes readiness probe
  • As admin: query returns all logs (no user_id filter injected)
  • As non-admin user: query results filtered to caller's user_id
  • Verify blocked endpoints (/labels, /series) return 403 for non-admin users
  • Verify LogQL injection attempts in username are escaped

Go-based reverse proxy that injects user_id filtering into LogQL queries
based on the caller's Kubernetes identity (via TokenReview API).
Bridges per-user Loki access until Perses adds native user-scoped querying.

- proxy-source-configmap.yaml: Go source (main, auth, rewriter, config)
- deployment-user.yaml: runs on stock ubi9/go-toolset, no custom image
- rbac.yaml: SA + ClusterRoleBindings (logging-view, tokenreview)
- service.yaml: ClusterIP on port 8080
- kustomization.yaml: namespace replacement for ClusterRoleBinding subjects
@coderabbitai

coderabbitai Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 58153dd1-0da0-457d-a1dc-eacd85b947d9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci

openshift-ci Bot commented Jun 15, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tgitelman
Once this PR has been reviewed and has the lgtm label, please assign jland-redhat for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@rhods-ci-bot

Copy link
Copy Markdown

@tgitelman: The following test has Failed:

OCI Artifact Browser URL

View in Artifact Browser

Inspecting Test Artifacts Manually

To inspect your test artifacts manually, follow these steps:

  1. Install ORAS (see the ORAS installation guide).
  2. Download artifacts with the following commands:
mkdir -p oras-artifacts
cd oras-artifacts
oras pull quay.io/opendatahub/odh-ci-artifacts:maas-group-test-dllxt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants