feat: add observability proxy

[floreks]

This pull request introduces a new Observability Proxy service to the Console project, providing infrastructure and configuration for deploying, building, and managing the proxy alongside the main application. The changes include Helm chart templates, Kubernetes manifests, CI/CD workflows, and supporting configuration for the new service, as well as updates to network policies and service definitions. The Observability Proxy is designed to handle observability-related traffic and integrates with the Console via gRPC.

Observability Proxy Integration

• Added Helm chart templates for the Observability Proxy, including deployment.yaml, service.yaml, and hpa.yaml, enabling deployment, service exposure, and autoscaling for the proxy. [1] [2] [3]
• Introduced configuration options in values.yaml to control proxy deployment, resource allocation, autoscaling, and connection settings.

CI/CD Workflow Enhancements

• Added new GitHub Actions workflows for CI (observability-proxy-ci.yaml) and CD (observability-proxy-cd.yaml) to automate testing, building, and publishing of the Observability Proxy Docker images for multiple architectures. [1] [2]

Kubernetes Networking and Routing

• Updated ingress.yaml to route observability endpoints (/ext/v1/ingest, /ext/v1/query) to the Observability Proxy service when enabled.
• Added a dedicated NetworkPolicy template to restrict access to internal Console ports, including the new gRPC port and proxy ports, with configuration exposed in values.yaml. [1] [2]

Console Service and Deployment Updates

• Added gRPC port configuration to Console's deployment and service templates to support communication with the Observability Proxy. [1] [2]

Supporting Files

• Added the Observability Proxy module to go.work and provided a .gitignore and Dockerfile for building and packaging the proxy. [1] [2] [3]

Test Plan

Chart: oci://ghcr.io/pluralsh/console-chart
Name: console
Version: 0.0.0_pr-3332

image:
    tag: pr-3332
    imagePullPolicy: Always
...
observabilityProxy:
    enabled: true
    image:
        tag: pr-3332
        pullPolicy: Always

Checklist

• If required, I have updated the Plural documentation accordingly.
• I have added tests to cover my changes.
• I have added a meaningful title and summary to convey the impact of this PR to a user.

Plural Flow: console

[linear]

PROD-4541 add observability proxy

[floreks]

@greptileai review this PR. ES/Prom credentials are ignored intentionally. Clients are reponsible for sending credentials in requests to proxy.

[greptile-apps]

Greptile Summary

This PR introduces a new Observability Proxy service — a Go reverse proxy that sits in front of Prometheus (VictoriaMetrics) and Elasticsearch backends, fetches credentials and host configuration from Console via gRPC, and meters ingested bytes back to Console. It ships with Helm chart templates (Deployment, Service, HPA, NetworkPolicy, Ingress routing), multi-arch CI/CD GitHub Actions workflows, and a complete Go module under go/observability-proxy/.

Several issues flagged in earlier review rounds have been addressed in this revision: the thundering-herd on config-cache expiry is solved with singleflight, the usage-reporter goroutine is properly drained before grpcClient.Close() via a done channel, the WriteTimeout is set to 60 s, method guards are present on all proxy endpoints, and the upload/download artifact versions are aligned in both workflows.

Remaining observations:

• The observability-proxy Deployment uses serviceAccountName: console, inheriting the Console service account's RBAC permissions. The proxy needs no Kubernetes API access, so a dedicated empty service account would be more appropriate.
• forward() in handler.go allocates a new httputil.ReverseProxy struct (and closure) on every request. Since the Transport and ErrorHandler are constant, a shallow copy of a pre-built base proxy would avoid the per-request allocation on the hot path.
• Port 50051 is added to charts/console/templates/service.yaml unconditionally (not gated on observabilityProxy.enabled), while the NetworkPolicy that restricts access to it is gated on the separate networkPolicy.consoleInternalPorts.enabled flag (defaults true). These two flags should be documented as a pair, or the service port exposure should be tied to observabilityProxy.enabled.

Confidence Score: 4/5

• Safe to merge with minor follow-up; no critical bugs or security regressions introduced.
• Core Go logic is well-tested (singleflight deduplication, byte-counting, method guards, graceful shutdown all have tests or have been verified in code). Previously critical issues (undefined flags, reporter race, artifact version mismatch, write timeout) are resolved. Remaining concerns are a least-privilege service account and a per-request allocation — neither blocks correctness or security in the current deployment model.
• charts/console/templates/observability-proxy/deployment.yaml (service account), go/observability-proxy/internal/proxy/handler.go (per-request ReverseProxy allocation)

Important Files Changed

Filename	Overview
go/observability-proxy/internal/proxy/handler.go	Core proxy logic with method guards, metering via countingReadCloser, and upstream error handling. A new ReverseProxy is allocated per request; the Transport is shared correctly. Query-string forwarding is preserved through the Rewrite func.
go/observability-proxy/internal/console/provider.go	Caching config provider with singleflight deduplication (addressing thundering-herd on TTL boundaries), stale-cache fallback on refresh errors, and a dedicated mutex. Context is intentionally discarded so refresh is not cancelled by individual request contexts.
go/observability-proxy/internal/metering/meter.go	Atomic byte accumulator that periodically flushes to Console. Failed flushes re-queue bytes correctly. Success log fires only on the non-error path. Final flush is correctly awaited before process exit via the reporterDone channel in main.go.
go/observability-proxy/cmd/proxy/main.go	Clean startup/shutdown sequence. WriteTimeout is set to 60s. The usage-reporter goroutine is properly synchronized with a done channel so the final metering flush completes before grpcClient.Close() is called.
charts/console/templates/observability-proxy/deployment.yaml	Proxy deployment template. Uses console's service account instead of a dedicated, minimal one — violates least-privilege. Uses console.containerSecurityContext for the proxy container, which is a minor coupling.
charts/console/templates/networkpolicy.yaml	Restricts Console's internal ports (4369, 50051) to same-namespace traffic via namespaceSelector. Service ports (3000, service.port, 9090) are intentionally left open. Protection depends on networkPolicy.consoleInternalPorts.enabled being true alongside observabilityProxy.enabled.
.github/workflows/observability-proxy-cd.yaml	Multi-arch CD pipeline for three registries. upload-artifact and download-artifact are aligned on v7 (previously reported mismatch is resolved).
charts/console/templates/service.yaml	Adds gRPC port 50051 unconditionally to the Console ClusterIP service (not gated on observabilityProxy.enabled). Combined with the independent networkPolicy.consoleInternalPorts.enabled flag this was flagged in a previous review thread.

_{Last reviewed commit: "fix(observability-pr..."}

[floreks]

@greptileai fixed issues. review again

[floreks]

@greptileai all fixed, review

[floreks]

@greptileai

[floreks]

@greptileai

[floreks]

@greptileai

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	k8s.io/​klog/​v2@​v2.140.0
	golang.org/​x/​sync@​v0.18.0

View full report