Skip to content

PR 3/4: PLG Monitoring Stack - Metrics, Helm Charts & Promtail Sidecars - #71

Merged
alex-struk merged 19 commits into
developfrom
pr/plg-monitoring-stack
Mar 25, 2026
Merged

PR 3/4: PLG Monitoring Stack - Metrics, Helm Charts & Promtail Sidecars#71
alex-struk merged 19 commits into
developfrom
pr/plg-monitoring-stack

Conversation

@alex-struk

@alex-struk alex-struk commented Mar 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds the PLG (Promtail/Loki/Grafana) monitoring stack with Prometheus metrics, Helm charts for OpenShift deployment, and Docker Compose for local development.

Merge order: This is PR 3 of 4. Merge after PR 2. Use "Update branch" before merging.

Changes

  • Add session ID, client IP, and API key identifier to structured log output
  • Expose Prometheus RED metrics endpoint (request rate, error rate, duration) with Node.js runtime metrics
  • Helm chart for Loki with NDJSON parsing, 30-day retention, and PVC storage
  • Helm chart additions for Prometheus with scrape configs for backend-services and Temporal
  • Helm chart additions for Grafana with auth and pre-configured data sources
  • Docker Compose monitoring stack for local development (Promtail, Loki, Prometheus, Grafana)
  • Promtail sidecar containers for all OpenShift application pods
  • Integration of PLG deployment into GitHub Actions workflow and scripts
  • PLG monitoring stack requirements and user stories

Testing

  • Unit tests for AppLoggerService, LoggingMiddleware, RequestLoggingInterceptor, and MetricsService
  • API key auth guard tests updated for key prefix logging
  • Local monitoring stack verified with Docker Compose

Checklist

By submitting this pull request, I acknowledge that I have attempted to meet the following:

  • a self-review of my code
  • commented code particularly in hard-to-understand areas
  • corresponding changes to the documentation where required
  • changes tested to the best of my ability
  • no new errors or non-functional code

strukalex and others added 12 commits March 17, 2026 16:41
@strukalex @claude
feat: add PLG monitoring stack requirements and improve requirements-…
07ba862 
…refiner skill

Add consolidated requirements for Prometheus, Loki, and Grafana (PLG)
observability stack covering deployment, log aggregation, session tracking,
client IP logging, metrics, and pre-built dashboards.

Update requirements-refiner skill to provide concrete answer options with
recommendations for each clarifying question.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@strukalex @claude
feat: add user stories for PLG monitoring stack
7465332 
13 user stories across 5 phases covering log enrichment (sessionId,
clientIp, apiKeyId), Prometheus metrics, Helm charts for Loki/Prometheus/
Grafana, Docker Compose and OpenShift deployment integration, and
pre-built Grafana dashboards.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@strukalex @claude
feat: implement US-001 - Add Session ID to Request Context and Log Ou…
34d2464 
…tput

Add sessionId (from Keycloak session_state claim) to request context,
NDJSON log output, and shared LogContext interface. Unauthenticated
requests naturally omit the field.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@strukalex @claude
feat: implement US-002 - Add Client IP to Log Output
4fe55dc 
Extract client IP from X-Forwarded-For > X-Real-IP > socket.remoteAddress
and include it in NDJSON log output and shared LogContext interface.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@strukalex @claude
feat: implement US-003 - Add API Key Identifier to Log Output
d261c60 
Log API key prefix as apiKeyId for API key-authenticated requests.
Ensures sessionId and apiKeyId are mutually exclusive and the full
API key value is never written to logs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@strukalex @claude
feat: implement US-004 - Expose Prometheus RED Metrics Endpoint
cff543a 
Add prom-client with RED metrics (http_requests_total, http_request_duration_seconds,
http_request_errors_total), Node.js runtime default metrics, and /metrics endpoint
excluded from auth guards and OpenShift route.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@strukalex @claude
feat: implement US-005 - Create Helm Chart with Loki for Log Aggregation
8cf1caf 
Add PLG Helm chart with Loki StatefulSet, NDJSON-compatible config,
30-day retention via compactor, PVC storage, and per-environment
values files (local/openshift).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@strukalex @claude
feat: implement US-006 - Add Prometheus to Helm Chart with Scrape Con…
1038284 
…figuration

Add Prometheus StatefulSet, ConfigMap with scrape targets for
backend-services and Temporal, configurable scrape interval and
PVC storage, per-environment values overrides.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@strukalex @claude
feat: implement US-007 - Add Grafana to Helm Chart with Auth and Data…
467f6c1 
… Sources

Add Grafana Deployment with username/password auth, pre-provisioned
Prometheus and Loki data sources, ClusterIP service on port 3001,
no OpenShift Route (access via port-forwarding).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@strukalex @claude
feat: implement US-008 - Create Docker Compose for Local PLG Stack
d719fa1 
Add opt-in docker-compose.monitoring.yml with Loki, Promtail, Prometheus,
and Grafana. Promtail auto-discovers containers via Docker socket.
Includes npm scripts (dev:monitoring) and VS Code task integration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@strukalex @claude
feat: implement US-009 - Add Promtail Sidecar Containers to OpenShift…
dd3c2a3 
… Deployments

Add Promtail sidecar containers to backend-services, temporal-worker,
temporal-server, frontend, and PostgreSQL pods. Each sidecar tails
log files from shared volumes and forwards to Loki with service labels.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@strukalex @claude
feat: implement US-010 - Integrate PLG Deployment with GitHub Actions…
372bdca 
… and Scripts

Add PLG Helm deployment to GitHub Actions workflow and oc-deploy.sh script.
PLG environment variables configurable per overlay. PLG is a separate Helm
release that does not affect existing Kustomize deployment.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@alex-struk
alex-struk marked this pull request as ready for review March 20, 2026 22:46
@strukalex @claude
merge: resolve conflicts with develop branch
fafd8d6 
Resolve merge conflicts in 4 test files by combining develop's mock-based
test structure with PLG branch's sessionId/clientIp/apiKeyId feature coverage.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@alex-struk
alex-struk force-pushed the pr/plg-monitoring-stack branch from 76e81b2 to fafd8d6 Compare March 24, 2026 21:54
strukalex and others added 3 commits March 24, 2026 15:53
@strukalex @claude
fix: correct reflector mock values in ApiKeyAuthGuard tests
330bbf0 
The reflector mock was returning `true`/`false` instead of
`{ allowApiKey: true }`/`undefined`, causing the guard to skip
API key auth checks and resolve instead of rejecting.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@strukalex @claude
refactor: move docker-compose.monitoring.yml to deployments/local/
59f8a36 
Co-locate the monitoring compose file with its config files in
deployments/local/. Update volume paths and all references in
package.json and docs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@strukalex @claude
style: apply biome lint fixes to backend services
b71aa0a 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dbarkowsky
dbarkowsky reviewed Mar 25, 2026
View reviewed changes
Comment thread apps/backend-services/src/auth/api-key-auth.guard.ts Outdated
dbarkowsky
dbarkowsky approved these changes Mar 25, 2026
View reviewed changes

@dbarkowsky dbarkowsky left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok to move forward and see how this works with the final part.
For the helm charts, in other projects we've kept them in a separate bcgov-c repo at the request of the DevOps folks. They would have a better idea as to why, but something to keep in mind if it's a security concern.

strukalex and others added 2 commits March 25, 2026 13:04
@strukalex @claude
refactor: consolidate API key request properties into single apiKey o…
5493646 
…bject

Replace individual request.apiKeyGroupId and request.apiKeyPrefix properties
with a single request.apiKey object (ValidatedApiKey interface), mirroring
the request.user pattern for JWT auth. This avoids growing one-off properties
each time downstream code needs a new field from the validated key.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@strukalex @claude
style: apply biome lint fixes to api-key service and identity guard t…
1b9aad9 
…ests

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@alex-struk

alex-struk commented Mar 25, 2026

Copy link
Copy Markdown
Collaborator Author

Ok to move forward and see how this works with the final part. For the helm charts, in other projects we've kept them in a separate bcgov-c repo at the request of the DevOps folks. They would have a better idea as to why, but something to keep in mind if it's a security concern.

Eventually all infra code will live in a separate argoCD repo, that's what Brandon has been working on, I don't think there are security concerns currently as long as we manage keys properly.

@alex-struk
alex-struk merged commit e0f5911 into develop Mar 25, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dbarkowsky dbarkowsky dbarkowsky approved these changes
@antsand antsand Awaiting requested review from antsand antsand is a code owner
@NoorChasib NoorChasib Awaiting requested review from NoorChasib
@kmandryk kmandryk Awaiting requested review from kmandryk kmandryk is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@alex-struk @dbarkowsky @strukalex