Skip to content

RECORDING.md

Latest commit

 

History

History
65 lines (40 loc) · 4.11 KB

RECORDING.md

File metadata and controls

65 lines (40 loc) · 4.11 KB
Raw

Recording rules overview

Recording rules pre-compute PromQL expressions and store the results as new time series in RHOBS. They are used to create derived metrics that would be too expensive or complex to compute at query time or to precompute them if they are in high demand — most commonly the konflux_up availability metric that powers the SLO dashboard and the Tactical Status Page.

Recording rules are defined as PrometheusRule resources in the rhobs/recording/ directory. They share the same Kubernetes kind, deployment pipeline, and testing toolchain as alerting rules.

Creating a new recording rule

  1. Create a new rule file in rhobs/recording/ — use the recording rule template as a starting point
  2. Write a corresponding test file in test/promql/tests/recording/ — use the test template and see Test file format
  3. Run validation — see Validation and testing
  4. Open a PR

Validation and testing

Recording rules share the same validation toolchain as alerting rules. All commands run inside a container — no local tool install needed.

make check-and-test

For faster iteration on a single rule:

make selective-check-and-test \
  RULE_FILES="rhobs/recording/my_recording_rules.yaml" \
  TEST_CASE_FILES="test/promql/tests/recording/my_recording_rules_test.yaml"

YAML linting:

make sync_pipenv   # first time only
make lint_yamls

Test file format

Tests use promtool's unit test format — use the recording rule test template as a starting point.

Test both the expected-up and expected-down cases. For rules with namespace or label filters, include a test case where the input data doesn't match the filter to verify the rule produces no output.

Architecture

Recording rules in this repository are evaluated by RHOBS (Red Hat Observability Service), not by in-cluster Prometheus. Any metric used in a recording rule expression must first be forwarded to RHOBS via infra-deployments. A rule referencing a metric that isn't forwarded will silently produce no data.

Cluster-level recording rules

Recording rules that need to be evaluated by in-cluster Prometheus (rather than RHOBS) are defined in infra-deployments.

Use cluster-level recording rules when the derived metric is only needed in-cluster, or the metric has a high cardinality that would use a large amount of our RHOBS metrics quota (quota is ~10 million in production and ~1 million in stage).

Deployment Model

Recording rules are deployed via the same pipeline as alerting rules — see ALERTS.md § Deployment Model. Both use the same app-interface saas file.

Conventions

PrometheusRule structure

Use the recording rule template as a starting point. All fields are documented with [required] / [optional] tags in the template.

The konflux_up availability pattern

konflux_up is a standardized boolean metric that every Konflux service should expose. It enables a unified SLO dashboard and the Tactical Status Page to show fleet-wide availability. See ALERTS.md § Availability alerts for the full specification of required labels and how signals are created.

Use the availability recording rule template for the standard deployment replica pattern. The template includes inline comments explaining each step.