Skip to content

CLAUDE.md

Latest commit

 

History

History
157 lines (114 loc) · 8.6 KB

CLAUDE.md

File metadata and controls

157 lines (114 loc) · 8.6 KB
Raw

Orion Regulus Development Guide

What This Tool Does

Automated regression detection for Regulus network performance tests. Wraps the cloud-bulldozer/orion changepoint detection tool to handle Regulus's hundreds of test variations.

Core workflow: Query ES by batch_id → discover unique fingerprints → run Orion with static template (configs/template.yaml) + --input-vars per fingerprint → Hunter changepoint detection → report regressions.

Architecture

Fingerprint Definition

Fingerprint fields are defined in the static template (configs/template.yaml). Each {{ field }} placeholder is a fingerprint field. At startup, the analyzer validates the template covers all ES mapping fields (minus NON_FINGERPRINT_FIELDS). A self-detect fallback (--use-self-detect) can generate configs without the template.

NON_FINGERPRINT_FIELDS is defined in three files — keep them in sync:

  • scripts/analyze-batch.py
  • scripts/verify-batch.py
  • scripts/verify-mapping.py

Dual Metric Tracking

Two metrics tracked per fingerprint, both with direction: 0 (flag changes in either direction):

Metric ES Field Aggregation Threshold
throughput mean avg 5%
cpu_cost busy_cpu avg 10%

Critical: Both metrics MUST have agg: {agg_type: avg} in the Orion config. Without it, Orion's standard batch path assigns all documents to the first metric only (due to a "first match wins" break in get_results_batch at matcher.py:631). The agg block routes metrics through the aggregation path which runs each independently.

Prow CI Integration

This directory is the source of truth for the Prow CI step. The Prow step (openshift-qe-orion-regulus) clones the Regulus repo and runs ORION/scripts/prow-entry.sh, which bridges Prow env vars to analyze-batch.py CLI args.

Prow commands.sh → clones Regulus repo → cd ORION → scripts/prow-entry.sh → scripts/analyze-batch.py

There is no bundled copy of the analyzer in the step registry.

Orion Invocation

analyze-batch.py auto-detects pip-installed orion CLI vs podman run-it script. Install orion from GitHub (not PyPI — pip install orion is a different package):

pip3 install git+https://github.com/cloud-bulldozer/orion.git

Key Files

scripts/
  analyze-batch.py        # Core analyzer — source of truth for dev and Prow
  prow-entry.sh           # Prow CI entry point (bridges env vars → CLI args)
  verify-mapping.py       # Verify ES index mapping compatibility
  verify-batch.py         # Verify batch data quality
  validate-test-results.sh # Single source of truth for test expectations
  list-batches.py         # Format batch listing output
  run-it                  # Podman wrapper for Orion container

unit-test/
  generate-batch-test-data.py  # Generate 6-fingerprint mock test data
  generate-mock-data.py        # Base mock data generator class
  json-to-bulk.py              # Convert JSON to ES bulk format

configs/
  template.yaml            # Static Orion config template with fingerprint placeholders
  README.md               # Config approach documentation
  CONFIG-TUTORIAL.md       # Orion config tutorial
  DESIGN-TEMPLATE.md       # Template design doc

Makefile                   # All targets (run `make help`)
FINGERPRINT-DEFINITION.md  # Fingerprint field definitions and tracked metrics
requirements.txt           # Python deps (orion from GitHub)

Make Targets

make help              # Show all targets with descriptions

# Production
make analyze BATCH_ID=... [MATCH=...] [IGNORE=...]
make list-batches      # List batches in ES
make show-mapping      # Raw ES mapping JSON
make verify-mapping    # Validate mapping compatibility
make verify-batch BATCH_ID=...

# Testing
make test-full         # Full cycle: create mock → push → analyze → validate
make test-prow         # Simulate Prow CI locally
make verify-test       # Re-validate last test results
make create-mock       # Generate mock data
make push-batch        # Push mock data to ES
make clean-mock        # Delete mock data from ES
make setup             # Install Python deps (skip-if-installed)

Test Expectations

Mock data has 6 fingerprints (threads=16/32/64/128/256 + multibench composite).

Test expectations live in ONE place: scripts/validate-test-results.sh. Both make test-full and make verify-test call it. Do not duplicate expectations elsewhere.

Testing Approaches Compared

Three ways to test the regression analysis pipeline, from narrowest to most realistic:

make test-full make test-prow test-prow-step.sh
What it tests Analyzer directly prow-entry.sh wrapper Actual commands.sh that Prow runs
Calls analyze-batch.py prow-entry.sh openshift-qe-orion-regulus-commands.sh
Data setup Creates & pushes mock data Expects data already in ES Expects data already in ES
ES credentials lab.config lab.config Interactive prompt; creates mock /secret/perfscale-prod
Secrets simulation None None Creates /tmp/prow-secret-perfscale-prod/ and symlinks to /secret/perfscale-prod
ES index regulus-mock-results regulus-mock-results regulus-mock-results
Validates results validate-test-results.sh Checks /tmp/prow-artifacts/ Checks timestamped ARTIFACT_DIR

Typical sequence: run test-full first (generates + pushes mock data, validates analyzer), then test-prow (reuses same data, validates Prow bridge). Use test-prow-step.sh for full end-to-end Prow simulation.

test-prow-step.sh usage: lives in ORION/scripts/ but must be copied to the step-registry directory (release/ci-operator/step-registry/openshift-qe/orion/regulus/) to run, since it calls commands.sh from that location.

Note: test-prow-step.sh creates /secret/perfscale-prod via symlink. This can cause make es-* commands in REPORT/ to fail because SOURCE_ES_CONFIG sees /secret/ and tries to read credentials from it instead of falling through to lab.config.

Common Pitfalls

  • pip install orion installs the wrong package (epistimio/orion). Use pip3 install git+https://github.com/cloud-bulldozer/orion.git.
  • Orion metrics without agg block: Both metrics sharing the same documents will break — second metric gets zero data. Always use agg: {agg_type: avg}.
  • Exit code 1 from make analyze: Expected when regressions are detected — this is the success signal for "found problems."
  • Exit code 3 from Orion: No matching data found. Check that metadata values in the config match what's in ES — a wrong type or literal "MISSING" value will cause zero matches.
  • Orion wildcard metadata: Putting * in a regular metadata value does NOT create a wildcard query — Orion uses Q("match") (exact match) for regular fields. Wildcard queries require the field to be under metadata.wildcard in the config. See matcher.py:139-152.
  • Orion appends test name to output filenames: --save-output-path foo.json with test name fingerprint-1 produces foo_fingerprint-1.json. If you change the test name, the output filename changes.
  • Orion loads env vars into template context: load_config merges all env vars (lowercased) with --input-vars before Jinja2 rendering. --input-vars values take precedence.
  • make setup noise: Uses skip-if-installed check and -q flag to stay quiet.
  • ES index for testing: regulus-mock-results (set via TEST_INDEX). Production: regulus-results-*.

ES Configuration

lab.config (in the Regulus repo root) is the single source of truth for ES credentials. Set these variables:

ES_PROTOCOL=https
ES_HOST=es-host:9200
ES_USER=your-user
ES_PASSWORD=your-password

The Makefile sources lab.config automatically and constructs the ES connection internally. Default index: ES_INDEX=regulus-results-*.

Prow Entry Point Details

scripts/prow-entry.sh accepts these env vars:

  • ES credentials — reads from /secret/perfscale-prod/{username,password,host} (Prow) or lab.config (local)
  • BATCH_ID — batch to analyze (empty = auto-discover latest)
  • ES_BENCHMARK_INDEX — index pattern (default: regulus-results-*)
  • MATCH, IGNORE, LOOKBACK, DEBUG — passed through to analyze-batch.py
  • ARTIFACT_DIR — copies configs/output here for Prow artifacts
  • DEBUG_IGNORE, DEBUG_MATCH — developer overrides hardcoded in the script (empty by default). If either is non-empty, both replace the Prow IGNORE/MATCH values. Edit these to iterate without Prow approval.