Skip to content

README.md

Latest commit

 

History

History
130 lines (90 loc) · 3.03 KB

README.md

File metadata and controls

130 lines (90 loc) · 3.03 KB

Live Canary Local and GitHub Setup

This directory contains the unified entrypoints for the live regression lanes:

  • run.sh dispatches named lanes and writes artifacts
  • scrub-artifacts.sh scans artifacts before upload
  • upgrade-canary.sh checks previous-release DB compatibility

The auth-focused Python runners remain the executors behind the auth lanes:

  • scripts/auth_canary/run_canary.py
  • scripts/auth_live_canary/run_live_canary.py
  • scripts/auth_browser_canary/run_browser_canary.py

Their shared auth canary setup, provider registry, and runtime helpers live in:

  • scripts/live_canary/common.py
  • scripts/live_canary/auth_registry.py
  • scripts/live_canary/auth_runtime.py

Future auth providers should be added through the shared registry and account guide, not by creating a new standalone runner shape.

Run commands from the repository root.

Lane Families

Upstream live LLM lanes

  • deterministic-replay
  • public-smoke
  • persona-rotating
  • private-oauth
  • provider-matrix
  • release-public-full
  • upgrade-canary

Auth lanes added on this branch

  • auth-smoke
  • auth-full
  • auth-channels
  • auth-live-seeded
  • auth-browser-consent

Local Commands

Run the public live smoke lane:

LANE=public-smoke scripts/live-canary/run.sh

Run the provider matrix lane:

LANE=provider-matrix \
PROVIDER=openai-compatible \
PROVIDER_TEST_TARGET=e2e_live_mission \
SCENARIO=mission_daily_news_digest_with_followup \
scripts/live-canary/run.sh

Run the auth smoke lane:

LANE=auth-smoke scripts/live-canary/run.sh

Run the seeded auth live lane:

LANE=auth-live-seeded scripts/live-canary/run.sh

Run the browser-consent auth lane:

LANE=auth-browser-consent scripts/live-canary/run.sh

Run selected auth provider cases:

LANE=auth-live-seeded CASES=gmail,github scripts/live-canary/run.sh
LANE=auth-live-seeded CASES=ops_workflow scripts/live-canary/run.sh
LANE=auth-browser-consent CASES=google,github scripts/live-canary/run.sh

Use CI-style browser installation for auth browser lanes:

LANE=auth-browser-consent PLAYWRIGHT_INSTALL=with-deps scripts/live-canary/run.sh

Reuse an existing build and Python environment:

LANE=auth-smoke SKIP_BUILD=1 SKIP_PYTHON_BOOTSTRAP=1 scripts/live-canary/run.sh

Run an upgrade canary:

LANE=upgrade-canary \
PREVIOUS_REF=v0.1.2 \
CURRENT_REF=HEAD \
scripts/live-canary/run.sh

Artifacts are written under:

artifacts/live-canary/<lane>/<provider>/<timestamp>/

Secrets And Account Material

Public live LLM lane secrets and variables are documented in docs/internal/live-canary.md.

Seeded auth live-provider credentials:

GitHub Workflow

GitHub Actions uses .github/workflows/live-canary.yml as the single scheduled and manual entrypoint. That workflow now contains both the upstream live LLM jobs and the auth-specific canary jobs.