feat(eval): GlitchTip event ingester for regression cases#1820
Conversation
Add scripts/scrape_glitchtip_events.py which lists open is:unresolved issues across the org, fetches each issue's latest event for the exception type and deepest in-app stack frame, and emits one JSON record per unique issue. The IncidentSynthesizer gains a GlitchTipIncident dataclass routed through the same _synthesize_eval_case seam, producing one P1 (warn-only) eval-case YAML per record keyed on glitchtip-issue:<issue_id>. Reuses the existing glitchtip_insights base-URL/DSN resolution and HTTP auth shape; no host is hardcoded, so the package and public artefacts stay backend-agnostic. Dedup spans the scraper output, emitted YAMLs, and the synthesizer content hash. A configurable allow-list filters the administrative wiring-probe issues. A daily workflow opens a PR with the emitted cases for operator review, cron disabled until smoke passes.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @chernistry, you have reached your weekly rate limit of 2500000 diff characters.
Please try again later or upgrade to continue using Sourcery
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: ⛔ Files ignored due to path filters (7)
📒 Files selected for processing (5)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Sonar insights (advisory, no merge-block)Snapshot of
Run This comment is a soft signal. The Sonar scan runs on push to |
Review-bot acknowledgement summary
All must-address findings are resolved or acknowledged. |
|
bernstein doctor observe for PR #1820 ( sonar -- WARN (project bernstein)
code-scanning -- WARN (33 open alert(s))
Skipped backends (credentials not configured)
See docs/observability/unified-doctor.md for backend setup notes. |
Contract drift detected - proposed patchInline autofix push failed ( Three contract tests act as drift detectors against the public CLI / API surface:
One or more failed on this PR. Files changed: How to applyEither run the regen script locally: uv run python scripts/regen_contract_drift.py --fixture all
git add -A && git commit -m "chore(ci): regenerate contract drift allow-lists"
git pushOr apply the patch directly: gh pr checkout 1820
git apply <<'PATCH'
diff --git a/tests/unit/test_readme_api_coverage.py b/tests/unit/test_readme_api_coverage.py
index fa6b08a6..90b3d71d 100644
--- a/tests/unit/test_readme_api_coverage.py
+++ b/tests/unit/test_readme_api_coverage.py
@@ -239,6 +239,8 @@ DOCUMENTED_COMMANDS: frozenset[str] = frozenset(
"desktop-register",
# Bot-added: drift autofix (regen_contract_drift.py)
"supervisor",
+ # Bot-added: drift autofix (regen_contract_drift.py)
+ "schedule",
}
)
PATCH
git add -A && git commit -m "chore(ci): regenerate contract drift allow-lists"
git pushFull diffdiff --git a/tests/unit/test_readme_api_coverage.py b/tests/unit/test_readme_api_coverage.py
index fa6b08a6..90b3d71d 100644
--- a/tests/unit/test_readme_api_coverage.py
+++ b/tests/unit/test_readme_api_coverage.py
@@ -239,6 +239,8 @@ DOCUMENTED_COMMANDS: frozenset[str] = frozenset(
"desktop-register",
# Bot-added: drift autofix (regen_contract_drift.py)
"supervisor",
+ # Bot-added: drift autofix (regen_contract_drift.py)
+ "schedule",
}
)Source CI run: https://github.com/sipyourdrink-ltd/bernstein/actions/runs/26253850668 Refs #1273. |
1 participant
Summary
Adds the inbound half of the GlitchTip integration: open unresolved
issues become P1 (warn-only) regression eval cases in the existing
incident-synthesis pipeline. The outbound DSN path already shipped; this
closes the loop so production exceptions feed the quality gate.
scripts/scrape_glitchtip_events.pylistsis:unresolvedissuesacross all projects in the org, follows
Link: rel="next"pagination,fetches each issue's latest event for the exception type/value and the
deepest in-app stack frame, reads the
environment/release/server_nametags, and emits one JSON record per unique issue.IncidentSynthesizergains aGlitchTipIncidentdataclass routedthrough the existing
_synthesize_eval_caseseam (the same seam usedby DLQ entries, post-mortems, and CI-failure post-mortems from feat(eval): ingest CI-failure post-mortems into regression cases #1808).
Each case carries
source_incident: glitchtip-issue:<issue_id>andseverity: P1.glitchtip_insightsbase-URL / DSN resolution and HTTPauth shape. No host is hardcoded anywhere; the
test_no_hardcoded_infraguard passes for the package, the new doc,and the new workflow.
Current GlitchTip issues (org
bernstein)Dedup contract
Re-runs are a pure no-op. Three axes merge before any write:
glitchtip_issue_id.source_incidentisglitchtip-issue:<id>.inc-<sha1[:12]>) plus thesource_incidentslug at write time.Wiring-probe filter
DEFAULT_WIRING_PROBE_ALLOW_LISTfilters the two administrative smokeissues (
glitchtip insights wiring probe,glitchtip smoke from operator finalisation). Matching iscase-insensitive and substring-based; add entries in the script or via
repeatable
--wiring-probeflags. Documented indocs/operations/glitchtip-ingester.md.Env-var matrix
BERNSTEIN_GLITCHTIP_TOKENGLITCHTIP_API_TOKENBERNSTEIN_GLITCHTIP_BASE_URLGLITCHTIP_BASE_URLBERNSTEIN_GLITCHTIP_ORGGLITCHTIP_ORG_SLUGbernsteinBase URL falls back to the host of
BERNSTEIN_GLITCHTIP_DSN/BERNSTEIN_TELEMETRY_DSN/GLITCHTIP_DSN. No host literal ships.Smoke run trace
Authenticated to the live server as the operator and resolved the two
administrative wiring-probe issues via the API:
Scraper graceful-exit path (no token configured):
Workflow
.github/workflows/glitchtip-ingester.ymlruns daily at23 6 * * *(6 min after the Sonar sweeper), with
workflow_dispatchseverity /max-per-day inputs mirroring the Sonar sweeper, and opens a PR with the
emitted cases. Cron is gated by
ENABLE_CRON(default0) until aclean dispatch smoke run.
Test plan
uv run pytest tests/unit/eval/ -q --no-cov --timeout=120(175 passed)uv run pytest tests/unit/observability/test_no_hardcoded_infra.py(2 passed)uv run ruff check . && uv run ruff format(clean)