Skip to content

Spec 038 — rule_fired trace event for Tier-3 rule fires#251

Merged
codemonkeychris merged 2 commits into
mainfrom
spec-038/rule-fired-trace-event
May 12, 2026
Merged

Spec 038 — rule_fired trace event for Tier-3 rule fires#251
codemonkeychris merged 2 commits into
mainfrom
spec-038/rule-fired-trace-event

Conversation

@codemonkeychris
Copy link
Copy Markdown
Collaborator

@codemonkeychris codemonkeychris commented May 12, 2026

Summary

  • Adds a structured {kind: \"rule_fired\", rule, code, confidence, evidence, file, line, mode} row to mur check --trace whenever a Tier-3 rule attaches a suggestion to a diagnostic.
  • Closes the EC3-final watch-item: per-rule firing-rate audits collapse from multi-step content scans against events.jsonl agent tool outputs to a 1-line jq '.kind==\"rule_fired\"' over the trace file.
  • Tier-2 hits deliberately don't emit this row — Tier-2 firing rate is visible via the opt-in MUR_TELEMETRY=1 channel; the trace event exists specifically to make Tier-3 firings discoverable without telemetry opt-in.

Changes

Code (3 files):

  • Suggestion record gains bool IsRule = false (positional default; existing call sites unchanged).
  • SuggesterOrchestrator sets IsRule: true only at the rule-winning return site.
  • TraceWriter.WriteRuleFired(...) with same 1024-char evidence truncation + path sanitization as diag rows.
  • CheckCommand.EmitDiagnostics invokes it on rule-flagged suggestions.

Tests (5 new):

  • TraceWriterTests: row-schema, evidence-truncation under 2 KB, file-path sanitization for <external> paths.
  • CheckCommandPipelineTests: end-to-end emit-on-rule + no-emit-on-tier2.
  • 227/227 in the CheckCommand test area pass.

Docs:

  • CHANGELOG entry under Spec 038 — mur check did-you-mean.
  • Spec/tasks watch-item marked LANDED with implementation summary.
  • docs/reference/mur-check-did-you-mean.md trace-schema section expanded to enumerate all four row kinds (diagnostic, command, rule_self_disabled, rule_fired).

Follow-up

Remaining EC3-final watch-items + Phase-4 cleanup work tracked in #252 (filed alongside this PR).

Test plan

  • dotnet build src/Reactor.Cli succeeds (.NET 10, warnings-as-errors clean).
  • CheckCommand tests pass (227/227, x64).
  • Reviewer spot-checks the new trace row shape against the existing rule_self_disabled schema for consistency.

🤖 Generated with Claude Code

Closes the EC3-final watch-item: per-rule firing-rate audits are now a
1-line `jq '.kind=="rule_fired"'` over the trace file instead of a
multi-step content scan against `events.jsonl` agent tool outputs.

- `Suggestion.IsRule` (default false) discriminates Tier-3 from Tier-2.
- `TraceWriter.WriteRuleFired` emits `{ts, kind, rule, code, confidence,
  evidence, file, line, mode}` with the same 1024-char evidence
  truncation + path sanitization as diag rows.
- `CheckCommand.EmitDiagnostics` writes the event whenever a
  rule-flagged suggestion attaches; Tier-2 hits stay off the trace
  (their firing rate is visible via the opt-in MUR_TELEMETRY channel).
- 5 new unit tests: schema, evidence-truncation, path-sanitization,
  end-to-end pipeline emit-on-rule + no-emit-on-tier2.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a new rule_fired JSONL trace row to mur check --trace so Tier-3 rule firings are directly discoverable from trace output (without requiring telemetry opt-in), and updates tests/docs to lock in the schema and behavior.

Changes:

  • Add Suggestion.IsRule (default false) and set it only when a Tier-3 rule wins in SuggesterOrchestrator.
  • Add TraceWriter.WriteRuleFired(...) and emit kind: "rule_fired" rows from CheckCommand.EmitDiagnostics when IsRule suggestions are attached.
  • Add unit + pipeline tests and update docs/CHANGELOG to describe the new trace row kind.
Show a summary per file
File Description
tests/Reactor.Tests/CheckCommandTests/TraceWriterTests.cs Adds schema/truncation/path-sanitization unit tests for the new rule_fired trace row.
tests/Reactor.Tests/CheckCommandTests/CheckCommandPipelineTests.cs Adds end-to-end tests verifying rule_fired emission for Tier-3 and non-emission for Tier-2.
src/Reactor.Cli/Check/TraceWriter.cs Implements WriteRuleFired + RuleFiredRow with evidence truncation and path sanitization.
src/Reactor.Cli/Check/SuggesterOrchestrator.cs Extends Suggestion with IsRule and marks only rule-winning suggestions as IsRule: true.
src/Reactor.Cli/Check/CheckCommand.cs Emits rule_fired trace rows when an IsRule suggestion is attached to a diagnostic.
docs/specs/tasks/038-mur-check-did-you-mean-implementation.md Marks the watch-item as landed and documents the implementation + tests.
docs/reference/mur-check-did-you-mean.md Updates trace docs to enumerate rule_fired alongside other row kinds.
CHANGELOG.md Adds a changelog entry describing the new rule_fired trace event.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 8/8 changed files
  • Comments generated: 2

Comment thread tests/Reactor.Tests/CheckCommandTests/CheckCommandPipelineTests.cs Outdated
Comment thread docs/specs/tasks/038-mur-check-did-you-mean-implementation.md Outdated
- Dispose JsonDocument instances explicitly in the new pipeline test;
  detach the rule_fired row via JsonElement.Clone() so assertions
  outlive the parsed doc without leaking pooled buffers.
- Fix the jq one-liner example: `'.kind=="rule_fired"'` evaluates the
  expression on every row (outputting a boolean), not filters to
  matching rows. Correct form is `'select(.kind=="rule_fired")'`.
  Fixed in both the spec/tasks watch-item entry and the TraceWriter
  test comment.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@codemonkeychris
Copy link
Copy Markdown
Collaborator Author

Addressed both Copilot CR comments in 739403e:

  1. JsonDocument disposal in pipeline test — refactored Emit_writes_rule_fired_trace_event_when_rule_suggestion_attached to parse each line inside a using block and detach the matching row via JsonElement.Clone() so assertions outlive the parsed document without leaking pooled buffers.
  2. jq one-liner'.kind=="rule_fired"' evaluates the expression per row (emits a boolean stream); correct filter is 'select(.kind=="rule_fired")'. Fixed in the spec/tasks watch-item entry, the TraceWriter test comment, and issue Spec 038 — Phase 4 cleanup: targeted-prompt batch, guardrail retrofit, Checkpoint D, ranker training #252's §A.4 (same example had propagated).

27/27 trace + pipeline tests still pass.

@codemonkeychris
Copy link
Copy Markdown
Collaborator Author

CI failure on Integration Tests is unrelated flake — TemplatePackageTestFixture ctor timed out at 3:42 wall on dotnet new install --debug:custom-hive (budget 120 s). Diagnosis:

  • Failure is in tests/Reactor.IntegrationTests/Packaging/CreateTemplateTests.cs:268 — template-engine install on a cold CI runner. Nothing in 739403e touches packaging.
  • Prior commit b2e6c8e in this PR had all 7 checks green including Integration Tests.
  • Recent main CI is 8/8 green.
  • dotnet new install cache thrash is the known flaky surface called out in spec 038 EC3-original; the recent template-typo fix (Spec 038 Phase 3 — Class-A rules, correctness fixes, EC3 results, template typo fix #250) addressed the duplicate-identity failure mode, but the underlying cold-runner install can still stall.

Re-triggered the failed job via gh run rerun --failed.

@codemonkeychris codemonkeychris merged commit 80381b0 into main May 12, 2026
11 of 12 checks passed
@codemonkeychris codemonkeychris deleted the spec-038/rule-fired-trace-event branch May 12, 2026 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants