Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,15 @@ to land under these conventions; subsequent specs follow this shape.
(Border / Image / ScrollView) with a back-reference to the Controls section
— the prior parenthetical Button carve-out was easy to miss mid-build.
(spec 038 §3 — agent-facing skill updates)
- Spec 038 EC3-final watch-item: `rule_fired` trace event. When a Tier-3
rule attaches a suggestion to a diagnostic, `mur check --trace` now writes
one structured row per fire:
`{kind: "rule_fired", rule, code, confidence, evidence, file, line, mode}`.
Per-rule firing-rate audits collapse from multi-step content scans against
`events.jsonl` agent tool outputs to a 1-line `jq` over the trace file.
Tier-2 suggestions deliberately do not emit this row — Tier-2 firing rates
are visible via the opt-in `MUR_TELEMETRY=1` channel. (spec 038 §0.3,
EC3-final watch-item)
- Spec 038 §3.1a residual: trace-channel structured warning hook for
self-disabled rules. `TraceWriter.WriteRuleSelfDisabled(rule, target)`
emits `{kind: "rule_self_disabled", rule, unresolved_target, mode}`.
Expand Down
8 changes: 6 additions & 2 deletions docs/reference/mur-check-did-you-mean.md
Original file line number Diff line number Diff line change
Expand Up @@ -375,7 +375,7 @@ CheckCommand.Run (CheckCommand.cs)
│ ├─ for each rule: TargetsResolve (else self-disable callback)
│ └─ TryMatch; pick highest confidence
│ RULE WINS over tier2Best (spec §6)
└─ trace.Write (optional, --trace; records all parsed diagnostics + rule self-disables)
└─ trace.Write (optional, --trace; records all parsed diagnostics + rule self-disables + rule fires)
```

The `Suggestion` (highest-confidence above threshold, rule preferred over Tier-2) is attached to the `Diag` for line formatting in `Diag.Format`.
Expand Down Expand Up @@ -456,7 +456,11 @@ Phases 0–2 are merged to `main`. Phase 3 is in flight on `eval/spec-038-ec3-20

### Phase 0 — instrumentation (merged)

- `--trace <path>` flag on `mur check`. Writes a JSONL stream of every parsed diagnostic, one row per diagnostic. Schema: `{ts, code, severity, file, line, col, msg, receiver_type?, member?, mode}`. Trace is opt-in, never written by default, never includes source code text, never includes absolute paths outside the project root.
- `--trace <path>` flag on `mur check`. Writes a JSONL stream, one row per parsed diagnostic (plus auxiliary structured events). Trace is opt-in, never written by default, never includes source code text, never includes absolute paths outside the project root. Row kinds:
- **diagnostic row** (default, no `kind` field): `{ts, code, severity, file, line, col, msg, receiver_type?, member?, mode}`.
- **command header** (`kind: "command"`): `{ts, kind, argv, mode}` — full effective `dotnet build` argv at the head of the trace.
- **rule self-disabled** (`kind: "rule_self_disabled"`): `{ts, kind, rule, unresolved_target, mode}` — emitted when a Tier-3 rule's declared target fails to resolve against the live compilation. Dedup'd per-invocation per-rule.
- **rule fired** (`kind: "rule_fired"`): `{ts, kind, rule, code, confidence, evidence, file, line, mode}` — emitted whenever a Tier-3 rule attaches a suggestion to a diagnostic. Tier-2 hits do not emit this row; their firing rate is visible via the opt-in `MUR_TELEMETRY=1` channel.
- Folder structure: `src/Reactor.Cli/Check/{Suggesters,Rules}/` with README pointers; mirrored test folders.
- A smoke fixture (`tests/Reactor.IntegrationTests/MurCheck/Fixtures/SmokeFixture/`) plus a smoke test that drives the end-to-end pipeline.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -545,7 +545,7 @@ Clean rerun after the EC3-original watch-items closed: template typo fix (`Micrs
- **Class-A rule exercise.** Targeted-prompt batch at `C:\temp\mur-targeted-prompt-spec.md` is still the empirical question on the three new Class-A rules. Author or curate prompts that surface CS1955/`GridSize.Auto`, CS0117/`GridSize.Pixel|Pixels|Fixed`, and CS1061/CS0117 on `TextBlockElement.Style`. The clean-PASS doesn't change that this batch left the rules' token impact unmeasured.
- **§11 risk-row guardrail retrofit.** Move EC3-final criterion #3 from "low-confidence audit" to "verified" by adding a post-run analysis pass that runs `mur check --final` against the run's final workspace and compares against iteration-mode suggestions. Same instrument the EC3-original results section called for; still open.
- **Tier-2 SKILL.md trims.** Now empirically de-risked by EC3-final's PASS. The drag-and-drop relocation to `reactor-input` and the Context-pattern trim could push `reactor-getting-started` another ~65 lines lighter. Separate PR; no blockers.
- **`rule_fired` trace event.** One-line addition to `TraceWriter.cs` + the orchestrator emission point would make per-rule firing-rate audits a one-line grep against the trace file instead of multi-step content scan against `events.jsonl` agent tool outputs. Cheap; lands before the targeted-prompt batch ideally.
- **`rule_fired` trace event — LANDED 2026-05-12.** `TraceWriter.WriteRuleFired(rule, code, confidence, evidence, file, line)` emits `{kind: "rule_fired", rule, code, confidence, evidence, file, line, mode}`. `Suggestion.IsRule` (default false) discriminates Tier-3 hits from Tier-2; `CheckCommand.EmitDiagnostics` writes the trace event whenever a rule-flagged suggestion attaches. Three new unit tests in `TraceWriterTests.cs` (schema, evidence-truncation, path-sanitization) + two pipeline tests in `CheckCommandPipelineTests.cs` (writes-on-rule, no-write-on-tier2) lock the contract. Tier-2 hits stay off the trace by design — telemetry channel still owns Tier-2 rate; the trace event exists specifically to make Tier-3 firings discoverable without telemetry opt-in. Targeted-prompt batch is now a 1-line `jq '.kind=="rule_fired"'` over the trace file.
Comment thread
codemonkeychris marked this conversation as resolved.
Outdated

---

Expand Down
19 changes: 18 additions & 1 deletion src/Reactor.Cli/Check/CheckCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,24 @@ internal static void EmitDiagnostics(IReadOnlyList<Diag> diagnostics, TextWriter
if (stdoutFilter is not null && !stdoutFilter(d)) continue;
var suggestion = suggest?.Invoke(d);
stdout.WriteLine(d.Format(suggestion));
if (suggestion is not null) Telemetry.OnSuggestionEmitted(d.Code, suggestion);
if (suggestion is not null)
{
Telemetry.OnSuggestionEmitted(d.Code, suggestion);
// Spec 038 EC3-final watch-item: structured rule-fire events
// make per-rule firing-rate audits a 1-line grep over the
// trace file. Only emitted for Tier-3 rule hits; Tier-2 fires
// are visible via the opt-in telemetry channel instead.
if (suggestion.IsRule)
{
trace?.WriteRuleFired(
ruleName: suggestion.SuggesterName,
code: d.Code,
confidence: suggestion.Confidence,
evidence: suggestion.Evidence,
file: d.File,
line: d.Line);
}
}
}
}

Expand Down
4 changes: 2 additions & 2 deletions src/Reactor.Cli/Check/SuggesterOrchestrator.cs
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@

namespace Microsoft.UI.Reactor.Cli.Check;

internal sealed record Suggestion(string Text, double Confidence, string Evidence, string SuggesterName);
internal sealed record Suggestion(string Text, double Confidence, string Evidence, string SuggesterName, bool IsRule = false);

internal sealed class SuggesterOrchestrator
{
Expand Down Expand Up @@ -180,7 +180,7 @@ public SuggesterOrchestrator(
var evidence = string.IsNullOrEmpty(s.Evidence)
? rule.Provenance
: $"{s.Evidence} ({rule.Provenance})";
return new Suggestion(s.Text!, s.Confidence, evidence, rule.Name);
return new Suggestion(s.Text!, s.Confidence, evidence, rule.Name, IsRule: true);
}
}

Expand Down
49 changes: 49 additions & 0 deletions src/Reactor.Cli/Check/TraceWriter.cs
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,44 @@ public void WriteRuleSelfDisabled(string ruleName, string unresolvedTarget)
writer.WriteLine(json);
}

/// <summary>
/// Spec 038 EC3-final watch-item: emit one structured row per Tier-3 rule
/// fire so per-rule firing-rate audits are a 1-line grep against the trace
/// JSONL instead of a multi-step content scan against agent tool outputs.
/// Schema:
///
/// { ts, kind: "rule_fired", rule, code, confidence, evidence, file, line, mode }
///
/// `evidence` is truncated to <see cref="MaxMessageChars"/> to keep the row
/// under the 2 KB cap the writer enforces. Tier-2 suggestions deliberately
/// do not emit this row — Tier-2 firing rates are derivable from telemetry
/// (~/.mur/telemetry/*.jsonl); the trace event exists specifically to make
/// Tier-3 rule firings discoverable without telemetry opt-in.
/// </summary>
public void WriteRuleFired(
string ruleName,
string code,
double confidence,
string evidence,
string file,
int line)
{
var ev = evidence;
if (ev.Length > MaxMessageChars) ev = ev[..MaxMessageChars];
var row = new RuleFiredRow(
ts: DateTime.UtcNow.ToString("o"),
kind: "rule_fired",
rule: ruleName,
code: code,
confidence: confidence,
evidence: ev,
file: SanitizePath(file, projectRoot),
line: line,
mode: mode);
var json = JsonSerializer.Serialize(row, JsonOpts);
writer.WriteLine(json);
}

static readonly JsonSerializerOptions JsonOpts = new()
{
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
Expand Down Expand Up @@ -187,4 +225,15 @@ internal sealed record RuleSelfDisabledRow(
string rule,
string unresolved_target,
string mode);

internal sealed record RuleFiredRow(
string ts,
string kind,
string rule,
string code,
double confidence,
string evidence,
string file,
int line,
string mode);
}
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,86 @@ public void Ranker_filter_suppresses_stdout_but_trace_still_records_every_row()
finally { try { File.Delete(tracePath); } catch { } }
}

[Fact]
public void Emit_writes_rule_fired_trace_event_when_rule_suggestion_attached()
{
// Spec 038 EC3-final watch-item: rule fires must show up in the trace
// as a structured `rule_fired` row so per-rule firing-rate audits are
// a 1-line grep. End-to-end through EmitDiagnostics: when the suggest
// callback returns a rule-flagged Suggestion, the trace file gains a
// rule_fired row alongside the diagnostic row.
var diags = CheckCommand.ParseDiagnostics("""
X.cs(1,1): error CS1955: Non-invocable member [P.csproj]
""");
var stdout = new StringWriter();
var tracePath = Path.Combine(Path.GetTempPath(), "mur-check-rulefire-" + Guid.NewGuid() + ".jsonl");
try
{
using (var trace = TraceWriter.Open(tracePath, Path.GetFullPath(".")))
{
CheckCommand.EmitDiagnostics(diags, stdout, trace,
suggest: _ => new Suggestion(
Text: ".Auto (drop parens)",
Confidence: 0.95,
Evidence: "GridSize.Auto is a property (cluster:C0004)",
SuggesterName: "GridSizeFactoryParensRule",
IsRule: true));
}

var lines = File.ReadAllLines(tracePath);
Assert.Equal(2, lines.Length); // 1 diag row + 1 rule_fired row

var ruleFired = lines
.Select(l => JsonDocument.Parse(l))
.Single(d => d.RootElement.TryGetProperty("kind", out var k) &&
k.GetString() == "rule_fired");
Assert.Equal("GridSizeFactoryParensRule", ruleFired.RootElement.GetProperty("rule").GetString());
Assert.Equal("CS1955", ruleFired.RootElement.GetProperty("code").GetString());
Assert.Equal(0.95, ruleFired.RootElement.GetProperty("confidence").GetDouble());
Assert.Equal("X.cs", ruleFired.RootElement.GetProperty("file").GetString());
Assert.Equal(1, ruleFired.RootElement.GetProperty("line").GetInt32());
Comment thread
codemonkeychris marked this conversation as resolved.
Outdated

// The stdout line still carries the → try: suffix — trace event
// is in addition to, not in place of.
Assert.Contains(".Auto (drop parens)", stdout.ToString());
}
finally { try { File.Delete(tracePath); } catch { } }
}

[Fact]
public void Emit_does_not_write_rule_fired_for_tier2_suggestions()
{
// Tier-2 firing rates are visible via the opt-in telemetry channel;
// the trace's rule_fired row is Tier-3-only by design. A Tier-2 hit
// here must leave no rule_fired marker in the trace.
var diags = CheckCommand.ParseDiagnostics("""
X.cs(1,1): error CS1061: missing member [P.csproj]
""");
var stdout = new StringWriter();
var tracePath = Path.Combine(Path.GetTempPath(), "mur-check-tier2-" + Guid.NewGuid() + ".jsonl");
try
{
using (var trace = TraceWriter.Open(tracePath, Path.GetFullPath(".")))
{
CheckCommand.EmitDiagnostics(diags, stdout, trace,
suggest: _ => new Suggestion(
Text: "Button(label, onClick: x)",
Confidence: 0.91,
Evidence: "factory has Action onClick parameter",
SuggesterName: "SymbolSuggester")); // IsRule default = false
}

var lines = File.ReadAllLines(tracePath);
// Exactly one row — the diagnostic — and it is NOT a rule_fired row.
Assert.Single(lines);
using var doc = JsonDocument.Parse(lines[0]);
var hasKind = doc.RootElement.TryGetProperty("kind", out var k);
if (hasKind)
Assert.NotEqual("rule_fired", k.GetString());
}
finally { try { File.Delete(tracePath); } catch { } }
}

[Fact]
public void Ranker_final_mode_emits_everything_to_stdout()
{
Expand Down
71 changes: 71 additions & 0 deletions tests/Reactor.Tests/CheckCommandTests/TraceWriterTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,77 @@ public void Rule_self_disabled_row_has_expected_schema()
Assert.True(line.Length <= 2048, $"trace row was {line.Length} bytes — exceeds 2 KB cap.");
}

[Fact]
public void Rule_fired_row_has_expected_schema()
{
// Spec 038 EC3-final watch-item: per-rule firing-rate audits should
// be a 1-line grep over the trace file. The row carries enough info
// to identify the rule, the diagnostic it fired on, the confidence,
// and the location — so a maintainer can `jq` for `.kind=="rule_fired"`
// and reconstruct firing rates without rejoining against the diag rows.
using var tmp = TempFile.Create();
using (var w = TraceWriter.Open(tmp.Path, Path.GetFullPath("."), mode: "iteration"))
{
w.WriteRuleFired(
ruleName: "GridSizeFactoryParensRule",
code: "CS1955",
confidence: 0.95,
evidence: "GridSize.Auto is a property, not a method (cluster:C0004)",
file: "Program.cs",
line: 42);
}

var line = File.ReadAllLines(tmp.Path).Single();
using var doc = JsonDocument.Parse(line);
Assert.Equal("rule_fired", doc.RootElement.GetProperty("kind").GetString());
Assert.Equal("GridSizeFactoryParensRule", doc.RootElement.GetProperty("rule").GetString());
Assert.Equal("CS1955", doc.RootElement.GetProperty("code").GetString());
Assert.Equal(0.95, doc.RootElement.GetProperty("confidence").GetDouble());
Assert.Contains("GridSize.Auto", doc.RootElement.GetProperty("evidence").GetString());
Assert.Equal("Program.cs", doc.RootElement.GetProperty("file").GetString());
Assert.Equal(42, doc.RootElement.GetProperty("line").GetInt32());
Assert.Equal("iteration", doc.RootElement.GetProperty("mode").GetString());
Assert.True(line.Length <= 2048, $"trace row was {line.Length} bytes — exceeds 2 KB cap.");
}

[Fact]
public void Rule_fired_evidence_is_truncated_under_2KB_cap()
{
// Same defensive truncation we apply to diag.msg: a runaway evidence
// string should never blow the 2 KB row budget.
var huge = new string('e', 8192);
using var tmp = TempFile.Create();
using (var w = TraceWriter.Open(tmp.Path, Path.GetFullPath(".")))
w.WriteRuleFired("R", "CS1061", 0.9, huge, "a.cs", 1);

var line = File.ReadAllLines(tmp.Path).Single();
Assert.True(line.Length <= 2048, $"trace row was {line.Length} bytes — exceeds 2 KB cap.");
}

[Fact]
public void Rule_fired_file_is_sanitized_like_diag_rows()
{
// Same path-leak protection that applies to diag rows: a rule fire
// outside the project root must not leak the absolute file path.
var root = Path.GetFullPath(Path.Combine(Path.GetTempPath(), "reactor-trace-rf-" + Guid.NewGuid()));
Directory.CreateDirectory(root);
try
{
var outside = Path.Combine(Path.GetTempPath(), "elsewhere-" + Guid.NewGuid(), "Bar.cs");
using var tmp = TempFile.Create();
using (var w = TraceWriter.Open(tmp.Path, root))
w.WriteRuleFired("R", "CS1061", 0.9, "ev", outside, 1);

var line = File.ReadAllLines(tmp.Path).Single();
using var doc = JsonDocument.Parse(line);
Assert.Equal("<external>", doc.RootElement.GetProperty("file").GetString());
}
finally
{
try { Directory.Delete(root, recursive: true); } catch { }
}
}

sealed class TempFile : IDisposable
{
public string Path { get; }
Expand Down
Loading