Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,15 @@ to land under these conventions; subsequent specs follow this shape.
(Border / Image / ScrollView) with a back-reference to the Controls section
— the prior parenthetical Button carve-out was easy to miss mid-build.
(spec 038 §3 — agent-facing skill updates)
- Spec 038 EC3-final watch-item: `rule_fired` trace event. When a Tier-3
rule attaches a suggestion to a diagnostic, `mur check --trace` now writes
one structured row per fire:
`{kind: "rule_fired", rule, code, confidence, evidence, file, line, mode}`.
Per-rule firing-rate audits collapse from multi-step content scans against
`events.jsonl` agent tool outputs to a 1-line `jq` over the trace file.
Tier-2 suggestions deliberately do not emit this row — Tier-2 firing rates
are visible via the opt-in `MUR_TELEMETRY=1` channel. (spec 038 §0.3,
EC3-final watch-item)
- Spec 038 §3.1a residual: trace-channel structured warning hook for
self-disabled rules. `TraceWriter.WriteRuleSelfDisabled(rule, target)`
emits `{kind: "rule_self_disabled", rule, unresolved_target, mode}`.
Expand Down
8 changes: 6 additions & 2 deletions docs/reference/mur-check-did-you-mean.md
Original file line number Diff line number Diff line change
Expand Up @@ -375,7 +375,7 @@ CheckCommand.Run (CheckCommand.cs)
│ ├─ for each rule: TargetsResolve (else self-disable callback)
│ └─ TryMatch; pick highest confidence
│ RULE WINS over tier2Best (spec §6)
└─ trace.Write (optional, --trace; records all parsed diagnostics + rule self-disables)
└─ trace.Write (optional, --trace; records all parsed diagnostics + rule self-disables + rule fires)
```

The `Suggestion` (highest-confidence above threshold, rule preferred over Tier-2) is attached to the `Diag` for line formatting in `Diag.Format`.
Expand Down Expand Up @@ -456,7 +456,11 @@ Phases 0–2 are merged to `main`. Phase 3 is in flight on `eval/spec-038-ec3-20

### Phase 0 — instrumentation (merged)

- `--trace <path>` flag on `mur check`. Writes a JSONL stream of every parsed diagnostic, one row per diagnostic. Schema: `{ts, code, severity, file, line, col, msg, receiver_type?, member?, mode}`. Trace is opt-in, never written by default, never includes source code text, never includes absolute paths outside the project root.
- `--trace <path>` flag on `mur check`. Writes a JSONL stream, one row per parsed diagnostic (plus auxiliary structured events). Trace is opt-in, never written by default, never includes source code text, never includes absolute paths outside the project root. Row kinds:
- **diagnostic row** (default, no `kind` field): `{ts, code, severity, file, line, col, msg, receiver_type?, member?, mode}`.
- **command header** (`kind: "command"`): `{ts, kind, argv, mode}` — full effective `dotnet build` argv at the head of the trace.
- **rule self-disabled** (`kind: "rule_self_disabled"`): `{ts, kind, rule, unresolved_target, mode}` — emitted when a Tier-3 rule's declared target fails to resolve against the live compilation. Dedup'd per-invocation per-rule.
- **rule fired** (`kind: "rule_fired"`): `{ts, kind, rule, code, confidence, evidence, file, line, mode}` — emitted whenever a Tier-3 rule attaches a suggestion to a diagnostic. Tier-2 hits do not emit this row; their firing rate is visible via the opt-in `MUR_TELEMETRY=1` channel.
- Folder structure: `src/Reactor.Cli/Check/{Suggesters,Rules}/` with README pointers; mirrored test folders.
- A smoke fixture (`tests/Reactor.IntegrationTests/MurCheck/Fixtures/SmokeFixture/`) plus a smoke test that drives the end-to-end pipeline.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -545,7 +545,7 @@ Clean rerun after the EC3-original watch-items closed: template typo fix (`Micrs
- **Class-A rule exercise.** Targeted-prompt batch at `C:\temp\mur-targeted-prompt-spec.md` is still the empirical question on the three new Class-A rules. Author or curate prompts that surface CS1955/`GridSize.Auto`, CS0117/`GridSize.Pixel|Pixels|Fixed`, and CS1061/CS0117 on `TextBlockElement.Style`. The clean-PASS doesn't change that this batch left the rules' token impact unmeasured.
- **§11 risk-row guardrail retrofit.** Move EC3-final criterion #3 from "low-confidence audit" to "verified" by adding a post-run analysis pass that runs `mur check --final` against the run's final workspace and compares against iteration-mode suggestions. Same instrument the EC3-original results section called for; still open.
- **Tier-2 SKILL.md trims.** Now empirically de-risked by EC3-final's PASS. The drag-and-drop relocation to `reactor-input` and the Context-pattern trim could push `reactor-getting-started` another ~65 lines lighter. Separate PR; no blockers.
- **`rule_fired` trace event.** One-line addition to `TraceWriter.cs` + the orchestrator emission point would make per-rule firing-rate audits a one-line grep against the trace file instead of multi-step content scan against `events.jsonl` agent tool outputs. Cheap; lands before the targeted-prompt batch ideally.
- **`rule_fired` trace event — LANDED 2026-05-12.** `TraceWriter.WriteRuleFired(rule, code, confidence, evidence, file, line)` emits `{kind: "rule_fired", rule, code, confidence, evidence, file, line, mode}`. `Suggestion.IsRule` (default false) discriminates Tier-3 hits from Tier-2; `CheckCommand.EmitDiagnostics` writes the trace event whenever a rule-flagged suggestion attaches. Three new unit tests in `TraceWriterTests.cs` (schema, evidence-truncation, path-sanitization) + two pipeline tests in `CheckCommandPipelineTests.cs` (writes-on-rule, no-write-on-tier2) lock the contract. Tier-2 hits stay off the trace by design — telemetry channel still owns Tier-2 rate; the trace event exists specifically to make Tier-3 firings discoverable without telemetry opt-in. Targeted-prompt batch is now a 1-line `jq 'select(.kind=="rule_fired")'` over the trace file.

---

Expand Down
19 changes: 18 additions & 1 deletion src/Reactor.Cli/Check/CheckCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,24 @@ internal static void EmitDiagnostics(IReadOnlyList<Diag> diagnostics, TextWriter
if (stdoutFilter is not null && !stdoutFilter(d)) continue;
var suggestion = suggest?.Invoke(d);
stdout.WriteLine(d.Format(suggestion));
if (suggestion is not null) Telemetry.OnSuggestionEmitted(d.Code, suggestion);
if (suggestion is not null)
{
Telemetry.OnSuggestionEmitted(d.Code, suggestion);
// Spec 038 EC3-final watch-item: structured rule-fire events
// make per-rule firing-rate audits a 1-line grep over the
// trace file. Only emitted for Tier-3 rule hits; Tier-2 fires
// are visible via the opt-in telemetry channel instead.
if (suggestion.IsRule)
{
trace?.WriteRuleFired(
ruleName: suggestion.SuggesterName,
code: d.Code,
confidence: suggestion.Confidence,
evidence: suggestion.Evidence,
file: d.File,
line: d.Line);
}
}
}
}

Expand Down
4 changes: 2 additions & 2 deletions src/Reactor.Cli/Check/SuggesterOrchestrator.cs
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@

namespace Microsoft.UI.Reactor.Cli.Check;

internal sealed record Suggestion(string Text, double Confidence, string Evidence, string SuggesterName);
internal sealed record Suggestion(string Text, double Confidence, string Evidence, string SuggesterName, bool IsRule = false);

internal sealed class SuggesterOrchestrator
{
Expand Down Expand Up @@ -180,7 +180,7 @@ public SuggesterOrchestrator(
var evidence = string.IsNullOrEmpty(s.Evidence)
? rule.Provenance
: $"{s.Evidence} ({rule.Provenance})";
return new Suggestion(s.Text!, s.Confidence, evidence, rule.Name);
return new Suggestion(s.Text!, s.Confidence, evidence, rule.Name, IsRule: true);
}
}

Expand Down
49 changes: 49 additions & 0 deletions src/Reactor.Cli/Check/TraceWriter.cs
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,44 @@ public void WriteRuleSelfDisabled(string ruleName, string unresolvedTarget)
writer.WriteLine(json);
}

/// <summary>
/// Spec 038 EC3-final watch-item: emit one structured row per Tier-3 rule
/// fire so per-rule firing-rate audits are a 1-line grep against the trace
/// JSONL instead of a multi-step content scan against agent tool outputs.
/// Schema:
///
/// { ts, kind: "rule_fired", rule, code, confidence, evidence, file, line, mode }
///
/// `evidence` is truncated to <see cref="MaxMessageChars"/> to keep the row
/// under the 2 KB cap the writer enforces. Tier-2 suggestions deliberately
/// do not emit this row — Tier-2 firing rates are derivable from telemetry
/// (~/.mur/telemetry/*.jsonl); the trace event exists specifically to make
/// Tier-3 rule firings discoverable without telemetry opt-in.
/// </summary>
public void WriteRuleFired(
string ruleName,
string code,
double confidence,
string evidence,
string file,
int line)
{
var ev = evidence;
if (ev.Length > MaxMessageChars) ev = ev[..MaxMessageChars];
var row = new RuleFiredRow(
ts: DateTime.UtcNow.ToString("o"),
kind: "rule_fired",
rule: ruleName,
code: code,
confidence: confidence,
evidence: ev,
file: SanitizePath(file, projectRoot),
line: line,
mode: mode);
var json = JsonSerializer.Serialize(row, JsonOpts);
writer.WriteLine(json);
}

static readonly JsonSerializerOptions JsonOpts = new()
{
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull,
Expand Down Expand Up @@ -187,4 +225,15 @@ internal sealed record RuleSelfDisabledRow(
string rule,
string unresolved_target,
string mode);

internal sealed record RuleFiredRow(
string ts,
string kind,
string rule,
string code,
double confidence,
string evidence,
string file,
int line,
string mode);
}
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,99 @@ public void Ranker_filter_suppresses_stdout_but_trace_still_records_every_row()
finally { try { File.Delete(tracePath); } catch { } }
}

[Fact]
public void Emit_writes_rule_fired_trace_event_when_rule_suggestion_attached()
{
// Spec 038 EC3-final watch-item: rule fires must show up in the trace
// as a structured `rule_fired` row so per-rule firing-rate audits are
// a 1-line grep. End-to-end through EmitDiagnostics: when the suggest
// callback returns a rule-flagged Suggestion, the trace file gains a
// rule_fired row alongside the diagnostic row.
var diags = CheckCommand.ParseDiagnostics("""
X.cs(1,1): error CS1955: Non-invocable member [P.csproj]
""");
var stdout = new StringWriter();
var tracePath = Path.Combine(Path.GetTempPath(), "mur-check-rulefire-" + Guid.NewGuid() + ".jsonl");
try
{
using (var trace = TraceWriter.Open(tracePath, Path.GetFullPath(".")))
{
CheckCommand.EmitDiagnostics(diags, stdout, trace,
suggest: _ => new Suggestion(
Text: ".Auto (drop parens)",
Confidence: 0.95,
Evidence: "GridSize.Auto is a property (cluster:C0004)",
SuggesterName: "GridSizeFactoryParensRule",
IsRule: true));
}

var lines = File.ReadAllLines(tracePath);
Assert.Equal(2, lines.Length); // 1 diag row + 1 rule_fired row

// Find the rule_fired row and detach its element so the parsed
// JsonDocument can be disposed inside the loop (Copilot CR: avoid
// leaking pooled buffers across the assertion block).
JsonElement ruleFired = default;
var found = false;
foreach (var line in lines)
{
using var doc = JsonDocument.Parse(line);
if (doc.RootElement.TryGetProperty("kind", out var k) &&
k.GetString() == "rule_fired")
{
ruleFired = doc.RootElement.Clone();
found = true;
break;
}
}
Assert.True(found, "trace did not contain a rule_fired row.");
Assert.Equal("GridSizeFactoryParensRule", ruleFired.GetProperty("rule").GetString());
Assert.Equal("CS1955", ruleFired.GetProperty("code").GetString());
Assert.Equal(0.95, ruleFired.GetProperty("confidence").GetDouble());
Assert.Equal("X.cs", ruleFired.GetProperty("file").GetString());
Assert.Equal(1, ruleFired.GetProperty("line").GetInt32());

// The stdout line still carries the → try: suffix — trace event
// is in addition to, not in place of.
Assert.Contains(".Auto (drop parens)", stdout.ToString());
}
finally { try { File.Delete(tracePath); } catch { } }
}

[Fact]
public void Emit_does_not_write_rule_fired_for_tier2_suggestions()
{
// Tier-2 firing rates are visible via the opt-in telemetry channel;
// the trace's rule_fired row is Tier-3-only by design. A Tier-2 hit
// here must leave no rule_fired marker in the trace.
var diags = CheckCommand.ParseDiagnostics("""
X.cs(1,1): error CS1061: missing member [P.csproj]
""");
var stdout = new StringWriter();
var tracePath = Path.Combine(Path.GetTempPath(), "mur-check-tier2-" + Guid.NewGuid() + ".jsonl");
try
{
using (var trace = TraceWriter.Open(tracePath, Path.GetFullPath(".")))
{
CheckCommand.EmitDiagnostics(diags, stdout, trace,
suggest: _ => new Suggestion(
Text: "Button(label, onClick: x)",
Confidence: 0.91,
Evidence: "factory has Action onClick parameter",
SuggesterName: "SymbolSuggester")); // IsRule default = false
}

var lines = File.ReadAllLines(tracePath);
// Exactly one row — the diagnostic — and it is NOT a rule_fired row.
Assert.Single(lines);
using var doc = JsonDocument.Parse(lines[0]);
var hasKind = doc.RootElement.TryGetProperty("kind", out var k);
if (hasKind)
Assert.NotEqual("rule_fired", k.GetString());
}
finally { try { File.Delete(tracePath); } catch { } }
}

[Fact]
public void Ranker_final_mode_emits_everything_to_stdout()
{
Expand Down
71 changes: 71 additions & 0 deletions tests/Reactor.Tests/CheckCommandTests/TraceWriterTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,77 @@ public void Rule_self_disabled_row_has_expected_schema()
Assert.True(line.Length <= 2048, $"trace row was {line.Length} bytes — exceeds 2 KB cap.");
}

[Fact]
public void Rule_fired_row_has_expected_schema()
{
// Spec 038 EC3-final watch-item: per-rule firing-rate audits should
// be a 1-line grep over the trace file. The row carries enough info
// to identify the rule, the diagnostic it fired on, the confidence,
// and the location — so a maintainer can `jq 'select(.kind=="rule_fired")'`
// and reconstruct firing rates without rejoining against the diag rows.
using var tmp = TempFile.Create();
using (var w = TraceWriter.Open(tmp.Path, Path.GetFullPath("."), mode: "iteration"))
{
w.WriteRuleFired(
ruleName: "GridSizeFactoryParensRule",
code: "CS1955",
confidence: 0.95,
evidence: "GridSize.Auto is a property, not a method (cluster:C0004)",
file: "Program.cs",
line: 42);
}

var line = File.ReadAllLines(tmp.Path).Single();
using var doc = JsonDocument.Parse(line);
Assert.Equal("rule_fired", doc.RootElement.GetProperty("kind").GetString());
Assert.Equal("GridSizeFactoryParensRule", doc.RootElement.GetProperty("rule").GetString());
Assert.Equal("CS1955", doc.RootElement.GetProperty("code").GetString());
Assert.Equal(0.95, doc.RootElement.GetProperty("confidence").GetDouble());
Assert.Contains("GridSize.Auto", doc.RootElement.GetProperty("evidence").GetString());
Assert.Equal("Program.cs", doc.RootElement.GetProperty("file").GetString());
Assert.Equal(42, doc.RootElement.GetProperty("line").GetInt32());
Assert.Equal("iteration", doc.RootElement.GetProperty("mode").GetString());
Assert.True(line.Length <= 2048, $"trace row was {line.Length} bytes — exceeds 2 KB cap.");
}

[Fact]
public void Rule_fired_evidence_is_truncated_under_2KB_cap()
{
// Same defensive truncation we apply to diag.msg: a runaway evidence
// string should never blow the 2 KB row budget.
var huge = new string('e', 8192);
using var tmp = TempFile.Create();
using (var w = TraceWriter.Open(tmp.Path, Path.GetFullPath(".")))
w.WriteRuleFired("R", "CS1061", 0.9, huge, "a.cs", 1);

var line = File.ReadAllLines(tmp.Path).Single();
Assert.True(line.Length <= 2048, $"trace row was {line.Length} bytes — exceeds 2 KB cap.");
}

[Fact]
public void Rule_fired_file_is_sanitized_like_diag_rows()
{
// Same path-leak protection that applies to diag rows: a rule fire
// outside the project root must not leak the absolute file path.
var root = Path.GetFullPath(Path.Combine(Path.GetTempPath(), "reactor-trace-rf-" + Guid.NewGuid()));
Directory.CreateDirectory(root);
try
{
var outside = Path.Combine(Path.GetTempPath(), "elsewhere-" + Guid.NewGuid(), "Bar.cs");
using var tmp = TempFile.Create();
using (var w = TraceWriter.Open(tmp.Path, root))
w.WriteRuleFired("R", "CS1061", 0.9, "ev", outside, 1);

var line = File.ReadAllLines(tmp.Path).Single();
using var doc = JsonDocument.Parse(line);
Assert.Equal("<external>", doc.RootElement.GetProperty("file").GetString());
}
finally
{
try { Directory.Delete(root, recursive: true); } catch { }
}
}

sealed class TempFile : IDisposable
{
public string Path { get; }
Expand Down
Loading