Skip to content

Commit fa64aba

Browse files
authored
test: add action_sequence grader tests (microsoft#110)
Related to microsoft#105 Proactive test cases for the action_sequence grader. Written from the spec while Linus implements. Covers: exact_match, in_order_match, any_order_match modes, score calculations, error cases, and factory integration. **Test breakdown (31 test cases):** - Constructor validation: 6 tests (missing actions, empty actions, invalid mode, valid modes) - exact_match mode: 6 tests (pass, extra step, missing step, wrong order, empty actual, single tool) - in_order_match mode: 5 tests (exact seq, extras interspersed, wrong order, missing step, subset order) - any_order_match mode: 5 tests (reordered, extras, missing, duplicate expected, insufficient duplicates) - Score calculations: 4 tests (perfect, partial, zero, details verification + f1 check) - Edge cases: 3 tests (nil session, nil tools_used, duration recording) - Factory integration: 2 tests (Create success, Create invalid params error) **Note:** These tests depend on the implementation in the companion PR. Merge the implementation PR first.
1 parent 9552207 commit fa64aba

15 files changed

Lines changed: 900 additions & 94 deletions

.ai-team/agents/linus/history.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,4 +21,8 @@
2121
- **Mock engine (`execution.NewMockEngine`)** — Returns deterministic responses. Useful for testing the full pipeline without a real Copilot SDK connection. Spec YAML with `executor: mock` triggers it.
2222
- **Fixture isolation** — The runner resolves `--context-dir` relative to CWD and task glob patterns relative to the spec file directory. Tests use `t.TempDir()` for full isolation.
2323
- **Shared workspace hazard** — Multiple agents may work concurrently in this repo. Always verify `git branch --show-current` before committing; stash operations can carry changes across branches.
24+
- **Nil session pattern (#105 review fix):** Graders that depend on `SessionDigest` must return a graceful zero-score `GraderResults` (not an error) when `session == nil`. The runner treats grader errors differently from zero-score results — errors may abort the run while zero-scores are recorded as failed validations. Follow `behavior_grader.go` as the canonical pattern. Fixed in `action_sequence_grader.go` per PR #110 review from Rusty.
25+
📌 Team update (2026-02-11): Grader nil-session error handling contract — all graders depending on SessionDigest must return zero-score GraderResults with nil error when session is nil, not (nil, error). — decided by Rusty
26+
27+
📌 Team update (2026-02-11): PR #103 azd extension packaging approved and merged. Registry URLs need update to spboyer/waza for production. — decided by Rusty
2428
<!-- Append new learnings below. -->

.ai-team/agents/rusty/history.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,5 @@
99
## Learnings
1010

1111
<!-- Append new learnings below. Each entry is something lasting about the project. -->
12+
📌 Team update (2026-02-11): Run command tests must reset package-level flag vars (contextDir, outputPath, verbose) at top of each test body to prevent state leakage. — decided by Linus
1213
<!-- Append new learnings below. -->

.ai-team/decisions.md

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,36 @@
88
**What:** All issues must follow: feature branch → commit → push → PR with `Closes #N`@copilot review → address feedback → merge. No direct commits to main.
99
**Why:** User directive
1010

11-
### 2026-02-09: Sensei reference repo
11+
### 2026-02-09: Sensei reference repo for E2 implementation
1212
**By:** Squad (Coordinator)
13-
**What:** E2 sensei engine must adopt patterns from https://github.com/spboyer/sensei. Scoring: Low/Medium/Medium-High/High. Ralph loop: READ→SCORE→CHECK→SCAFFOLD→IMPROVE→VERIFY→TOKENS→SUMMARY→PROMPT→REPEAT. Token management: count/check/suggest/compare with .token-limits.json config.
13+
**What:** The sensei engine (E2: #32-38) must adopt functionality from https://github.com/spboyer/sensei. Key patterns to port to Go:
14+
15+
**Scoring Algorithm:** Low (description < 150 chars OR no triggers) → Medium (description >= 150 chars AND trigger keywords) → Medium-High (USE FOR + DO NOT USE FOR) → High (Medium-High + routing clarity). Rule-based checks: name validation, description length, trigger/anti-trigger detection, routing clarity, MCP integration.
16+
17+
**Ralph Loop:** READ → SCORE → CHECK → SCAFFOLD → IMPROVE FRONTMATTER → IMPROVE TESTS → VERIFY → CHECK TOKENS → SUMMARY → PROMPT → REPEAT (max 5 iterations, target: Medium-High).
18+
19+
**Token Management:** count/check/suggest/compare with .token-limits.json config. SKILL.md: 500 soft / 5000 hard. References: 2000 each. --strict exits 1 on limit breach.
1420
**Why:** User directive — spboyer/sensei is the reference implementation
1521

1622
### 2026-02-09: Monitor human engineer comments
1723
**By:** Shayne Boyer (via Copilot)
18-
**What:** Periodically check for comments from Charles and Richard on open issues. Follow up on responses.
24+
**What:** Periodically check for comments from Charles (@chlowell) and Richard (@richardpark-msft) on open issues. Follow up on responses.
1925
**Why:** User directive
26+
27+
### 2026-02-09: Run command test patterns
28+
**By:** Linus (Backend Dev)
29+
**Related:** #24
30+
**What:** Each test that calls `cmd.Execute()` must reset package-level flag vars (`contextDir`, `outputPath`, `verbose`) at the top of the test body to prevent state leakage. If new flags are added to `newRunCommand()`, the reset block must be updated accordingly.
31+
**Why:** Package-level Cobra flag bindings persist across test cases in the same process.
32+
33+
### 2026-02-11: PR #103 — azd extension packaging approved
34+
**By:** Rusty (Lead)
35+
**Related:** E7 (#62), PR #103
36+
**What:** Wallace Breza's azd extension PR approved and merged. Adds `extension.yaml`, build scripts, `registry.json`, `version.txt`. Purely additive — no changes to existing Go code or CI. Follow-up: update registry URLs to `spboyer/waza` for production; add trailing newlines (POSIX nit); consider CI for automated extension builds.
37+
**Why:** E7 (AZD Extension) P2 roadmap item — establishes foundation without disruption.
38+
39+
### 2026-02-11: Grader nil-session error handling contract
40+
**By:** Rusty (Lead)
41+
**Related:** PR #110
42+
**What:** All graders depending on `SessionDigest` must return a zero-score `GraderResults` with `nil` error when session is nil — not `(nil, error)`. Matches `behavior_grader.go` pattern (lines 56-62). Returning an error would abort the entire run instead of gracefully recording a zero score.
43+
**Why:** Runner distinguishes grader errors (may abort) from zero-score results (contribute to scoring). Affects `action_sequence_grader.go` and future session-dependent graders.

.ai-team/decisions/inbox/coordinator-sensei-reference.md

Lines changed: 0 additions & 40 deletions
This file was deleted.

.ai-team/decisions/inbox/copilot-directive-2026-02-09-git-workflow.md

Lines changed: 0 additions & 4 deletions
This file was deleted.

.ai-team/decisions/inbox/copilot-directive-2026-02-09-sensei-ref.md

Lines changed: 0 additions & 9 deletions
This file was deleted.

.ai-team/decisions/inbox/linus-run-command.md

Lines changed: 0 additions & 30 deletions
This file was deleted.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Session Log: 2026-02-11 — PR Reviews
2+
3+
**Requested by:** Shayne Boyer
4+
5+
## Events
6+
7+
1. **Rusty reviewed PR #110** (action_sequence grader): requested changes — nil session bug must return zero-score result not error; alignment nit on constant naming.
8+
9+
2. **Rusty reviewed PR #103** (Wallace's azd extension): approved — clean, purely additive, aligns with E7 roadmap. Registry URLs need update before production.
10+
11+
3. **Linus fixed PR #110 review feedback**: added graceful nil session handling (zero-score return), updated test assertions, aligned constants.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
## Orchestration Entry
2+
3+
| Field | Value |
4+
|-------|-------|
5+
| **Agent** | Linus (Backend Dev) |
6+
| **Routed because** | Reviewer rejection lockout: Basher authored PR #110, Rusty requested changes → Linus assigned revision |
7+
| **Mode** | background |
8+
| **Files authorized** | action_sequence_grader.go, action_sequence_grader_test.go, outcome.go |
9+
| **Files produced** | Updated 3 files, pushed to squad/105-action-sequence-tests |
10+
| **Outcome** | ✅ Fixed nil session handling (graceful zero-score), test assertions, and constant alignment per Rusty's review |
11+
| **PR** | #110 |
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
## Orchestration Entry
2+
3+
| Field | Value |
4+
|-------|-------|
5+
| **Agent** | Rusty (Lead) |
6+
| **Routed because** | Code review of PR #103 (Wallace's azd extension) |
7+
| **Mode** | background |
8+
| **Files authorized** | PR #103 diff (read-only) |
9+
| **Files produced** | PR review comment on #103 (approved), decisions/inbox/rusty-pr103-review.md |
10+
| **Outcome** | ✅ Approved. Clean, purely additive, aligns with E7. Non-blocking notes about registry URLs and trailing newlines. |
11+
| **PR** | #103 |

0 commit comments

Comments
 (0)