feat(scaffold): per-repo pre-commit tools registry (L2 additive merge) by guyoron1 · Pull Request #83 · guyoron1/fullsend

guyoron1 · 2026-06-25T16:03:21Z

Summary

Extends the pre-commit tools registry (fullsend-ai#1055) with per-repo additive merge:

L2 additive merge: Target repos place .pre-commit-tools.yaml at repo root to extend the upstream/org registry. New entries append, matching (repo, hook_id) entries override, exclude: true suppresses.
Supply-chain security: Per-repo registries are read from the base branch (git show origin/${TARGET_BRANCH}:...), not the PR head. Changes take effect only after merge.
P1 fix: Add uv match entry so hooks with entry: "uv run ..." are recognized alongside uvx.
P2 fix: Document that shellcheck-py is auto-managed by pre-commit (no registry entry needed).

Three-layer resolution

upstream defaults (fullsend-ai/fullsend)
  → org override: customized/scripts/.pre-commit-tools.yaml  (L1, full replacement)
    → per-repo additive: .pre-commit-tools.yaml at repo root (L2, additive merge)

Changes

File	Change
`docs/ADRs/0056-per-repo-precommit-tools-registry.md`	New ADR documenting design, security model, merge semantics
`resolve-precommit-tools.py`	Refactor `resolve()` to accept dict, add `merge_registries()`, add `--local-registry` CLI arg
`.pre-commit-tools.yaml`	Add `uv` match entry (P1), shellcheck-py note (P2), update header docs
`post-code.sh`, `post-fix.sh`, `pre-code.sh`, `pre-fix.sh`	Extract base-branch registry and pass `--local-registry`
`customizing-agents.md`	New "Customizing Pre-commit Tool Dependencies" section
`test_resolve_precommit_tools.py`	13 unit tests for merge, exclude, dedup, uv match, malformed input

Test plan

python3 internal/scaffold/scripts-tests/test_resolve_precommit_tools.py — 13/13 pass
go test ./internal/scaffold/... — existing Go tests pass
make lint — passes
Manual: hook with entry: "uv run mypy" matches the uv registry entry
Manual: exclude: true on gitleaks suppresses the upstream entry
Manual: local registry extends upstream without replacing it

Closes fullsend-ai#1270

Move max_turns and max_cost_usd checks from custom shell code in run-functional.sh into deterministic check judges in eval.yaml. The harness's score.py now enforces these via min_pass_rate: 1.0 thresholds. Extract the pre-flight annotation validation into a standalone eval/lint-cases.sh linter, wired up as `make lint-eval-cases` and included in `make test`. This runs cheaply without executing agents. Net effect: ~90 lines removed from run-functional.sh. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

Extend lint-cases.sh to verify that eval.yaml declares max_turns and max_cost judges, not just that annotations.yaml declares the thresholds. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

The CLI runner receives {output_dir} which is workspace/output. Writing metrics.json to $OUTPUT_DIR/output/ created a double-nested path that score.py couldn't find. Write to $OUTPUT_DIR/metrics.json instead so the file appears at the expected output/metrics.json key. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

The triage agent consistently takes ~23 turns on this case. The previous threshold of 15 was too tight. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

Add three new triage eval cases covering the remaining major outcomes: - 002-needs-info-vague-crash: vague issue with no repro steps, expects action "insufficient" and needs-info label - 003-feature-request: clear feature request, expects action "sufficient" with category "feature" and triaged+feature labels (not ready-to-code) - 004-duplicate-issue: issue duplicating a seed issue, expects action "duplicate" with duplicate label Supporting changes: - setup-fixture.sh: support seed_issues in input.yaml for pre-populating issues before the main fixture (needed by duplicate test) - eval.yaml: add forbidden_labels judge to verify wrong labels are NOT applied (needs-info must not get ready-to-code, etc.) - lint-cases.sh: check for forbidden_labels judge presence Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

Cases use isolated ephemeral repos with UUID suffixes, so there's no shared state. Sequential execution took ~15 min of agent time across 4 cases; parallel should bring wall-clock down to ~6 min. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

ADR 0045 was taken on main by forge-portable-harness-schema while this branch was out of date. Renumber both branch ADRs and update all cross-references. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

ADR 0046-host-side-api-server-design was merged to main while this branch was open, creating a duplicate number. Renumber 0046-functional-tests-for-agent-pipelines to 0048 and update all cross-references. Also fix stale heading numbers in ADRs 0047 and 0048. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

- Write partial metrics.json on agent failure so downstream judges can inspect behavioral data even when the agent errors out - Add CLOUD_ML_REGION to runner.env in eval.yaml (was only in execution.env) - Let teardown-fixture.sh fail visibly instead of swallowing errors with 2>/dev/null || true — the harness on_failure: continue already handles non-fatal teardown - Distinguish infrastructure failures from agent failures in run-functional.sh by checking for case output before proceeding to scoring - Fix stale reference to eval/fullsend-runner.sh in ADR 0047 (actual path is eval/scripts/run-fullsend.sh) Signed-off-by: Ralph Bean <rbean@redhat.com> Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

Main acquired ADR numbers 0047-0050 while this branch was open. Renumber the PR's ADRs and update all cross-references. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

feat: add functional test framework for agent pipelines

Wrap each script-test invocation in a run-timed macro that emits ::debug:: annotations with elapsed seconds. Hidden by default; visible when re-running a job with "Enable debug logging" checked. Local run shows post-prioritize-test.sh dominates at ~213s of ~225s total. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

The test set GITHUB_CSMA_SLOT_MAX_MS=0 and GITHUB_CSMA_BACKOFF_CAP_SEC=1 but forgot GITHUB_CSMA_SPREAD_MAX_SEC, which defaults to 60. Every retry called _github_csma_post_reset_spread sleeping up to 60s, totaling ~213s of pointless sleep across the retry test cases. Setting GITHUB_CSMA_SPREAD_MAX_SEC=0 drops the test from ~213s to ~10s. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

The code agent hardcodes TARGET_BRANCH to "main", which breaks repos whose default branch has a different name (e.g. redhat-3.17, master, develop). The post-code script uses this value for git merge-base and gh pr create --base, so PR creation fails silently on non-main repos. Add a "Resolve target branch" step that: 1. Parses --branch <ref> from the /fs-code comment (explicit override) 2. Falls back to the repo's default branch via the GitHub API 3. Falls back to "main" if the API call fails This follows the same pattern the fix agent already uses to resolve the PR base branch dynamically. Closes fullsend-ai#2345 Signed-off-by: Marcus Kok <mkok@redhat.com>

Propose moving branch-targeting logic from the hardcoded TARGET_BRANCH workflow env to the agent (via structured output) with post-script policy enforcement. This supersedes the workflow-level fix in the previous commit. The agent determines the target branch from issue context, writes it to code-result.json, and the post-script validates it against CODE_ALLOWED_TARGET_BRANCHES before creating the PR. See ADR 0047 for the full decision record. Signed-off-by: Marcus Kok <mkok@redhat.com>

…en-scaffold fix(scaffold): use __FULLSEND_AI_REF__ for mint-token action refs

Scope automerge to patch bumps and pin updates via packageRules. Combined with repo-level allow_auto_merge and the ruleset bypass for renovate-fullsend, these low-risk updates merge automatically after CI passes. Closes fullsend-ai#2506 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Hector Martinez <hemartin@redhat.com>

…te-automerge feat(renovate): enable automerge for low-risk PRs

Add a dedicated macOS runner in lint.yml to exercise darwin-specific sandbox behavior. Add TestUploadDir_SuppressesAppleDoubleInTarball on darwin: verifies COPYFILE_DISABLE=1 prevents ._* files in tarballs using python3 tarfile inspection, with a negative control to confirm xattr application actually triggers AppleDouble generation without the flag. Signed-off-by: Hector Martinez <hemartin@redhat.com>

…test ci(sandbox): add macOS runner for darwin-specific tar behavior test

…ools feat(scaffold): auto-detect and install pre-commit tool dependencies

…r-based-changelog fix(ci): use GitHub PR titles for release notes

…path-filter fix(ci): skip e2e/functional tests in merge queue for irrelevant paths

Documents the L2 additive merge design: per-repo .pre-commit-tools.yaml at repo root extends upstream/org defaults. Covers three-layer resolution order, merge semantics (extend/override/exclude), and base-branch-only reads for supply-chain security. Relates: fullsend-ai#1270 Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>

…erge Refactor resolve() to accept a parsed dict instead of a file path. Add merge_registries() that merges a per-repo .pre-commit-tools.yaml with upstream/org defaults: new entries extend, matching (repo, hook_id) entries override, and exclude: true suppresses. Add --local-registry CLI arg. Caller scripts extract the base-branch registry via git show (not the PR head) for supply-chain safety and pass it to the resolver. Also fixes fullsend-ai#1270 P1: add uv match_entry so hooks with entry "uv run ..." are recognized alongside the existing "uvx" match. Closes: fullsend-ai#1270 Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>

Add "Customizing Pre-commit Tool Dependencies" section to the customizing-agents guide. Covers three-layer resolution, examples for adding and suppressing entries, and the base-branch security model. Add unit tests for merge_registries() and resolve() covering extend, override, exclude, dedup, uv/uvx match, malformed input, and end-to-end merged resolution. Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>

guyoron1 · 2026-06-25T16:06:41Z

/fs-qf

fullsend-ai-review · 2026-06-25T16:06:44Z

🤖 Finished Review · ✅ Success · Started 4:06 PM UTC · Completed 4:09 PM UTC
Commit: db171e5 · View workflow run →

fullsend-ai-review · 2026-06-25T16:09:10Z

Review

Reason: token-limit

This PR was NOT reviewed. Do not count this as an approval.

Previous run

Review

Reason: token-limit

This PR was NOT reviewed. Do not count this as an approval.

Previous run (2)

Review

Reason: stale-head

The review agent reviewed commit 4e4e558ab9f7e909aedfc18757caca62ebdd4e44 but the PR HEAD is now 06fa1cf3aa1db283dc0ef325b86bf8a55dc3d02e. This review was discarded to avoid approving unreviewed code.

Previous run (3)

Review

Reason: token-limit

This PR was NOT reviewed. Do not count this as an approval.

Previous run (4)

Review

Reason: token-limit

This PR was NOT reviewed. Do not count this as an approval.

github-actions · 2026-06-25T16:09:26Z

QualityFlow Pipeline Summary

Stage	Agent	Status
1	STP Builder	✅
2	STP Reviewer	✅
3	STP Refiner	✅
4	STD Builder	✅
5	STD Reviewer	✅
6	STD Refiner	✅
7	Test Generator	✅

Test Output

Language	Count	Location
Go	5 files	`internal/scaffold/qf_*_test.go`

Issue: GH-1270

Generated by QualityFlow

guyoron1 · 2026-06-25T16:30:33Z

/fs-qf

fullsend-ai-review · 2026-06-25T16:42:02Z

🤖 Finished Review · ✅ Success · Started 4:42 PM UTC · Completed 4:44 PM UTC
Commit: 2719e94 · View workflow run →

fullsend-ai-review · 2026-06-25T16:47:10Z

🤖 Finished Review · ✅ Success · Started 4:47 PM UTC · Completed 5:06 PM UTC
Commit: 2719e94 · View workflow run →

fullsend-ai-review · 2026-06-25T17:06:46Z

/fs-review

fullsend-ai-review · 2026-06-25T17:09:40Z

🤖 Finished Review · ✅ Success · Started 5:09 PM UTC · Completed 5:33 PM UTC
Commit: 2719e94 · View workflow run →

…p ci] - Fixed metadata count mismatch (unit: 24→26, functional: 10→8) - Removed related_prs from STD document_metadata (belongs in STP) - Fixed PSE classification: moved verification from Steps to Expected - Added [NEGATIVE] tags to 5 negative-behavior test stubs - Review verdict: APPROVED_WITH_FINDINGS (0 critical, 0 major)

fullsend-ai-review · 2026-06-25T17:36:32Z

🤖 Finished Review · ✅ Success · Started 5:36 PM UTC · Completed 5:38 PM UTC
Commit: 2719e94 · View workflow run →

Removes intermediate pipeline artifacts (STP, STD, reviews). Test files (5) are co-located in source tree with qf_ prefix. Jira: fullsend-aiGH-1270 [skip ci]

ralphbean and others added 30 commits June 22, 2026 15:06

fix(eval): raise max_turns threshold to 30 for triage case

9041b54

The triage agent consistently takes ~23 turns on this case. The previous threshold of 15 was too tight. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

fix(eval): raise max_turns threshold to 30 for triage case

8129761

The triage agent consistently takes ~23 turns on this case. The previous threshold of 15 was too tight. Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

chore: fix gofmt in claude_progress.go

3e5e578

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

chore: fix gofmt in claude_progress.go

a1dfdc2

Assisted-by: Claude claude-opus-4-6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

fix(docs): renumber ADRs 0047/0048 to 0051/0052 to avoid collision

8299019

Main acquired ADR numbers 0047-0050 while this branch was open. Renumber the PR's ADRs and update all cross-references. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

fix(docs): renumber ADRs 0047/0048 to 0051/0052 to avoid collision

722af5c

Main acquired ADR numbers 0047-0050 while this branch was open. Renumber the PR's ADRs and update all cross-references. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ralph Bean <rbean@redhat.com>

Merge pull request fullsend-ai#1682 from fullsend-ai/ci/functional-evals

2893fcc

feat: add functional test framework for agent pipelines

Merge pull request fullsend-ai#1682 from fullsend-ai/ci/functional-evals

e66df00

feat: add functional test framework for agent pipelines

waynesun09 and others added 11 commits June 24, 2026 21:32

Merge pull request fullsend-ai#2628 from fullsend-ai/fix-pin-mint-tok…

db7892b

…en-scaffold fix(scaffold): use __FULLSEND_AI_REF__ for mint-token action refs

Merge pull request fullsend-ai#2546 from fullsend-ai/feat/2506-renova…

5b6d704

…te-automerge feat(renovate): enable automerge for low-risk PRs

Merge pull request fullsend-ai#2063 from fullsend-ai/fix/macos-files-…

bd8a21e

…test ci(sandbox): add macOS runner for darwin-specific tar behavior test

Merge pull request fullsend-ai#1055 from fullsend-ai/auto-precommit-t…

397e330

…ools feat(scaffold): auto-detect and install pre-commit tool dependencies

Merge pull request fullsend-ai#2627 from fullsend-ai/fix/goreleaser-p…

780abad

…r-based-changelog fix(ci): use GitHub PR titles for release notes

Merge pull request fullsend-ai#2617 from fullsend-ai/fix/merge-queue-…

f502487

…path-filter fix(ci): skip e2e/functional tests in merge queue for irrelevant paths

QualityFlow added 2 commits June 25, 2026 16:39

Add QualityFlow output for fullsend-aiGH-1270 [skip ci]

14ba854

Add STP output for fullsend-aiGH-1270 [skip ci]

9ceeaf4

QualityFlow added 2 commits June 25, 2026 16:47

Add QualityFlow output for fullsend-aiGH-1270 [skip ci]

4e4e558

Add QualityFlow output for fullsend-aiGH-1270 [skip ci]

06fa1cf

Add QualityFlow output for fullsend-aiGH-1270 [skip ci]

cbd8fde

QualityFlow added 2 commits June 25, 2026 17:15

Add QualityFlow output for fullsend-aiGH-1270 [skip ci]

27cb049

QualityFlow and others added 2 commits June 25, 2026 17:41

Add QualityFlow tests for fullsend-aiGH-1270 [skip ci]

7beecf5

Clean QualityFlow artifacts for fullsend-aiGH-1270

3537246

Removes intermediate pipeline artifacts (STP, STD, reviews). Test files (5) are co-located in source tree with qf_ prefix. Jira: fullsend-aiGH-1270 [skip ci]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(scaffold): per-repo pre-commit tools registry (L2 additive merge)#83

feat(scaffold): per-repo pre-commit tools registry (L2 additive merge)#83
guyoron1 wants to merge 550 commits into
mainfrom
mirror-2663-precommit-registry

guyoron1 commented Jun 25, 2026

Uh oh!

guyoron1 commented Jun 25, 2026

Uh oh!

fullsend-ai-review Bot commented Jun 25, 2026 •

edited

Loading

Uh oh!

fullsend-ai-review Bot commented Jun 25, 2026 •

edited

Loading

Review

Review

Review

Review

Uh oh!

github-actions Bot commented Jun 25, 2026 •

edited

Loading

Uh oh!

guyoron1 commented Jun 25, 2026

Uh oh!

fullsend-ai-review Bot commented Jun 25, 2026 •

edited

Loading

Uh oh!

fullsend-ai-review Bot commented Jun 25, 2026 •

edited

Loading

Uh oh!

fullsend-ai-review Bot commented Jun 25, 2026

Uh oh!

fullsend-ai-review Bot commented Jun 25, 2026 •

edited

Loading

Uh oh!

fullsend-ai-review Bot commented Jun 25, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Conversation

guyoron1 commented Jun 25, 2026

Summary

Three-layer resolution

Changes

Test plan

Uh oh!

guyoron1 commented Jun 25, 2026

Uh oh!

fullsend-ai-review Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-review Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review

Review

Review

Review

Review

Uh oh!

github-actions Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

QualityFlow Pipeline Summary

Test Output

Uh oh!

guyoron1 commented Jun 25, 2026

Uh oh!

fullsend-ai-review Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-review Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-review Bot commented Jun 25, 2026

Uh oh!

fullsend-ai-review Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-review Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

fullsend-ai-review Bot commented Jun 25, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jun 25, 2026 •

edited

Loading

github-actions Bot commented Jun 25, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jun 25, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jun 25, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jun 25, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jun 25, 2026 •

edited

Loading