auto-label: teach prompt about area::ci + test coverage ≠ documentation#2129
Open
ramkrishna2910 wants to merge 1 commit into
Open
auto-label: teach prompt about area::ci + test coverage ≠ documentation#2129ramkrishna2910 wants to merge 1 commit into
ramkrishna2910 wants to merge 1 commit into
Conversation
…tation Follow-up to #2076. After running the first backfill of open items, two small gaps showed up: 1. Four CI / infrastructure items (#2116, #2037, #1960, and one more) had no place to land. Created `area::ci` on the repo and now teach the prompt about it, including covering test infrastructure (CI fixtures, harness changes, runner cleanup). 2. The model applied `documentation` to two test-coverage items (#2032, #2040), interpreting "missing test coverage" or "documents the gap" loosely. Add an explicit rule that `documentation` is for human-readable docs only, and test coverage / infrastructure items are `enhancement` (or `bug` if a test is broken), never `documentation`. Also adds `area::ci` to KNOWN_LABELS so the parser admits it. Verified the at-most-one-area rule still applies (`area::ci, area::api` → just `area::ci` passes through; `area::ci` against an existing area::* is dropped). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fl0rianr
approved these changes
Jun 7, 2026
Collaborator
fl0rianr
left a comment
There was a problem hiding this comment.
Thanks, looks good code-wise; this is a clean follow-up to #2076.
One small note on the test plan / rollout wording: schedule and issue_comment runs are priority-only, so they will not exercise the updated LLM prompt for already-open items. Existing issues that previously got (none) because area::ci did not exist would need a manual workflow_dispatch in all mode, either targeted or as a fresh backfill. Newly opened issues/PRs will pick this up automatically.
I don't think this blocks the code change, but it might be worth adjusting the PR description/test plan so the rollout expectation is accurate. Your call.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2076.
After running the first backfill of open items via #2076's workflow, two small gaps showed up in the prompt:
1. CI / infrastructure items had no label
Four open items (#2116, #2037, #1960, plus one more on the website side) were about CI workflows, runner cleanup, or test infrastructure. None of
area::cli,area::installer,area::api,area::trayfit, so the model returned(none).This PR:
area::cito theAreablock ofSYSTEM_PROMPTwith explicit coverage for: CI / GitHub Actions workflows, self-hosted runner infrastructure, and test infrastructure (fixtures, harness, CI cleanup)."area::ci"toKNOWN_LABELSso the parser admits it. Thearea::cilabel already exists on the repo (created out-of-band during the backfill triage).area::rule still applies:area::ci, area::api→ justarea::cipasses through;area::ciagainst an existingarea::*is dropped.2.
documentationwas over-applied to test-coverage itemsThe model labeled #2032 and #2040 with
documentationeven though they were a quality bug and a test-coverage gap respectively. The connection seems to be the model treating "documents the missing coverage" or "missing test docs" loosely.This PR adds an explicit rule to the prompt:
Test plan
area::ciadmission, at-most-one-area::enforcement, and co-occurrence with engine + type labels.area::ci-shaped existing issue to spot-check.🤖 Generated with Claude Code