Skip to content

auto-label: teach prompt about area::ci + test coverage ≠ documentation#2129

Open
ramkrishna2910 wants to merge 1 commit into
mainfrom
auto-label-prompt-area-ci
Open

auto-label: teach prompt about area::ci + test coverage ≠ documentation#2129
ramkrishna2910 wants to merge 1 commit into
mainfrom
auto-label-prompt-area-ci

Conversation

@ramkrishna2910
Copy link
Copy Markdown
Contributor

Follow-up to #2076.

After running the first backfill of open items via #2076's workflow, two small gaps showed up in the prompt:

1. CI / infrastructure items had no label

Four open items (#2116, #2037, #1960, plus one more on the website side) were about CI workflows, runner cleanup, or test infrastructure. None of area::cli, area::installer, area::api, area::tray fit, so the model returned (none).

This PR:

  • Adds area::ci to the Area block of SYSTEM_PROMPT with explicit coverage for: CI / GitHub Actions workflows, self-hosted runner infrastructure, and test infrastructure (fixtures, harness, CI cleanup).
  • Adds "area::ci" to KNOWN_LABELS so the parser admits it. The area::ci label already exists on the repo (created out-of-band during the backfill triage).
  • Verified the at-most-one-area:: rule still applies: area::ci, area::api → just area::ci passes through; area::ci against an existing area::* is dropped.

2. documentation was over-applied to test-coverage items

The model labeled #2032 and #2040 with documentation even though they were a quality bug and a test-coverage gap respectively. The connection seems to be the model treating "documents the missing coverage" or "missing test docs" loosely.

This PR adds an explicit rule to the prompt:

documentation is for human-readable docs only — READMEs, user guides, installation instructions, doc-comments. Items about test coverage, missing tests, test infrastructure, or test harness changes are enhancement (or bug if the test itself is broken), NEVER documentation.

Test plan

  • Local parser verification — 5 cases covering area::ci admission, at-most-one-area:: enforcement, and co-occurrence with engine + type labels.
  • After merge, no fresh backfill needed — the schedule cron + event triggers will exercise the new prompt on the next opened issue / commented item. Optionally dispatch a dry-run on an area::ci-shaped existing issue to spot-check.

🤖 Generated with Claude Code

…tation

Follow-up to #2076.

After running the first backfill of open items, two small gaps showed up:

1. Four CI / infrastructure items (#2116, #2037, #1960, and one more)
   had no place to land. Created `area::ci` on the repo and now teach
   the prompt about it, including covering test infrastructure (CI
   fixtures, harness changes, runner cleanup).

2. The model applied `documentation` to two test-coverage items
   (#2032, #2040), interpreting "missing test coverage" or "documents
   the gap" loosely. Add an explicit rule that `documentation` is for
   human-readable docs only, and test coverage / infrastructure items
   are `enhancement` (or `bug` if a test is broken), never
   `documentation`.

Also adds `area::ci` to KNOWN_LABELS so the parser admits it. Verified
the at-most-one-area rule still applies (`area::ci, area::api` → just
`area::ci` passes through; `area::ci` against an existing area::* is
dropped).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added enhancement New feature or request documentation Improvements or additions to documentation labels Jun 7, 2026
@ramkrishna2910 ramkrishna2910 requested a review from fl0rianr June 7, 2026 04:12
Copy link
Copy Markdown
Collaborator

@fl0rianr fl0rianr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks good code-wise; this is a clean follow-up to #2076.

One small note on the test plan / rollout wording: schedule and issue_comment runs are priority-only, so they will not exercise the updated LLM prompt for already-open items. Existing issues that previously got (none) because area::ci did not exist would need a manual workflow_dispatch in all mode, either targeted or as a fresh backfill. Newly opened issues/PRs will pick this up automatically.

I don't think this blocks the code change, but it might be worth adjusting the PR description/test plan so the rollout expectation is accurate. Your call.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants