Skip to content

Conversation

ascorbic
Copy link
Contributor

@ascorbic ascorbic commented Oct 17, 2025

Changes

  • changeset errors don't fail checks – they just add a comment
  • instead of relying on LLM to check heading levels (it's bad at it) it instead uses marked to parse it, along with some awk and jq trickery (thanks Claude)
  • uses pull_request_target for running the tests. This allows it to comment on PRs from forks. As this is more risky, security-wise, this also refactors a lot to make it safer – it checks-out main and just gets the changeset content from the branch so no user-generated code is run. It also adds a few more checks for things like excessively large changesets.
  • rewrote the prompt with some help from Claude. It seems to give more reliable results.

Testing

Tested on a branch using pull_request, before switching to pull_request_target for this PR. As ever with pull_request_target PRs, this part will only be tested when it's actually merged and a new PR opened. Here's the PR with test changesets and comments: ascorbic#2

Example of the output:

image

Docs

Copy link

changeset-bot bot commented Oct 17, 2025

⚠️ No Changeset found

Latest commit: 20a45d6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions github-actions bot added the 🚨 action Modifies GitHub Actions label Oct 17, 2025
- IMPORTANT: Ensure there is a newline before and after the code blocks in your feedback. Having triple backticks on the same line as other text WILL cause rendering issues.
- ALWAYS check your feedback for proper markdown formatting, particularly code fences and backticks.
- Use markdown codeblocks with triple backticks: 'yaml' for YAML examples, 'markdown' for changeset body examples
- IMPORTANT: Ensure there is a newline before and after code blocks. Triple backticks on the same line as other text WILL cause rendering issues
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noting that it's not uncommon for us to have triple backticks and diff on the same line! (Niche case, but not sure whether worth mentioning here)

Copy link
Member

@sarah11918 sarah11918 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving for the updated validation rules! (Noted one possible exception in a comment)

@ascorbic ascorbic merged commit 8f4562b into main Oct 17, 2025
20 checks passed
@ascorbic ascorbic deleted the changesets-fix branch October 17, 2025 12:47
Comment on lines +17 to +18
✓ Correct: "Fixes", "Adds", "Updates", "Removes"
✗ Incorrect: "Fix", "Add", "Update", "Remove", "Fixed", "Added"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do symbols really help here? Geniune question

Comment on lines +46 to +63
```markdown
---
'astro': minor
---

Adds support for custom error pages in middleware. You can now return a custom error response:

\`\`\`js
export function onRequest(context, next) {
if (!context.locals.user) {
return new Response('Unauthorized', { status: 401 });
}
return next();
}
\`\`\`
```

```markdown
Copy link
Member

@ematipico ematipico Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very hard to provide a suggestion with markdown and UI, but usually here you can create "nested" code blocks:

``````md

## Heading

```js
const foo = "bar";
```

```````

- name: Set up Node.js
uses: actions/setup-node@2028fbc5c25fe9cf00d9f06a71cc4710d4507903 # v6.0.0
with:
node-version: '20'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
node-version: '20'
node-version: 'lts/*'

So we don't need to update it

Comment on lines +235 to +276
# Check for shallow headings in each changeset
echo "::debug::Checking for shallow headings (h1, h2, h3) in changesets"
while IFS= read -r file; do
if [ -f "$file" ]; then
# Make path absolute for the script
abs_file="$(pwd)/$file"
# Run the script and capture output
if heading_output=$(pnpm --filter astro-scripts has-shallow-headings "$abs_file" 2>&1); then
echo "::debug::$file: No shallow headings found"
else
# Filter output to only lines that look like markdown headings (start with #)
# This removes pnpm wrapper noise
heading_output=$(pnpm --filter astro-scripts has-shallow-headings "$abs_file" 2>&1 | grep '^#' || true)
if [ -n "$heading_output" ]; then
echo "::warning::$file: Contains forbidden shallow headings"
# Add heading errors to the validation response
# The script outputs the offending headings, one per line
while IFS= read -r heading; do
if [ -n "$heading" ]; then
error_msg="Forbidden heading found: '$heading'. Headings shallower than h4 (####) break changelog formatting. Please use h4 or deeper."
echo "::debug::Adding heading error for $file: $heading"
# Add error to the file's general_errors array in the JSON
jq --arg file "$file" --arg error "$error_msg" '
.overall_valid = false |
.files |= map(
if .file == $file then
.valid = false |
.general_errors += [$error]
else . end
)
' validation_response.json > validation_response.tmp.json
mv validation_response.tmp.json validation_response.json
fi
done <<< "$heading_output"
else
echo "::debug::$file: Script failed but no headings found (might be other error)"
fi
fi
fi
done < changeset_files.txt
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if it's best to have a JS script instead. I doubt any of us will be able to maintain it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🚨 action Modifies GitHub Actions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants