ci: auto-create GitHub release when release PR is merged#3023
Conversation
Add create-release.yml workflow that triggers when a rel-* branch PR is merged into main. It extracts the version from the branch name, creates a GitHub release with auto-generated notes, which then chains into the existing pypi-release and version-bump-prs workflows. Also update prepare-release.yml PR template and the sdk-release skill to reflect the fully automated post-merge pipeline. Co-authored-by: openhands <openhands@all-hands.dev>
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED |
Using github.event.pull_request.merge_commit_sha ensures the tag points at exactly the release PR merge commit, even if other PRs land on main before the workflow runs. Co-authored-by: openhands <openhands@all-hands.dev>
Use --notes-start-tag with the latest existing release tag so auto-generated notes compare exactly from the previous release, matching the behavior of manually selecting 'Previous tag' in the GitHub UI. Co-authored-by: openhands <openhands@all-hands.dev>
✅ End-to-end test passed in xingyaoww/release-expTested the full flow: Workflow run
Verified
Workflow log highlightsThis comment was created by an AI agent (OpenHands) on behalf of the user. |
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
Clean automation that removes a manual release step. The workflow is well-designed with proper validation, idempotency, and error handling. One minor documentation note inline about Slack notifications.
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
✅ QA Report: PASS
This PR successfully automates GitHub release creation. The workflow correctly triggers on merged release PRs, validates inputs, creates releases with auto-generated notes, and chains into the existing PyPI publication pipeline.
Does this PR achieve its stated goal?
Yes. The PR eliminates the manual "create GitHub release" step that previously took ~25 minutes between merging the release PR and PyPI publication. The new create-release.yml workflow automatically detects merged rel-* PRs, extracts the version, validates the format, checks for duplicate releases (idempotent), and creates the release with the correct target commit and auto-generated notes. The workflow integrates correctly with pypi-release.yml (which triggers on release.published) and the documentation accurately reflects the new automated flow.
| Phase | Result |
|---|---|
| Environment Setup | ✅ gh CLI v2.89.0 available, YAML syntax valid |
| CI Status | ✅ All core checks passing (1 unrelated docker build failure) |
| Functional Verification | ✅ All workflow logic verified |
Functional Verification
Test 1: Version Extraction and Validation
Baseline — Understanding the requirement:
The workflow must extract version numbers from branch names like rel-1.19.1 and validate they match the semver format X.Y.Z. Invalid formats should fail early.
Verification:
Ran version extraction script testing 5 scenarios:
/tmp/test_version_extraction.shOutput:
=== Testing Version Extraction Logic ===
✅ Test 1 PASSED: Extracted version '1.19.1' from branch 'rel-1.19.1'
✅ Test 2 PASSED: Extracted version '2.0.0' from branch 'rel-2.0.0'
✅ Test 3 PASSED: Correctly rejected invalid version '1.19' from branch 'rel-1.19'
✅ Test 4 PASSED: Correctly rejected invalid version 'alpha' from branch 'rel-alpha'
✅ Test 5 PASSED: Correctly rejected non-release branch 'feature-foo'
=== All version extraction tests passed! ===
This confirms the regex ^[0-9]+\.[0-9]+\.[0-9]+$ correctly validates semver versions and rejects malformed input.
Test 2: GitHub CLI Commands
Baseline — Command syntax must be valid:
The workflow uses gh release list, gh release view, and gh release create with specific flags. These commands must exist and accept the provided arguments.
Verification:
Tested each command pattern used in the workflow:
# Test 1: List releases to find previous tag
gh release list --repo OpenHands/software-agent-sdk \
--exclude-drafts --exclude-pre-releases --limit 1 \
--json tagName --jq '.[0].tagName'Output: v1.19.1 ✅
# Test 2: Check if a release exists (existing release)
gh release view "v1.19.1" --repo OpenHands/software-agent-sdk > /dev/null 2>&1 && echo "exists"Output: exists ✅
# Test 3: Check if a release exists (non-existent release)
gh release view "v99.99.99" --repo OpenHands/software-agent-sdk > /dev/null 2>&1 || echo "does not exist"Output: does not exist ✅
# Test 4: Verify gh release create flags
gh release create --help | grep -E "(--target|--title|--generate-notes|--notes-start-tag)"Confirmed all flags exist and match the workflow's usage ✅
All commands execute successfully with correct syntax.
Test 3: NOTES_FLAG Array Logic
Baseline — Optional flag handling:
The workflow conditionally adds --notes-start-tag <PREV_TAG> only when a previous release exists. For the first release, the flag should be omitted.
Verification:
Tested bash array expansion logic:
# Test with PREV_TAG set
PREV_TAG="v1.19.0"
NOTES_FLAG=()
if [ -n "$PREV_TAG" ]; then
NOTES_FLAG=(--notes-start-tag "$PREV_TAG")
fi
echo "${NOTES_FLAG[*]}"Output: --notes-start-tag v1.19.0 ✅
# Test with PREV_TAG empty
PREV_TAG=""
NOTES_FLAG=()
if [ -n "$PREV_TAG" ]; then
NOTES_FLAG=(--notes-start-tag "$PREV_TAG")
fi
echo "${NOTES_FLAG[*]}"Output: `` (empty, as expected) ✅
The array expansion "${NOTES_FLAG[@]}" correctly handles both cases without syntax errors.
Test 4: Idempotency
Baseline — Prevent duplicate releases:
If a release already exists (e.g., from a manual creation or workflow re-run), the workflow should skip release creation gracefully.
Verification:
VERSION="1.19.1"
if gh release view "v${VERSION}" --repo OpenHands/software-agent-sdk > /dev/null 2>&1; then
echo "Release exists, would skip"
fiOutput: Release exists, would skip ✅
The workflow's steps.check.outputs.exists condition correctly prevents duplicate releases.
Test 5: Integration with pypi-release Workflow
Baseline — Workflow chaining:
The new workflow must trigger the existing pypi-release.yml workflow, which is configured to run on release.types: [published].
Verification:
Checked pypi-release.yml trigger:
grep -A5 "^on:" .github/workflows/pypi-release.ymlOutput:
on:
workflow_dispatch:
release:
types: [published]Confirmed the chain: create-release.yml creates a release → GitHub emits release.published event → pypi-release.yml runs automatically ✅
Test 6: Complete Workflow Simulation
Baseline — End-to-end execution:
Simulated what happens when a rel-1.20.0 PR is merged.
Verification:
/tmp/test_complete_workflow.shOutput:
=== Simulating Complete Workflow Execution ===
Step 1: Extract version from branch name
✅ Extracted version: 1.20.0
Step 2: Check if release already exists
✅ Release v1.20.0 does not exist, proceeding
Step 3: Find previous release tag
✅ Previous tag: v1.19.1
Step 4: Prepare release creation command
Would execute:
gh release create v1.20.0 \
--repo OpenHands/software-agent-sdk \
--target <MERGE_COMMIT_SHA> \
--title v1.20.0 \
--generate-notes \
--notes-start-tag v1.19.1
✅ Command syntax is valid
=== Workflow simulation completed successfully! ===
All steps execute in the correct order with valid logic ✅
Test 7: Before/After Comparison
Before this PR:
Checked recent release PR #3019 (rel-1.19.1):
- PR merged:
2026-04-30T16:41:16Z - Release created:
2026-04-30T17:06:25Z(by xingyaoww) - Manual delay: 25.1 minutes
After this PR:
The workflow triggers immediately on merge, eliminating the manual step and 25-minute delay.
Documentation changes verified:
prepare-release.ymlPR template: Manual checklist items replaced with "What happens on merge" section explaining automation ✅sdk-releaseskill: Phase 6 changed from "Create and Publish the GitHub Release" (manual) to "Automated Release Pipeline (no action needed)" ✅- Checklist updated: Manual "create GitHub release" step marked as
_(Automated)_✅
Issues Found
None.
Move Slack notification out of the automated Phase 6 pipeline and into a new Phase 7 (Post-Release Announcements) as a human-performed step. Update the Quick Reference checklist accordingly. Co-authored-by: openhands <openhands@all-hands.dev>
Keep the copy-pasteable Slack message template so the human can easily compose the post-release announcement. Co-authored-by: openhands <openhands@all-hands.dev>
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
🟡 Acceptable - Solid automation design with one misleading output issue and a minor validation observation.
[IMPROVEMENT OPPORTUNITIES]
- Summary step should be conditional to avoid misleading output
[RISK ASSESSMENT]
- [Overall PR]
⚠️ Risk Assessment: 🟡 MEDIUM
This PR shifts the release safety checkpoint from "create GitHub release" to "merge release PR". Once merged, the full pipeline (GitHub release → PyPI publish → version bump PRs) executes automatically. The risk is mitigated by:
- Explicit warnings in the skill doc emphasizing that merging is effectively irreversible
- Idempotent release creation (won't duplicate if run multiple times)
- Standard GitHub APIs and safe workflow patterns
The MEDIUM rating reflects that this changes a production release workflow and removes a manual checkpoint, but the design is sound and includes appropriate safeguards.
VERDICT:
✅ Worth merging after addressing the Summary step issue
KEY INSIGHT:
Automating release creation is a solid workflow improvement, but the Summary step needs to be conditional to accurately reflect whether a release was created or skipped due to idempotency.
| echo "" >> "$GITHUB_STEP_SUMMARY" | ||
| echo "- **Tag**: v${VERSION}" >> "$GITHUB_STEP_SUMMARY" | ||
| echo "- **Release**: https://github.com/${{ github.repository }}/releases/tag/v${VERSION}" >> "$GITHUB_STEP_SUMMARY" | ||
| echo "" >> "$GITHUB_STEP_SUMMARY" |
There was a problem hiding this comment.
🟠 Important: The Summary step runs unconditionally, even when the release already exists and was skipped. This produces misleading output saying "Release v{VERSION} Created" when it was actually skipped.
Fix: Add a condition to this step:
- name: Summary
if: steps.check.outputs.exists == 'false'
env:
VERSION: ${{ steps.version.outputs.version }}
run: |Or add a conditional message that says "Release v{VERSION} already exists, skipped" when steps.check.outputs.exists == 'true'.
| BRANCH="${{ github.event.pull_request.head.ref }}" | ||
| VERSION="${BRANCH#rel-}" | ||
|
|
||
| if ! [[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then |
There was a problem hiding this comment.
🟡 Suggestion: The version regex ^[0-9]+\.[0-9]+\.[0-9]+$ only accepts semantic versions with exactly 3 parts (e.g., 1.19.1).
If the project ever uses pre-release versions like 1.0.0-rc1 or 1.0.0-alpha.1, this will fail.
Given the current versioning scheme uses simple semantic versions, this is probably fine. Just noting it for future reference in case versioning conventions change.
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
✅ QA Report: PASS WITH LIMITATION
All workflow components verified successfully through functional testing; full end-to-end verification requires a real release PR merge.
Does this PR achieve its stated goal?
Yes, with high confidence based on component testing. The PR successfully automates GitHub release creation when a release PR is merged. I verified all individual workflow components function correctly: version extraction, validation, release existence checks, previous release tag detection, and GitHub CLI command syntax. The workflow is now live in the repository and correctly triggered (then skipped) when this PR was merged to main, confirming the trigger mechanism works. While I could not execute the full release creation flow (which would require merging an actual rel-* branch and creating a real release), all testable components demonstrate the workflow will function as designed.
| Phase | Result |
|---|---|
| Environment Setup | ✅ Repository checked out, gh CLI available |
| CI Status | ✅ 21 checks passing, 1 unrelated review thread check failing, 7 builds pending |
| Functional Verification | ✅ All workflow components verified, integration confirmed |
Functional Verification
Test 1: Version Extraction and Validation
Step 1 — Test the workflow's version extraction logic:
Ran a test script simulating the workflow's bash logic for extracting versions from release branch names:
#!/bin/bash
BRANCH="rel-1.19.1"
VERSION="${BRANCH#rel-}"
if ! [[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "Invalid version"
exit 1
fi
echo "Version: $VERSION"Output:
PASS: Branch 'rel-1.19.1' → Version '1.19.1'
PASS: Branch 'rel-0.0.1' → Version '0.0.1'
PASS: Branch 'rel-10.20.30' → Version '10.20.30'
PASS: Correctly rejected 'rel-1.19' (missing patch version)
PASS: Correctly rejected 'rel-v1.19.1' (has 'v' prefix)
PASS: Correctly rejected 'feature-branch' (not a release branch)
✅ All version extraction tests passed!
Interpretation: The version extraction logic correctly handles valid semver formats and rejects invalid patterns, confirming the workflow will properly validate release branch names.
Test 2: Release Existence Check (Idempotency)
Step 1 — Verify the workflow can detect existing releases:
Ran gh release view commands to test the idempotency logic:
VERSION="1.19.1"
gh release view "v${VERSION}" --repo "OpenHands/software-agent-sdk" > /dev/null 2>&1
echo "exists=true"Output:
✅ PASS: Successfully detected that release v1.19.1 exists
✅ PASS: Correctly identified that release v999.999.999 does not exist
✅ All idempotency tests passed!
Interpretation: The workflow's idempotency check correctly identifies existing releases and would skip duplicate creation, preventing errors from re-running the workflow.
Test 3: Previous Release Tag Detection
Ran the workflow's command to find the latest release:
PREV_TAG=$(gh release list --repo "OpenHands/software-agent-sdk" \
--exclude-drafts --exclude-pre-releases --limit 1 \
--json tagName --jq '.[0].tagName')
echo "Previous release tag: ${PREV_TAG}"Output:
📌 Previous release tag: v1.19.1
✅ PASS: Found previous release tag: v1.19.1
Interpretation: The workflow successfully retrieves the previous release tag, which will be used for generating accurate release notes that cover changes since the last release.
Test 4: GitHub CLI Command Syntax Validation
Verified all flags used in the workflow are valid:
gh release create --help | grep -E "(--target|--title|--generate-notes|--notes-start-tag)"Output:
--generate-notes Automatically generate title and notes for the release
--notes-start-tag string Tag to use as the starting point for generating release notes
--target branch Target branch or full commit SHA
-t, --title string Release title
Interpretation: All flags used in the gh release create command are valid and supported by the GitHub CLI.
Test 5: Workflow Trigger Conditions
Simulated the workflow's trigger conditions:
if [ "$pr_merged" = "true" ] && [[ "$head_ref" == rel-* ]]; then
echo "TRIGGER"
else
echo "SKIP"
fiOutput:
✅ PASS: merged=true, head.ref=rel-1.19.1 → TRIGGER
✅ PASS: merged=false, head.ref=rel-1.19.1 → SKIP
✅ PASS: merged=true, head.ref=feature/add-something → SKIP
✅ All workflow trigger condition tests passed!
Interpretation: The workflow trigger logic correctly identifies when to run (merged release PRs) and when to skip (closed-but-not-merged PRs, non-release branches).
Test 6: Workflow YAML Syntax
Validated workflow syntax:
yamllint -d relaxed .github/workflows/create-release.ymlOutput:
✅ PASS: YAML syntax is valid
Interpretation: The workflow file is syntactically correct and will be parsed properly by GitHub Actions.
Test 7: Integration with Existing Pipeline
Verified pypi-release.yml trigger configuration:
grep -A 10 "^on:" .github/workflows/pypi-release.ymlOutput:
on:
workflow_dispatch:
release:
types: [published]Interpretation: The pypi-release.yml workflow is configured to trigger on release: types: [published], confirming it will automatically run when create-release.yml publishes a new release. The pipeline chain is correctly established:
merge rel-* PR → create-release.yml → pypi-release.yml → version-bump-prs.yml
Test 8: Workflow Permissions
Verified required permissions:
grep -A 2 "permissions:" .github/workflows/create-release.ymlOutput:
permissions:
contents: writeInterpretation: The workflow has the required contents: write permission to create releases.
Test 9: Live Workflow Trigger Verification
After this PR was merged to main, checked if the workflow triggered:
gh run list --repo OpenHands/software-agent-sdk \
--workflow "Create GitHub Release" --limit 5Output:
{
"conclusion": "skipped",
"createdAt": "2026-05-04T14:05:22Z",
"event": "pull_request",
"headBranch": "ci/auto-create-release-on-merge"
}Interpretation: The workflow was successfully triggered when this PR was merged (4 seconds after merge at 14:05:18Z), and correctly skipped execution because the head branch ci/auto-create-release-on-merge does not start with rel-. This confirms the trigger mechanism and branch filtering work correctly in production.
Test 10: Documentation Consistency
Verified all documentation updates align with the automation:
prepare-release.ymlPR template — Removed manual release creation checklist items, added "What happens on merge" section describing automationsdk-releaseskill — Updated Phase 6 from "Create and Publish the GitHub Release" to "Automated Release Pipeline (no action needed)", removed human approval checkpoints for release creation- Workflow description comments — Clearly document the workflow's purpose and integration point
Interpretation: All documentation consistently reflects the automated workflow and correctly removes manual release creation instructions.
Unable to Verify
Full End-to-End Release Creation
What could not be verified: Actually creating a GitHub release by merging a real rel-* branch PR.
What was attempted: I verified all individual components (version extraction, validation, release checks, CLI syntax, trigger conditions, permissions, integration points) and confirmed the workflow is live and correctly triggered when this PR was merged. However, actually executing the release creation step would require:
- Creating a
rel-X.Y.Zbranch with real version bumps - Opening a release PR
- Merging that PR to main
- Observing the workflow create a GitHub release
This would create a real release in the repository, which I should not do without explicit permission.
Confidence level: Very high. All testable components work correctly, the workflow is syntactically valid, the trigger mechanism is confirmed working in production, and the integration with pypi-release.yml is properly configured. The only untested step is the final gh release create execution, but since all its inputs (version, target commit, previous tag, flags) are verified correct, there is no reason to expect failure.
Suggested AGENTS.md guidance for future QA runs:
## QA Testing GitHub Actions Workflows
For workflow changes that trigger on specific events (e.g., PR merge, release publish):
1. Validate YAML syntax with `yamllint`
2. Test individual bash/shell logic in isolation
3. Verify GitHub CLI commands with `--help` to confirm flag validity
4. Check workflow trigger conditions with simulated variables
5. For workflows already merged: verify the workflow triggered correctly using `gh run list`
6. If full end-to-end testing requires destructive actions (creating releases, publishing packages), verify all components individually and document limitationsIssues Found
None.
3 participants
Summary
Automates the manual "create GitHub release" step that currently sits between merging the release PR and PyPI publication.
New workflow:
create-release.ymlTriggers when a
rel-*branch PR is merged intomain. It:rel-1.19.1→1.19.1)v<version>, targetingmain, with auto-generated release notesThis chains into the existing pipeline:
Other changes
prepare-release.yml: Updated the PR body template to explain the automated post-merge flow instead of listing manual release creation stepssdk-releaseskill: Updated to document the automated pipeline; checklist now marks post-merge steps as (Automated)Safety consideration
The safety guardrail shifts entirely to the merge decision — once the PR is merged, the full release pipeline fires automatically. The skill doc reflects this by emphasizing human approval at the merge step.
This PR was created by an AI agent (OpenHands) on behalf of the user.
@xingyaoww can click here to continue refining the PR
Agent Server images for this PR
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22-slimgolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:04e850d-pythonRun
All tags pushed for this build
About Multi-Architecture Support
04e850d-python) is a multi-arch manifest supporting both amd64 and arm6404e850d-python-amd64) are also available if needed