Skip to content
Draft
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 101 additions & 17 deletions .github/workflows/fix-test-failures.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

51 changes: 43 additions & 8 deletions .github/workflows/fix-test-failures.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@ tools:
safe-outputs:
create-issue:
labels: [test-reliability]
update-issue:
status:
target: "*"
add-comment:
max: 1
target: "*"
Comment thread
jeremymeng marked this conversation as resolved.
Outdated
network:
allowed:
- defaults
Expand Down Expand Up @@ -54,20 +60,34 @@ issue** (https://github.com/Azure/azure-sdk-for-js/issues/37864) for pre-existin
failures.

1. Fetch the body of issue #37864 using the GitHub API.
2. For each failing check run identified later in "Step 1 — Identify Failing
2. Fetch the **comments** on issue #37864 using the GitHub API
(`GET /repos/{owner}/{repo}/issues/37864/comments`). Previous workflow runs
append tracking summaries as comments to this issue; these comments also count
as known pre-existing failures.
3. For each failing check run identified later in "Step 1 — Identify Failing
Packages", check whether the **service directory** (from the check-run name,
e.g. `attestation` in `js - attestation (Build UnitTest ...)`) or the **npm
package name** (from annotations/file paths) and the **error pattern** match an
entry in the known failures list.
3. If a failure matches a known pre-existing issue:
entry in the known failures list (body **or** comments).
4. If a failure matches a known pre-existing issue:
- **Exclude** it from the new GitHub issue entirely.
- Do **not** attempt to reproduce or root-cause it — it is already tracked.
4. If **all** detected failures are known pre-existing issues, **stop immediately** —
do **not** create a GitHub issue. Simply report that all failures are known and
already tracked.
5. If some failures are new and some are known, create the issue for **new failures
- **Check the state of the tracked issue.** Each tracking comment records the
number/link of the issue that was filed for the failure (see Step 6). Fetch
that issue with the GitHub API and inspect its `state`. If the tracked issue
is **closed**, the failure has recurred after the issue was resolved, so
**re-open it**: call `update_issue` with `issue_number` set to the tracked
issue and `status` set to `open`. Add a brief note in the "Additional Notes"
section of any new issue (or in the run summary when no new issue is created)
listing which closed issues were re-opened. If the tracked issue is already
**open**, leave it untouched.
5. If **all** detected failures are known pre-existing issues, **stop immediately** —
do **not** create a GitHub issue. Re-open any closed tracked issues as described
above, then report that all failures are known and already tracked.
6. If some failures are new and some are known, create the issue for **new failures
only**. Add a brief note in the "Additional Notes" section listing which known
failures were excluded and linking to their tracking issues.
failures were excluded and linking to their tracking issues, and which closed
tracked issues were re-opened.

## Step 1 — Identify Failing Packages

Expand Down Expand Up @@ -202,3 +222,18 @@ multiple failures, or packages where CI failed but tests pass locally.>
```

Labels are applied automatically via the `safe-outputs.create-issue` configuration.

## Step 6 — Record Created Issue in Tracking Log

After creating the GitHub issue, add a comment to the **known failures tracking
issue** (#37864) so that future runs can detect this issue and avoid filing a
duplicate. Use `add_comment` on issue #37864 with the following format:

```
<!-- workflow-created-issue -->
**Service(s)/Package(s):** <affected service directories or npm package names, comma-separated>
**Error patterns:** <brief description of the error types seen, e.g. `TypeError: cannot read ...`, `AssertionError`>
**Issue title:** <exact title of the issue just created>
**Issue:** #<number of the issue just created> (record the number/link so future runs can check its state and re-open it if it gets closed while the failure persists)
**Date:** <today's date in YYYY-MM-DD format>
```
Loading