Skip to content

Commit 3834853

Browse files
konardkonard
authored
Merge pull request #262 from link-assistant/issue-261-71e2fe0bd25b
fix(ci): handle crates.io propagation delays and already-exists as success
2 parents e68be6d + b5098bb commit 3834853

6 files changed

Lines changed: 296 additions & 48 deletions

File tree

‎.github/workflows/rust.yml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -262,7 +262,7 @@ jobs:
262262
run: node scripts/publish-to-crates.mjs --should-pull
263263

264264
- name: Create GitHub Release
265-
if: steps.publish.outputs.published == 'true'
265+
if: steps.check.outputs.should_release == 'true' && (steps.publish.outputs.published == 'true' || steps.publish.outcome == 'success')
266266
env:
267267
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
268268
run: |
@@ -272,7 +272,7 @@ jobs:
272272
--prefix "rust-"
273273
274274
- name: Format GitHub release notes
275-
if: steps.publish.outputs.published == 'true'
275+
if: steps.check.outputs.should_release == 'true' && (steps.publish.outputs.published == 'true' || steps.publish.outcome == 'success')
276276
env:
277277
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
278278
run: node scripts/format-github-release.mjs --release-version "${{ steps.current_version.outputs.version }}" --repository "${{ github.repository }}" --commit-sha "${{ github.sha }}" --prefix "rust-"
@@ -339,7 +339,7 @@ jobs:
339339
run: node scripts/publish-to-crates.mjs
340340

341341
- name: Create GitHub Release
342-
if: steps.publish.outputs.published == 'true'
342+
if: steps.publish.outputs.published == 'true' || steps.publish.outcome == 'success'
343343
env:
344344
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
345345
run: |
@@ -349,7 +349,7 @@ jobs:
349349
--prefix "rust-"
350350
351351
- name: Format GitHub release notes
352-
if: steps.publish.outputs.published == 'true'
352+
if: steps.publish.outputs.published == 'true' || steps.publish.outcome == 'success'
353353
env:
354354
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
355355
run: node scripts/format-github-release.mjs --release-version "${{ steps.version.outputs.new_version }}" --repository "${{ github.repository }}" --commit-sha "${{ github.sha }}" --prefix "rust-"

‎docs/case-studies/issue-261/case-study.md‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# Case Study: Issue #261 - Rust CI/CD Pipeline Failure (No Crates Release, No GitHub Release)
2+
3+
## Summary
4+
5+
The Rust CI/CD auto-release pipeline successfully published crate `link-assistant-agent@0.9.2` to crates.io but then failed verification due to crates.io propagation delay. This caused the pipeline to exit with error, preventing GitHub Release creation.
6+
7+
## Timeline of Events
8+
9+
**CI Run:** [#24334488286](https://github.com/link-assistant/agent/actions/runs/24334488286/job/71048047065)
10+
**Date:** 2026-04-13
11+
12+
| Time (UTC) | Event |
13+
|---|---|
14+
| 08:54:25 | Auto Release job starts |
15+
| 08:54:32 | Detects 2 changelog fragments (patch bump) |
16+
| 08:54:34 | Bumps version to 0.9.2, commits and tags `rust-v0.9.2` |
17+
| 08:54:35 | Pushes changes and tags to origin |
18+
| 08:55:42 | `publish-to-crates.mjs` starts, detects crate doesn't exist yet on crates.io |
19+
| 08:55:42 | Publish attempt 1 starts (`cargo publish`) |
20+
| 08:56:19 | `cargo publish` succeeds (exit code 0), waits 5s for propagation |
21+
| 08:56:24 | Verification fails - crate not found on crates.io API yet (propagation delay) |
22+
| 08:56:24 | Script treats verification failure as publish failure, waits 10s |
23+
| 08:56:34 | Publish attempt 2 - gets `error: crate link-assistant-agent@0.9.2 already exists on crates.io index` |
24+
| 08:56:34 | Script matches `error: ` failure pattern, doesn't recognize "already exists" as success |
25+
| 08:56:44 | Publish attempt 3 - same "already exists" error |
26+
| 08:56:44 | Script exits with code 1, `published=false` |
27+
| 08:56:44 | GitHub Release step skipped (gated on `published == 'true'`) |
28+
29+
## Root Causes
30+
31+
### Root Cause 1: Insufficient crates.io propagation wait time
32+
- Script waited only 5 seconds for crates.io API propagation
33+
- The crate was actually published but the API hadn't updated yet
34+
- crates.io can take 10-30+ seconds to propagate depending on load
35+
36+
### Root Cause 2: "already exists" error not recognized as success
37+
- On retry, `cargo publish` returns: `error: crate link-assistant-agent@0.9.2 already exists on crates.io index`
38+
- The `detectPublishFailure()` function matched `error: ` pattern first
39+
- The `crate already uploaded` pattern didn't match this different error wording
40+
- The script had no concept of "already exists on index" being a success case
41+
42+
### Root Cause 3: No verification retries
43+
- Verification was a single check after a fixed 5s delay
44+
- No retry mechanism for verification (only for the publish command itself)
45+
46+
### Root Cause 4: GitHub Release gated solely on publish output
47+
- Workflow condition `steps.publish.outputs.published == 'true'` meant any publish failure blocked GitHub Release
48+
- Even when the crate WAS published, the script exited with error so the output was `false`
49+
50+
### Root Cause 5: No graceful handling of existing GitHub releases
51+
- `create-github-release.mjs` would fail fatally if the release tag already existed
52+
- No recovery path for re-running the pipeline
53+
54+
## Solutions Applied
55+
56+
### Fix 1: Recognize "already exists" as successful publish
57+
- Added `ALREADY_EXISTS_PATTERNS` array with common crates.io "already exists" messages
58+
- `detectAlreadyExists()` checks these patterns before `detectPublishFailure()`
59+
- When detected, script sets `published=true` and `already_published=true`
60+
61+
### Fix 2: Improved verification with retries
62+
- Increased initial propagation delay from 5s to 15s
63+
- Added verification retry loop (3 attempts with 10s between retries)
64+
- If cargo publish exits with code 0 but verification can't confirm, treats as success (trusting cargo's exit code)
65+
66+
### Fix 3: Graceful GitHub Release creation
67+
- `create-github-release.mjs` now catches "already exists" / "Validation Failed" errors
68+
- Skips creation silently instead of failing
69+
70+
### Fix 4: Decoupled workflow conditions
71+
- GitHub Release step now runs if `should_release == 'true'` AND either `published == 'true'` OR `publish.outcome == 'success'`
72+
- This ensures GitHub Release is created even in edge cases
73+
74+
## Best Practices Applied (from reference repos)
75+
76+
Referenced from:
77+
- [rust-ai-driven-development-pipeline-template](https://github.com/link-foundation/rust-ai-driven-development-pipeline-template)
78+
- [mem-rs](https://github.com/linksplatform/mem-rs)
79+
80+
1. **Treat crates.io as authoritative source** - check actual API, not just git tags
81+
2. **"Already exists" is success** - following mem-rs graceful pattern
82+
3. **Trust cargo exit code** - if `cargo publish` exits 0, the publish succeeded even if API is slow
83+
4. **Idempotent release creation** - GitHub Release creation handles "already exists" gracefully
84+
5. **Verification with backoff** - multiple verification attempts with increasing delays

‎experiments/test-publish-logic.mjs‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
#!/usr/bin/env node
2+
3+
/**
4+
* Test script to verify publish-to-crates.mjs logic changes.
5+
* Tests the "already exists" detection and failure pattern detection.
6+
*/
7+
8+
const ALREADY_EXISTS_PATTERNS = [
9+
'already exists on crates.io index',
10+
'crate already uploaded',
11+
'already exists on the registry',
12+
];
13+
14+
const FAILURE_PATTERNS = [
15+
'error[E',
16+
'error: ',
17+
'403 Forbidden',
18+
'401 Unauthorized',
19+
'the remote server responded with an error',
20+
];
21+
22+
function detectAlreadyExists(output) {
23+
for (const pattern of ALREADY_EXISTS_PATTERNS) {
24+
if (output.includes(pattern)) {
25+
return true;
26+
}
27+
}
28+
return false;
29+
}
30+
31+
function detectPublishFailure(output) {
32+
if (detectAlreadyExists(output)) {
33+
return null;
34+
}
35+
for (const pattern of FAILURE_PATTERNS) {
36+
if (output.includes(pattern)) {
37+
return pattern;
38+
}
39+
}
40+
return null;
41+
}
42+
43+
// Test cases
44+
const tests = [
45+
{
46+
name: '"already exists on crates.io index" should NOT be a failure',
47+
input: ` Updating crates.io index
48+
Credential cargo:token get crates-io
49+
error: crate link-assistant-agent@0.9.2 already exists on crates.io index`,
50+
expectAlreadyExists: true,
51+
expectFailure: null,
52+
},
53+
{
54+
name: '"crate already uploaded" should NOT be a failure',
55+
input: 'error: crate already uploaded',
56+
expectAlreadyExists: true,
57+
expectFailure: null,
58+
},
59+
{
60+
name: 'real error[E should be detected as failure',
61+
input: 'error[E0433]: failed to resolve',
62+
expectAlreadyExists: false,
63+
expectFailure: 'error[E',
64+
},
65+
{
66+
name: '403 Forbidden should be detected as failure',
67+
input: '403 Forbidden: invalid token',
68+
expectAlreadyExists: false,
69+
expectFailure: '403 Forbidden',
70+
},
71+
{
72+
name: 'generic "error: " without already exists should be failure',
73+
input: 'error: failed to verify package',
74+
expectAlreadyExists: false,
75+
expectFailure: 'error: ',
76+
},
77+
{
78+
name: 'clean output should not be a failure',
79+
input: ' Compiling link-assistant-agent v0.9.2\n Uploading link-assistant-agent v0.9.2',
80+
expectAlreadyExists: false,
81+
expectFailure: null,
82+
},
83+
];
84+
85+
let passed = 0;
86+
let failed = 0;
87+
88+
for (const test of tests) {
89+
const alreadyExists = detectAlreadyExists(test.input);
90+
const failure = detectPublishFailure(test.input);
91+
92+
const alreadyExistsOk = alreadyExists === test.expectAlreadyExists;
93+
const failureOk = failure === test.expectFailure;
94+
95+
if (alreadyExistsOk && failureOk) {
96+
console.log(`PASS: ${test.name}`);
97+
passed++;
98+
} else {
99+
console.log(`FAIL: ${test.name}`);
100+
if (!alreadyExistsOk) {
101+
console.log(` alreadyExists: expected=${test.expectAlreadyExists}, got=${alreadyExists}`);
102+
}
103+
if (!failureOk) {
104+
console.log(` failure: expected=${JSON.stringify(test.expectFailure)}, got=${JSON.stringify(failure)}`);
105+
}
106+
failed++;
107+
}
108+
}
109+
110+
console.log(`\n${passed} passed, ${failed} failed`);
111+
process.exit(failed > 0 ? 1 : 0);

‎js/.changeset/fix-crates-propagation.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@link-assistant/agent': patch
3+
---
4+
5+
Handle crates.io propagation delays and treat "already exists" as successful publish in CI/CD scripts

‎scripts/create-github-release.mjs‎

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -108,11 +108,23 @@ try {
108108
body: releaseNotes,
109109
});
110110

111-
await $`gh api repos/${repository}/releases -X POST --input -`.run({
112-
stdin: payload,
113-
});
114-
115-
console.log(`\u2705 Created GitHub release: ${tag}`);
111+
try {
112+
await $`gh api repos/${repository}/releases -X POST --input -`.run({
113+
stdin: payload,
114+
});
115+
console.log(`\u2705 Created GitHub release: ${tag}`);
116+
} catch (releaseError) {
117+
const errorMsg = releaseError.message || '';
118+
if (
119+
errorMsg.includes('already exists') ||
120+
errorMsg.includes('already_exists') ||
121+
errorMsg.includes('Validation Failed')
122+
) {
123+
console.log(`Release ${tag} already exists, skipping creation`);
124+
} else {
125+
throw releaseError;
126+
}
127+
}
116128
} catch (error) {
117129
console.error('Error creating release:', error.message);
118130
process.exit(1);

0 commit comments

Comments
 (0)