Skip to content

feat: add retry logic with exponential backoff for URL timeout checks #237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
NebraskaCoder merged 2 commits into from
Nov 26, 2025
+72 −6
Merged

feat: add retry logic with exponential backoff for URL timeout checks #237

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d564470
feat: add retry logic with exponential backoff for URL timeout checks
NebraskaCoder Nov 26, 2025
ff5d619
refactor: remove unnecessary checkUrl wrapper function
NebraskaCoder Nov 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 72 additions & 6 deletions scripts/check-download-urls.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ function extractUrls(obj, path = "") {
return urls;
}

// Check if URL is accessible
function checkUrl(urlInfo) {
// Check if URL is accessible (single attempt)
function checkUrlOnce(urlInfo) {
return new Promise((resolve) => {
const url = new URL(urlInfo.url);
const protocol = url.protocol === "https:" ? https : http;
Expand Down Expand Up @@ -76,6 +76,58 @@ function checkUrl(urlInfo) {
});
}

// Retry timed out URLs with exponential backoff
async function retryTimeouts(timedOutResults) {
if (timedOutResults.length === 0) return [];

const maxRetries = 10;
const baseDelay = 1000; // Start with 1 second
let remaining = [...timedOutResults];
const successful = [];
Copy link

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Misleading variable name: successful contains all non-timeout results (including "FAILED" and "ERROR" statuses), not just successful results. Consider renaming to retried or completed to better reflect its actual content.

Copilot uses AI. Check for mistakes.

for (
let attempt = 1;
attempt <= maxRetries && remaining.length > 0;
attempt++
) {
// Exponential backoff: 1s, 2s, 4s, 8s, 16s, 32s, 60s (capped), 60s, 60s, 60s
const delay = Math.min(baseDelay * Math.pow(2, attempt - 1), 60000);

console.log(
`\n${colors.yellow}Retry attempt ${attempt}/${maxRetries} for ${remaining.length} timed out URL(s) (waiting ${delay / 1000}s)...${colors.reset}`
);

await new Promise((resolve) => setTimeout(resolve, delay));

const retryResults = await Promise.all(
remaining.map((r) => checkUrlOnce({ url: r.url, path: r.path }))
);

const stillTimedOut = [];
for (const result of retryResults) {
if (result.status === "TIMEOUT") {
stillTimedOut.push(result);
} else {
successful.push(result);
console.log(
` ${colors.green}✓${colors.reset} ${result.url} - ${result.status === "OK" ? "OK" : result.status}`
);
Comment on lines +112 to +114
Copy link

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Misleading success indicator: A green checkmark is displayed for retried URLs even when they return "FAILED" or "ERROR" status. This gives the false impression that these URLs are now working. Only URLs with status === "OK" should receive a success checkmark.

Suggested change
console.log(
` ${colors.green}✓${colors.reset} ${result.url} - ${result.status === "OK" ? "OK" : result.status}`
);
if (result.status === "OK") {
console.log(
` ${colors.green}✓${colors.reset} ${result.url} - OK`
);
} else if (result.status === "FAILED" || result.status === "ERROR") {
console.log(
` ${colors.red}✗${colors.reset} ${result.url} - ${result.status}`
);
} else {
console.log(
` ${colors.yellow}!${colors.reset} ${result.url} - ${result.status}`
);
}

Copilot uses AI. Check for mistakes.
}
Comment on lines +106 to +115
Copy link

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Non-timeout failures during retries are incorrectly treated as successful. When a previously timed-out URL is retried and returns "FAILED" or "ERROR" status (not "TIMEOUT"), it gets added to the successful array and is logged with a success checkmark. This means these failed URLs won't be reported as failures in the final results.

The logic should only add results with status === "OK" to the successful array. Failed and error results should be tracked separately and returned as failures.

Copilot uses AI. Check for mistakes.
}

remaining = stillTimedOut;

if (remaining.length === 0) {
console.log(
`${colors.green}All timed out URLs recovered after ${attempt} retry attempt(s)!${colors.reset}`
);
Comment on lines +121 to +123
Copy link

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Misleading success message: The message claims "All timed out URLs recovered" but this isn't accurate. A URL that was retried and returned "FAILED" or "ERROR" status hasn't truly "recovered" - it just didn't timeout again. The message should distinguish between URLs that became accessible (status "OK") versus those that simply returned a different failure status.

Suggested change
console.log(
`${colors.green}All timed out URLs recovered after ${attempt} retry attempt(s)!${colors.reset}`
);
const okCount = successful.filter(r => r.status === "OK").length;
const nonOkCount = successful.length - okCount;
let msg = `${colors.green}${okCount} timed out URL(s) recovered (status OK) after ${attempt} retry attempt(s).${colors.reset}`;
if (nonOkCount > 0) {
msg += ` ${colors.yellow}${nonOkCount} URL(s) returned non-timeout failure status (e.g. FAILED, ERROR).${colors.reset}`;
}
console.log(msg);

Copilot uses AI. Check for mistakes.
}
}

// Return both successful retries and still-failed ones
return [...successful, ...remaining];
}

// Progress bar
function updateProgress(current, total) {
const percentage = Math.round((current / total) * 100);
Expand Down Expand Up @@ -106,7 +158,7 @@ async function main() {
// Process in batches to avoid overwhelming servers
for (let i = 0; i < urls.length; i += batchSize) {
const batch = urls.slice(i, i + batchSize);
const batchResults = await Promise.all(batch.map(checkUrl));
const batchResults = await Promise.all(batch.map(checkUrlOnce));
results.push(...batchResults);

processed += batch.length;
Expand All @@ -121,16 +173,30 @@ async function main() {
// Clear progress bar
process.stdout.write("\r" + " ".repeat(80) + "\r");

// Separate timeouts from other failures for retry logic
const timedOut = results.filter((r) => r.status === "TIMEOUT");
const otherResults = results.filter((r) => r.status !== "TIMEOUT");

// Retry timed out URLs with exponential backoff
let finalResults = [...otherResults];
if (timedOut.length > 0) {
console.log(
`\n${colors.yellow}${timedOut.length} URL(s) timed out. Retrying with exponential backoff...${colors.reset}`
);
const retryResults = await retryTimeouts(timedOut);
finalResults = [...finalResults, ...retryResults];
}

// Report results
const failed = results.filter((r) => r.status !== "OK");
const successful = results.filter((r) => r.status === "OK");
const failed = finalResults.filter((r) => r.status !== "OK");
const successful = finalResults.filter((r) => r.status === "OK");

console.log("\n" + colors.blue + "=" + "=".repeat(50) + colors.reset);
console.log(`${colors.blue}URL Check Results${colors.reset}`);
console.log(colors.blue + "=" + "=".repeat(50) + colors.reset + "\n");

console.log(
`Total URLs checked: ${colors.yellow}${results.length}${colors.reset}`
`Total URLs checked: ${colors.yellow}${finalResults.length}${colors.reset}`
);
console.log(`Successful: ${colors.green}${successful.length}${colors.reset}`);
console.log(`Failed: ${colors.red}${failed.length}${colors.reset}\n`);
Expand Down
Loading