Resilient Link Checking for Nightly CI

[SeanTAllen]

Problem

The nightly link-checking job fails frequently due to transient external link failures (rate limiting, temporary outages, sites blocking CI bots). This makes the check unreliable — it cries wolf so often that real link rot gets ignored.

Internal and external links have very different reliability profiles, but the mkdocs tooling does not support splitting them into separate checks.

Proposed Approach: Failure Streak Tracking

Instead of failing on any broken link, track consecutive failures per URL and only alert when a link has been broken for N consecutive nightly runs (e.g., 3). This filters out transient failures while still catching genuine link rot.

Mechanism

Use a pinned GitHub issue as the persistence layer. The nightly job would:

1. Run the link checker, capture which URLs failed
2. Read the tracking data from the pinned issue body (JSON)
3. For each failed URL, increment its consecutive failure count
4. For each URL that passed, reset its count to 0
5. Update the issue body with the new tracking data via gh api
6. Exit non-zero only if any URL has hit the threshold (e.g., 3 consecutive failures)

Tracking Data Format

{
  "https://example.com/dead-page": {
    "consecutive_failures": 3,
    "first_failure": "2026-02-01",
    "last_failure": "2026-02-06"
  },
  "https://flaky.com/page": {
    "consecutive_failures": 1,
    "first_failure": "2026-02-06",
    "last_failure": "2026-02-06"
  }
}

Why a Pinned Issue?

Alternatives considered:

• File in the repo: Works but creates noisy daily commits to the git history.
• GitHub Actions cache: Caches get evicted frequently (ponylang repos often run at or over the cache limit), which would reset streak counts and delay alerts.
• GitHub Actions artifacts: Same eviction problem as cache.
• Pinned issue: No repo noise, no eviction, visible to anyone who wants to see link health status. The only cost is one API call per nightly run to read/update the issue body.

Additional Improvements

• Ignore list: Maintain a list of domains known to block CI bots (LinkedIn, Twitter/X, etc.) that should be excluded from checking entirely.
• Reduced frequency: Link rot happens over weeks, not hours. Weekly checks might be sufficient and would reduce noise further.
• PR checks: Only check internal/relative links on PRs (if the tooling can be configured to do so). External links are only checked on the nightly/weekly schedule.

Applies To

This approach would work for all ponylang repos with link checking:

• ponylang-website
• pony-tutorial
• pony-patterns