Skip to content

A GitHub request that fails below the HTTP layer discards its cause, so a fold failure cannot be diagnosed #213

Description

@Nitjsefnie

Description

A GitHub REST request that fails below the HTTP layer is reported as Error: GitHub request failed. with nothing else — no cause, no URL, no underlying error. The original error is caught and discarded rather than attached, so the log cannot say whether the fold hit a DNS failure, a TLS error, a socket reset, a proxy refusal, or a bug in the caller.

It happened in production tonight. /srv/overflow was on 12a0409, and one repository's fold failed and then succeeded on the next drain sixty-three seconds later with no intervention:

Sep 07 03:47:02 overflow[3724099]: Reconciliation of repository 245cf511-1434-48f3-a106-6bfcb2cc1f69 failed. Error: GitHub request failed.
    at l.request (...)
    at async l.getPullRequestDiff (...)
    at async Object.withRepositoryReconciliation (...)
Sep 07 03:47:02 overflow[3724099]: Reconciliation drain { RETRY_SCHEDULED: 1 }
Sep 07 03:48:05 overflow[3724099]: Reconciliation drain { RECONCILED: 1 }

Everything needed to classify that failure was in the caught error and was thrown away. A transient network fault and a defect in getPullRequestDiff produce byte-identical log output, so no amount of reading the journal can separate them.

The contrast inside the same function is what makes this a defect rather than a missing nicety: an HTTP-level failure is reported richly, as a GitHubApiError carrying status, rate-limit classification and response body. Only the non-HTTP path is silent, and that is the path a transport problem takes.

Two sites, both on src/lib/github/client.ts at 12a0409:

  • line 589, the REST request() catch-all
  • line 278, the paginating requestWorkflow catch

Both end throw new Error("GitHub request failed.") with the caught error unreferenced.

Expected Behavior

A failed request should report what failed. The thrown error should carry the original as its cause, plus the request path, so the journal distinguishes a network fault from a caller defect without a reproduction. Node's Error supports { cause } and the journal already renders a [cause]: chain — the surrounding code prints one for the wrapping Unable to reconcile repository. error.

Reproduction Steps

  1. Point GitHubClient at an API URL whose host does not resolve, or otherwise make fetchImplementation reject with a network-level error rather than an HTTP response.
  2. Call any method that goes through request()getPullRequestDiff for instance.
  3. Observe the thrown error is GitHub request failed. with no cause and no indication of which request or what went wrong.

Environment / Context

Observed on /srv/overflow at 12a0409, overflow.service MainPID 3724099. The repository reconciled normally on the following drain, so the underlying condition was transient; which transient condition it was cannot be recovered from the log.

Note the separate, louder failure earlier the same night — RATE_LIMIT: API rate limit already exceeded for user ID 75166987 at 02:46 — is a different path that reports itself properly, and is a symptom of the shared account's secondary rate limit rather than of this.

Discovered During

Flagged by the session that landed PR 204 (issue 202), which saw the two repository fold failures in the journal and correctly declined to file, since it could not tell a revoked token or a deleted repository from a code defect. Investigated and reproduced against the code by the manager session; the inability to tell those apart from the log IS this defect.

Suggested Fix

Unverified. At both sites, construct the error with the caught one as its cause and include the path, for example new Error(\GitHub request failed: ${path}`, { cause: error }). The existing tests that assert on the message "GitHub request failed."would need to match a prefix rather than the whole string; checktests/github/` before changing the text, since #203 added cases that pin messages on this path.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: infraDeployment, database, migrations and opsbugSomething isn't workingoffered: smallOpening catalog · comparison 3 · reserve 3

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions