BUG: CI-Tests misses successful CI when checkSuites exceed the first 30 #5149

Describe the bug

CI-Tests reports 0 out of 30 merged PRs when every sampled PR has successful GitHub Actions test check runs, if the PR head commit has more than 30 check suites and the successful github-actions suites fall after the first GraphQL page.

This is reproducible with Scorecard v5.3.0 and the current main implementation.

Reproduction

Public repository: github.com/oaslananka/easyeda-mcp-pro

$ scorecard version
GitVersion: v5.3.0
GitCommit: c22063e786c11f9dd714d777a687ff7c4599b600

$ GITHUB_AUTH_TOKEN=... scorecard \
    --repo=github.com/oaslananka/easyeda-mcp-pro \
    --checks=CI-Tests \
    --show-details \
    --verbosity=debug

CI-Tests: 0 / 10
Reason: 0 out of 30 merged PRs checked by a CI test

The same result is produced by ossf/scorecard-action in workflow run 30498645165, so this is not limited to the CLI invocation.

Example PR: oaslananka/easyeda-mcp-pro#466

• PR head SHA: 98deba2249aaf487cd8b609939f75b91a54b5bd7
• REST GET /repos/oaslananka/easyeda-mcp-pro/commits/<sha>/check-runs returns 19 completed check runs, including successful github-actions runs for:
  • quality (24)
  • Linux test matrix
  • Windows test matrix
  • macOS test matrix
• GraphQL pullRequest.commits(last:1).commit.checkSuites(first:100) returns 37 suites.
• The first successful github-actions suite is at one-based index 34.
• None of the first 30 suites is both successful and recognized as CI by testsRunInCI.

I audited the same data for the 30 merged PRs considered by Scorecard:

The extra suites are created by installed integrations. Many are queued/empty suites; the actual successful GitHub Actions suites sort after them.

Root cause

The current GitHub client defines:

checksToAnalyze = 30

and queries:

graphql:"checkSuites(first: $checksToAnalyze)"

parseCheckRuns treats that partial result as a complete cache entry for the PR head SHA. Because the SHA is present in checkRunsByRef, listCheckRunsForRef does not fall back to REST. Successful CI suites after position 30 are therefore invisible to CI-Tests and SAST.

The same limit and non-paginated query are still present on the current default branch as of 2026-07-30.

Expected behavior

CI-Tests should detect the successful github-actions suites and score these PRs as tested regardless of how many unrelated check suites exist on the commit.

Suggested fix

Paginate the checkSuites connection until all pages are read, or treat a truncated GraphQL result as a cache miss and use the existing REST fallback. A regression test could provide more than 30 suites with a successful github-actions suite after index 30 and assert that testsRunInCI returns true.

Duplicate check

I reviewed #2750 and #5112 before filing:

• BUG: CI-Tests doesn't detect all recent PRs in a repo #2750 concerns commit/PR association and missing recent PRs.
• fix: fallback to REST API for fine-grained PAT in CheckRuns #5112 concerns GraphQL authorization failure with fine-grained PATs and REST fallback.

Neither covers a successful GraphQL response truncated before the relevant check suites.