Skip to content

Conversation

@dvcorreia
Copy link

What kind of change does this PR introduce?

Introduces a new --skip-unsupported-checks CLI flag to skip checks that are not supported by a repo type (e.g. when running against a GitLab repo, it will skip checks that don't have GitLab support, not including them in the final report and score).

For a bit of context, we run Scorecard in an air-gapped environment to score private repositories hosted on a self-hosted GitLab CE instance. Since most of the checks are not supported by GitLab, the score will never be 10/10, being a bit deceptive to our developers in that they cannot do anything to fix that. This change allows the scores to be more representative of what developers can actually achieve for their repository.

What is the current behavior?

When running Scorecard against non-GitHub repositories (e.g., GitLab, Azure DevOps, local), all checks are executed regardless of whether they support the repository type. This results in misleading scores where checks that are incompatible with the repo type fail, producing gaps in the score that developers cannot fix or address since their repository hosting platform doesn't support those checks.

What is the new behavior (if this is a feature change)?**

With the new --skip-unsupported-checks flag, Scorecard will:

  1. Detect the repository type (GitHub, GitLab, Azure DevOps, local)
  2. Check each check's supported repository types in checks.yaml
  3. Skip any checks that don't explicitly support the detected repository type
  4. Only run supported checks, providing a score that accurately reflects what's achievable for that repository type

When the flag is enabled with the default format, skipped checks are printed to stderr for visibility: Skipping (<repo-uri>) [<check-name>]

  • Tests for the changes have been added (for bug fixes/features)

Which issue(s) this PR fixes

Fixes #4795

Special notes for your reviewer

When using this flag, the number of checks run may be significantly lower than the full suite, particularly for non-GitHub repositories. For GitLab, most checks already have experimental support but are not listed as supported in checks.yaml. This can probably be addressed in a follow-up PR (related issue #4700).

Does this PR introduce a user-facing change?

For user-facing changes, please add a concise, human-readable release note to
the release-note

(In particular, describe what changes users might need to make in their
application as a result of this pull request.)

A new `--skip-unsupported-checks` CLI flag is now available to skip checks that don't support the repository type. When enabled, Scorecard will only run checks that explicitly support the detected repository type (GitHub, GitLab, Azure DevOps, or local). This provides more accurate scoring for non-GitHub repositories by excluding incompatible checks from the report and final score.

Introduces `--skip-unsupported-checks` to skip checks that are not
supported by a repo.

Closes ossf#4795

Signed-off-by: Diogo Correia <diogo.correia@siemens.com>
@dvcorreia dvcorreia requested a review from a team as a code owner January 27, 2026 19:16
@dvcorreia dvcorreia requested review from jeffmendoza and justaugustus and removed request for a team January 27, 2026 19:16
@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jan 27, 2026
@dvcorreia
Copy link
Author

This is a first iteration so if there's a better way to approach this, I'm happy to adjust things! I went with this implementation as it seemed like the least disruptive way to add the flag 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L This PR changes 100-499 lines, ignoring generated files.

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Feature: Filter-out incompatible repository checks

1 participant