add socket tier 1 reachability analysis - #201
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new GitHub Actions workflow intended to run Socket reachability scanning on a weekly schedule (and on-demand) for this repository, including setup for Go/Rust/Node and log parsing to surface Tier-2 fallback scenarios as GitHub Actions warnings.
Changes:
- Introduces a scheduled + manual workflow to run
socket scan create --reachusing a Socket API token. - Installs required toolchains (Rust, Go, Node) and enables Corepack before running the scan.
- Captures scan output to detect Tier-2 fallbacks and emit a
::warning::annotation while preserving the scan exit code.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Drop the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 env block: every action here already declares `using: node24`, and runners have defaulted to Node 24 since 2026-06-16 (Node 20 is removed 2026-09-16) - Remove the unused Corepack step; corepack only shims yarn/pnpm, and this repo uses neither - Replace dtolnay/rust-toolchain with `rustup update`, matching this repo's other workflows and dropping a third-party action from a security workflow - Read the Go version from go.mod (go 1.25) instead of hardcoding 1.26.4, so the scan matches what this repo targets and cannot drift Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.
Suppressed comments (2)
.github/workflows/socket-scan.yml:94
- A
::warning::command creates an annotation only; it does not give the job or workflow a yellow conclusion, so this run still appears green/success. If degraded reachability must be visible in status checks, it needs to fail or be reported through a separate mechanism; otherwise update the GREEN/YELLOW documentation to describe the annotation accurately.
echo "::warning::Socket scan completed with Tier 2 fallbacks - some vulnerabilities used precomputed reachability instead of full Tier 1"
.github/workflows/socket-scan.yml:94
- The known 15.2.7 result (
Found 0 projects across 0 ecosystems) is not one of these patterns, so the expected SBOM-only run exits successfully without any degradation warning and is incorrectly presented as full Tier 1 coverage. Detect that signature too and make the annotation describe both fallback and zero-project outcomes.
if [ $rc -eq 0 ] && grep -qE "Reachability falls back to Tier 2|fallback to the results from the pre-computed|Reachability falls back to precomputed" /tmp/scan.log; then
echo "::warning::Socket scan completed with Tier 2 fallbacks - some vulnerabilities used precomputed reachability instead of full Tier 1"
The `--reach-version=15.2.7` pin is no longer needed, and it was actively hiding the fact that no reachability analysis was running. Background: Coana 15.3.15 had made Socket's compute-artifacts endpoint fail with "internal error (code=unknown)" during the Tier 1 fetching-artifacts step, so the scan exited 1. Pinning to 15.2.7 dodged that, but 15.2.7 then reported "Found 0 projects across 0 ecosystems" for this repo — leaving it SBOM-only while still exiting 0, so the job rendered green as though a full Tier 1 scan had succeeded. Both problems are resolved. Two workflow_dispatch runs on 2026-09-03 with CLI v1.1.167 confirmed it: pinned 15.2.7 and unpinned (Coana 15.10.36) each found the Go project and ran code-aware analysis with identical results — 18 vulnerabilities, 1 reachable, 16 unreachable, 1 unsupported, 89% noise reduction. Unpinned was ~2x faster (17.1s vs 33.7s) and also ran the import-graph pre-filter and the `coana-finalized` backend step, neither of which 15.2.7 emitted. Changes: - Remove --reach-version=15.2.7 so Coana runs at the CLI default. - Add a check that emits a ::warning:: when the log reports 0 analyzable projects, so a silent recurrence shows up yellow instead of green. The pattern is "Found 0 projects( across 0 ecosystems)?" rather than a bare "0 projects across 0 ecosystems", which would also match a line like "Found 10 projects across 0 ecosystems". - Replace the stale day-one SBOM-only note with the resolved history, and document both yellow cases in the status legend. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
.github/workflows/socket-scan.yml:52
- This note contradicts the resolved history above and the command below: the
--reach-versionflag has already been removed. Keeping this instruction makes maintainers believe the workflow is still pinned.
Removes the RESOLVED block describing the old --reach-version=15.2.7 pin and the 15.3.15 regression. The pin is gone and both problems are fixed, so the history is no longer useful in the workflow itself; it remains in the git log. Also removes two now-dangling references that pointed at it: - the trailing "`--reach-version=15.2.7` pins Coana ... described in the day-one note above" line, which survived the earlier edit and described a flag that no longer exists - "Not expected for this repo — see the note above" in the YELLOW legend Comment-only; the scan step is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (1)
.github/workflows/socket-scan.yml:85
::warning::creates an annotation but does not create a yellow job or workflow conclusion; the run remains successful/green. These branches therefore cannot distinguish full Tier 1 coverage from zero-project or Tier 2 fallback by the job state as documented. If degraded scans must be visible in status, fail the job or publish a separate check/notification; otherwise revise the GREEN/YELLOW guidance to require inspecting annotations.
if [ $rc -eq 0 ] && grep -qE "Found 0 projects( across 0 ecosystems)?" /tmp/scan.log; then
echo "::warning::Socket found 0 analyzable projects - no Tier 1 reachability ran. CVE detection from the SBOM still applies."
fi
if [ $rc -eq 0 ] && grep -qE "Reachability falls back to Tier 2|fallback to the results from the pre-computed|Reachability falls back to precomputed" /tmp/scan.log; then
echo "::warning::Socket scan completed with Tier 2 fallbacks - some vulnerabilities used precomputed reachability instead of full Tier 1"
…ding Replaces the log-grep for "Found 0 projects across 0 ecosystems" with a structural check. That string is Coana/CLI-version-specific wording, so it can silently stop matching on an upgrade — which would put us back to a green run that did no reachability analysis. - Pass --reach-retain-facts-file so the .socket.facts.json report survives. - Read `(.components // []) | length` with jq. When Coana cannot perform Tier 1 the report is still written but "components" is an empty array; a successful run populates it. - Warn separately when the report is absent, unparseable, or has zero components. The check is warn-only — a healthy scan prints nothing extra. - Add `rm -f .socket.facts.json` before the scan. The CLI help warns that a stale facts file "is picked up as a pre-generated input and silently overrides fresh analysis". A CI checkout is clean, so this is defensive. Verified on the test branch: a forced 0-components run produced the expected yellow annotation with the job still exiting 0, and all five paths (success, success+Tier 2 fallback, zero components, absent report, unparseable report) were checked offline against real report payloads. One known limit: "components" reflects the dependency graph rather than whether reachability was computed, so it catches the observed failure by correlation (no projects -> no graph). A run with --reach-ecosystems set to an absent ecosystem still populates components while skipping analysis; that cannot happen here, since the flag is not used in this workflow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.
Suppressed comments (2)
.github/workflows/socket-scan.yml:10
::warning::creates an annotation; it does not become the job's state or change a successful job/workflow conclusion from green. This GREEN/YELLOW/RED runbook therefore misleads operators into expecting a yellow workflow result. Describe the degraded cases as warning annotations, or use a separate failing/check mechanism if a distinct status is required.
# Socket scan — reading the job status. (The scan step below produces this: an
# exit code + an optional ::warning:: annotation, which GitHub Actions renders
# as the job's state.)
.github/workflows/socket-scan.yml:67
- A warning annotation is visible in the run UI, but it does not produce a yellow run-level status; the successful job remains green. Correct this description so maintainers do not rely on a status that GitHub Actions does not provide.
# succeeded but logged a Tier 2 fallback, emit a ::warning::
# annotation that GitHub Actions renders as a yellow run-level
# warning without failing the job.
More info: https://stellarorg.atlassian.net/wiki/spaces/SCRT/pages/5689311233/Socket+Tier+1+Reachability+Analysis