Skip to content

add socket tier 1 reachability analysis - #201

Merged
kanwalpreetd merged 7 commits into
stellar:mainfrom
kanwalpreetd:main
Sep 3, 2026
Merged

add socket tier 1 reachability analysis#201
kanwalpreetd merged 7 commits into
stellar:mainfrom
kanwalpreetd:main

Conversation

@kanwalpreetd

Copy link
Copy Markdown
Contributor

Copilot AI balanced review requested due to automatic review settings June 29, 2026 19:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new GitHub Actions workflow intended to run Socket reachability scanning on a weekly schedule (and on-demand) for this repository, including setup for Go/Rust/Node and log parsing to surface Tier-2 fallback scenarios as GitHub Actions warnings.

Changes:

  • Introduces a scheduled + manual workflow to run socket scan create --reach using a Socket API token.
  • Installs required toolchains (Rust, Go, Node) and enables Corepack before running the scan.
  • Captures scan output to detect Tier-2 fallbacks and emit a ::warning:: annotation while preserving the scan exit code.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/socket-scan.yml Outdated
Comment thread .github/workflows/socket-scan.yml
Comment thread .github/workflows/socket-scan.yml
Comment thread .github/workflows/socket-scan.yml
kanwalpreetd and others added 2 commits August 26, 2026 02:26
- Drop the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 env block: every action here
  already declares `using: node24`, and runners have defaulted to Node 24
  since 2026-06-16 (Node 20 is removed 2026-09-16)
- Remove the unused Corepack step; corepack only shims yarn/pnpm, and this
  repo uses neither
- Replace dtolnay/rust-toolchain with `rustup update`, matching this repo's
  other workflows and dropping a third-party action from a security workflow
- Read the Go version from go.mod (go 1.25) instead of hardcoding
  1.26.4, so the scan matches what this repo targets and cannot drift

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings September 1, 2026 16:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

.github/workflows/socket-scan.yml:94

  • A ::warning:: command creates an annotation only; it does not give the job or workflow a yellow conclusion, so this run still appears green/success. If degraded reachability must be visible in status checks, it needs to fail or be reported through a separate mechanism; otherwise update the GREEN/YELLOW documentation to describe the annotation accurately.
            echo "::warning::Socket scan completed with Tier 2 fallbacks - some vulnerabilities used precomputed reachability instead of full Tier 1"

.github/workflows/socket-scan.yml:94

  • The known 15.2.7 result (Found 0 projects across 0 ecosystems) is not one of these patterns, so the expected SBOM-only run exits successfully without any degradation warning and is incorrectly presented as full Tier 1 coverage. Detect that signature too and make the annotation describe both fallback and zero-project outcomes.
          if [ $rc -eq 0 ] && grep -qE "Reachability falls back to Tier 2|fallback to the results from the pre-computed|Reachability falls back to precomputed" /tmp/scan.log; then
            echo "::warning::Socket scan completed with Tier 2 fallbacks - some vulnerabilities used precomputed reachability instead of full Tier 1"

Comment thread .github/workflows/socket-scan.yml Outdated
Comment thread .github/workflows/socket-scan.yml Outdated
The `--reach-version=15.2.7` pin is no longer needed, and it was actively
hiding the fact that no reachability analysis was running.

Background: Coana 15.3.15 had made Socket's compute-artifacts endpoint fail
with "internal error (code=unknown)" during the Tier 1 fetching-artifacts
step, so the scan exited 1. Pinning to 15.2.7 dodged that, but 15.2.7 then
reported "Found 0 projects across 0 ecosystems" for this repo — leaving it
SBOM-only while still exiting 0, so the job rendered green as though a full
Tier 1 scan had succeeded.

Both problems are resolved. Two workflow_dispatch runs on 2026-09-03 with CLI
v1.1.167 confirmed it: pinned 15.2.7 and unpinned (Coana 15.10.36) each found
the Go project and ran code-aware analysis with identical results — 18
vulnerabilities, 1 reachable, 16 unreachable, 1 unsupported, 89% noise
reduction. Unpinned was ~2x faster (17.1s vs 33.7s) and also ran the
import-graph pre-filter and the `coana-finalized` backend step, neither of
which 15.2.7 emitted.

Changes:
- Remove --reach-version=15.2.7 so Coana runs at the CLI default.
- Add a check that emits a ::warning:: when the log reports 0 analyzable
  projects, so a silent recurrence shows up yellow instead of green. The
  pattern is "Found 0 projects( across 0 ecosystems)?" rather than a bare
  "0 projects across 0 ecosystems", which would also match a line like
  "Found 10 projects across 0 ecosystems".
- Replace the stale day-one SBOM-only note with the resolved history, and
  document both yellow cases in the status legend.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings September 3, 2026 17:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.github/workflows/socket-scan.yml:52

  • This note contradicts the resolved history above and the command below: the --reach-version flag has already been removed. Keeping this instruction makes maintainers believe the workflow is still pinned.

Comment thread .github/workflows/socket-scan.yml Outdated
Removes the RESOLVED block describing the old --reach-version=15.2.7 pin and
the 15.3.15 regression. The pin is gone and both problems are fixed, so the
history is no longer useful in the workflow itself; it remains in the git log.

Also removes two now-dangling references that pointed at it:
- the trailing "`--reach-version=15.2.7` pins Coana ... described in the
  day-one note above" line, which survived the earlier edit and described a
  flag that no longer exists
- "Not expected for this repo — see the note above" in the YELLOW legend

Comment-only; the scan step is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings September 3, 2026 18:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/workflows/socket-scan.yml:85

  • ::warning:: creates an annotation but does not create a yellow job or workflow conclusion; the run remains successful/green. These branches therefore cannot distinguish full Tier 1 coverage from zero-project or Tier 2 fallback by the job state as documented. If degraded scans must be visible in status, fail the job or publish a separate check/notification; otherwise revise the GREEN/YELLOW guidance to require inspecting annotations.
          if [ $rc -eq 0 ] && grep -qE "Found 0 projects( across 0 ecosystems)?" /tmp/scan.log; then
            echo "::warning::Socket found 0 analyzable projects - no Tier 1 reachability ran. CVE detection from the SBOM still applies."
          fi
          if [ $rc -eq 0 ] && grep -qE "Reachability falls back to Tier 2|fallback to the results from the pre-computed|Reachability falls back to precomputed" /tmp/scan.log; then
            echo "::warning::Socket scan completed with Tier 2 fallbacks - some vulnerabilities used precomputed reachability instead of full Tier 1"

Comment thread .github/workflows/socket-scan.yml Outdated
…ding

Replaces the log-grep for "Found 0 projects across 0 ecosystems" with a
structural check. That string is Coana/CLI-version-specific wording, so it can
silently stop matching on an upgrade — which would put us back to a green run
that did no reachability analysis.

- Pass --reach-retain-facts-file so the .socket.facts.json report survives.
- Read `(.components // []) | length` with jq. When Coana cannot perform Tier
  1 the report is still written but "components" is an empty array; a
  successful run populates it.
- Warn separately when the report is absent, unparseable, or has zero
  components. The check is warn-only — a healthy scan prints nothing extra.
- Add `rm -f .socket.facts.json` before the scan. The CLI help warns that a
  stale facts file "is picked up as a pre-generated input and silently
  overrides fresh analysis". A CI checkout is clean, so this is defensive.

Verified on the test branch: a forced 0-components run produced the expected
yellow annotation with the job still exiting 0, and all five paths (success,
success+Tier 2 fallback, zero components, absent report, unparseable report)
were checked offline against real report payloads.

One known limit: "components" reflects the dependency graph rather than
whether reachability was computed, so it catches the observed failure by
correlation (no projects -> no graph). A run with --reach-ecosystems set to an
absent ecosystem still populates components while skipping analysis; that
cannot happen here, since the flag is not used in this workflow.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings September 3, 2026 21:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

.github/workflows/socket-scan.yml:10

  • ::warning:: creates an annotation; it does not become the job's state or change a successful job/workflow conclusion from green. This GREEN/YELLOW/RED runbook therefore misleads operators into expecting a yellow workflow result. Describe the degraded cases as warning annotations, or use a separate failing/check mechanism if a distinct status is required.
# Socket scan — reading the job status. (The scan step below produces this: an
# exit code + an optional ::warning:: annotation, which GitHub Actions renders
# as the job's state.)

.github/workflows/socket-scan.yml:67

  • A warning annotation is visible in the run UI, but it does not produce a yellow run-level status; the successful job remains green. Correct this description so maintainers do not rely on a status that GitHub Actions does not provide.
          # succeeded but logged a Tier 2 fallback, emit a ::warning::
          # annotation that GitHub Actions renders as a yellow run-level
          # warning without failing the job.

Comment thread .github/workflows/socket-scan.yml
@kanwalpreetd
kanwalpreetd merged commit 44b4558 into stellar:main Sep 3, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants