Skip to content

test(chore): use OCLIF config caching testing method stolen from heroku CLI#1399

Draft
filmaj wants to merge 7 commits into
mainfrom
unit-test-cfg-caching
Draft

test(chore): use OCLIF config caching testing method stolen from heroku CLI#1399
filmaj wants to merge 7 commits into
mainfrom
unit-test-cfg-caching

Conversation

@filmaj

@filmaj filmaj commented Jun 27, 2026

Copy link
Copy Markdown
Member

Description

Heroku's CLI explicitly exercises the Command classes instead of shelling out. See their code for details: https://github.com/heroku/heroku-cli-test-utils/blob/main/src/run-command.ts

This PR shamelessly copies their approach. I've noticed that the Command class-based testing approach takes significant time on Windows in CI - want to see if this approach is better.

While the unit tests take ~2 mins on Ubuntu to complete in CI, they take 5-6 mins on Windows (see e.g. this recent main CI run).

Roughly, the results of this approach (based on a sample size of one!) in CI:

|Environment|Branch|

Locally, on an M4 Macbook Pro, here are a variety of tasks and their results. I wanted to compare individual test files as well as all unit tests, as not all 'unit tests' run in pnpm test:unit are converted to using a mock SanityCommand class (a majority still use the OCLIF test utils that shell out and collect stdout/stderr):

Task Branch Vitest Results
pnpm test:unit This PR Duration 19.65s (transform 13.25s, setup 1.79s, import 157.24s, tests 18.41s, environment 11ms)
pnpm test:unit main Duration 18.92s (transform 12.94s, setup 1.82s, import 148.12s, tests 19.00s, environment 11ms)
pnpm test:unit packages/@sanity/cli/src/commands/datasets/__tests__/copy.test.ts This PR Duration 1.68s (transform 161ms, setup 7ms, import 1.37s, tests 162ms, environment 0ms)
pnpm test:unit packages/@sanity/cli/src/commands/datasets/__tests__/copy.test.ts main Duration 931ms (transform 153ms, setup 7ms, import 488ms, tests 311ms, environment 0ms)
  • Longer vitest "import" time. Not sure what this represents, but I believe this would be a per-test-file tax.

@filmaj filmaj self-assigned this Jun 27, 2026
@github-actions

github-actions Bot commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

📦 Bundle Stats — @sanity/cli

Compared against main (9f50d269)

@sanity/cli

Metric Value vs main (9f50d26)
Internal (raw) 2.7 KB -
Internal (gzip) 1.0 KB -
Bundled (raw) 11.16 MB -
Bundled (gzip) 2.10 MB -
Import time 882ms +5ms, +0.6%

bin:sanity

Metric Value vs main (9f50d26)
Internal (raw) 782 B -
Internal (gzip) 423 B -
Bundled (raw) 9.87 MB -
Bundled (gzip) 1.78 MB -
Import time 2.26s +11ms, +0.5%

🗺️ View treemap · Artifacts

Details
  • Import time regressions over 10% are flagged with ⚠️
  • Sizes shown as raw / gzip 🗜️. Internal bytes = own code only. Total bytes = with all dependencies. Import time = Node.js cold-start median.

📦 Bundle Stats — @sanity/cli-core

Compared against main (9f50d269)

Metric Value vs main (9f50d26)
Internal (raw) 106.7 KB -
Internal (gzip) 26.7 KB -
Bundled (raw) 21.72 MB -
Bundled (gzip) 3.46 MB -
Import time 783ms +5ms, +0.6%

🗺️ View treemap · Artifacts

Details
  • Import time regressions over 10% are flagged with ⚠️
  • Sizes shown as raw / gzip 🗜️. Internal bytes = own code only. Total bytes = with all dependencies. Import time = Node.js cold-start median.

📦 Bundle Stats — create-sanity

Compared against main (9f50d269)

Metric Value vs main (9f50d26)
Internal (raw) 908 B -
Internal (gzip) 483 B -
Bundled (raw) 931 B -
Bundled (gzip) 491 B -
Import time ❌ ChildProcess denied: node -
Details
  • Import time regressions over 10% are flagged with ⚠️
  • Sizes shown as raw / gzip 🗜️. Internal bytes = own code only. Total bytes = with all dependencies. Import time = Node.js cold-start median.

@socket-security

socket-security Bot commented Jun 27, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addednpm/​@​heroku-cli/​test-utils@​1.0.0751009994100

View full report

@socket-security

socket-security Bot commented Jun 27, 2026

Copy link
Copy Markdown

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action Severity Alert  (click "▶" to expand/collapse)
Warn High
Obfuscated code: npm @sentry/node-core is 90.0% likely obfuscated

Confidence: 0.90

Location: Package overview

From: pnpm-lock.yamlnpm/@heroku-cli/test-utils@1.0.0npm/@sentry/node-core@10.60.0

ℹ Read more on: This package | This alert | What is obfuscated code?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/@sentry/node-core@10.60.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn High
Obfuscated code: npm es-abstract is 90.0% likely obfuscated

Confidence: 0.90

Location: Package overview

From: pnpm-lock.yamlnpm/eslint-plugin-import-x@4.16.2npm/eslint-import-resolver-typescript@4.4.5npm/@heroku-cli/test-utils@1.0.0npm/es-abstract@1.24.2

ℹ Read more on: This package | This alert | What is obfuscated code?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/es-abstract@1.24.2. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

Warn High
Obfuscated code: npm eslint-plugin-unicorn is 90.0% likely obfuscated

Confidence: 0.90

Location: Package overview

From: pnpm-lock.yamlnpm/@heroku-cli/test-utils@1.0.0npm/eslint-plugin-unicorn@56.0.1

ℹ Read more on: This package | This alert | What is obfuscated code?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/eslint-plugin-unicorn@56.0.1. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

@github-actions

Copy link
Copy Markdown
Contributor

Coverage Delta

No covered files changed in this PR.

Overall Coverage

Metric Coverage
Statements 74.3% (±0%)
Branches 64.2% (±0%)
Functions 68.8% (±0%)
Lines 74.9% (±0%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant