Skip to content

engineering: use tar+zstd for win32 node_modules cache#260131

Closed
tmm1 wants to merge 7 commits intomicrosoft:mainfrom
tmm1:tmm1/faster-win32-nodemodules
Closed

engineering: use tar+zstd for win32 node_modules cache#260131
tmm1 wants to merge 7 commits intomicrosoft:mainfrom
tmm1:tmm1/faster-win32-nodemodules

Conversation

@tmm1
Copy link
Contributor

@tmm1 tmm1 commented Aug 6, 2025

this is significantly faster in my tests, down to 60s vs 120-180s before in the Extract node_modules phase

GHA and Azure Pipelines images have tar.exe and zstandard bundled since a few years ago

(will need to bump cachesalt if merging, to invalidate old cache)

cc @lszomoru @deepak1556

Copy link
Collaborator

@deepak1556 deepak1556 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! should also mirror this in .github/workflows/pr-node-modules.yml and .github/workflows/pr-win32-test.yml that is used for pr builds

@tmm1
Copy link
Contributor Author

tmm1 commented Aug 7, 2025

should also mirror this in .github/workflows/pr-node-modules.yml and .github/workflows/pr-win32-test.yml that is used for pr builds

Those paths already use tar but may see added speedups from zstd. For some reason whatever 7z does in the azure-pipeline is very slow.

@deepak1556
Copy link
Collaborator

Those paths already use tar but may see added speedups from zstd.

The windows jobs use 7z

exec { 7z.exe a .build/node_modules_cache/cache.7z -mx3 `@.build/node_modules_list.txt }
and
exec { 7z.exe a .build/node_modules_cache/cache.7z -mx3 `@.build/node_modules_list.txt }

@deepak1556
Copy link
Collaborator

@joaomoreno actually wrote the archive cache steps, I couldn't find from commit history why 7z was preferred for windows, maybe it was one of the reasons you had pointed out but lets wait for his review before pushing this forward. He is oof this week :)

I know win32 ships their own tar.exe but I think its bsd not gnutar and the options may not match.

@joaomoreno joaomoreno added this to the December / January 2026 milestone Dec 16, 2025
@deepak1556 deepak1556 removed their assignment Jan 6, 2026
@deepak1556 deepak1556 modified the milestones: December 2025, January 2026 Jan 6, 2026
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Speeds up Win32 node_modules caching by switching from 7z archives to tar+zstd, aiming to reduce cache extraction time in CI.

Changes:

  • Replace 7z.exe archive creation/extraction with tar.exe + zstd on Win32 (Azure Pipelines + GitHub Actions).
  • Update cache artifact name from cache.7z to cache.tzst.
  • Bump build/.cachesalt to invalidate existing caches.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
build/azure-pipelines/win32/steps/product-build-win32-compile.yml Switch restore/extract + create cache archive steps to tar+zstd for win32 builds.
build/azure-pipelines/win32/product-build-win32-node-modules.yml Switch node-modules cache archive creation to tar+zstd.
build/.cachesalt Cache invalidation bump to force regeneration under new archive format.
.github/workflows/pr-win32-test.yml Update Win32 PR workflow cache extract/create steps to tar+zstd.
.github/workflows/pr-node-modules.yml Update Win32 node-modules workflow cache archive creation to tar+zstd.
Comments suppressed due to low confidence (2)

build/azure-pipelines/win32/steps/product-build-win32-compile.yml:113

  • The archive creation uses tar’s -P (preserve absolute paths). Since listNodeModules.ts emits relative paths, -P isn’t needed and can make the resulting archive unsafe to extract if absolute/path-traversal entries ever get introduced. Prefer omitting -P so the archive contains only relative paths.
      exec { & "C:\Program Files\Git\usr\bin\tar.exe" --posix -cf .build/node_modules_cache/cache.tzst --exclude cache.tzst -P -C . --files-from .build/node_modules_list.txt --force-local --use-compress-program "zstd -T0" }

.github/workflows/pr-win32-test.yml:89

  • The create-archive tar command includes -P (preserve absolute paths). Given the archive inputs are relative, -P is unnecessary and increases the risk of writing outside the workspace if unexpected entries ever make it into the cache archive. Prefer omitting -P for safer archives.
          exec { & "C:\Program Files\Git\usr\bin\tar.exe" --posix -cf .build/node_modules_cache/cache.tzst --exclude cache.tzst -P -C . --files-from .build/node_modules_list.txt --force-local --use-compress-program "zstd -T0" }

joaomoreno and others added 4 commits January 29, 2026 16:04
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@joaomoreno
Copy link
Member

Moving to #291624 due to engineering checks protections

@joaomoreno joaomoreno closed this Jan 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants