[CI] Follow-up improvements for Windows ccache setup

[subodh-dubey-amd]

Follow-up items from #4161 review comments.

Tasks

• Move local ccache storage to B: drive on Windows runners — current CACHE_DIR is on C: (OS disk, 30GB) which may cause disk space issues with high cache miss rates. Should use the temp disk on B: like the build directory.
  Ref: Enable ccache + bazel-remote caching for Windows builds #4161 (comment)

• Fix set vs export shell detection in setup_ccache.py — Windows CI uses bash (not cmd), so set might be wrong. Detect shell type instead of OS, or remove the output since workflows set CCACHE_CONFIGPATH as an env var directly.
  Ref: Enable ccache + bazel-remote caching for Windows builds #4161 (comment)

• Consider Windows-compatible compiler_check — current POSIX check uses ldd/sha256sum; a Windows equivalent could improve cache robustness beyond the default mtime check.
  Ref: Enable ccache + bazel-remote caching for Windows builds #4161 (comment)

• Refine ccache preset selection for multi-arch workflows — split presubmit/postsubmit/release presets based on trigger type (pull_request vs push vs schedule) when multi-arch replaces old CI pipelines.
  Ref: Enable ccache + bazel-remote caching for Windows builds #4161 (review)

• Monitor GHA cache usage — with actions/cache removed from Windows builds, check https://github.com/ROCm/TheRock/actions/caches to confirm reduced pressure on the 10GB limit.
  Ref: Enable ccache + bazel-remote caching for Windows builds #4161 (review)

[ScottTodd]

Cache hit rates for Windows math-libs stage on multi-arch CI seem pretty low. I'd say that's highest priority to fix compared to the other items.

For example: https://github.com/ROCm/TheRock/actions/runs/23663263447/job/68945135123?pr=4209#step:17:68

Cacheable calls:                7636 / 7702 (99.14%)
  Hits:                          123 / 7636 ( 1.61%)
    Direct:                       55 /  123 (44.72%)
    Preprocessed:                 68 /  123 (55.28%)
  Misses:                       7513 / 7636 (98.39%)
Uncacheable calls:                66 / 7702 ( 0.86%)

Compared to Linux on the same run: https://github.com/ROCm/TheRock/actions/runs/23663263447/job/68952946155?pr=4209#step:14:21

Cacheable calls:        1482 / 1505 (98.47%)
  Hits:                 1344 / 1482 (90.69%)
    Direct:             1344 / 1344 (100.0%)
    Preprocessed:          0 / 1344 ( 0.00%)
  Misses:                138 / 1482 ( 9.31%)
Uncacheable calls:        23 / 1505 ( 1.53%)

We did land a flurry of changes to multi-arch CI today, but I'm not yet seeing ccache logs for Windows. Maybe step 1 is collecting and uploading those? I wonder if the CCACHE_CONFIGPATH env var is not getting threaded through completely

TheRock/build_tools/setup_ccache.py

Lines 43 to 60 in 0041897

	# See https://ccache.dev/manual/4.6.1.html#_configuration
	CONFIG_PRESETS_MAP = {
	"local": {},
	# Moving build_*_packages.yml CCACHE Env variables to here (linux for now)
	# For initial implementation, pre and post submit will be the same
	"github-oss-presubmit": {
	"secondary_storage": CACHE_SRV_DEV,
	"log_file": REPO_ROOT / "build/logs/ccache/ccache.log",
	"stats_log": REPO_ROOT / "build/logs/ccache/ccache_stats.log",
	"max_size": "5G",
	},
	"github-oss-postsubmit": {
	"secondary_storage": CACHE_SRV_REL,
	"log_file": REPO_ROOT / "build/logs/ccache/ccache.log",
	"stats_log": REPO_ROOT / "build/logs/ccache/ccache_stats.log",
	"max_size": "5G",
	},
	}

Aaaaaah. That's not actually the build dir on Windows, so we aren't including it in the upload:

TheRock/.github/workflows/multi_arch_build_windows_artifacts.yml

Line 63 in 0041897

BUILD_DIR: B:\build

[subodh-dubey-amd]

Windows math-libs low cache hit rate

Dug into the ccache logs from #4234. Here's what's going on:

98% of math-libs compilations use clang++ built from source during the compiler-runtime stage. Only 87 out of 7,688 use system MSVC (cl.exe). The ~51 remote cache hits we see every run? Those are all from cl.exe.

The rebuilt clang++ binary is not byte-identical between runs. I tested both
compiler_check = mtime and compiler_check = content — neither helps because the binary itself differs each time (likely MSVC linker timestamps or non-deterministic linking). Linux doesn't have this problem because GCC/ld produces deterministic binaries.

compiler_check	Remote hits (math-libs)
mtime (default)	51 / 7,550
content (tested)	51 / 7,550

Test runs: Run 1 | Run 2

cc @ScottTodd @amd-justchen

[ScottTodd]

diff'd clang.exe between two runs with 010editor (https://sweetscape.com/010editor/), looks like just a single timestamp is different, otherwise bit identical?

[amd-justchen]

diff'd clang.exe between two runs with 010editor (https://sweetscape.com/010editor/), looks like just a single timestamp is different, otherwise bit identical?

What's that third line in the result that is partially obscured, does that say Diff? i guess with another 2 bytes somewhere with an address far down, perhaps another timestamp?

I suppose we could create/modify the compiler check cache script to binary diff but excluding those bytes.

[amd-nicknick]

@ScottTodd, @amd-justchen, this looks like just the COFF timestamp being different.
Adding a linker flag /Brepro for MSVC should resolve this, a quick hack I put together: #4419

We should always have this enabled for the CI builds; Windows kernel itself is built with this flag as well.

[amd-justchen]

@ScottTodd, @amd-justchen, this looks like just the COFF timestamp being different.
Adding a linker flag /Brepro for MSVC should resolve this, a quick hack I put together: #4419

We should always have this enabled for the CI builds; Windows kernel itself is built with this flag as well.

Great find! Also that seems like a fun binary inspector tool. Will have to keep that in mind!

[ScottTodd]

We're really going to want Windows ccache hit rates above 1-2% for the math-libs stage soon. Sample run across all GPU families: https://github.com/ROCm/TheRock/actions/runs/24258568482

platform	build job	time	cache hit rate
windows	math-libs gfx101X-dgpu	🔴 1h47m	2%
windows	math-libs gfx1151	🔴 1h37m	1.5%
windows	math-libs gfx110X-all	💥 2h40m+ (still running)	TBD
windows	math-libs gfx120X-all	💥 2h53m	1.5%
linux	math-libs gfx101X-dgpu	🟡 1h28m	41%
linux	math-libs gfx1151	✅ 15m (!!)	96%
linux	math-libs gfx110X-all	✅ 23m (!!)	95%
linux	math-libs gfx120X-all	🟡 42m	96%

Can someone help test #4419 and finish it if it works?

[astrelsky]

We're really going to want Windows ccache hit rates above 1-2% for the math-libs stage soon. Sample run across all GPU families: https://github.com/ROCm/TheRock/actions/runs/24258568482

platform build job time cache hit rate
windows math-libs gfx101X-dgpu 🔴 1h47m 2%
windows math-libs gfx1151 🔴 1h37m 1.5%
windows math-libs gfx110X-all 💥 2h40m+ (still running) TBD
windows math-libs gfx120X-all 💥 2h53m 1.5%
linux math-libs gfx101X-dgpu 🟡 1h28m 41%
linux math-libs gfx1151 ✅ 15m (!!) 96%
linux math-libs gfx110X-all ✅ 23m (!!) 95%
linux math-libs gfx120X-all 🟡 42m 96%
Can someone help test #4419 and finish it if it works?

I added the changes from #4419 locally this morning and performed the following steps:

• Deleted my local CCACHE_DIR.
• Deleted the existing TheRock build directory to force a clean build.
• Configured TheRock with cmake -B build -G Ninja . -DTHEROCK_AMDGPU_TARGETS=gfx1150 -DTHEROCK_DIST_AMDGPU_FAMILIES=gfx1150 -DTHEROCK_BUILD_TESTING=OFF -DTHEROCK_MIOPEN_USE_COMPOSABLE_KERNEL=ON -DCMAKE_C_COMPILER=cl.exe -DCMAKE_CXX_COMPILER=cl.exe -DCMAKE_LINKER=link.exe -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
• Started the build using cmake --build build --target therock-archives therock-dist
• Packed up the results with the wheel scripts and copied them out.
• Deleted the previous TheRock build directory to again force a clean build.
• Moved the existing ccache.log and ccache_stats.log to my desktop for fresh logs on the second run.
• Reconfigured the same as above and started the build.
• After 4 hours it finally finished.

There are some cache hits, but a lot of them miss.

Something noteworthy from the logs.

First build

[2026-04-11T06:02:34.998070 13624] Command line: ccache D:\\TheRock\\build\\core\\clr\\dist\\lib\\llvm\\bin\\clang++.exe -DUSE_PROF_API=1 -D__HIP_PLATFORM_AMD__=1 -Drocsparse_EXPORTS -ID:/TheRock/rocm-libraries/projects/rocsparse/library/src/include -ID:/TheRock/rocm-libraries/projects/rocsparse/library/include -ID:/TheRock/build/math-libs/BLAS/rocSPARSE/build/include/rocsparse -ID:/TheRock/build/math-libs/BLAS/rocSPARSE/build/include -isystem D:/TheRock/build/math-libs/rocPRIM/dist/include -isystem D:/TheRock/build/core/clr/dist/include -isystem D:/TheRock/build/math-libs/BLAS/rocBLAS/dist/include -DWIN32 -DWIN32_LEAN_AND_MEAN -D_CRT_SECURE_NO_WARNINGS -DNOMINMAX -fms-extensions -fms-compatibility -D_ENABLE_EXTENDED_ALIGNED_STORAGE -Wno-documentation-unknown-command -Wno-documentation-pedantic -Wno-unused-command-line-argument -Wno-explicit-specialization-storage-class -Wno-ignored-attributes -Wno-unknown-attributes -Wno-duplicate-decl-specifier --hip-path=D:/TheRock/build/core/clr/dist --hip-device-lib-path=D:/TheRock/build/core/clr/dist/lib/llvm/amdgcn/bitcode -O3 -DNDEBUG -std=c++17 -D_DLL -D_MT -Xclang --dependent-lib=msvcrt -fvisibility-inlines-hidden --offload-compress -DROCSPARSE_WITH_ROCBLAS -Wno-unused-command-line-argument -Wall -x hip --offload-arch=gfx1150 -MD -MT library/CMakeFiles/rocsparse.dir/src/conversion/rocsparse_gcoosort.cpp.obj -MF library\\CMakeFiles\\rocsparse.dir\\src\\conversion\\rocsparse_gcoosort.cpp.obj.d -o library/CMakeFiles/rocsparse.dir/src/conversion/rocsparse_gcoosort.cpp.obj -c D:/TheRock/rocm-libraries/projects/rocsparse/library/src/conversion/rocsparse_gcoosort.cpp
[2026-04-11T06:02:34.998070 13624] Hostname: geekom
[2026-04-11T06:02:34.998070 13624] Working directory: D:\TheRock\build\math-libs\BLAS\rocSPARSE\build
[2026-04-11T06:02:35.006406 13624] Compiler: D:\TheRock\build\core\clr\dist\lib\llvm\bin\clang++.exe
[2026-04-11T06:02:35.006432 13624] Compiler type: clang
[2026-04-11T06:02:35.067291 13624] Detected input file: D:/TheRock/rocm-libraries/projects/rocsparse/library/src/conversion/rocsparse_gcoosort.cpp
[2026-04-11T06:02:35.067627 13624] Source file: D:/TheRock/rocm-libraries/projects/rocsparse/library/src/conversion/rocsparse_gcoosort.cpp
[2026-04-11T06:02:35.067649 13624] Dependency file: library\CMakeFiles\rocsparse.dir\src\conversion\rocsparse_gcoosort.cpp.obj.d
[2026-04-11T06:02:35.067660 13624] Object file: library/CMakeFiles/rocsparse.dir/src/conversion/rocsparse_gcoosort.cpp.obj
[2026-04-11T06:02:35.068832 13624] Trying direct lookup
[2026-04-11T06:02:35.069217 13624] Manifest key: 11b0hh8s3r8mpohhdbde32d8almqc5d5a
[2026-04-11T06:02:35.069401 13624] No 11b0hh8s3r8mpohhdbde32d8almqc5d5a in local storage

Second build

[2026-04-11T10:49:52.048001 400  ] Command line: ccache D:\\TheRock\\build\\core\\clr\\dist\\lib\\llvm\\bin\\clang++.exe -DUSE_PROF_API=1 -D__HIP_PLATFORM_AMD__=1 -Drocsparse_EXPORTS -ID:/TheRock/rocm-libraries/projects/rocsparse/library/src/include -ID:/TheRock/rocm-libraries/projects/rocsparse/library/include -ID:/TheRock/build/math-libs/BLAS/rocSPARSE/build/include/rocsparse -ID:/TheRock/build/math-libs/BLAS/rocSPARSE/build/include -isystem D:/TheRock/build/math-libs/rocPRIM/dist/include -isystem D:/TheRock/build/core/clr/dist/include -isystem D:/TheRock/build/math-libs/BLAS/rocBLAS/dist/include -DWIN32 -DWIN32_LEAN_AND_MEAN -D_CRT_SECURE_NO_WARNINGS -DNOMINMAX -fms-extensions -fms-compatibility -D_ENABLE_EXTENDED_ALIGNED_STORAGE -Wno-documentation-unknown-command -Wno-documentation-pedantic -Wno-unused-command-line-argument -Wno-explicit-specialization-storage-class -Wno-ignored-attributes -Wno-unknown-attributes -Wno-duplicate-decl-specifier --hip-path=D:/TheRock/build/core/clr/dist --hip-device-lib-path=D:/TheRock/build/core/clr/dist/lib/llvm/amdgcn/bitcode -O3 -DNDEBUG -std=c++17 -D_DLL -D_MT -Xclang --dependent-lib=msvcrt -fvisibility-inlines-hidden --offload-compress -DROCSPARSE_WITH_ROCBLAS -Wno-unused-command-line-argument -Wall -x hip --offload-arch=gfx1150 -MD -MT library/CMakeFiles/rocsparse.dir/src/conversion/rocsparse_gcoosort.cpp.obj -MF library\\CMakeFiles\\rocsparse.dir\\src\\conversion\\rocsparse_gcoosort.cpp.obj.d -o library/CMakeFiles/rocsparse.dir/src/conversion/rocsparse_gcoosort.cpp.obj -c D:/TheRock/rocm-libraries/projects/rocsparse/library/src/conversion/rocsparse_gcoosort.cpp
[2026-04-11T10:49:52.048001 400  ] Hostname: geekom
[2026-04-11T10:49:52.048001 400  ] Working directory: D:\TheRock\build\math-libs\BLAS\rocSPARSE\build
[2026-04-11T10:49:52.054070 400  ] Compiler: D:\TheRock\build\core\clr\dist\lib\llvm\bin\clang++.exe
[2026-04-11T10:49:52.054095 400  ] Compiler type: clang
[2026-04-11T10:49:52.054689 400  ] Detected input file: D:/TheRock/rocm-libraries/projects/rocsparse/library/src/conversion/rocsparse_gcoosort.cpp
[2026-04-11T10:49:52.054915 400  ] Source file: D:/TheRock/rocm-libraries/projects/rocsparse/library/src/conversion/rocsparse_gcoosort.cpp
[2026-04-11T10:49:52.054939 400  ] Dependency file: library\CMakeFiles\rocsparse.dir\src\conversion\rocsparse_gcoosort.cpp.obj.d
[2026-04-11T10:49:52.054950 400  ] Object file: library/CMakeFiles/rocsparse.dir/src/conversion/rocsparse_gcoosort.cpp.obj
[2026-04-11T10:49:52.055898 400  ] Trying direct lookup
[2026-04-11T10:49:52.056303 400  ] Manifest key: 2fdaipi6t6b1frae6oadn3e41res8v3r0
[2026-04-11T10:49:52.056512 400  ] No 2fdaipi6t6b1frae6oadn3e41res8v3r0 in local storage

I compared the clang-cl.exe from the first build and the second build and they were identical. I then compared the hipblas.dll from the first build and the second build and they were different due to the TimeDateStamp in the IMAGE_NT_HEADERS. However, the real question is why the hell isn't ccache just ignoring this to begin with?

ccache_log.zip
ccache_stats.log