[CI] Windows ccache hit rate improvement

[amd-nicknick]

Suggestion Description

Improve Windows ccache hit rate

Operating System

Windows

GPU

No response

ROCm Component

TheRock

Work Items

1. Add /Brepro for reproducible builds to ensure compiler and components are stable.
   Added to all component via TheRock init.
2. The compiler check on Windows needs work.
   Switch to use content compiler check.
3. Improve compiler bootstrap cache hit rate.
4. Documentation update to guide users to enable ccache on Windows

[amd-nicknick]

@ScottTodd, @amd-justchen, @astrelsky, I pushed a new fix into #4419 that adds the /Brepro flag to all subprojects. There are 2 items that needs to be addressed:
<Moved checklist upstairs>

By default, the compiler check uses mtime, but after a recompile, even if the binary is equivalent (it is after /Brepro), the modified time will change. Testing with compiler_check = content gets promising result:

Compile rocblas target
Cacheable calls:   7943 /   7996 (99.34%)
  Hits:            7907 /   7943 (99.55%)
    Direct:        7907 /   7907 (100.0%)
    Preprocessed:     0 /   7907 ( 0.00%)
  Misses:            36 /   7943 ( 0.45%)
Uncacheable calls:   53 /   7996 ( 0.66%)

Next step: Implement a similar compiler artifact hashing on Windows, or we could just use compiler_check = content, I'll investigate.

[astrelsky]

For clarification, that compiler_check = content goes in the ccache configuration? The windows support document that details building on windows and quirks with using ccache on windows should be updated to cover it. This way people can have it configured correctly that want to build it locally.

Oh also, I still have the pull request open in porch that reapplies the using ccache (reverted because it broke windows builds but I included a fix in the reapply).

[amd-nicknick]

@astrelsky yeah it goes to ccache config. I added it just now into build_tools\setup_ccache.py (commit f5c4f87).
If this setup goes well in CI, what's left would be docs update.

Could you give this PR a try? I tried locally again and can confirm all binaries are stable & high hit rate for libs compilation.

Cacheable calls:   7889 /   7946 (99.28%)
  Hits:            7853 /   7889 (99.54%)
    Direct:        7853 /   7853 (100.0%)
    Preprocessed:     0 /   7853 ( 0.00%)
  Misses:            36 /   7889 ( 0.46%)
Uncacheable calls:   57 /   7946 ( 0.72%)

[astrelsky]

@astrelsky yeah it goes to ccache config. I added it just now into build_tools\setup_ccache.py (commit f5c4f87).
If this setup goes well in CI, what's left would be docs update.

Could you give this PR a try? I tried locally again and can confirm all binaries are stable & high hit rate for libs compilation.

Cacheable calls:   7889 /   7946 (99.28%)
  Hits:            7853 /   7889 (99.54%)
    Direct:        7853 /   7853 (100.0%)
    Preprocessed:     0 /   7853 ( 0.00%)
  Misses:            36 /   7889 ( 0.46%)
Uncacheable calls:   57 /   7946 ( 0.72%)

I can but unfortunately I won't be able to start until tonight. I should have results by tomorrow.

I'm not sure what the advantage of me trying it locally if you've already done so though 😂. I'm going to anyway before it's merged because 4-6 hour build times are brutal.

[amd-nicknick]

@astrelsky yeah no worries no hurry, I'd like to make sure it's actually working and there are no weird environment funnies here. We could wait for CI as well.

[astrelsky]

@astrelsky yeah no worries no hurry, I'd like to make sure it's actually working and there are no weird environment funnies here. We could wait for CI as well.

36 minutes with a clean build and a populated cache. I'm not sure how you get the clean statistics, so I'll just attach the logs here.

ccache_stats.log
ccache_log.zip

[amd-nicknick]

One more thing worth investigating: For the compiler phase, the cache rate is still low due to LLVM's use of precompiled headers. This is turned on for Windows builds by default and cause ccache to consider LLVM to be uncacheable.

Cacheable calls:                     6412 / 19871 (32.27%)
  Hits:                              5539 /  6412 (86.38%)
    Direct:                          1695 /  5539 (30.60%)
    Preprocessed:                    3844 /  5539 (69.40%)
  Misses:                             873 /  6412 (13.62%)
Uncacheable calls:                  13455 / 19871 (67.71%)
  Compilation failed:                  13 / 13455 ( 0.10%)
  Could not use precompiled header:  3929 / 13455 (29.20%)
  Preprocessing failed:                28 / 13455 ( 0.21%)
  Unsupported compiler option:       9481 / 13455 (70.46%)
  Unsupported source encoding:          4 / 13455 ( 0.03%)

Again, 2 directions ahead:

1. Disable PCH outright for everything, in theory cached hits should likely be sufficient to cover most of the compile time gains from PCH.
2. Try to get ccache + MSVC + PCH working.

Let me spend some time on this topic :)

[amd-nicknick]

For our setup, only compiling LLVM on Windows is the only (current) scenario that uses PCH. Since we configure LLVM with CMake, and LLVM uses the proper CMake directives (target_precompile_headers) to generate the compiler flags, ccache is able to understand and work with PCH if we add in sloppiness option.

ccache recently resolved a couple of MSVC PCH related issues, and my testing also shows that it works well with our configuration. (We always use latest ccache from chocolatey, which is great!)

Updated PR with the change that add the sloppiness. Full gfx1201 rebuild with fully populated cache:

(therock-build-venv) D:\Nick\TheRock\TheRock>ccache -s -v
Cache directory:                D:\Nick\TheRock\TheRock\.ccache\local
Config file:                    D:\Nick\TheRock\TheRock\.ccache\ccache.conf
System config file:
Stats updated:                  04/17/26 05:11:03
Cacheable calls:                28464 /  30423 (93.56%)
  Hits:                         28128 /  28464 (98.82%)
    Direct:                     28128 /  28128 (100.0%)
    Preprocessed:                   0 /  28128 ( 0.00%)
  Misses:                         336 /  28464 ( 1.18%)
Uncacheable calls:               1959 /  30423 ( 6.44%)
  Compilation failed:              13 /   1959 ( 0.66%)
  Compiler output file missing:  1149 /   1959 (58.65%)
  No input file:                  371 /   1959 (18.94%)
  Preprocessing failed:            69 /   1959 ( 3.52%)
  Unsupported code directive:       3 /   1959 ( 0.15%)
  Unsupported compiler option:    354 /   1959 (18.07%)
Successful lookups:
  Direct:                       28128 /  29698 (94.71%)
  Preprocessed:                     0 /   1498 ( 0.00%)
Local storage:
  Cache size (GB):                7.2 / 1024.0 ( 0.70%)
  Files:                        68376
  Hits:                         28128 /  28464 (98.82%)
  Misses:                         336 /  28464 ( 1.18%)
  Reads:                        59324
  Writes:                         672

[ScottTodd]

(We always use latest ccache from chocolatey, which is great!)

fyi that's been a consistent source of build breaks, since remote servers sometimes go down for maintenance. We're hoping to bake that into the runner images at some point (similar to how linux users a manylinux build docker container)