Fix the combination preprocessor cache mode + distributed compilation #2392
+255
−148
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2392 +/- ##
==========================================
- Coverage 67.06% 66.92% -0.15%
==========================================
Files 64 64
Lines 34718 34586 -132
==========================================
- Hits 23285 23145 -140
- Misses 11433 11441 +8 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
nice |
ahartmetz
force-pushed
the
fix_direct_mode_distributed
branch
from
100c577 to
422cf17
Compare
|
This seems really annoying to auto-test, and to be honest, I haven't even tested it manually yet ;) |
It will need to call itself (in the next commit), and the "consumes and moves values out of the instance" model doesn't work for that.
...and disable the workaround that disabled the former in case of the latter. Under the following conditions: - Build with preprocessor cache mode enabled - Distributed build - Preprocessor cache hit - Main cache miss the compile that was run as "fallback" after the cache miss did not have preprocessed sources passed to it since skipping the preprocessing step is the whole point of preprocessor cache mode. But local preprocessing is what enables distributed compilation. To fix this, when the problem occurs, recursively re-invoke get_cached_or_compile() with ForceRecache(*) so that the preprocessor cache hit is (mostly) ignored, which makes get_cached_or_compile() invoke the local preprocessing codepath, which yields code suitable for distributed compilation. (*) which also prevents further recursion
This reverts the "main" code changes of commit c0a0b6e but keeps the added test. A better fix will work as follows: - Always store the dep file in the cache - In non-preprocessor cache mode, which doesn't consider things like file name and included file names, so it can confuse cache entries with same preprocessed content but different dep files: do not restore the dep file. It is not necessary anyway because local preprocessing will produce the file. - In preprocessor cache mode, which uses input file names and hashes to distinguish cache entries, but does not run the preprocessor: *do* restore the dep file.
ahartmetz
force-pushed
the
fix_direct_mode_distributed
branch
from
422cf17 to
d97d906
Compare
|
So I did do some practical testing and ran into another blocker for preprocessor cache mode, this time that it didn't work with -MD (which is used by at least Make and Ninja backends of CMake, so very widely I'd say). It is curious that |
ahartmetz
force-pushed
the
fix_direct_mode_distributed
branch
from
d97d906 to
b813cb8
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.