Skip to content

Make virtual include paths shorter #26005

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Make virtual include paths shorter #26005

wants to merge 2 commits into from

Conversation

meteorcloudy
Copy link
Member

@meteorcloudy meteorcloudy commented May 5, 2025 •
edited
Loading

Fixes #18683

RELNOTES: Virtual include header files are linked under bin/_v_inc/<hash of target path> instead of bin/<target package path>/_virtual_include/<target name>. This shortens the virtual include paths which is critical for mitigating long path issue with MSVC on Windows.

@meteorcloudy meteorcloudy marked this pull request as draft May 5, 2025 13:27
@github-actions github-actions bot added the awaiting-review PR is awaiting review from an assigned reviewer label May 5, 2025
@meteorcloudy
Copy link
Member Author

Downstream test: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4650

@meteorcloudy meteorcloudy marked this pull request as ready for review May 5, 2025 14:36
@meteorcloudy meteorcloudy requested review from a team, lberki, comius, pzembrod and trybka as code owners May 5, 2025 14:36
@meteorcloudy meteorcloudy requested review from gregestren and removed request for a team May 5, 2025 14:36
@github-actions github-actions bot added team-Configurability platforms, toolchains, cquery, select(), config transitions team-Rules-CPP Issues for C++ rules labels May 5, 2025
katre
katre reviewed May 5, 2025
View reviewed changes
src/test/java/com/google/devtools/build/lib/rules/cpp/CcCommonTest.java Outdated
@@ -1071,12 +1072,12 @@ public void testIncludeManglingSmoke() throws Exception {
ConfiguredTarget lib = getConfiguredTarget("//third_party/a");
CcCompilationContext ccCompilationContext = lib.get(CcInfo.PROVIDER).getCcCompilationContext();
assertThat(ActionsTestUtil.prettyArtifactNames(ccCompilationContext.getDeclaredIncludeSrcs()))
.containsExactly("third_party/a/_virtual_includes/a/lib/b/c.h", "third_party/a/v1/b/c.h");
.containsExactly("_v_inc/207132b2/lib/b/c.h", "third_party/a/v1/b/c.h");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this hash stable? Will we see test failures (and one-off updates) in otherwise unrelated CLs?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the hash function is stable, input is basically the target path and name.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to only have this happen for Windows? I doubt it, but the loss in expressiveness of file paths is unfortunate.

Copy link
Collaborator

@fmeum fmeum May 5, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be a use case for ctx.exec_platform_has_constraint. But since clang-cl doesn't have this issue, it could be more appropriate to match on the compiler attribute of the toolchain or, even better, a new hash_virtual_includes feature that the MSVC toolchain sets.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a generic hash_virtual_includes feature makes sense as needing this support is a property of the toolchain itself, rather than the target (or exec) platform (should another toolchain run into this issue it would be available), and MSVC can add it.

@katre katre requested a review from lberki May 5, 2025 16:38
trybka
trybka reviewed May 5, 2025
View reviewed changes
src/main/starlark/builtins_bzl/common/cc/compile/cc_compilation_helper.bzl Outdated
@@ -21,7 +21,7 @@ load(":common/paths.bzl", "paths")

cc_internal = _builtins.internal.cc_internal

_VIRTUAL_INCLUDES_DIR = "_virtual_includes"
_VIRTUAL_INCLUDES_DIR = "_v_inc"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ISTM the important piece is hashing the potentially long set of path fragments that could accumulate.

Saving 11 chars on the constant piece is hopefully less relevant?

inc is clear enough, I suppose... maybe this could be "_virtual_inc" for everything, and then use a feature to control the hashing of path fragments, optionally?

Copy link
Collaborator

@fmeum fmeum May 5, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only rarely work on Windows builds, but even I got into situations where every single character counts. What do you think of also gating this abbreviation behind the new feature and giving it a more generic name that doesn't promise a particular scheme, say shorten_virtual_includes?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll defer to folks who have more MSVC experience than I do. I mostly work with clang-cl for windows builds.

I would certainly prefer to keep the more descriptive root when path length is not an issue (i.e. gate it behind a common feature as you suggest)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Saving 11 chars on the constant piece is hopefully less relevant?

The path length limit for MSVC is 260, 11 chars is almost 4% of that, I would say every char is precious on windows unfortunately.

I really like the shorten_virtual_includes idea, will give it a try.

@meteorcloudy
Copy link
Member Author

meteorcloudy commented May 7, 2025 •
edited
Loading

@comius has pointed out this approach depends on the actions.declare_shareable_artifact API which is unsafe and we want to remove eventually.

Another approach to solve this problem is to have a C++ wrapper for the MSVC toolchain to shorten paths that look too long.
Pros:

  • Allow us to remove actions.declare_shareable_artifact in future
  • More flexible on shortening any paths we want, not just for object files and include files.
  • Could detect and warning users about long paths

Cons:

@meteorcloudy @copybara-github
Make virtual include paths shorter
6e10c0e 
Fixes bazelbuild#18683

RELNOTES[INC]: Virtual include header files are linked under `bin/_v_inc/<hash of target path>` instead of `bin/<target package path>/_virtual_include/<target name>`. This shortens the virtual include paths which is critical for mitigating long path issue with MSVC on Windows.

Closes bazelbuild#26005.

PiperOrigin-RevId: 754930166
Change-Id: Ie1e5bad06a9f945c0cf71c5e122d444e527f7ca6
@meteorcloudy
Add back some proto library tests which was accidentally deleted in 3…
7841b51 
…7e0d23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@katre katre katre left review comments

@fmeum fmeum fmeum left review comments

@trybka trybka trybka left review comments

@comius comius Awaiting requested review from comius comius is a code owner

@pzembrod pzembrod Awaiting requested review from pzembrod pzembrod is a code owner

@lberki lberki Awaiting requested review from lberki lberki is a code owner

Assignees
No one assigned
Labels
awaiting-review PR is awaiting review from an assigned reviewer team-Configurability platforms, toolchains, cquery, select(), config transitions team-Rules-CPP Issues for C++ rules
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Alternative _virtual_includes organization to solve MSVC issues
4 participants
@meteorcloudy @katre @fmeum @trybka