depfile: reconciling absolute paths with relative paths in build manifest

[bradking]

This issue supersedes #1251 as requested in #1251 (comment).

---

The ninja manual warns against mixing absolute and relative paths, but it is a serious limitation in common use cases, such as generated header files.

People often want their build systems to invoke the compiler with absolute paths to sources files and include directories. This produces output for IDEs to match error messages that reliably refer back to the original files. It also may affect paths recorded in debug information. While there may be tool-specific alternatives in some cases, this approach is simple and widely used. However, it means that depfiles report absolute paths that, in the case of generated headers, need to be reconciled with relative paths in the build manifest. Currently this does not happen.

Here is a script demonstrating the issue: ninja-bug.bash.txt

CMake currently works around this issue by naming both absolute and relative paths to the same file as outputs of the build statement producing it. It works at the cost of doubling the number of stat calls needed. However, #2667 proposes making such build statements an error. Therefore we need an alternative solution to support existing use cases.

[bradking]

If ninja is taught to reconcile absolute and relative paths for diagnosing such errors, it can use the same reconciliation logic to correctly hook up depfile absolute paths to build manifest relative paths.

[bradking]

In order to reconcile relative paths with absolute paths, ninja needs to model an absolute path to the working directory to which paths are relative. This could be done with getcwd(), but that resolves symbolic links. get_current_dir_name() may be useful where available. However, only the build manifest generator knows how it expresses absolute paths that appear in compiler command lines. In #1924 I propose a ninja_workdir binding that the generator can write to tell ninja the exact absolute path to the working directory it used during generation.

[jhasse]

In Ninja's current state: What would break if you (CMake) only specified absolute paths in the build manifest?

[bradking]

CMake currently generates absolute paths on build manifest nodes only in two main cases:

• Paths outside the top of the build tree.
• Redundantly on custom command outputs that are otherwise relative paths, specifically to work around this Ninja issue.

If CMake generated a build manifest expressing outputs only by absolute path, it would also have to express dependencies only by absolute path in order for them to be hooked up. That means build artifact paths on command lines coming from $in and $out will be absolute too, making them longer than necessary and more likely to trip over command-line length limits. Furthermore, running ninja relative/path/to/something would no longer work. Overall this would be a huge regression, and is not an acceptable solution for CMake.

[jhasse]

I don't understand. If I run ninja -v on my build/ in-tree ninja build generated by CMake I can already see that most paths are already passed absolute to the compiler (no custom commands involved).

Furthermore, running ninja relative/path/to/something would no longer work.

Okay that's #2720. Let's assume (for this issue), that this was fixed.

[bradking]

most paths are already passed absolute to the compiler

Paths to sources and include directories, yes, to improve diagnostics and debug information. Most other paths are relative, including paths to object files and in-tree libraries.

[jhasse]

Path to object files and in-tree libraries don't end up in the depfile, do they?

[bradking]

They can with ld --dependency-file=depfile, though hopefully they'd be in the same form as on the command line.

[bradking]

CMake's custom commands can generate arbitrary files, not just header files. We don't know a given file's purpose at generate time, so we can't know whether a depfile may mention it by absolute path or not. Furthermore, add_custom_command also supports a DEPFILE for dependencies. The command lines are controlled by project code. We don't know whether paths will appear as relative or absolute in the generated depfiles. So, in general we do not have enough information to know whether a given node must be expressed as relative or absolute in the build manifest to match depfile appearances. That's why they must be reconciled at build time by ninja.

[jhasse]

User have to make sure that there commands output the paths in the depfile in the same way that it gets passed. This is something GCC does too and is the correct way. Given that there are already quite a few restrictions on DEPFILE I don't think it's an unreasonable thing to ask from users.