Should we use CMake globbing to find srcs and headers?

[swolchok]

Currently, ExecuTorch CMake builds invoke buck2 to extract source file information at CMake generation time. I am considering removing buck2 from the ExecuTorch OSS build process in favor of a CMake-native build.

The CMake project famously "do[es] not recommend using GLOB to collect a list of source files from your source tree". The reason is that CMake generates configuration for another build system. Because the underlying build system may not support globbing, the glob happens at CMake time, not build time, so if you manage to rebuild without rerunning CMake, your new file will be missed.

However, we have several reasons to use GLOB:

• internal builds use buck2, which supports globs, which developers are very used to using.
• it is inconvenient to have to update the build system configuration every time you add a new file, and people will invariably forget. This is exacerbated by internal familiarity with globbing in buck2.

Note that the CONFIGURE_DEPENDS option to GLOB is supposed to cause CMake to attempt to configure the generated build files (e.g., Makefiles) to re-run CMake during builds if glob results change. However, the ninja-build version in both focal (current Ubuntu LTS, support ends in April of this year, 2025) and jammy (next Ubuntu LTS) is 1.10.0 or 1.10.1, both of which, per Alex Reinking, have a bug affecting globbing. For what it's worth, PyTorch core uses GLOB but does not use CONFIGURE_DEPENDS.

Personally, I am inclined to follow PyTorch core and use GLOB without CONFIGURE_DEPENDS, for the following reasons:

• motivation to use GLOB as outlined above
• As a general rule, we should match PyTorch core when in doubt; ExecuTorch is PyTorch
• both parts of the decision are reversible: we can always either add CONFIGURE_DEPENDS or remove GLOB in the future
• our current setup is essentially GLOB-via-buck2-query anyway

However, if we reach a decision that manually listing source/header files is best, I will not change my plans to set up a CMake build for that reason. Thoughts?

[orionr]

Tagging @malfet as my CMake expert, but I feel like glob is fine. One question I have is how we are using *.bzl files right now to share across CMake, buck2, and Bazel builds for core PyTorch? I feel like we had some shared file listing system there, but @swolchok saw blobs. Tagging @larryliu0820 too when back.

[malfet]

IMO globbing is fine as well, though I though our internal guides recommend avoiding it as well due to the complex package boundary resolution logic that exists in buck2

Also, please note that ones does not have to invoke buck just to extract filelists, as .bzl files are written in starlark, which is a subset of Python, i.e. one can execute bzl file to extract required info using regular python interpreter. In PyTorch it is done using append_filelist function
https://github.com/pytorch/pytorch/blob/4499d60d56d5c88010dbc95c9fcafacae5cb8a1d/cmake/Codegen.cmake#L411

[swolchok]

In PyTorch it is done using append_filelist function

This looks great, but why does PyTorch mix explicit filelists with globs?

[malfet]

Historical reasons(and i think different executions of buck vs CMake) :)

[dbort]

Globbing seems like it would require us to change the layout of our source tree: to avoid putting multiple targets in the same directory, or to rename source files to have a common prefix/substring that can be globbed on.

For the long-term goal of decoupling buck2 from the cmake build, I think that managing source lists in common .bzl files would be the most straightforward.