[WIP] bazel: Add compdb extractor

[phlax]

No description provided.

[repokitteh-read-only]

CC @envoyproxy/dependency-shepherds: Your approval is needed for changes made to (bazel/.*repos.*\.bzl)|(bazel/dependency_imports\.bzl)|(api/bazel/.*\.bzl)|(.*/requirements\.txt)|(.*\.patch).
envoyproxy/dependency-shepherds assignee is @moderation

🐱

Caused by: #42428 was opened by phlax.

see: more, trace.

[phlax]

cc @tsaarni

[phlax]

these dont want to be here - but we are going to struggle to fit all of these into our current workspace fun

[mmorel-35]

Are you sure this transitives bring anything ? I thought they were empty

[phlax]

not sure - just followed the docs

[mmorel-35]

I believe you can remove those imports they are noops

[tsaarni]

How should I run this? I get the same error that I had when I tried this myself:

$ bazel run @hedron_compile_commands//:refresh_all

WARNING: The following configs were expanded more than once: [clang, clang-common, libc++]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Invocation ID: c426a032-7888-4e97-918c-48195c4cbad1
DEBUG: Rule 'dynamic_modules_rust_sdk_crate_index' indicated that a canonical reproducible form can be obtained by modifying arguments _action_listener = <unknown object com.google.devtools.build.lib.packages.Attribute$LabelListLateBoundDefault>, _config_dependencies = [], _configure = False, _environ = ["CARGO_BAZEL_GENERATOR_URL", "CARGO_BAZEL_GENERATOR_SHA256", "CARGO_BAZEL_REPIN", "REPIN", "CARGO_BAZEL_REPIN_ONLY", "CARGO_BAZEL_ISOLATED", "CARGO_BAZEL_DEBUG", "CARGO_BAZEL_TIMEOUT"], _original_name = ""
DEBUG: Repository dynamic_modules_rust_sdk_crate_index instantiated at:
  /workspaces/envoy/WORKSPACE:36:25: in <toplevel>
  /workspaces/envoy/bazel/dependency_imports.bzl:74:24: in envoy_dependency_imports
  /workspaces/envoy/bazel/dependency_imports.bzl:214:22: in crates_repositories
Repository rule crates_repository defined at:
  /home/envoybuild/.cache/bazel/_bazel_envoybuild/2d35de14639eaad1ac7060a4dd7e3351/external/rules_rust/crate_universe/private/crates_repository.bzl:157:36: in <toplevel>
WARNING: The following configs were expanded more than once: [clang, clang-common, libc++]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Analyzed target @@hedron_compile_commands//:refresh_all (159 packages loaded, 11019 targets configured).
INFO: Found 1 target...
Target @@hedron_compile_commands//:refresh_all up-to-date:
  bazel-bin/external/hedron_compile_commands/refresh_all
  bazel-bin/external/hedron_compile_commands/refresh_all.check_python_version.py
  bazel-bin/external/hedron_compile_commands/refresh_all.py
INFO: Elapsed time: 128.581s, Critical Path: 1.16s
INFO: 14 processes: 12 internal, 2 processwrapper-sandbox.
INFO: Build completed successfully, 14 total actions
INFO: Running command line: bazel-bin/external/hedron_compile_commands/refresh_all
/home/envoybuild/.cache/bazel/_bazel_envoybuild/2d35de14639eaad1ac7060a4dd7e3351/execroot/envoy/bazel-out/k8-fastbuild/bin/external/hedron_compile_commands/refresh_all.runfiles/python3_12_x86_64-unknown-linux-gnu/bin/python3: can't open file '/home/envoybuild/.cache/bazel/_bazel_envoybuild/2d35de14639eaad1ac7060a4dd7e3351/execroot/envoy/bazel-out/k8-fastbuild/bin/external/hedron_compile_commands/refresh_all.runfiles/envoy/%interpreter_args%': [Errno 2] No such file or directory

[phlax]

yeah - this is a bug in their setup (hedronvision/bazel-compile-commands-extractor#168 (comment))

for some reason they switched to rules_python a couple of years ago and the reverted it (afaict because they didnt like the dumb no-root rule) - that is never going to work out of the box - struggling to understand how it would work for anyone

that said - fix should be pretty easy

[phlax]

hedronvision/bazel-compile-commands-extractor#268 (comment)

[phlax]

@tsaarni with some warnings (not sure how important), and using the pr version, this kinda works

would be good to know if what it produces is what we want

[tsaarni]

Is the correct way to run this bazel run //:refresh_compile_commands? It seemed to do a lot of work,but in the end it failed and didn’t generate a compile_commands.json for me

Traceback (most recent call last):
  File "/home/envoybuild/.cache/bazel/_bazel_envoybuild/2d35de14639eaad1ac7060a4dd7e3351/execroot/envoy/bazel-out/k8-fastbuild/bin/refresh_compile_commands.runfiles/envoy/refresh_compile_commands.py", line 1433, in <module>
    main()
    ~~~~^^
  File "/home/envoybuild/.cache/bazel/_bazel_envoybuild/2d35de14639eaad1ac7060a4dd7e3351/execroot/envoy/bazel-out/k8-fastbuild/bin/refresh_compile_commands.runfiles/envoy/refresh_compile_commands.py", line 1415, in main
    compile_command_entries.extend(_get_commands(target, flags))
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/envoybuild/.cache/bazel/_bazel_envoybuild/2d35de14639eaad1ac7060a4dd7e3351/execroot/envoy/bazel-out/k8-fastbuild/bin/refresh_compile_commands.runfiles/envoy/refresh_compile_commands.py", line 1277, in _get_commands
    yield from _convert_compile_commands(parsed_aquery_output)
  File "/home/envoybuild/.cache/bazel/_bazel_envoybuild/2d35de14639eaad1ac7060a4dd7e3351/execroot/envoy/bazel-out/k8-fastbuild/bin/refresh_compile_commands.runfiles/envoy/refresh_compile_commands.py", line 1157, in _convert_compile_commands
    for source_files, header_files, compile_command_args in outputs:
                                                            ^^^^^^^
  File "/usr/lib/python3.13/concurrent/futures/_base.py", line 619, in result_iterator
    yield _result_or_cancel(fs.pop())
          ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/usr/lib/python3.13/concurrent/futures/_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
           ~~~~~~~~~~^^^^^^^^^
  File "/usr/lib/python3.13/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib/python3.13/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/envoybuild/.cache/bazel/_bazel_envoybuild/2d35de14639eaad1ac7060a4dd7e3351/execroot/envoy/bazel-out/k8-fastbuild/bin/refresh_compile_commands.runfiles/envoy/refresh_compile_commands.py", line 1121, in _get_cpp_command_for_files
    source_files, header_files = _get_files(compile_action)
                                 ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/envoybuild/.cache/bazel/_bazel_envoybuild/2d35de14639eaad1ac7060a4dd7e3351/execroot/envoy/bazel-out/k8-fastbuild/bin/refresh_compile_commands.runfiles/envoy/refresh_compile_commands.py", line 625, in _get_files
    assert source_file_candidates, f"No source files found in compile args: {compile_action.arguments}.\nPlease file an issue with this information!"
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: No source files found in compile args: ['external/llvm_toolchain/bin/cc_wrapper.sh', '-xc++-header', '-fsyntax-only', '-U_FORTIFY_SOURCE', '--target=x86_64-unknown-linux-gnu', '-U_FORTIFY_SOURCE', '-fstack-protector', '-fno-omit-frame-pointer', '-fcolor-diagnostics', '-Wall', '-Wthread-safety', '-Wself-assign', '-std=c++20', '-stdlib=libc++', '-Xclang', '-fno-cxx-modules', '-Wno-module-import-in-extern-c', '-MD', '-MF', 'bazel-out/k8-fastbuild/bin/external/com_google_absl/absl/types/_objs/optional/optional.h.d', '-iquote', 'external/com_google_absl', '-iquote', 'bazel-out/k8-fastbuild/bin/external/com_google_absl', '-DABSL_MIN_LOG_LEVEL=4', '-Wno-deprecated-declarations', '-fdebug-types-section', '-fPIC', '-std=c++20', '-fsized-deallocation', '-Wall', '-Wextra', '-Wc++98-compat-extra-semi', '-Wcast-qual', '-Wconversion', '-Wdeprecated-pragma', '-Wfloat-overflow-conversion', '-Wfloat-zero-conversion', '-Wfor-loop-analysis', '-Wformat-security', '-Wgnu-redeclared-enum', '-Winfinite-recursion', '-Winvalid-constexpr', '-Wliteral-conversion', '-Wmissing-declarations', '-Wnullability-completeness', '-Woverlength-strings', '-Wpointer-arith', '-Wself-assign', '-Wshadow-all', '-Wshorten-64-to-32', '-Wsign-conversion', '-Wstring-conversion', '-Wtautological-overlap-compare', '-Wtautological-unsigned-zero-compare', '-Wthread-safety', '-Wundef', '-Wuninitialized', '-Wunreachable-code', '-Wunused-comparison', '-Wunused-local-typedefs', '-Wunused-result', '-Wvla', '-Wwrite-strings', '-Wno-float-conversion', '-Wno-implicit-float-conversion', '-Wno-implicit-int-float-conversion', '-Wno-unknown-warning-option', '-Wno-unused-command-line-argument', '-DNOMINMAX', '--sysroot=external/sysroot_linux_amd64/', '-c', 'external/com_google_absl/absl/types/optional.h', '-o', 'bazel-out/k8-fastbuild/bin/external/com_google_absl/absl/types/_objs/optional/optional.h.processed', '-no-canonical-prefixes', '-Wno-builtin-macro-redefined', '-D__DATE__="redacted"', '-D__TIMESTAMP__="redacted"', '-D__TIME__="redacted"'].
Please file an issue with this information!

Full log is here https://gist.github.com/tsaarni/53dbc62dcebb13777d6f3039b7940a9a