[clangd] "Assertion 'LocalDeclID < LocalNumDecls' failed." when using clangd with `--experimental-modules-support`

[jiixyj]

I'm trying out the new experimental module support of clangd with a ~25 KLOC large code base. On some files (~5 out of ~170), clangd crashes with an assertion failure in clang/lib/Serialization/ASTReader.cpp: "Assertion `LocalDeclID < LocalNumDecls' failed." With assertions disabled there are instead "random" crashes in the parser (I guess because of UB?).

The git hash is ff993f9 .

I haven't found a minimal, standalone reproducer yet, sadly. I have attached the output of clangd --compile-commands-dir=. --experimental-modules-support --check=src/ldgr/core.cppm and the backtrace in gdb.

Any ideas how to further debug this? I haven't found any pattern in the ~5 files that crash, although the crashes are reproducible.

clangd-output.txt
gdb-log.txt

[jiixyj]

This is the assertion in question:

llvm-project/clang/lib/Serialization/ASTReader.cpp

Line 963 in ff993f9

assert(LocalDeclID < LocalNumDecls);

It isn't hit if I compile the project normally with clang/cmake.

[llvmbot]

@llvm/issue-subscribers-clangd

Author: Jan Kokemüller (jiixyj)

I'm trying out the new experimental module support of clangd with a ~25 KLOC large code base. On some files (~5 out of ~170), clangd crashes with an assertion failure in `clang/lib/Serialization/ASTReader.cpp`: "Assertion `LocalDeclID < LocalNumDecls' failed." With assertions disabled there are instead "random" crashes in the parser (I guess because of UB?).

The git hash is ff993f9 .

I haven't found a minimal, standalone reproducer yet, sadly. I have attached the output of clangd --compile-commands-dir=. --experimental-modules-support --check=src/ldgr/core.cppm and the backtrace in gdb.

Any ideas how to further debug this? I haven't found any pattern in the ~5 files that crash, although the crashes are reproducible.

clangd-output.txt
gdb-log.txt

[llvmbot]

@llvm/issue-subscribers-clang-modules

Author: Jan Kokemüller (jiixyj)

I'm trying out the new experimental module support of clangd with a ~25 KLOC large code base. On some files (~5 out of ~170), clangd crashes with an assertion failure in `clang/lib/Serialization/ASTReader.cpp`: "Assertion `LocalDeclID < LocalNumDecls' failed." With assertions disabled there are instead "random" crashes in the parser (I guess because of UB?).

The git hash is ff993f9 .

I haven't found a minimal, standalone reproducer yet, sadly. I have attached the output of clangd --compile-commands-dir=. --experimental-modules-support --check=src/ldgr/core.cppm and the backtrace in gdb.

Any ideas how to further debug this? I haven't found any pattern in the ~5 files that crash, although the crashes are reproducible.

clangd-output.txt
gdb-log.txt

[shafik]

CC @HighCommander4 @ChuanqiXu9

[ChuanqiXu9]

Did you use the correct compilation databases?

[jiixyj]

Did you use the correct compilation databases?

I think so. I've symlinked it into the main project folder, but it also crashes if I directly run clangd inside the build folder where the compilation database lives.

I've had a quick look at the output of strace -ttt -f and found that sometimes, the BMI's of the build folder are used, and only sometimes the built ones in /tmp. Could this be a problem?

...
[pid  7100] 1741585644.430668 openat(AT_FDCWD, "/home/jan/git/ldgr/build/RelWithDebInfo/src/CMakeFiles/ldgr.dir/ldgr.util.strong_types.pcm", O_RDONLY|O_CLOEXEC) = 3
[pid  7100] 1741585644.430738 readlink("/proc/self/fd/3", "/home/jan/git/ldgr/build/RelWith"..., 4096) = 90
[pid  7100] 1741585644.430811 fstat(3, {st_mode=S_IFREG|0644, st_size=20976, ...}) = 0
[pid  7100] 1741585644.430897 fstat(3, {st_mode=S_IFREG|0644, st_size=20976, ...}) = 0
[pid  7100] 1741585644.430958 mmap(NULL, 20976, PROT_READ, MAP_PRIVATE|MAP_NORESERVE, 3, 0) = 0x7f828800e000
[pid  7100] 1741585644.431025 rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
[pid  7100] 1741585644.431081 close(3)  = 0
[pid  7100] 1741585644.431134 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid  7100] 1741585644.431241 openat(AT_FDCWD, "/home/jan/git/ldgr/build/RelWithDebInfo/src/CMakeFiles/ldgr.dir/ldgr.util.utf.pcm", O_RDONLY|O_CLOEXEC) = 3
[pid  7100] 1741585644.431304 readlink("/proc/self/fd/3", "/home/jan/git/ldgr/build/RelWith"..., 4096) = 81
[pid  7100] 1741585644.431369 fstat(3, {st_mode=S_IFREG|0644, st_size=2064280, ...}) = 0
[pid  7100] 1741585644.431460 fstat(3, {st_mode=S_IFREG|0644, st_size=2064280, ...}) = 0
[pid  7100] 1741585644.431517 mmap(NULL, 2064280, PROT_READ, MAP_PRIVATE|MAP_NORESERVE, 3, 0) = 0x7f827ee08000
[pid  7100] 1741585644.431576 rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
[pid  7100] 1741585644.431633 close(3)  = 0
[pid  7100] 1741585644.431685 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid  7100] 1741585644.453950 newfstatat(AT_FDCWD, "/tmp/clangd/module_files/primitives.cppm9961093566810085741-f2-90-4b-3d-e6-a5", {st_mode=S_IFDIR|0750, st_size=60, ...}, 0) = 0
[pid  7100] 1741585644.454126 openat(AT_FDCWD, "/tmp/clangd/module_files/primitives.cppm9961093566810085741-f2-90-4b-3d-e6-a5/ldgr.primitives.pcm", O_RDONLY|O_CLOEXEC) = 3
[pid  7100] 1741585644.454225 readlink("/proc/self/fd/3", "/tmp/clangd/module_files/primiti"..., 4096) = 97
[pid  7100] 1741585644.454340 fstat(3, {st_mode=S_IFREG|0644, st_size=27556, ...}) = 0
[pid  7100] 1741585644.454432 fstat(3, {st_mode=S_IFREG|0644, st_size=27556, ...}) = 0
[pid  7100] 1741585644.454542 mmap(NULL, 27556, PROT_READ, MAP_PRIVATE|MAP_NORESERVE, 3, 0) = 0x7f8288007000
[pid  7100] 1741585644.454639 rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
[pid  7100] 1741585644.454730 close(3)  = 0
[pid  7100] 1741585644.454809 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid  7100] 1741585644.454944 openat(AT_FDCWD, "/home/jan/git/ldgr/build/RelWithDebInfo/src/CMakeFiles/ldgr.dir/ldgr.precision.pcm", O_RDONLY|O_CLOEXEC) = 3
[pid  7100] 1741585644.455036 readlink("/proc/self/fd/3", "/home/jan/git/ldgr/build/RelWith"..., 4096) = 82
[pid  7100] 1741585644.455130 fstat(3, {st_mode=S_IFREG|0644, st_size=26680, ...}) = 0
[pid  7100] 1741585644.455216 fstat(3, {st_mode=S_IFREG|0644, st_size=26680, ...}) = 0
[pid  7100] 1741585644.455300 mmap(NULL, 26680, PROT_READ, MAP_PRIVATE|MAP_NORESERVE, 3, 0) = 0x7f8288000000
[pid  7100] 1741585644.455382 rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
[pid  7100] 1741585644.455470 close(3)  = 0
[pid  7100] 1741585644.455543 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
...

When I delete the .pcm files from the build folder with find src -type f -name '*.pcm' -delete I'm getting a different crash, ending in:

E[07:01:57.080] Failed to build module ldgr.balance_sheet.value; due to Compilation failed
I[07:01:57.080] Built prerequisite modules for file /home/jan/git/ldgr/src/ldgr/report.cpp in 6.72 seconds
I[07:01:57.080] Indexing headers...
I[07:01:57.085] Building AST...
clangd: /freebsd/data/linux/git-net/llvm-project/clang/lib/Tooling/Syntax/Tokens.cpp:717: TokenBuffer clang::syntax::TokenCollector::Builder::build() &&: Assertion `Result.ExpandedTokens.back().kind() == tok::eof' failed.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: clangd --compile-commands-dir=. --experimental-modules-support --check=src/ldgr/report.cpp
 #0 0x000055a7499016b8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /freebsd/data/linux/git-net/llvm-project/llvm/lib/Support/Unix/Signals.inc:804:13
 #1 0x000055a7498ff900 llvm::sys::RunSignalHandlers() /freebsd/data/linux/git-net/llvm-project/llvm/lib/Support/Signals.cpp:106:18
 #2 0x000055a749901ec1 SignalHandler(int, siginfo_t*, void*) /freebsd/data/linux/git-net/llvm-project/llvm/lib/Support/Unix/Signals.inc:0:3
 #3 0x00007f2c324558c0 __restore_rt libc_sigaction.c:0:0
 #4 0x00007f2c324a462c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
 #5 0x00007f2c32455822 gsignal ./signal/../sysdeps/posix/raise.c:27:6
 #6 0x00007f2c3243e4af abort ./stdlib/abort.c:81:7
 #7 0x00007f2c3243e3d7 _nl_load_domain.cold ./intl/loadmsgcat.c:1177:9
 #8 0x00007f2c3244e342 (/usr/lib/libc.so.6+0x38342)
 #9 0x000055a74aa2ad0b (/freebsd/data/linux/git-net/llvm-project/prefix/bin/clangd+0x15aad0b)
#10 0x000055a74aa2a802 llvm::DenseMap<clang::SourceLocation, clang::SourceLocation, llvm::DenseMapInfo<clang::SourceLocation, void>, llvm::detail::DenseMapPair<clang::SourceLocation, clang::SourceLocation>>::~DenseMap() /freebsd/data/linux/git-net/llvm-project/llvm/include/llvm/ADT/DenseMap.h:766:23
#11 0x000055a74aa2a802 clang::syntax::TokenCollector::Builder::~Builder() /freebsd/data/linux/git-net/llvm-project/clang/lib/Tooling/Syntax/Tokens.cpp:706:23
#12 0x000055a74aa2a802 clang::syntax::TokenCollector::consume() && /freebsd/data/linux/git-net/llvm-project/clang/lib/Tooling/Syntax/Tokens.cpp:898:3
#13 0x000055a74ac570e2 clang::clangd::ParsedAST::build(llvm::StringRef, clang::clangd::ParseInputs const&, std::unique_ptr<clang::CompilerInvocation, std::default_delete<clang::CompilerInvocation>>, llvm::ArrayRef<clang::clangd::Diag>, std::shared_ptr<clang::clangd::PreambleData const>) /freebsd/data/linux/git-net/llvm-project/clang-tools-extra/clangd/ParsedAST.cpp:726:10
#14 0x000055a74aa3b5bd clang::clangd::(anonymous namespace)::Checker::buildAST() /freebsd/data/linux/git-net/llvm-project/clang-tools-extra/clangd/tool/Check.cpp:262:5
#15 0x000055a74aa3b5bd clang::clangd::check(llvm::StringRef, clang::clangd::ThreadsafeFS const&, clang::clangd::ClangdLSPServer::Options const&) /freebsd/data/linux/git-net/llvm-project/clang-tools-extra/clangd/tool/Check.cpp:522:10
#16 0x000055a74aa30761 clang::clangd::clangdMain(int, char**) /freebsd/data/linux/git-net/llvm-project/clang-tools-extra/clangd/tool/ClangdMain.cpp:1014:12
#17 0x00007f2c3243fc4c __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:74:3
#18 0x00007f2c3243fd05 call_init ./csu/../csu/libc-start.c:128:20
#19 0x00007f2c3243fd05 __libc_start_main@GLIBC_2.2.5 ./csu/../csu/libc-start.c:347:5
#20 0x000055a7497dd9e1 _start /builddir/glibc-2.39/csu/../sysdeps/x86_64/start.S:117:0
zsh: IOT instruction  clangd --compile-commands-dir=. --experimental-modules-support 

Maybe those errors are to blame somehow?

error: unable to handle compilation, expected exactly one compiler job in '/freebsd/data/linux/git-net/llvm-project/prefix/bin/clang++ --driver-mode=g++ -I/freebsd/data/linux/git-net/llvm-project/prefix/bin/../lib /x86_64-unknown-linux-gnu/../../share/libc++/v1 -m64 -stdlib=libc++ -fexperimental-library -O2 -g -fcolor-diagnostics -Wno-reserved-module-identifier -c -resource-dir=/freebsd/data/linux/git-net/llvm-project/prefi x/lib/clang/21 -- '

[jiixyj]

Here is the compilation database in question, generated by CMake 3.30.1.

compile_commands.json

[ChuanqiXu9]

Did you use the correct compilation databases?

I think so. I've symlinked it into the main project folder, but it also crashes if I directly run clangd inside the build folder where the compilation database lives.

I've had a quick look at the output of strace -ttt -f and found that sometimes, the BMI's of the build folder are used, and only sometimes the built ones in /tmp. Could this be a problem?

I guess so. The assertion says, the loaded decls are more than what expected. And may be it is due to inconsistent setups.

Did you use -fmodules-reduced-bmi in your normal build? If not, I think maybe it worth to give it a try.

...
[pid  7100] 1741585644.430668 openat(AT_FDCWD, "/home/jan/git/ldgr/build/RelWithDebInfo/src/CMakeFiles/ldgr.dir/ldgr.util.strong_types.pcm", O_RDONLY|O_CLOEXEC) = 3
[pid  7100] 1741585644.430738 readlink("/proc/self/fd/3", "/home/jan/git/ldgr/build/RelWith"..., 4096) = 90
[pid  7100] 1741585644.430811 fstat(3, {st_mode=S_IFREG|0644, st_size=20976, ...}) = 0
[pid  7100] 1741585644.430897 fstat(3, {st_mode=S_IFREG|0644, st_size=20976, ...}) = 0
[pid  7100] 1741585644.430958 mmap(NULL, 20976, PROT_READ, MAP_PRIVATE|MAP_NORESERVE, 3, 0) = 0x7f828800e000
[pid  7100] 1741585644.431025 rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
[pid  7100] 1741585644.431081 close(3)  = 0
[pid  7100] 1741585644.431134 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid  7100] 1741585644.431241 openat(AT_FDCWD, "/home/jan/git/ldgr/build/RelWithDebInfo/src/CMakeFiles/ldgr.dir/ldgr.util.utf.pcm", O_RDONLY|O_CLOEXEC) = 3
[pid  7100] 1741585644.431304 readlink("/proc/self/fd/3", "/home/jan/git/ldgr/build/RelWith"..., 4096) = 81
[pid  7100] 1741585644.431369 fstat(3, {st_mode=S_IFREG|0644, st_size=2064280, ...}) = 0
[pid  7100] 1741585644.431460 fstat(3, {st_mode=S_IFREG|0644, st_size=2064280, ...}) = 0
[pid  7100] 1741585644.431517 mmap(NULL, 2064280, PROT_READ, MAP_PRIVATE|MAP_NORESERVE, 3, 0) = 0x7f827ee08000
[pid  7100] 1741585644.431576 rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
[pid  7100] 1741585644.431633 close(3)  = 0
[pid  7100] 1741585644.431685 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid  7100] 1741585644.453950 newfstatat(AT_FDCWD, "/tmp/clangd/module_files/primitives.cppm9961093566810085741-f2-90-4b-3d-e6-a5", {st_mode=S_IFDIR|0750, st_size=60, ...}, 0) = 0
[pid  7100] 1741585644.454126 openat(AT_FDCWD, "/tmp/clangd/module_files/primitives.cppm9961093566810085741-f2-90-4b-3d-e6-a5/ldgr.primitives.pcm", O_RDONLY|O_CLOEXEC) = 3
[pid  7100] 1741585644.454225 readlink("/proc/self/fd/3", "/tmp/clangd/module_files/primiti"..., 4096) = 97
[pid  7100] 1741585644.454340 fstat(3, {st_mode=S_IFREG|0644, st_size=27556, ...}) = 0
[pid  7100] 1741585644.454432 fstat(3, {st_mode=S_IFREG|0644, st_size=27556, ...}) = 0
[pid  7100] 1741585644.454542 mmap(NULL, 27556, PROT_READ, MAP_PRIVATE|MAP_NORESERVE, 3, 0) = 0x7f8288007000
[pid  7100] 1741585644.454639 rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
[pid  7100] 1741585644.454730 close(3)  = 0
[pid  7100] 1741585644.454809 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid  7100] 1741585644.454944 openat(AT_FDCWD, "/home/jan/git/ldgr/build/RelWithDebInfo/src/CMakeFiles/ldgr.dir/ldgr.precision.pcm", O_RDONLY|O_CLOEXEC) = 3
[pid  7100] 1741585644.455036 readlink("/proc/self/fd/3", "/home/jan/git/ldgr/build/RelWith"..., 4096) = 82
[pid  7100] 1741585644.455130 fstat(3, {st_mode=S_IFREG|0644, st_size=26680, ...}) = 0
[pid  7100] 1741585644.455216 fstat(3, {st_mode=S_IFREG|0644, st_size=26680, ...}) = 0
[pid  7100] 1741585644.455300 mmap(NULL, 26680, PROT_READ, MAP_PRIVATE|MAP_NORESERVE, 3, 0) = 0x7f8288000000
[pid  7100] 1741585644.455382 rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
[pid  7100] 1741585644.455470 close(3)  = 0
[pid  7100] 1741585644.455543 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
...

When I delete the .pcm files from the build folder with find src -type f -name '*.pcm' -delete I'm getting a different crash, ending in:

E[07:01:57.080] Failed to build module ldgr.balance_sheet.value; due to Compilation failed
I[07:01:57.080] Built prerequisite modules for file /home/jan/git/ldgr/src/ldgr/report.cpp in 6.72 seconds
I[07:01:57.080] Indexing headers...
I[07:01:57.085] Building AST...
clangd: /freebsd/data/linux/git-net/llvm-project/clang/lib/Tooling/Syntax/Tokens.cpp:717: TokenBuffer clang::syntax::TokenCollector::Builder::build() &&: Assertion `Result.ExpandedTokens.back().kind() == tok::eof' failed.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: clangd --compile-commands-dir=. --experimental-modules-support --check=src/ldgr/report.cpp
 #0 0x000055a7499016b8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /freebsd/data/linux/git-net/llvm-project/llvm/lib/Support/Unix/Signals.inc:804:13
 #1 0x000055a7498ff900 llvm::sys::RunSignalHandlers() /freebsd/data/linux/git-net/llvm-project/llvm/lib/Support/Signals.cpp:106:18
 #2 0x000055a749901ec1 SignalHandler(int, siginfo_t*, void*) /freebsd/data/linux/git-net/llvm-project/llvm/lib/Support/Unix/Signals.inc:0:3
 #3 0x00007f2c324558c0 __restore_rt libc_sigaction.c:0:0
 #4 0x00007f2c324a462c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
 #5 0x00007f2c32455822 gsignal ./signal/../sysdeps/posix/raise.c:27:6
 #6 0x00007f2c3243e4af abort ./stdlib/abort.c:81:7
 #7 0x00007f2c3243e3d7 _nl_load_domain.cold ./intl/loadmsgcat.c:1177:9
 #8 0x00007f2c3244e342 (/usr/lib/libc.so.6+0x38342)
 #9 0x000055a74aa2ad0b (/freebsd/data/linux/git-net/llvm-project/prefix/bin/clangd+0x15aad0b)
#10 0x000055a74aa2a802 llvm::DenseMap<clang::SourceLocation, clang::SourceLocation, llvm::DenseMapInfo<clang::SourceLocation, void>, llvm::detail::DenseMapPair<clang::SourceLocation, clang::SourceLocation>>::~DenseMap() /freebsd/data/linux/git-net/llvm-project/llvm/include/llvm/ADT/DenseMap.h:766:23
#11 0x000055a74aa2a802 clang::syntax::TokenCollector::Builder::~Builder() /freebsd/data/linux/git-net/llvm-project/clang/lib/Tooling/Syntax/Tokens.cpp:706:23
#12 0x000055a74aa2a802 clang::syntax::TokenCollector::consume() && /freebsd/data/linux/git-net/llvm-project/clang/lib/Tooling/Syntax/Tokens.cpp:898:3
#13 0x000055a74ac570e2 clang::clangd::ParsedAST::build(llvm::StringRef, clang::clangd::ParseInputs const&, std::unique_ptr<clang::CompilerInvocation, std::default_delete<clang::CompilerInvocation>>, llvm::ArrayRef<clang::clangd::Diag>, std::shared_ptr<clang::clangd::PreambleData const>) /freebsd/data/linux/git-net/llvm-project/clang-tools-extra/clangd/ParsedAST.cpp:726:10
#14 0x000055a74aa3b5bd clang::clangd::(anonymous namespace)::Checker::buildAST() /freebsd/data/linux/git-net/llvm-project/clang-tools-extra/clangd/tool/Check.cpp:262:5
#15 0x000055a74aa3b5bd clang::clangd::check(llvm::StringRef, clang::clangd::ThreadsafeFS const&, clang::clangd::ClangdLSPServer::Options const&) /freebsd/data/linux/git-net/llvm-project/clang-tools-extra/clangd/tool/Check.cpp:522:10
#16 0x000055a74aa30761 clang::clangd::clangdMain(int, char**) /freebsd/data/linux/git-net/llvm-project/clang-tools-extra/clangd/tool/ClangdMain.cpp:1014:12
#17 0x00007f2c3243fc4c __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:74:3
#18 0x00007f2c3243fd05 call_init ./csu/../csu/libc-start.c:128:20
#19 0x00007f2c3243fd05 __libc_start_main@GLIBC_2.2.5 ./csu/../csu/libc-start.c:347:5
#20 0x000055a7497dd9e1 _start /builddir/glibc-2.39/csu/../sysdeps/x86_64/start.S:117:0
zsh: IOT instruction  clangd --compile-commands-dir=. --experimental-modules-support 

Maybe those errors are to blame somehow?

error: unable to handle compilation, expected exactly one compiler job in '/freebsd/data/linux/git-net/llvm-project/prefix/bin/clang++ --driver-mode=g++ -I/freebsd/data/linux/git-net/llvm-project/prefix/bin/../lib /x86_64-unknown-linux-gnu/../../share/libc++/v1 -m64 -stdlib=libc++ -fexperimental-library -O2 -g -fcolor-diagnostics -Wno-reserved-module-identifier -c -resource-dir=/freebsd/data/linux/git-net/llvm-project/prefi x/lib/clang/21 -- '

I had no clue. Did you restart the clangd server after removing these .pcm in build dir?