[cpullvm] - Enable LIBCXX testing for aarch64/arm/riscv if LIBC tests are enabled

[pawan-nirpal-031]

Enable LIBCXX tests for aarch64/arm/riscv arch, if the corresponding LIBC tests are enabled.

Signed-off-by: Pawan Nirpal pnirpal@qti.qualcomm.com

[pawan-nirpal-031]

@jonathonpenix Could you please review and add other reviewers as you see fit.

[jonathonpenix]

At a glance, it looks like this is headed in the right direction, thanks Pawan!

Few misc. thoughts:

The libc++abi/libunwind tests are also included under the broader "enable libc++ tests". Right now they're running as we just run all enabled tests:

cpullvm-toolchain/qualcomm-software/scripts/test.sh

Lines 20 to 24 in 8f6aeba

	# Run all relevant test targets. This might be too broad eventually,
	# but while we have a limited number of variants (and no compiler-rt
	# or libc++ testing enabled) we can run everything.
	cd "${REPO_ROOT}"/build
	ninja check-all-llvm-toolchain

I'm fine if we enable this stuff incrementally, but I think we have a few options to do so. Things that make the most sense to me:

1. If we get down to just test failures, we can xfail and gradually re-enable tests and retain the check-all*
2. If we want to ex: merge this and address libcxxabi/libunwind build failures later, we can break up the check-all-llvm-toolchain into smaller components and exclude abi/unwind.

@pawan-nirpal-031 do you know/remember how long the libc++ tests were taking to run on your local machine?

Arm has these split out as they apparently take a long time to run, so we might want to do # 2 above even if we resolve build/test failures.

[lenary]

I am not 100% sure how this ends up working, but are we correctly setting long_tests and large_tests? I think we should be if we're using qualcomm-software/embedded-runtimes/test-support/llvm-libc++-picolibc.cfg.in but I'm not sure if we are using that config or something else.

These prevent some of the really slow tests from running.

[apazos]

They are already turned off in
embedded-runtimes/test-support/llvm-libc++-picolibc.cfg.inllvm-libc++-picolibc.cfg.in:
config.long_tests = False
config.large_tests = False

@pawan-nirpal-031, let's focus on fixing the issues first. We can merge the fixes without enabling the tests. Before activating the tests, we may need to split them into separate runs, as they seem to still take considerable time even with long/large tests disabled.

[pawan-nirpal-031]

@apazos Yes I think that would be preferable. I've fixed the tests, some of them have to be unsupported due to memory limitations set to 2MB IIRC, those tests run into memory issues and I suppose aren't meant for embedded. Long double tests have to be xfailed through and through. But I will update this PR and we can discuss on the approach more.

CC @jonathonpenix @lenary

[pawan-nirpal-031]

Large test

These tests allocate very large amounts of memory (1,000,000 element containers, 1M-element vectors) that exceed what's available in the QEMU semihosting environment for bare-metal targets. The process aborts when malloc fails or the heap overflows.
Specifically:
- flat.map/size.pass.cpp: inserts 1,000,000 elements into a flat_map
- sort.heap/complexity.pass.cpp: allocates a 1M-element vector (1 << 20) and runs std::mt19937 over it
- eval_param.pass.cpp and simd_copy.pass.cpp: similarly large workloads
The bare-metal QEMU machine has a fixed, limited RAM (defined by the linker script picolibcpp.ld). When the test tries to allocate more than what's available, malloc returns null or the heap collides with the stack, causing an abort (signal 6 → exit 134).

std/algorithms/alg.sorting/alg.heap.operations/sort.heap/complexity.pass.cpp
 libc++-aarch64a_tlsie :: std/algorithms/alg.sorting/alg.heap.operations/sort.heap/ranges_sort_heap.pass.cpp
  libc++-aarch64a_tlsie :: std/containers/container.adaptors/flat.map/flat.map.capacity/size.pass.cpp
  libc++-aarch64a_tlsie :: std/containers/container.adaptors/flat.set/flat.set.capacity/size.pass.cpp
  libcxx/test/std/experimental/simd/simd.class/simd_copy.pass.cpp

UNSUPPORTED : Picolibc doesn't yet support stdio for long doubles for aarch64/arm/riscv

std/input.output/iostream.format/output.streams/ostream.formatted/ostream.inserters.arithmetic/long_double.pass.cpp
		  libc++-aarch64a_tlsie :: std/language.support/support.start.term/quick_exit.pass.cpp
		  libc++-aarch64a_tlsie :: std/localization/locale.categories/category.monetary/locale.money.get/locale.money.get.members/get_long_double_overlong.pass.cpp
		  libc++-aarch64a_tlsie :: std/localization/locale.categories/category.numeric/locale.nm.put/facet.num.put.members/put_long_double.hex.pass.cpp
		  libc++-aarch64a_tlsie :: std/localization/locale.categories/category.numeric/locale.nm.put/facet.num.put.members/put_long_double.pass.cpp
		std/strings/string.conversions/to_string.pass.cpp
		std/strings/string.conversions/stold.pass.cpp
                

Picolibc simply doesn't implement quick_exit or at_quick_exit yet. 
libcxx/test/std/language.support/support.start.term/quick_exit.pass.cpp

yet to get a hold on this one : libcxx/test/std/strings/c.strings/no_c8rtomb_mbrtoc8.verify.cpp

[apazos]

Hi @pawan-nirpal-031,

Besides tests failing, there are several ways to disable tests to control the time it takes to run the tests. Bringing them up here so we select the most appropriate one depending on the tests running time and the tests failing, so we can progressively enable them as we fix the issues:

1. Select the ninja check command to run in /qualcomm-software/qualcomm-software/scripts/test.sh
   Right now, we enable ninja check-all-llvm-toolchain (checks all LLVM/clang binaries and runtimes but excludes lib C++/abi/undwind tests)

We can consider enabling tests for libc++ in parts, as done in ATfe, they skip ninja check-cxx which takes long, see ATfE /embedded/scripts/test.sh
ninja check-all (checks all LLVM/clang binaries and runtimes including compiler-rt, lib C++/abi/undwind)
ninja check-all-llvm-toolchain (excludes lib C++ runtimes tests)
ninja check-cxxabi
ninja check-unwind
ninja check-cxx

2. ATfE has a separate /embedded/scripts/test_libcxx.sh where they only enable ninja check-cxx

3. This alternative allows to specify the particular project, variants and tests to disable
   https://github.com/qualcomm/cpullvm-toolchain/blob/qualcomm-software/qualcomm-software/embedded-runtimes/test-support/xfails.py
   We can check the tests disabled in ATfE /embedded/arm-runtimes/test_support/xfails.py

For example, if you only enable ninja check-cxxabi or ninja check-unwind or both, are there failures?

[pawan-nirpal-031]

@apazos Thanks for the detailed description, Yes I feel incrementally enabling would be the best option, ninja check-cxx are the ones that I am focusing on this PR. I imagine we don't want any source changes apart from changes in qualcomm-sotfware/* is that right? If yes I don't think there's much we can do to disable failing tests that need source changes, I'm wrestling with some riscv32* tests that are need support from upstream, Do you think we can push some of these libcxx source changes upstream first? Or we can disable the tests now and then start upstreaming fixes into libcxx?

[apazos]

We have no custom LLVM changes in qualcomm-software and it keeps up with latest LLVM tip.
Which issue are you running into with riscv32* tests?
I am not sure the changes to tests are upstreamable (they reflect that in the particular runners/builders we are using they take a long time) and the way I see to disable the tests is along what ATfE does in /embedded/arm-runtimes/test_support/xfails.py.

[pawan-nirpal-031]

@apazos @jonathonpenix @lenary I have worked libcxx into a different test_libcxx.sh test script file similar to ATFE, All the tests locally pass, however I am omitting check unwind, abi and picolibc tests. The ETA for the libcxx tests locally is around 6-6:30 hours. The failure now seems to be purely CI timeout. The job has exceeded the maximum execution time of 6h0m0s

[apazos]

@pawan-nirpal-031 this patch will have to be further cleaned up as some of these changes you are pushing in a separate PR. Also, might make sense to combine the xfailed test patch with the patch that enables the tests.

[pawan-nirpal-031]

@pawan-nirpal-031 this patch will have to be further cleaned up as some of these changes you are pushing in a separate PR. Also, might make sense to combine the xfailed test patch with the patch that enables the tests.

Hi @apazos I'm sorry I didn't quite understand, you said we were only going to push the fixes and defer the enablement to a later PR. I will clean up the infra changes alright. Then as per me this PR should have xfails.py (fixes) and the separation of test_libcxx.sh. So if we also push the xfails to the enablement patch, this is just a CI change patch which separates the ninja check, is that what we intend to achieve? Apologies for the confusion.