[cpullvm] Run full suite of tests, dump to xml logs

With automerge now enabled, we have had more consistent failures, even
in Linux x86 builds. And, we've now seen failures that have been masked
by other failures since we stop after the first one.

So let's try what seems to be done in ATfE and use lit's `--ignore-fail` and
dump all results into xml.

A few things to note:
1. Windows builds are not impacted by this--they will still fail at the first
   error. We don't have QEMU testing enabled there, so we're really only
   running the lit tests anyway, so I don't think this is as large of an issue.
   We can re-evaluate this as needed.
2. We might need to think about how this works for ex: PRs. As-is, if
   there are failures I *think* the precheckin build will still succeed. So
   either we need an extra step, or people need to be diligent about
   checking the test reports.
3. I'm also not sure how this will work with eld? Given that its lit tests
   are split with different defaults. There might be some work needed to make
   this work well.

Note that the code added here is from or is derived from ATfE. All
license/copyright info we already have in the modified files should already be
correct/appropriate.

Signed-off-by: Jonathon Penix <jpenix@qti.qualcomm.com>

Author: jonathonpenix

.github/workflows/native-runtime-nightly.yml

@@ -104,3 +104,10 @@ jobs:
       - name: Test
         if: matrix.test_script != ''
         run: ./qualcomm-software/scripts/${{ matrix.test_script }}
+
+      - name: Upload test results
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
+        if: always()
+        with:
+          name: test-results-${{ matrix.build_script }}-${{ matrix.target_os }}
+          path: build/**/lit_results.junit.xml

.github/workflows/nightly.yml

@@ -62,6 +62,13 @@ jobs:
       - name: Test
         run: ./qualcomm-software/scripts/test.sh
 
+      - name: Upload test results
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
+        if: always()
+        with:
+          name: test-results-build.sh-Linux-x86
+          path: build/**/lit_results.junit.xml
+
   build-and-test:
     if: github.repository == 'qualcomm/cpullvm-toolchain'
     name: ${{ matrix.build_script }} on ${{ matrix.target_os }}
@@ -142,3 +149,10 @@ jobs:
       - name: Test
         if: matrix.test_script != ''
         run: ./qualcomm-software/scripts/${{ matrix.test_script }}
+
+      - name: Upload test results
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
+        if: always()
+        with:
+          name: test-results-${{ matrix.build_script }}-${{ matrix.target_os }}
+          path: build/**/lit_results.junit.xml

qualcomm-software/scripts/test.sh

@@ -17,6 +17,16 @@ set -ex
 SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 REPO_ROOT=$( git -C "${SCRIPT_DIR}" rev-parse --show-toplevel )
 
+# If a test fails, lit will ordinarily return a non-zero result,
+# which prevents further testing. Setting the --ignore-fail option
+# will cause testing to continue, so that CI systems can get a
+# full set of results.
+# The lit test suites do not generate xml results by default.
+# This can be enabled with the --xunit-xml-output option. The file
+# written will be relative to the individual suite's build directly,
+# so the same name can be used for all files for consistency.
+export LIT_OPTS="--ignore-fail --xunit-xml-output=lit_results.junit.xml"
+
 # Run all relevant test targets. This might be too broad eventually,
 # but while we have a limited number of variants (and no compiler-rt
 # or libc++ testing enabled) we can run everything.