Skip to content

[cpullvm] Don't stop on first test failure in Linux builds, write test results to xml logs#360

Draft
jonathonpenix wants to merge 1 commit into
qualcomm:qualcomm-softwarefrom
jonathonpenix:pr/xml_test_logs
Draft

[cpullvm] Don't stop on first test failure in Linux builds, write test results to xml logs#360
jonathonpenix wants to merge 1 commit into
qualcomm:qualcomm-softwarefrom
jonathonpenix:pr/xml_test_logs

Conversation

@jonathonpenix

Copy link
Copy Markdown
Contributor

With automerge now enabled, we have had more consistent failures, even in Linux x86 builds. And, we've now seen failures that have been masked by other failures since we stop after the first one.

So let's try what seems to be done in ATfE and use lit's --ignore-fail and dump all results into xml.

A few things to note:

  1. Windows builds are not impacted by this--they will still fail at the first error. We don't have QEMU testing enabled there, so we're really only running the lit tests anyway, so I don't think this is as large of an issue. We can re-evaluate this as needed.
  2. We might need to think about how this works for ex: PRs. As-is, if there are failures I think the precheckin build will still succeed. So either we need an extra step, or people need to be diligent about checking the test reports.
  3. I'm also not sure how this will work with eld? Given that its lit tests are split with different defaults. There might be some work needed to make this work well.

Note that the code added here is from or is derived from ATfE. All license/copyright info we already have in the modified files should already be correct/appropriate.

@jonathonpenix jonathonpenix changed the title [cpullvm] Run full suite of tests, dump to xml logs [cpullvm] Don't stop on first test failure in Linux builds, write test results to xml logs May 15, 2026
…t results to xml logs

With automerge now enabled, we have had more consistent failures, even
in Linux x86 builds. And, we've now seen failures that have been masked
by other failures since we stop after the first one.

So let's try what seems to be done in ATfE and use lit's `--ignore-fail` and
dump all results into xml.

A few things to note:
1. Windows builds are not impacted by this--they will still fail at the first
   error. We don't have QEMU testing enabled there, so we're really only
   running the lit tests anyway, so I don't think this is as large of an issue.
   We can re-evaluate this as needed.
2. We might need to think about how this works for ex: PRs. As-is, if
   there are failures I *think* the precheckin build will still succeed. So
   either we need an extra step, or people need to be diligent about
   checking the test reports.
3. I'm also not sure how this will work with eld? Given that its lit tests
   are split with different defaults. There might be some work needed to make
   this work well.

Note that the code added here is from or is derived from ATfE. All
license/copyright info we already have in the modified files should already be
correct/appropriate.

Signed-off-by: Jonathon Penix <jpenix@qti.qualcomm.com>
@jonathonpenix

jonathonpenix commented May 15, 2026

Copy link
Copy Markdown
Contributor Author

This might need more work as mentioned in the description, but just creating this for discussion about what Arm does, to see what the results would look like, etc.

Edit: Also confirming that we're going to need to do something for eld to handle the whole "invoke lit multiple times" thing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant