merge main into amd-staging - #3553
Merged
rocm-cciapp[bot] merged 32 commits intoJul 24, 2026
Merged
Conversation
…m#184466) The CoroSplit and CoroFrame generate several control flow instructions (`br`, `select`, `switch`) that previously lacked `!prof` branch weight metadata. When a coroutine is split, most basic blocks are simply cloned into the generated functions (`.resume`, `.destroy`, and `.cleanup`) and don't actually change their internal structure or layout. This PR changes: 1. Dispatch Branch Weights: Since the original coroutine already ran through PGO, we already have the `SuspendFreqs` records telling us exactly how many times each suspension point was executed in the real world. In the new clone `.resume` functions, the compiler generates new routing `switch` instruction to direct the traffic. Instead of guessing the probabilities for these new routes, we simply look back at the `SuspendFreqs` records, calculate the exact execution frequency and directly onto the new routing `switch` instructions as `!prof` data. 2. Function Entry Count: We compute the entry count for the generated `.resume` function by summing up the frequencies of all suspension target blocks, ensuring the `.resume` function's entry metadata accurately reflects its true invocation frequency rather than inheriting original counts. Two new test cases is added see: ``` llvm/test/Transforms/Coroutines/coro-split-resume-entry-count-no-suspend.ll llvm/test/Transforms/Coroutines/coro-split-resume-entry-count.ll ``` 3. Others updates add branch weight metadata in several locations within these passes: * **CoroSplit.cpp**: * `handleFinalSuspend`: Added an equal (50/50) weight to conditional branches handling final-suspend conditions via `setExplicitlyUnknownBranchWeightsIfProfiled`. * `updateCoroFrame`: Adds an equal (50/50) weight to the select instruction that chooses either the `destroy` or `cleanup` function by using `setExplicitlyUnknownBranchWeightsIfProfiled`. * **CoroFrame.cpp**: * `rewritePHIsForCleanupPad`: Adds a branch weight of {0, 1, 1} to two-way switch instructions driven by PHI nodes generated in the EH dispatch block (.corodispatch). This provides accurate probabilistic information for `SimplifyCFG`, which typically translates such `switch` ``` cleanup2.corodispatch: ; preds = %catch.dispatch.2, %handler2 %1 = phi i8 [ 0, %handler2 ], [ 1, %catch.dispatch.2 ] %2 = cleanuppad within %h1 [] switch i8 %1, label %unreachable [ i8 0, label %cleanup2.from.handler2 i8 1, label %cleanup2.from.catch.dispatch.2 ] ``` into `icmp` + `br`. ``` cleanup2.corodispatch: ; preds = %catch.dispatch.2, %handler2 %1 = phi i8 [ 0, %handler2 ], [ 1, %catch.dispatch.2 ] %2 = cleanuppad within %h1 [] %3 = icmp eq i8 %1, 0 br i1 %3, label %cleanup2.from.handler2, label %cleanup2.from.catch.dispatch.2 ``` * **CoroCleanup.cc** * Setting the `unknown` profile to `NoopFn`/`__NoopCoro_ResumeDestroy` to avoid the false warnings. because this function is generated by this the pass, PGO has no profile count for it. Co-authored-by: Jin Huang <jingold@google.com>
These are still failing after 4f8de76.
- 16x16x64 FP8/BF8 WMMA run in 4 cycles, not 8. Add a 4-cycle WriteXDL1PassWMMA write and split the FP8/BF8 InstRW by shape so that 16x16x128 FP8/BF8 (and F16/BF16) keep the 8-cycle latency. - f8f6f4 WMMA run in 4 cycles when both matrix inputs are f4 and 8 cycles when any input is f6 or f8, instead of the previous 8/16. Update the scheduling predicate accordingly.
After llvm#207274, libc++ fails __assume_aligned assertion in __assume_valid_range. https://lab.llvm.org/buildbot/#/builders/25/builds/18994 `DenseArrayAttrImpl<T>::operator ArrayRef<T>()` is just a cast of `raw.data()`. `raw` is aligned copy of range from from BytecodeReader done by Builtin_DenseArrayRawDataParameter allocator. However, if range is empty, aligned copying was omitted, leaving unaligned `ArrayRef<char>`. The fix is to replace unaligned `ArrayRef<>` with aligned default constructed.
…ding table-based cttz (llvm#211202) Do not attempt to zext (or truncate) the table's first element into the cttz input type, as the element type may be wider than the input type. Instead, convert the cttz result and emit the select directly in the table's access type, which is the final expected type as well. Minor opportunity to favor using APInt when the first element has more than 64 active bits to avoid crashing. Fixes: llvm#210890.
Add statistics for the numbers of explicit and synthesized register classes, along with the number of register pressure sets. This makes growth in the generated register model easier to spot since it can be surprising. Assisted-by: codex
…me tests" (llvm#211484) This reverts llvm#210889 (labath@b7f642f, "Revert recent changes to the hermetic tests"), re-applying llvm#209999 (labath@9e2e9b3, "[libc] Port process utilities to hermetic mode and enable some tests") and llvm#210715 (labath@4bd1a44, "[libc] Make hermetic test syscall deps linux-only"), which was reverted due to failures on the GPU bots. There are a couple of small changes w.r.t the original patches: - This version disables the exit tests on builds that do not support subprocess tests (via the newly introduced LIBC_TEST_SUBPROCESS_TESTS). This is necessary because the GPU builds contain the relevant entry points, which means their tests are not skipped automatically. - fix a typo in a variable name - make the dependencies conditional on LIBC_TEST_SUBPROCESS_TESTS instead of listing the target OSs directly. This is a no-op, as that's exactly how the variable is defined, but it makes the code cleaner.
Customized DOWNLOAD_COMMAND ran a raw `git clone`, which fails with "destination path already exists" whenever ninja reruns the download step against an already cloned checkout. Use the native GIT_REPOSITORY/GIT_TAG (with default DOWNLOAD_COMMAND) step instead, which handles this case correctly
…ctor. NFC (llvm#211734) Drop the field from the descriptor and keep the labels in a propagator-local SmallVector indexed by block number.
BOLT currently assumes the DWARF32 format when it rewrites debug sections. As a result, rewriting a binary that was compiled with `-gdwarf64` either produces a corrupted output or fails `llvm-dwarfdump --verify`. This patch adds DWARF64 support to the BOLT DWARF rewriter, honoring each unit's DwarfFormat and emitting correct encodings DWARF64 for binaries, including mix DWARF32 and DWARF64.
To make this work, I needed to remove the getauxval definition from HermeticTestUtils.cpp. Even though it was weak, it still prevented the real one from being used because the linker will not extract an archive member if the dependency has already been satisfied by a weak definition. I *think* I've removed the need for it by changing how the getauxval dependency is declared in cmake (it fixes the errors on the presubmit aarch64 bot), but I don't know if that was the only issue.
…m#211666) Similar to llvm#207406 - the middleend (SLP/VectorCombine/InstCombine) should no longer generate any mixed "addsub" scalar buildvector patterns for the backend. We can remove the lowerToAddSubOrFMAddSub build vector code path and rely on the combineShuffleToAddSubOrFMAddSub path. The vector code created by SLP/VectorCombine isn't yet perfect (and I'm still working on improving it), but lowerToAddSubOrFMAddSub is unrelated and of no more use to us. The CodeGen buildvector tests can be removed - equivalent test coverage is present in Transforms/PhaseOrdering/X86 that generate shuffle patterns that we test for in fmaddsub-combine.ll, fmsubadd-combine.ll and sse3-avx-addsub.ll (and sse3-avx-addsub-2.ll can be deleted). Fixes llvm#144489
…lvm#211670) Remove unnecessary uses for CM from LVP::executePlan.
…s and returns (llvm#211226) PseudoProbe insertion can generate invalid IR by inserting a probe between a musttail/llvm.experimental.deoptimize call and its following ret. Insert the probe before these calls to preserve the required instruction ordering.
…1507) Noalias applies to accesses on other threads, but this was not very clear in existing wording, because "during the execution of the function" is somewhat ambiguous. Explicitly mention that it applies to accesses from other threads. This means that conflicting accesses (i.e. not read-read) need to either happen-before function entry, or function exit needs to happen-before them, otherwise behavior is undefined. For accesses not based on the noalias pointer, this requires synchronization *outside* the function.
…lvm#210077) The SPF float min/max matching skipped the check that either the select is nsz or one of the ops non-zero in the case where the zero used in the fcmp and select is the same one. This does not make any sense, because which zero is used in the fcmp is just completely irrelevant. This leads to miscompiles where the SPF pattern is lowered to hardware minnum/maxnum operations. These have ordered zero, rather than picking whichever zero is specified in the select. It can still be accidentally correct if the zero happens to be the right one for min/max. I believe the current assumption is that SPF only matches in cases where signed zero behavior does not matter. An alternative way to fix this would be to match the specific zero that's required to match ordered zero semantics. Though I'd rather we match that in DAGCombine, not the SPF based SDAGBuilder code. Fixes llvm#93414.
…#210725) Using a pipe avoids races between "hermetic" and "unit" versions of the same test and avoids leaving filesystem artifacts around if the test fails. If I'm successful, we won't have the worry about the first problem anymore, but I think it's still a nice cleanup. Assisted by Gemini.
…211572) Reverts llvm#186093 Issue llvm#206839 shows that EliminateSpillageCopies is not currently ready to be enabled by default on AArch64. llvm#207169 was raised to fix the issue, but concerns were raised around complexity of the function and change. Until a suitable solution can be found, EliminateSpillageCopy will be disabled on AArch64 by default. For those who wish to still use the pass, `-enable-spill-copy-elim=true` will ensure the pass is run.
) I think this part was missed in llvm#210826.
…1564) GFX13 uses the num_flt suffix as the default assembler name for the 32-bit FP image atomic min/max instructions. Keep flt as a backward-compatible alias. GFX12 is unchanged (flt default, num_flt alias). --------- Co-authored-by: Petar Avramovic <Petar.Avramovic@amd.com>
…zer splitter (llvm#211544) Based on the discussion in llvm#211198
llvm#211538) Splitter never used its LoopInfo reference Based on the discussion in llvm#211198
There is a known issue with these tests (see llvm#35661). It looks that the behaviour is very architecture specific and it was somehow solved for aarch64 but no mention to riscv64.
Reserved registers are not allocatable and since these are singleton classes, they should be marked as such. Assisted-by: codex
Leave them NYI now.
…lvm#195059) Extend optimizeInductionLiveOutUsers to handle complex induction variable by using SCEV analysis. When an induction variable forms an affine AddRec {Start,+,Step}, compute the final value as Start + (ResumeTripCount - 1) * Step using VPDerivedIVRecipe. This patch eliminates unnecessary vector widening for induction variables only used outside the loop. Pre-commit test llvm#195055
dpalermo
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.