Worker-to-Self Channel Example

[hunhoffe]

I am working on writing a worker-to-worker data transfer example for channels (as part of the grouping of examples that exercise various features of channels, #648).

Draft PR is here: #653

I am basing it off the code in the channel_size example (PR waiting to be merged here: #642)

The channel_size example works well for me. As an intermediate step to adding worker-to-worker communication to that example, I tried to have each worker send data to itself over a channel. That is the version of the code that is pushed in the draft PR #653. The particular file of interest is this one.

When I run with this intermediate step, I get the following error:

Using aiecc.py from:  /scratch/ehunhoff/mlir-air/mlir-aie/install/bin/..
Running: builtin.module(air-insert-launch-and-segment-around-herd,func.func(air-lower-herd-parallel),air-dma-to-channel,canonicalize,cse,air-specialize-channel-wrap-and-stride,func.func(air-renumber-dma),func.func(convert-linalg-to-loops),air-place-herds{num-rows=6 num-cols=4 row-anchor=2 col-anchor=0})
Running: builtin.module(air-to-aie{emit-while-loop=false row-offset=2 col-offset=0 device=npu1_4col})
python3: /scratch/ehunhoff/mlir-air/mlir/lib/Conversion/AIRToAIESchedulingUtils.cpp:956: void xilinx::air::simpleDMAChannelAllocation(std::vector<MemcpyBundleAsFlow> &, ShimDMAAllocator &, MemTileDMAAllocator &, TileDMAAllocator &): Assertion `core' failed.
Aborted (core dumped)
make: *** [Makefile:7: run] Error 134

My question is:

• Is a worker allowed to put/get data to/from a channel to itself?
• Or is this a bug (either in my example code or the air compiler)?

[erwei-xilinx]

Thanks for investigating into worker-to-worker communication scenario. This scenario hasn't been exercised, so a bug at air compiler is possible. The simpleDMAChannelAllocation is implemented with a naive logic to allocate DMA channels to data movements with endpoints of different memory space; in your case the endpoints are both L1, which might lead to unexpected behaviour.

[erwei-xilinx]

And this assumption could affect other methods and passes in air compiler.

[hunhoffe]

Ah, I see!

As a path forward, would it make sense to:

• Create a channel-to-self example with two sub-examples:
  • L1->L1 (mark this as fail for now, and leave an issue up to address this later)
  • L2->L1 (this should presumably succeed)
• Go ahead and ignore this intermediate error and continue with my plans for a worker-to-worker example, since L1 on one worker is a different memory space than L1 on a different worker?

[hunhoffe]

An update on this plan:

• I did create a channel to self (l1-l1) that fails in the PR here: Channel Examples: Worker-to-self, Worker-to-worker #653
• l2-l1 is not possible using only operations in a herd, since the herd is unable to directly create l2 allocations

I will leave this issue up until the l1-l1 worker-to-self example is supported (and the lit test passes) or it is decided this is definitely an illegal action, which can be caught by the verifier (hopefully with a helpful error).

[hunhoffe]

This is an issue that directly addresses accessing l2 allocations in the channel: #666

This PR takes that fix and attempts to fix the worker-to-self example using this new capability; however, it still fails with:

python3: mlir-air/mlir/lib/Conversion/AIRToAIESchedulingUtils.cpp:956: void xilinx::air::simpleDMAChannelAllocation(std::vector<MemcpyBundleAsFlow> &, ShimDMAAllocator &, MemTileDMAAllocator &, TileDMAAllocator &): Assertion `core' failed.
Aborted (core dumped)