Commit 84de900
authored
Update EDM <-> EDM handshake in fabric_erisc_datamover (#23813)
### Ticket
No Ticket.
### Problem description
- The current EDM <-> EDM handshake approach for
`fabric_erisc_datamover` kernels relies on all devices in the Mesh being
initialized before fabric is loaded
- This assumption is true for single host scenarios, but breaks down in
the multi-host case, since each host will initialize it's local mesh
asynchronous of all other hosts
- With delays added to device startup on some hosts, the handshake step
hangs. This is because the delayed host clears the handshake message
written to it
### What's changed
- Add new _single step_ handshake APIs to `edm_handshake.hpp`:
`sender_side_handshake` and `receiver_side_handshake` (protocol
described in header)
- Use them in `fabric_erisc_datamover.cpp` to ensure that the multi-host
scenario described above is supported
- Mark handshake APIs used by legacy CCLs as deprecated. These will be
removed once legacy CCLs are purged.
- General cleanup - remove unused args from `EthChannelBuffer` ctor
### Checklist
- [ ] [All post
commit](https://github.com/tenstorrent/tt-metal/actions/workflows/all-post-commit-workflows.yaml)
CI passes
- [ ] [Blackhole Post
commit](https://github.com/tenstorrent/tt-metal/actions/workflows/blackhole-post-commit.yaml)
CI with demo tests passes (if applicable)
- [ ] [Model
regression](https://github.com/tenstorrent/tt-metal/actions/workflows/perf-models.yaml)
CI passes (if applicable)
- [ ] [Device performance
regression](https://github.com/tenstorrent/tt-metal/actions/workflows/perf-device-models.yaml)
CI passes (if applicable)
- [ ] (For models and ops writers) [Single-card demo
tests](https://github.com/tenstorrent/tt-metal/actions/workflows/single-card-demo-tests.yaml)
CI passes (if applicable) See [recommended dev
flow](https://github.com/tenstorrent/tt-metal/blob/main/models/MODEL_ADD.md#a-recommended-dev-flow-on-github-for-adding-new-models).
- [ ] New/Existing tests provide coverage for changes1 parent 30110bc commit 84de900
File tree
10 files changed
+111
-60
lines changed- tests/tt_metal/tt_metal/perf_microbenchmark/routing/kernels
- tt_metal/fabric
- hw/inc/edm_fabric
- impl/kernels
- edm_fabric
- ttnn/cpp/ttnn/operations/ccl
- barrier/device/kernels
- kernels/edm
10 files changed
+111
-60
lines changedLines changed: 1 addition & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
| 43 | + | |
48 | 44 | | |
49 | 45 | | |
50 | 46 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
808 | 808 | | |
809 | 809 | | |
810 | 810 | | |
811 | | - | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
812 | 815 | | |
813 | 816 | | |
814 | 817 | | |
| |||
915 | 918 | | |
916 | 919 | | |
917 | 920 | | |
| 921 | + | |
918 | 922 | | |
919 | 923 | | |
920 | 924 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
295 | 295 | | |
296 | 296 | | |
297 | 297 | | |
298 | | - | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
299 | 302 | | |
300 | 303 | | |
301 | 304 | | |
| |||
382 | 385 | | |
383 | 386 | | |
384 | 387 | | |
385 | | - | |
386 | 388 | | |
387 | 389 | | |
388 | 390 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
9 | 10 | | |
10 | 11 | | |
11 | 12 | | |
| |||
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | 20 | | |
25 | 21 | | |
26 | 22 | | |
| |||
32 | 28 | | |
33 | 29 | | |
34 | 30 | | |
| 31 | + | |
35 | 32 | | |
36 | 33 | | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
37 | 43 | | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
38 | 109 | | |
39 | 110 | | |
40 | 111 | | |
| |||
47 | 118 | | |
48 | 119 | | |
49 | 120 | | |
50 | | - | |
| 121 | + | |
51 | 122 | | |
52 | 123 | | |
53 | 124 | | |
| |||
66 | 137 | | |
67 | 138 | | |
68 | 139 | | |
69 | | - | |
| 140 | + | |
70 | 141 | | |
71 | 142 | | |
72 | 143 | | |
| |||
79 | 150 | | |
80 | 151 | | |
81 | 152 | | |
82 | | - | |
| 153 | + | |
83 | 154 | | |
84 | 155 | | |
85 | 156 | | |
86 | | - | |
| 157 | + | |
87 | 158 | | |
88 | 159 | | |
89 | 160 | | |
| |||
99 | 170 | | |
100 | 171 | | |
101 | 172 | | |
102 | | - | |
| 173 | + | |
103 | 174 | | |
104 | 175 | | |
105 | 176 | | |
106 | 177 | | |
107 | 178 | | |
108 | 179 | | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
109 | 183 | | |
110 | 184 | | |
111 | 185 | | |
| |||
Lines changed: 1 addition & 8 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
| 56 | + | |
62 | 57 | | |
63 | 58 | | |
64 | 59 | | |
| |||
134 | 129 | | |
135 | 130 | | |
136 | 131 | | |
137 | | - | |
138 | 132 | | |
139 | 133 | | |
140 | 134 | | |
| |||
144 | 138 | | |
145 | 139 | | |
146 | 140 | | |
147 | | - | |
148 | 141 | | |
149 | 142 | | |
150 | 143 | | |
| |||
Lines changed: 5 additions & 19 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1425 | 1425 | | |
1426 | 1426 | | |
1427 | 1427 | | |
1428 | | - | |
1429 | | - | |
1430 | 1428 | | |
1431 | 1429 | | |
1432 | 1430 | | |
| |||
1466 | 1464 | | |
1467 | 1465 | | |
1468 | 1466 | | |
1469 | | - | |
1470 | | - | |
1471 | | - | |
1472 | | - | |
1473 | | - | |
1474 | | - | |
1475 | | - | |
1476 | | - | |
1477 | 1467 | | |
1478 | 1468 | | |
1479 | 1469 | | |
| |||
1875 | 1865 | | |
1876 | 1866 | | |
1877 | 1867 | | |
1878 | | - | |
1879 | 1868 | | |
1880 | 1869 | | |
1881 | 1870 | | |
1882 | 1871 | | |
1883 | 1872 | | |
1884 | 1873 | | |
1885 | 1874 | | |
1886 | | - | |
1887 | 1875 | | |
1888 | 1876 | | |
1889 | 1877 | | |
1890 | 1878 | | |
1891 | | - | |
1892 | | - | |
1893 | | - | |
1894 | | - | |
1895 | | - | |
| 1879 | + | |
1896 | 1880 | | |
1897 | 1881 | | |
1898 | 1882 | | |
| |||
1921 | 1905 | | |
1922 | 1906 | | |
1923 | 1907 | | |
1924 | | - | |
| 1908 | + | |
| 1909 | + | |
1925 | 1910 | | |
1926 | | - | |
| 1911 | + | |
| 1912 | + | |
1927 | 1913 | | |
1928 | 1914 | | |
1929 | 1915 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
| 58 | + | |
63 | 59 | | |
64 | 60 | | |
65 | 61 | | |
| |||
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
58 | | - | |
59 | | - | |
| 58 | + | |
| 59 | + | |
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| |||
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
49 | | - | |
| 49 | + | |
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
63 | | - | |
| 63 | + | |
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
| |||
Lines changed: 6 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
145 | 145 | | |
146 | 146 | | |
147 | 147 | | |
148 | | - | |
| 148 | + | |
149 | 149 | | |
150 | | - | |
| 150 | + | |
151 | 151 | | |
152 | 152 | | |
153 | 153 | | |
| |||
184 | 184 | | |
185 | 185 | | |
186 | 186 | | |
187 | | - | |
| 187 | + | |
188 | 188 | | |
189 | | - | |
| 189 | + | |
190 | 190 | | |
191 | 191 | | |
192 | 192 | | |
| |||
224 | 224 | | |
225 | 225 | | |
226 | 226 | | |
227 | | - | |
| 227 | + | |
228 | 228 | | |
229 | 229 | | |
230 | | - | |
| 230 | + | |
231 | 231 | | |
232 | 232 | | |
233 | 233 | | |
| |||
0 commit comments