Add multi-mesh tests to Galaxy CI (#36005)

### Ticket
[Link to Github
Issue](#34551)

### Problem description
Currently there are a lack of multi-mesh stability tests in existing
pipelines

### What's changed
Added stability tests for multi-mesh tests for two arrangements of
all-to-all mesh configurations, each having 3 meshes total. Each added
suite takes ~24 minutes to run.

https://github.com/tenstorrent/tt-metal/actions/runs/21050162315 <--
Passing run on CI

### Checklist

- [ ] [![All post-commit
tests](https://github.com/tenstorrent/tt-metal/actions/workflows/all-post-commit-workflows.yaml/badge.svg?branch={{branch_name}})](https://github.com/tenstorrent/tt-metal/actions/workflows/all-post-commit-workflows.yaml?query=branch:{{branch_name}})
- [ ] [![Blackhole Post
commit](https://github.com/tenstorrent/tt-metal/actions/workflows/blackhole-post-commit.yaml/badge.svg?branch={{branch_name}})](https://github.com/tenstorrent/tt-metal/actions/workflows/blackhole-post-commit.yaml?query=branch:{{branch_name}})
- [ ]
[![cpp-unit-tests](https://github.com/tenstorrent/tt-metal/actions/workflows/tt-metal-l2-nightly.yaml/badge.svg?branch={{branch_name}})](https://github.com/tenstorrent/tt-metal/actions/workflows/tt-metal-l2-nightly.yaml?query=branch:{{branch_name}})
- [ ] New/Existing tests provide coverage for changes


#### Model tests

If your changes cover model-related code, you should run tests
corresponding to affected models and platforms (Single card, T3K,
Galaxy). "Choose your pipeline" workflows facilitate running multiple
kinds of tests in a single run. Each offers `models-mandatory` and
`models-extended` presets.
The former includes a minimal set of tests, to be run always. The latter
extends that with additional ones - use your best judgement in deciding
which is the most appropriate for your PR.

- [ ] [![(Single) Choose your
pipeline](https://github.com/tenstorrent/tt-metal/actions/workflows/pipeline-select.yaml/badge.svg?branch={{branch_name}})](https://github.com/tenstorrent/tt-metal/actions/workflows/pipeline-select.yaml?query=branch:{{branch_name}})
- [ ] `models-mandatory` preset (runs: [Device perf
regressions](https://github.com/tenstorrent/tt-metal/actions/workflows/perf-device-models.yaml)
and [Frequent model and ttnn
tests](https://github.com/tenstorrent/tt-metal/actions/workflows/fast-dispatch-full-regressions-and-models.yaml))
- [ ] `models-extended` preset (runs: the mandatory tests, plus
[Demo](https://github.com/tenstorrent/tt-metal/actions/workflows/single-card-demo-tests.yaml)
and [Model
perf](https://github.com/tenstorrent/tt-metal/actions/workflows/perf-models.yaml)
tests)
  - [ ] other selection - specify runs

- [ ] [![(T3K) Choose your
pipeline](https://github.com/tenstorrent/tt-metal/actions/workflows/pipeline-select-t3k.yaml/badge.svg?branch={{branch_name}})](https://github.com/tenstorrent/tt-metal/actions/workflows/pipeline-select-t3k.yaml?query=branch:{{branch_name}})
- [ ] `models-mandatory` preset (runs: [Unit
tests](https://github.com/tenstorrent/tt-metal/actions/workflows/t3000-unit-tests.yaml))
- [ ] `models-extended` preset (runs: the mandatory tests, plus
[Demo](https://github.com/tenstorrent/tt-metal/actions/workflows/t3000-demo-tests.yaml)
and [Model
perf](https://github.com/tenstorrent/tt-metal/actions/workflows/t3000-model-perf-tests.yaml)
tests)
  - [ ] other selection - specify runs

- [ ] [![(Galaxy) Choose your
pipeline](https://github.com/tenstorrent/tt-metal/actions/workflows/pipeline-select-galaxy.yaml/badge.svg?branch={{branch_name}})](https://github.com/tenstorrent/tt-metal/actions/workflows/pipeline-select-galaxy.yaml?query=branch:{{branch_name}})
- [ ] `models-mandatory` preset (runs: [Quick
tests](https://github.com/tenstorrent/tt-metal/actions/workflows/galaxy-quick.yaml))
- [ ] `models-extended` preset (runs: the mandatory tests, plus
[Demo](https://github.com/tenstorrent/tt-metal/actions/workflows/galaxy-demo-tests.yaml)
and [Model
perf](https://github.com/tenstorrent/tt-metal/actions/workflows/galaxy-model-perf-tests.yaml)
tests)
  - [ ] other selection - specify runs

Author: nnyamagoudar-TT

.github/workflows/galaxy-nightly-tests-impl.yaml

@@ -38,6 +38,8 @@ jobs:
           { name: "Galaxy CCL tests", arch: wormhole_b0, cmd: "pytest tests/nightly/tg/ccl", timeout: 40, owner_id: U05ACKAJTHS}, # Naif Tarafdar
           { name: "Galaxy Fabric unit tests", arch: wormhole_b0, cmd: build/test/tt_metal/tt_fabric/fabric_unit_tests --gtest_filter="NightlyFabric*Fixture.*", timeout: 25, owner_id: U08UBDUKH6Z}, # Neel Nyamagoudar
           { name: "Galaxy Fabric 2D Torus Nightly Tests", arch: wormhole_b0, cmd: ./build/test/tt_metal/perf_microbenchmark/routing/test_tt_fabric --test_config tests/tt_metal/tt_metal/perf_microbenchmark/routing/test_fabric_2d_torus_nightly.yaml, timeout: 45, owner_id: U08UBDUKH6Z}, # Neel Nyamagoudar
+          { name: "Galaxy Fabric Multi-Mesh 4x4 and 2x4s stability tests", arch: wormhole_b0, cmd: python tests/tt_metal/tt_fabric/utils/generate_rank_bindings.py && tt-run --rank-binding 4x4_2x4_3_mesh_rank_binding.yaml --mpi-args "--allow-run-as-root --tag-output" ./build/test/tt_metal/perf_microbenchmark/routing/test_tt_fabric --test_config tests/tt_metal/tt_metal/perf_microbenchmark/routing/test_fabric_multi_mesh_stability_short_running.yaml, timeout: 15, owner_id: U08UBDUKH6Z}, # Neel Nyamagoudar
+          { name: "Galaxy Fabric Multi-mesh 2x8 and 2x4s stability tests", arch: wormhole_b0, cmd: python tests/tt_metal/tt_fabric/utils/generate_rank_bindings.py && tt-run --rank-binding 2x8_2x4_3_mesh_rank_binding.yaml --mpi-args "--allow-run-as-root --tag-output" ./build/test/tt_metal/perf_microbenchmark/routing/test_tt_fabric --test_config tests/tt_metal/tt_metal/perf_microbenchmark/routing/test_fabric_multi_mesh_stability_short_running.yaml, timeout: 15, owner_id: U08UBDUKH6Z}, # Neel Nyamagoudar
           # {
           #   name: "Llama Galaxy Accuracy Test",
           #   arch: wormhole_b0,

tests/tt_metal/tt_fabric/custom_mesh_descriptors/wh_galaxy_split_2x8_2x4_3_mesh.textproto

@@ -0,0 +1,48 @@
+# --- Meshes ---------------------------------------------------------------
+
+mesh_descriptors {
+  name: "MESH4"
+  arch: WORMHOLE_B0
+  device_topology { dims: [ 2, 8 ] }
+  host_topology   { dims: [ 1, 1 ] }
+  channels {
+    count: 4
+    policy: RELAXED
+  }
+}
+mesh_descriptors {
+  name: "MESH2"
+  arch: WORMHOLE_B0
+  device_topology { dims: [ 2, 4 ] }
+  host_topology   { dims: [ 1, 1 ] }
+  channels {
+    count: 4
+    policy: RELAXED
+  }
+}
+graph_descriptors {
+  name: "G0"
+  type: "FABRIC"
+  instances { mesh { mesh_descriptor: "MESH2" mesh_id: 0 } }
+  instances { mesh { mesh_descriptor: "MESH2" mesh_id: 1 } }
+  instances { mesh { mesh_descriptor: "MESH4" mesh_id: 2 } }
+
+
+  connections {
+    nodes { mesh { mesh_descriptor: "MESH2" mesh_id: 0 } }
+    nodes { mesh { mesh_descriptor: "MESH2" mesh_id: 1 } }
+    channels { count: 4 policy: RELAXED }
+  }
+  connections{
+    nodes { mesh { mesh_descriptor: "MESH2" mesh_id: 0 } }
+    nodes { mesh { mesh_descriptor: "MESH4" mesh_id: 2 } }
+    channels { count: 4 policy: RELAXED }
+  }
+  connections{
+    nodes { mesh { mesh_descriptor: "MESH2" mesh_id: 1 } }
+    nodes { mesh { mesh_descriptor: "MESH4" mesh_id: 2 } }
+    channels { count: 4 policy: RELAXED }
+  }
+}
+# --- Instantiation ----------------------------------------------------------
+top_level_instance { graph { graph_descriptor: "G0" graph_id: 0 } }

tests/tt_metal/tt_fabric/custom_mesh_descriptors/wh_galaxy_split_4x4_2x4_3_mesh.textproto

@@ -0,0 +1,48 @@
+# --- Meshes ---------------------------------------------------------------
+
+mesh_descriptors {
+  name: "MESH4"
+  arch: WORMHOLE_B0
+  device_topology { dims: [ 4, 4 ] }
+  host_topology   { dims: [ 1, 1 ] }
+  channels {
+    count: 4
+    policy: RELAXED
+  }
+}
+mesh_descriptors {
+  name: "MESH2"
+  arch: WORMHOLE_B0
+  device_topology { dims: [ 2, 4 ] }
+  host_topology   { dims: [ 1, 1 ] }
+  channels {
+    count: 4
+    policy: RELAXED
+  }
+}
+graph_descriptors {
+  name: "G0"
+  type: "FABRIC"
+  instances { mesh { mesh_descriptor: "MESH4" mesh_id: 0 } }
+  instances { mesh { mesh_descriptor: "MESH2" mesh_id: 1 } }
+  instances { mesh { mesh_descriptor: "MESH2" mesh_id: 2 } }
+
+  connections {
+    nodes { mesh { mesh_descriptor: "MESH4" mesh_id: 0 } }
+    nodes { mesh { mesh_descriptor: "MESH2" mesh_id: 1 } }
+    channels { count: 4 policy: RELAXED }
+  }
+  connections{
+    nodes { mesh { mesh_descriptor: "MESH4" mesh_id: 0 } }
+    nodes { mesh { mesh_descriptor: "MESH2" mesh_id: 2 } }
+    channels { count: 4 policy: RELAXED }
+  }
+  connections{
+    nodes { mesh { mesh_descriptor: "MESH2" mesh_id: 1 } }
+    nodes { mesh { mesh_descriptor: "MESH2" mesh_id: 2 } }
+    channels { count: 4 policy: RELAXED }
+  }
+}
+
+# --- Instantiation ----------------------------------------------------------
+top_level_instance { graph { graph_descriptor: "G0" graph_id: 0 } }

tests/tt_metal/tt_fabric/utils/generate_rank_bindings.py

@@ -112,6 +112,18 @@ def generate_supported_rank_bindings():
         2: [4],
         3: [2],
     }
+    # Process Rank ID To Tray ID Mapping for 4x4 + 2x4 + 2x4 (3 mesh) configuration
+    WH_GLX_4X4_2X4_3_MESH_RANK_TO_TRAY_MAPPING = {
+        0: [1, 2],  # 4x4 mesh needs 2 trays (16 devices)
+        1: [3],  # 2x4 mesh needs 1 tray (8 devices)
+        2: [4],  # 2x4 mesh needs 1 tray (8 devices)
+    }
+    # Process Rank ID To Tray ID Mapping for 2x4 + 2x4 + 2x8 (3 mesh) configuration
+    WH_GLX_2X8_2X4_3_MESH_RANK_TO_TRAY_MAPPING = {
+        0: [1],  # 2x4 mesh needs 1 tray (8 devices)
+        1: [3],  # 2x4 mesh needs 1 tray (8 devices)
+        2: [2, 4],  # 2x8 mesh needs 2 trays (16 devices)
+    }
 
     # Rank bindings for Dual Mesh Setup (1 process per mesh)
     DUAL_MESH_RANK_BINDINGS = [
@@ -197,6 +209,44 @@ def generate_supported_rank_bindings():
         },
     ]
 
+    # Rank bindings for Tri Mesh Setup: 4x4 + 2x4 + 2x4 (1 process per mesh, 3 meshes)
+    TRI_MESH_4X4_2X4_RANK_BINDINGS = [
+        {
+            "rank": 0,
+            "mesh_id": 0,
+            "mesh_host_rank": 0,
+        },
+        {
+            "rank": 1,
+            "mesh_id": 1,
+            "mesh_host_rank": 0,
+        },
+        {
+            "rank": 2,
+            "mesh_id": 2,
+            "mesh_host_rank": 0,
+        },
+    ]
+
+    # Rank bindings for Tri Mesh Setup: 2x4 + 2x4 + 2x8 (1 process per mesh, 3 meshes)
+    TRI_MESH_2X8_2X4_RANK_BINDINGS = [
+        {
+            "rank": 0,
+            "mesh_id": 0,
+            "mesh_host_rank": 0,
+        },
+        {
+            "rank": 1,
+            "mesh_id": 1,
+            "mesh_host_rank": 0,
+        },
+        {
+            "rank": 2,
+            "mesh_id": 2,
+            "mesh_host_rank": 0,
+        },
+    ]
+
     mapping_file = "tray_to_pcie_device_mapping.yaml"
     generate_tray_to_pcie_device_mapping(mapping_file)
     with open(mapping_file, "r") as f:
@@ -231,6 +281,20 @@ def generate_supported_rank_bindings():
         "tests/tt_metal/tt_fabric/custom_mesh_descriptors/wh_galaxy_2x4_mesh_graph_descriptor.textproto",
         "2x4_multi_mesh_cyclic_rank_binding.yaml",
     )
+    generate_rank_binding_yaml(
+        tray_to_pcie_device_mapping,
+        TRI_MESH_4X4_2X4_RANK_BINDINGS,
+        WH_GLX_4X4_2X4_3_MESH_RANK_TO_TRAY_MAPPING,
+        "tests/tt_metal/tt_fabric/custom_mesh_descriptors/wh_galaxy_split_4x4_2x4_3_mesh.textproto",
+        "4x4_2x4_3_mesh_rank_binding.yaml",
+    )
+    generate_rank_binding_yaml(
+        tray_to_pcie_device_mapping,
+        TRI_MESH_2X8_2X4_RANK_BINDINGS,
+        WH_GLX_2X8_2X4_3_MESH_RANK_TO_TRAY_MAPPING,
+        "tests/tt_metal/tt_fabric/custom_mesh_descriptors/wh_galaxy_split_2x8_2x4_3_mesh.textproto",
+        "2x8_2x4_3_mesh_rank_binding.yaml",
+    )
 
 
 if __name__ == "__main__":

tests/tt_metal/tt_metal/perf_microbenchmark/routing/test_fabric_multi_mesh_stability_short_running.yaml

@@ -0,0 +1,36 @@
+Tests:
+  - name: "FlowControlMultiMesh"
+    enable_flow_control: true
+    fabric_setup:
+      topology: Mesh
+
+    parametrization_params:
+      ntype: [unicast_write, atomic_inc, fused_atomic_inc, unicast_scatter_write]
+      num_links: [1, 2, 3, 4]
+      size: [1024, 2048, 4096]
+
+    defaults:
+      ftype: unicast
+      num_packets: 10000
+
+    patterns:
+      - type: all_to_all
+
+  - name: "MeshLowLatencyLooped"
+    fabric_setup:
+      topology: Mesh
+
+    top_level_iterations: 5
+
+    parametrization_params:
+      ntype: [unicast_write, atomic_inc , fused_atomic_inc, unicast_scatter_write]
+      num_links: [1, 2, 3, 4]
+
+    defaults:
+      ftype: unicast
+      size: 32
+      num_packets: 10000
+
+    patterns:
+      - type: all_to_all
+        iterations: 2