[Question] I'm confused about what the "_xd" suffix means in the all_to_all benchmark configs

[jimoosciuc]

When reading the accelerator-microbenchmarks code, I'm very confused about one part. Below is the benchmark test configuration for all_to_all communication:

all_to_all_3d:

• mesh_shape: "4x4x8"
  sharding_strategy: "4x4x8"
  op_dimension: 3

all_to_all_2d:

• mesh_shape: "4x32"
  sharding_strategy: "1x32"
  op_dimension: 2

all_to_all_1d:
mesh_shape: "4x4x8"
sharding_strategy: "1x1x4"
op_dimension: 1

I originally thought that "_xd" describes data communicating across x dimensions, but it seems that both 1d and 2d only need communication in one dimension? Is it determined by op_dimension for how many dimensions? The all_to_all_benchmark function in the code also doesn't seem to use the op_dimension parameter.

[hylin2002]

The "_xd" suffix here means "x" dimensional toruses (i.e. wrap-around links) are involved in the collective operation (e.g. AllToAll). For example, in the 2D case,
all_to_all_2d:

• mesh_shape: "4x32"
• sharding_strategy: "1x32"

This means the AllToAll is performed over 32 devices (i.e. 16 chips). And there are 4 groups performing AllToAll independently (inferred from the "4x" in the mesh).

op_dimension doesn't really have any effects. It's just convenient for logging the number of dimensions that toruses are (supposed to be) used in the collective operation.

We are currently developing official documentation that will detail Ironwood's technical specifications and will update this section once it is published.