Design a first-class multi-host launch API for Emily.Distributed.Launcher

[ausimian]

Type: design / feature
Priority: do after the other open distributed issues — this is the Elixir crux of the feature and deserves the polished API once the scaffold bugs (#110–#116) are settled.

Motivation

Emily.Distributed.Launcher today only launches local peers over loopback: start_peer/4 hardcodes connection: :standard_io + local -pa code paths, and write_hostfile/2 only ever emits 127.0.0.1. The multi-Mac story (ssh exec + Thunderbolt-bridge IPs) exists solely as an aspirational snippet in the moduledoc — there is no option a client can actually pass to launch remote ranks.

This is the user-facing surface of the distributed feature in Elixir. It should be excellent: launching N ranks across M Macs should be as clean as the current single-node call.

Core insight

The hostfile entry and the :peer.start_link spec are both pure functions of a per-rank placement. So the launcher collapses to: build a list of placements → map each to (a) its hostfile line and (b) its peer spec. Localhost is just the degenerate placement.

%{control: :local,       address: "127.0.0.1",  port: 18_000 + rank}
%{control: {:ssh, host}, address: "<bridge-ip>", port: 18_000 + rank}

defp hostfile_entry(%{address: a, port: p}), do: ["#{a}:#{p}"]

defp peer_spec(%{control: :local}, rank, code_args, env),
  do: %{name: :"emily_rank#{rank}", connection: :standard_io, args: code_args, env: env}

defp peer_spec(%{control: {:ssh, host}}, rank, _code_args, env),
  do: %{name: :"emily_rank#{rank}", connection: :standard_io,
        exec: {~c"/usr/bin/ssh", [String.to_charlist(host), ~c"erl"]}, env: env}

start/2/run/3 then map placements → write hostfile, map placements → start_peer; the existing partial-failure cleanup (#110) is unchanged. Current behavior becomes placements = for r <- 0..n-1, do: %{control: :local, address: "127.0.0.1", port: base + r}.

Two things that are NOT just a cosmetic mapping

1. Control address ≠ data address

On a Thunderbolt rig, ssh rides the management LAN while MLX rings over separate Thunderbolt-bridge IPs (typically IPv4 link-local 169.254.x.x, often nameless). So a host entry carries two addresses: the ssh target (control) and the bridge IP that lands in the hostfile (data). They coincide only for localhost / single-network clusters.

2. Remote code loading

The local case works because args passes -pa <this node's paths> — valid only on the same filesystem. A remote erl can't use those; the host needs the emily build + NIF already present (identical path, or a release). This is the real reason remote isn't a one-line change. (See moduledoc "Caveat".)

MLX hostfile parsing — findings (verified in deps/mlx_src)

ring.cpp:load_nodes → detail::parse_address (distributed/utils.cpp):

• Hostnames resolve. getaddrinfo(ip, port, hints) with hints.ai_family = AF_UNSPEC and no AI_NUMERICHOST — so a bare "mac1:18000" in the hostfile is valid; MLX resolves it via DNS//etc/hosts/mDNS.
• But AF_UNSPEC lookup feeds an AF_INET socket. parse_address keeps the first getaddrinfo result, while the socket is hardcoded socket(AF_INET, ...) for both bind and connect. A name that resolves IPv6-first → IPv6 sockaddr in an IPv4 socket → connect/bind fails. (.local Bonjour names are exactly the IPv6-first risk.)
• First-colon split. ip_port.find(":") → IPv6 literals don't parse. Effectively IPv4-literal-or-hostname only.

Implications for the API

• Single-address mode: passing the bare hostname is legal (MLX resolves it), but to dodge the IPv6-first trap we should resolve it ourselves to IPv4 (:inet.getaddr(name, :inet)) and write the literal.
• Split/Thunderbolt mode: bridge IPs are IPv4 link-local literals — drop straight in, no resolution, no IPv6 ambiguity. The clean path.

Open design decisions

1. Host entry shape — single string "host" (ssh + hostfile share it, resolve to IPv4 ourselves) vs. split %{ssh: ..., data_ip: ...} (handles Thunderbolt). Probably support both: a bare string is the shorthand, a map is the full form.
2. Remote code provisioning — assume identical path + reuse -pa; or :code_paths/:release per host; or wire only the launch mechanism and document the requirement for now.
3. rank ↔ host mapping — one rank per host (likely, one GPU per Mac) vs. hosts-with-slot-count.
4. API ergonomics — :hosts keyword on start/2/run/3; default (no :hosts) preserves today's N-local-ranks behavior so nothing breaks.

Acceptance sketch

• run(n, fun) unchanged (N local ranks).
• run(hosts: [...], fun) (or similar) launches across the listed hosts, writes a correct hostfile (right data IPs), and tears down cleanly on partial failure (per Launcher.start/2 leaks peers and hostfile on partial startup failure #110).
• Clear error when a remote host lacks the emily build / NIF.
• Tests cover placement → hostfile-entry and placement → peer-spec projections without needing real remote hosts.

Context: emerged from the code review + design discussion on feature/distributed-mlx-scaffold.