how to distribute to different erlang nodes?

[benoitc]

Is there a way to distribute to different Erlang nodes?

[jaynel]

Initially I am concentrating on a single VM node to avoid complexity and to test the ideas. There is nothing that prevents distribution. I don't use supervision hierarchies, but instead rely on spawn_link to tightly bind all related processes. A single failure should take down a full Co-op (but no more than a single Co-op unless you explicitly tie dynamically spawned processes to the Co-op), and that feature should work across VM nodes.

Probably the most useful technique would be to build a fanout pattern that had workers on different nodes. I am also considering a built-in capability to have a DHT approach to concurrency, but that is much further off in the future.

The approach to making it distributed will involve having proplist options on the coop constructor. Requiring .erlang.hosts / net_adm:world() and so on will complicate the configuration and setup, but once that is present you should be able to spawn processes anywhere. I can see separating the Co-op Head from the Co-op Body, or having a Fanout across multiple nodes, or even a pipeline that keeps similar data on the same VM.

You can do it very simply now, by keeping a single Co-op on a single VM, but feeding the resulting data to a different process or Co-op on a different VM. The coop_node:node_task_add_downstream_pids/2 function accepts a list of Pids and adds them to the output queue without linking to the Kill_Switch or any of the other Co-op processes, so network instability shouldn't crash your Co-ops.