Remote node doesn't receive last update right before process terminates

[tverlaan]

We use Phoenix.Tracker for keeping the state of currently active calls. Each call has a certain state in the metadata of the tracker. When a process updates the state in the terminate callback, the update of the metadata is only published locally. The remote node doesn't see this updated metadata but only sees the process leave since it terminated within the broadcast_window.

I made an example project that demoes the above scenario. You can find it here: https://github.com/tverlaan/presence_multinode

We discussed possible improvements and alternative solutions during ElixirConfEU, but we didn't come to a conclusion just yet.

[josevalim]

Thanks for the report! To add more info, @chrismccord believes this happen somewhere in presence when we are squeezing the joins and leaves together to push to the client (or in the client itself). We currently treat the changes in state as a leave+join so anything that happens in between is condensed.

[tverlaan]

Would it be possible to update meta when leave+join is condensed? Or will it break when another node has a presence update for the same key?

Can you point me to some documentation or a paper where I can read into the implementation by Phoenix a bit more?

[josevalim]

Another node cannot update the same key because the keys are always per node. It’s hard to think about updates because they do not exist at the low level right now as they are leaves and joins. It is also very important to note that there is no delivery guarantee, so relying on any update change to affect other state can be misleading. For example, imagine this scenario: 1. Process is present 2. Process updates meta 3. Node crashes or process dies Local clients will see the meta update but remote ones. Can you please remind us what was the state you wanted to see in an update? Paper: We implement a CRDT right now but we can very likely simplify the abstraction. So reading the CRDT paper will present you the right ideas but the implementation is ultimately simpler. -- *José Valimwww.plataformatec.com.br <http://www.plataformatec.com.br/>Founder and Director of R&D*

[tverlaan]

We were "updating" meta to reflect the process is about to exit clear or fail.

I'll try to get a better understanding of the implementation to see if and/or how we can improve it. Outcome could be documentation as well I think.

[michalmuskala]

One way to solve this would be to differentiate (I'm not really sure how) in handle_diff between process explicitly calling untrack and being removed after it died.

[josevalim]

@tverlaan actually, I think when you leave, you should have the latest meta:

phoenix_pubsub/lib/phoenix/tracker.ex

Lines 64 to 78 in 1185fdb

	def handle_diff(diff, state) do
	for {topic, {joins, leaves}} <- diff do
	for {key, meta} <- joins do
	IO.puts "presence join: key \"#{key}\" with meta #{inspect meta}"
	msg = {:join, key, meta}
	Phoenix.PubSub.direct_broadcast!(state.node_name, state.pubsub_server, topic, msg)
	end
	for {key, meta} <- leaves do
	IO.puts "presence leave: key \"#{key}\" with meta #{inspect meta}"
	msg = {:leave, key, meta}
	Phoenix.PubSub.direct_broadcast!(state.node_name, state.pubsub_server, topic, msg)
	end
	end
	{:ok, state}
	end

This latest meta should have your latest changes. Are you using this information in the client?

In case the leave information does not have the latest meta, then that would be a bug. @michalmuskala, this should also provide you a way to pass the exit information, we could add it to the meta under a reserved key.