Ending a session gracefully

[meejah]

Consider magic-wormhole connections as having two layers: the generic setup of a channel (or many subchannels in Dilation) and an "application layer" on top (such as file-transfer).

Most "application layer" protocols will want a way to gracefully end their session.
(I use "session" here to include Dilation because those may span many TCP connections).

The file-transfer protocol has an implicit end-of-session: any offer that is exchanged is the only one, so no more mailbox communication will happen after that.

However, the next-generation file transfer (see magic-wormhole/magic-wormhole-protocols#23 ) for example needs some way to indicate that the entire session is over.

I expect all Dilation protocols will have a similar need. In Dilation specifically, we could choose to declare that closing the control-channel is this "session is over" signal. However: that doesn't work for non-Dilated protocols; and the control-channel may never exist (because of network conditions). So, we need a mailbox-level message here.

Another consideration (thanks @warner) is synchronization around close signals. In subchannels in Dilation, there is a "half-closed" state where side A has sent all the data they intend to, and sent a CLOSE message. So side A knows side B will see all their data before the CLOSE. Similarly, since side A will wait for side B's CLOSE message, they now they'll see all of side B's data. Maybe it is possible to extend this synchronization to the mailbox itself.

Currently, the w.close() API sends a CLOSE message to the relay, which responds with a CLOSED (but doesn't, for example, send a CLOSED to the other side). So an option could be to re-use this signal -- amend the protocol so that either side could receive a CLOSED message that isn't in response to a CLOSE. This kind of breaks the symmetry of close/closed though.

I think an explicit mailbox-level message + phase should be added to the protocol. This can be used by both "classic" Transit-using (or even mailbox-only) protocols and by Dilation. It also allows proper recover in "journaled" mode, I believe.

The rough idea is this:

• one side (call it A) sends its final mailbox message
• it then sends a phase=closing mailbox message with body {"closing": true}
• after sending such a message, that side MUST NOT post any more mailbox messages (that is, no more number-phase application messages)
• upon seeing the phase=closing message, side B posts any remaining mailbox messages as then a final phase=closing message with body {"closed": true}.
• side B MUST close its mailbox immediately after this (sends CLOSE to mailbox-server, gets CLOSED back) -- it knows side A will produce no new messages
• side A receives any additional mailbox messages from side B (maybe none)
• side A gets a phase=closing with {"closed": true} from side B
• side A now knows it has all mailbox messages, and MUST immediately close its mailbox (because it knows the other side has done this also)
• the session is now concluded

[meejah]

(If we agree this sounds plausible, I will instead express it as a PR against the protocol specs alongside Python implementations)

[piegamesde]

I'd like to leave this up to the application level protocol how to deal with session closing, as the needs are really diverse. Between rendezvous messages, Dilation and Transit, there are seven different combinations of communication channels that may be used, of which five are practicable (using Dilation and Transit connections at once makes no sense). Each application should decide on one communication channel as their "source of truth", and implement a proper closing mechanism there.

However, I agree that there is need to close a mailbox for all participants (instead of having a per-client counter and only closing after the last one is gone), see for example #19

[meejah]

"Close a mailbox for all participants" is precisely the use-case being addressed here.

(And as a bonus, it should serve as a "source of truth" for many protocols -- of course, other protocols could choose a different way to shut down if desired).

[meejah]

I'd like to leave this up to the application level protocol how to deal with session closing, [..]

The reason I don't want to do this is precisely because of possible confusion, and because I believe all application protocols need this feature. Therefore, it makes sense to have a built-in way to do it.

The mailbox is the correct communication channel because it's the only one that definitely exists. Transit or Dilation may never be established. Since an application protocol that only uses the mailbox is possible, it has to be the mailbox channel.

Note that the existing file-transfer application protocol does the session-closing implicitly: only a single offer is allowed, and only a single file-transfer will occur. Thus, the Transit connection never needs to stay open and there is no ambiguity as to when the each side should close the mailbox. There is a remaining problem, though: if a transfer is cancelled. There is no explicit way to signal this, and the protocol suffers some ambiguous behavior in those cases. An explicit "close" signal like the one described here could answer that need.

For Dilation the above is no longer true: the session may stay open for an arbitrary amount of time.

The only other application protocols I know of are magic-folder and tahoe-lafs Grid invitations: both of those also have a non-ambiguous end, so don't strictly need this mechanism. However, they do not have a way to cancel an ongoing invite -- this could be used for that.

[piegamesde]

The rendezvous server already enforces that at most two parties may be in one room. Therefore I'd like to expand the semantics some more on that premise:

• If one side sends CLOSE the mailbox should actually be closed (instead of waiting for the other side to CLOSE it too). The other side should be notified about that, either through CLOSED, a custom message or some error message.
  • This should not cause any breakage with current implementations, if anything it might even fix some hanging/deadlock bugs
• We could even make it so that a closed mailbox cannot be reopened (for some reasonable amount of time), this would solve the "cancel the invitation" problem. (Actually, this should probably be done on the nameplate level, because then the original file transfer would profit from it too).

The only other application protocols I know of are magic-wormhole and tahoe-lafs Grid invitations

There are also my port forwarding and transfer v2 experiments (the former being released but marked as experimental feature), both automatically close the wormhole as soon as a transit connection is established and use that for subsequent error handling and everything else.

[meejah]

Changing the semantics of CLOSE I think has a problem with giving both sides a chance to send any "final" messages. It's more like pulling the ethernet out of the wall to close TCP.

That is, if side A sends a CLOSE it may be that side B has messages in-flight that will now never get delivered (that is, side A never has a chance to let side B finish). The phase approach outlined above does give each side a chance to synchronize and come to agreement about closing (i.e. there's an end-to-end messaging flow instead of one side arbitrarily shutting down the mailbox). In spirit, this is like TCP shutdown with FIN then ACK -- the "fin" is the "phase=closing" from side A (which then enters a state similar to closing in TCP) and the "fin-ack" is the "phase=closing" from side B back to side A (TCP is somewhat more complex than this, so it's just an analogy).

Note that applications don't have to use the above proposed flow as the only way to shut down. This would still work with the port-forwarding flow you outline, I think? That is, you follow the above phased close once Transit is established. (In fact, you could ignored the phased close entirely and have each side implicitly know when to CLOSE their side of the mailbox -- that is, when Transit is established).

[meejah]

(Note that I'm not totally against using CLOSE and/or changing its semantics, but I am against making those semantics that one side can just immediately shut off the mailbox w/o any end-to-end flow / acknowledgement).

[piegamesde]

Hm, do you have an example of a protocol that might make use of that close behavior on the happy path? I understand your proposal, but it adds some significant complexity to the protocol so I need a compelling use-case to convince me it is worth the cost.

[meejah]

Dilated File Transfer

edit: to expand, there isn't a way to "end the session" in the proposed Dilated File Transfer application protocol. This would be it.

[piegamesde]

Actually, Dilated file transfer can use that same mechanism on the Dilation level: closing the control channel will already do the closing/closed dance and then once the control channel is closed the Wormhole may safely be closed without need for further negotiation.

[meejah]

Right, the control-channel could be used (however, this is more general -- and covers the cases where we've never actually succeeded at opening a Dilation channel -- which may never succeed). That is, the only communications channel the definitely exists is the mailbox.

[piegamesde]

cases where we've never actually succeeded at opening a Dilation channel -- which may never succeed

That's why I was specifically talking about the happy path. On the unhappy path, I think just closing in the face and triggering errors on the other side is fine, because it would have been an error anyways.

For example, in a dilated file transfer session one side decides that it wants to close. If there are any ongoing transfers those will be awaited, but it will reject all new offers and make no send offers either. Once the last transfer completed, it can simply close the connection (ignoring graceful shutdown of the Dilation control channel for now). If the other side was to make an offer attempt concurrently it would simply get a connection lost error, which would be fine and expected.

[meejah]

Again, I'm not saying we can't use the control-connection (in fact, that was my first attempt at this) but it's not as general (i.e. it doesn't help mailbox-only protocols). (It's Dilation-only -- and of course depends on that channel being established in the first place.

Application-protocols inventing baroque ways of shutdown is precisely why I wanted a more-general way (we've already got 3 different ways to shut down with just 2 protocols under consideration here).

The "ongoing transfers awaited" part is why I believe we need an intermediate state between "open" and "closed" (i.e. "closing", where no new messages / offers / etc are sent but ongoing things including in-flight mailbox messages can be completed).

There's a few options proposed, maybe it makes sense to summarize the pro/con of them and missing features etc. Are there any besides:

• unilateral mailbox close from either side (i.e. change semantics of CLOSE)
• custom application-protocol messaging
  • implicit (somehow implied by other messaging / state of the protocol)
  • close control subchannel (Dilation-only)
  • (custom message?)
• a "phase" approach (phase=closing)

There's another edge-case here which might need more-careful wording of the spec: side A has decided to stop the Dilated File Transfer session (i.e. entered phase=closing), but there's at least one ongoing transfer. Then, one side loses their dilation network connection. So, you should be allowed to do more dilation-* phase messaging (the text above does already prohibit just number-phase messaging) once you're in "closing" phase. For Dilation we need to be clear about control messages (i.e. a side in phase=closing can't send any more?).

This allows the subchannel with the ongoing transfer to be re-established (via a further dilation-* generation) and that transfer can conclude properly. Then the other side enters phase=closing etc etc and it shuts down gracefully by sending CLOSE on all subchannels (including control) and awaiting the ACK.

[meejah]

Maybe this is easier to analyze as a pull-request or better write-up than a github comment ... do we at least agree that a unilateral CLOSE doesn't cover a happy-path shutdown of Dilated File Transfer..?