Specify recovery semantics

[mmusgrov]

Section 3.2.9. Recovery Requirements of the spec document discusses recovery but it is under specified. This issue is to formalise our stance on recovery. The preference is to leave it unspecified since the spec is already clear that the outcome of closing/cancelling an LRA MUST either have an eventual end state otherwise it MUST be logged.

[tomjenkinson]

I think it is clear from 3.2.9 that the responsibility of infrastructure and applications is to be able to satisfy the requirements of the normal LRA life cycle in the presence of failure.

[mmusgrov]

Note that there is a test in the TCK called acceptTest that tests that participants which return 202 Accepted are eventually completed/compensated. The test does this by triggering a "recovery scan" on a "recovery coordinator endpoint" and the config for this endpoint need removing and an alternative way of checking for completion/compensation is required (such as waiting for a configurable period).

[mmusgrov]

@tomjenkinson you are right, I think this issue can be closed as "already fixed"

[rdebusscher]

@mmusgrov @tomjenkinson
Is the LRA_HTTP_RECOVERY_HEADER linked to the recovery requirements described in 3.2.9? The header is only briefly mentioned in 3.2.2 Start and ending LRA but not explaining what the Participant should do with the Recovery Header.

[mmusgrov]

The LRA_HTTP_RECOVERY_HEADER was added to the Java client API (and was in the original REST part of the spec when the project was in the sandbox) to allow participants to tell the coordinator to contact them on different URLs. This was to support participant recovery. Since we no longer have the REST endpoints and Java Client API it is not so simple to incorporate the same functionality.

If we loose the LRA_HTTP_RECOVERY_HEADER in milestone 1.0 then we loose participant recovery but I am fine with moving it out to 1.x.

[mmusgrov]

BTW even without participant recovery functionality the LRA_HTTP_RECOVERY_HEADER header is still required - it is returned on responses to client requests that result in the enlistment of a participant - it enables the caller and the participant to know that a participant has been enlisted in the LRA. Also note that the header is included when participant callbacks are issued (compensate/complete/etc). I will check that this is clear in the spec.

Actually participant recovery functionality is quite easy to keep if we add an @Recover annotation:

@Recover
Response participantHasMoved(
    @HeaderParam(LRA_HTTP_RECOVERY_HEADER) String recoveryId) {...}

@Compensate
aMethod(...) {}

etc

The implementation sees the presence of the recovery header and replaces the participant (ie uses this new resource to complete/compensate instead of the old one) and puts the new recovery URI in the LRA_HTTP_RECOVERY_HEADER response header so that the caller can move it again if required.

This would also fix the requirement I mentioned above about needing something to trigger a recovery attempt (the TCK test called acceptTest): if the LRA is completing then the implementation SHOULD replay the completion notification on this participant (similarly for compensating).

[tomjenkinson]

+1, except we must not (nor should we need to) be changing the recovery header ID

[tomjenkinson]

My understanding is that business logic iterates outstanding recoveryIds and calls them during the processes recovery sequence.

[mmusgrov]

@rdebusscher @ochaloup @mstefank and others, I'd like to get agreement on this before Rudy goes on PTO (which I think means by the end of the week).

During this weeks hangout the primary objection came from Ondra stating that recovery should not be exposed in the business logic. But this should not be a problem since the method does not need to be exposed to other services because issue #35 can be used to hide it from such users.

Martin's objection was that applications should be using service URLs so should never need to change their endpoints. But unless we have good reason we should not restrict which endpoints a service can be contacted on.

Note that the reason I have called it recovery and raised the proposal on this github issue is that the feature would typically be used if the participant crashes during the final phase of the protocol (ie when the LRA is in the state completing/compensating) and wishes to come up on a different IP address or port. The fact that participants can also use the feature to trigger replay of the end phase of the protocol on the participant in order to discover the outcome is just a bonus.

[ochaloup]

My objection is about giving the user a new annotation. It does not make any difference if the @Recover is used as the JAX-RS endpoint or if it's defined as standalone @Recover annotation without the JAX-RS endpoint being defined.

I think the ability to redefine the location for the participant is reasonable. But it would be better not enforcing it with annotation at the participant side. The programmatic API would be a better fit for this.

With this proposal the behaviour of the coordinator regarding LRA_HTTP_RECOVERY_HEADER should be clarified. If the LRA_HTTP_RECOVERY_HEADER will be left in the spec then it would be good to say that coordinator has to return when participant is enlisted. Then it should be clarified what is the REST call format which the coordinator expects from the participant when invoked to LRA_HTTP_RECOVERY_HEADER url. Then only the direct REST call could be used to invoke to functionality at the coordinator.

It could be worthy to consider splitting the capability of the participant location enlistment into two - one for changing location and the other for recovery invocation.

[rdebusscher]

@mmusgrov @ochaloup @xstefank and all,

I have created a scenario using the Participant recovery in this gist https://gist.github.com/rdebusscher/ad914ba528dbe638f926e6bf2fdbd3e9

I cannot see how the LRA gets properly cancelled without specific code in the client (which is not desired)

[mmusgrov]

In response to Ondra:

To your first comment:

The programmatic API would be a better fit for this.

The two APIs (programmatic and annotations) are intended to be semantically equivalent.

And to the remainder of your comments, we do already do what you are asking for:

Your second comment:

would be good to say that coordinator has to return [the recovery header] when participant is enlisted

is already covered in the specification document

If the method is to run in the context of an LRA and the annotated class
also contains methods annotated with @Compensate and @Complete then the bean
(aka participant or compensator) will be enlisted with the LRA. Enlisting with
an LRA means that the bean will be notified when the current LRA is
subsequently cancelled or closed. In addition the implementation should generate a
URI for this participant that can be used
for recovery purposes and it should make it available to the participant via a JAX-RS
header param with name defined by the Java constant LRA_HTTP_RECOVERY_HEADER as defined

and the LRA javadoc:

/**
 * the name of the HTTP header field that contains a recovery URI corresponding
 * to a participant enlistment in an LRA
 */
String LRA_HTTP_RECOVERY_HEADER = "Long-Running-Action-Recovery";

The Red Hat implementation exposes this header whenever our coordinator invokes participant callbacks (though I have not said that in the spec yet since we still have this current open issue).

Your third comment is invalid:

it should be clarified what is the REST call format which the coordinator expects from the participant when invoked

The LRA specification is focused purely on the annotations and the coordinator is private to our implementation of the spec. All that needs to happen is for the implementation (of the LRA spec) to notice the Recover annotation and check two things 1) that the recovery header is present on the incoming request and 2) that the enclosing resource conforms to our requirements to be a participant (ie has a Compensate method) - it is very similar to how the implementation should enlist a participant (the only difference would be that it should update it private mapping of participants to recovery ids which it can do because the incoming JAX-RS call to the Recover method will carry the recovery header that was provided by the implementation when the original participant was enlisted).

To your last comment:

It could be worthy to consider splitting the capability of the participant location enlistment into two - one for changing location and the other for recovery invocation.

I don't see why that would be necessary: the application is saying "I would like to be asked to compensate at a new location please". Since typically the application uses this recovery behaviour when the LRA is in the state compensating or completing, then after a crash it makes eminent sense for the implementation to replay the termination phase at the new location (it would be surprising if we specified it otherwise).

In response to Rudy:

I am not altogether clear about your scenario so my response may be off. You say

service A is crashed at that moment and restored at a different URL.
How is the cancelStepA() called eventually?

If service A crashes and comes back up on a different endpoint the method annotated with @recover should be called by something. This will cause the coordinator/implementation to replay the compensation invocation on the participant where the @recover annotation is present.

[ochaloup]

@mmusgrov :

The two APIs (programmatic and annotations) are intended to be semantically equivalent.

I don't know what is behind this intention but I don't see it as a good idea here. If there is a programmatic API then there is no much reason to provide this functionality with the annotation.

Your second comment is already covered in the specification document

I see. When I was reading it I thought it would be good to be clarified but as you summarized that for me I have no more objections.

The LRA specification is focused purely on the annotations and the coordinator is private to our implementation of the spec.

I do understand. I wrote my comment rather for a consideration and discussion. I understand that this comment nearly touches the interoperability. Here we talk about API not about the message communication protocol.

I don't see why that would be necessary: the application is saying "I would like to be asked to compensate at a new location please".

Ok. It makes sense to me.