Skip to content

UCP/FT: probe-gated lane recovery via aux uct_ep_check - #11563

Merged
gleon99 merged 9 commits into
openucx:masterfrom
evgeny-leksikov:ft-split/13-probe-gated-recovery
Jul 30, 2026
Merged

UCP/FT: probe-gated lane recovery via aux uct_ep_check#11563
gleon99 merged 9 commits into
openucx:masterfrom
evgeny-leksikov:ft-split/13-probe-gated-recovery

Conversation

@evgeny-leksikov

@evgeny-leksikov evgeny-leksikov commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

What?

Gate recovery of failed point-to-point lanes on an auxiliary endpoint probe before reconnecting the rebuilt lane.

Add coverage for successful probe-gated recovery, endpoint teardown with an outstanding probe, and recovery retry exhaustion while unaffected lanes remain usable.

Why?

Repeatedly reconnecting an endpoint over an unavailable route creates unnecessary churn. Silently resetting the failed transport endpoint can also leave inconsistent outstanding-operation accounting.

Probing first allows UCP to reconnect only when the route is usable while preserving communication over unaffected lanes.

How?

When a point-to-point lane fails:

  • discard the failed transport endpoint and rebuild the lane with a wireup proxy containing a fresh inner endpoint;
  • select a same-device auxiliary transport that supports endpoint checking;
  • create an auxiliary endpoint using the packed peer interface address;
  • keep the rebuilt endpoint encapsulated while the probe is outstanding;
  • connect the rebuilt endpoint and replace the proxy only after probe success;
  • retry recovery after probe failure up to the configured limit;
  • cancel and release outstanding probe state during endpoint teardown via common discard flow.

Comment thread src/ucp/core/ucp_ep.c Outdated
Comment thread src/ucp/core/ucp_ep.c Outdated
Comment thread src/ucp/wireup/wireup.c Outdated
@evgeny-leksikov
evgeny-leksikov force-pushed the ft-split/13-probe-gated-recovery branch 3 times, most recently from 8489e40 to ee42e90 Compare July 10, 2026 15:16
Probe failed RC routes through an auxiliary endpoint before reconnecting lanes, and cover recovery, retry exhaustion, and teardown behavior.
@evgeny-leksikov
evgeny-leksikov force-pushed the ft-split/13-probe-gated-recovery branch from 3a429b1 to 4cbbf04 Compare July 19, 2026 18:49
@evgeny-leksikov
evgeny-leksikov marked this pull request as ready for review July 20, 2026 08:16
@gleon99

gleon99 commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

@jeynmann please review

Comment thread src/ucp/core/ucp_ep.c Outdated
Comment thread src/ucp/core/ucp_ep.c
Comment thread src/ucp/core/ucp_ep.c
Comment thread src/ucp/core/ucp_ep.c Outdated
Comment thread test/gtest/ucp/test_ucp_fault_tolerance.cc
Comment thread test/gtest/ucp/test_ucp_fault_tolerance.cc Outdated
@evgeny-leksikov evgeny-leksikov changed the title UCP/FT: probe-gated lane recovery via UD aux uct_ep_check (POC) UCP/FT: probe-gated lane recovery via UD aux uct_ep_check Jul 20, 2026
Comment thread src/ucp/core/ucp_ep.c Outdated
wiface = ucp_worker_iface(worker, select_info.rsc_index);
uct_ep_params.field_mask = UCT_EP_PARAM_FIELD_IFACE |
UCT_EP_PARAM_FIELD_DEV_ADDR |
UCT_EP_PARAM_FIELD_IFACE_ADDR;

This comment was marked as resolved.

Comment thread src/ucp/core/ucp_ep.c

if (ucp_ep_get_live_lanes(ep) == 0) {
ucs_error("ep %p: recovery retries exhausted", ep);
ucp_ep_set_lanes_failed_schedule(ep, 0, UCS_ERR_ENDPOINT_TIMEOUT);

This comment was marked as resolved.

Keep recovery state and auxiliary endpoint lifetime synchronized so transient probes and missing replies advance retries without stalling.
Pass the interface-address length when creating the recovery auxiliary endpoint and keep suppressing keepalive after recovery exhaustion schedules endpoint failure.
Comment thread src/ucp/core/ucp_ep.c
goto done;
}

if (status != UCS_ERR_NO_RESOURCE) {

This comment was marked as resolved.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Comment thread src/ucp/core/ucp_ep.c

/* Reset counter by new event. */
arg->retries_left = context->config.ext.recovery_retries;
arg->state = UCP_EP_RECOVERY_STATE_IDLE;

This comment was marked as resolved.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, sequential multi-lane failures are supported. The recovery state and retry budget are per EP intentionally: a new lane failure restarts the address-exchange round for the current failed-lane set. The actual probe state is per lane (probe[UCP_MAX_LANES]), so the second failure does not overwrite the first lane’s probe.

A lane that becomes ready meanwhile is cleared independently. Reply processing intersects provided_lane_map with the current failed-lane map, so addresses for an already-restored lane are ignored, while newly failed lanes are restored from the subsequent exchange. Resetting retries_left for a new failure event is also intentional.

Comment thread src/ucp/core/ucp_ep.c Outdated
@evgeny-leksikov
evgeny-leksikov requested a review from gleon99 July 24, 2026 09:55
@evgeny-leksikov evgeny-leksikov changed the title UCP/FT: probe-gated lane recovery via UD aux uct_ep_check UCP/FT: probe-gated lane recovery via aux uct_ep_check Jul 27, 2026
Comment thread src/ucp/wireup/wireup.c Outdated
@gleon99
gleon99 enabled auto-merge (squash) July 28, 2026 07:34
@gleon99
gleon99 disabled auto-merge July 28, 2026 07:50
@gleon99
gleon99 enabled auto-merge (squash) July 28, 2026 07:50
@gleon99
gleon99 merged commit db208ee into openucx:master Jul 30, 2026
161 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants