2121//! `(pool, lane)` — is good and is not rewritten here. This module is the seam that lets a plane
2222//! reach it.
2323//!
24- //! **What core owns:** what a candidate SET is, the ORDER candidates are walked in, the
25- //! interchangeability CHECK, the retry-safety RULE, the admission (through the one breaker), and the
26- //! refusal it produces. **What a plane owns:** what a CANDIDATE is ([`Candidate`]) and what makes two
27- //! of them interchangeable (the pin it hands back from
28- //! [`Candidate::interchange_key`]). [`crate::egress_auth::gate`] is the precedent this copies rather
24+ //! **What core owns:** what a candidate SET is, the LOOP that walks it, the interchangeability
25+ //! CHECK, the retry-safety RULE, the admission (through the one breaker), and the refusal it
26+ //! produces. **What a plane owns:** what a CANDIDATE is ([`Candidate`]), what makes two of them
27+ //! interchangeable (the pin it hands back from [`Candidate::interchange_key`]), the ORDER they are
28+ //! offered in ([`Order`]) and which admission primitive its dispatch needs — and nothing else.
29+ //! [`crate::egress_auth::gate`] is the precedent this copies rather
2930//! than a new idea: a plane supplies a grant kind and keeps its refusal wording; it does not keep its
3031//! own decision. [`crate::audit`] is the nearer one still: core owns the mechanism, a stream supplies
3132//! one record type.
8586//! for a plane busbar does not have and shows it selects, admits, trips and reroutes with NO second
8687//! breaker, NO second walk and NO error type written for it.
8788
89+ // ## ONE SELECTION LOOP, ON ALL THREE PLANES (owner ruling R-I: "Unify now — R-B should be true on
90+ // ## selection too in 1.6.0")
91+ //
92+ // [`walk_with`] is that loop, and it is the only one. R-B was already true of the BREAKER — one FSM,
93+ // one `breaker::classify`, one disposition pipeline, no plane-local state machine — and this is what
94+ // made it true of SELECTION as well. Until it landed the model plane had its OWN admission loop in
95+ // `proxy::select::pick_among`, and this module's own definition of the capability said "the one
96+ // selection loop" while two existed. Both are now the same function.
97+ //
98+ // WHY THIS DIRECTION. The model plane's loop carried SWRR weighting, routing policy, session
99+ // affinity, queueing and `on_exhausted`, and the natural fear was that folding it in would drag all
100+ // of that into core. It did not, because those are not selection:
101+ //
102+ // * WEIGHTING / ROUTING POLICY / AFFINITY are ORDER — they decide who is ASKED first, never who is
103+ // ALLOWED. They are now an [`Order`] implementation on the model plane, and nothing they return
104+ // can admit a candidate the breaker refused, because they perform no admission at all.
105+ // * QUEUEING and `on_exhausted` (503 / `fallback_pool` / `least_bad` / `queue`) are what that plane
106+ // does AFTER the loop finds nothing. `proxy::engine::walk`'s own header already said so: the
107+ // queue wait "lives HERE in on_exhausted dispatch, never inside `pick_among` — selection stays
108+ // non-blocking". They never moved and they were never a second selection.
109+ // * The CONCURRENCY PERMIT is the one real difference, and it is a difference of one field:
110+ // `try_admit` is `try_admit_breaker` plus a permit acquisition, over the same cell, the same
111+ // verdict decoder and the same single-flight probe CAS. It rides in as the plane's ADMISSION.
112+ //
113+ // The opposite direction — making `pick_among` the one loop and moving MCP and A2A onto it — was
114+ // rejected on the code: it is welded to `App`, to `WeightedLane` and to lane indices into
115+ // `app.lanes`, which MCP servers and A2A agents are not members of (they run on `PlaneBreakers`'
116+ // own runtime); and it has neither the pin check nor the repeat-safety rule, so those two safety
117+ // rules would have had to be bolted onto it or lost on the two planes that need them most.
118+ //
88119// MOUNTED. The reroute-parity unit (owner ruling R-B: "llm mcp a2a are identical") consumes this
89120// seam on both non-LLM planes: `mcp::reroute` walks a `tool_pools:` candidate set before every
90121// `tools/call` leg, and `a2a::receive` walks an `agent_pools:` set at submission admission. Both
@@ -312,33 +343,36 @@ impl std::fmt::Display for Refusal {
312343 }
313344}
314345
315- /// A CANDIDATE THAT MAY BE DISPATCHED TO, and the probe it now owns.
346+ /// A CANDIDATE THAT MAY BE DISPATCHED TO, and the admission token it now owns.
316347///
317- /// Only [`walk`] can build one, so a dispatch that skipped the breaker does not compile. `probe_epoch`
318- /// is the owner token [`LaneRuntime::try_admit_breaker`] handed back — the caller releases it
319- /// OWNER-CHECKED via `release_probe_owned_in` after recording an outcome, exactly the discipline the
320- /// model plane's queue dispatch uses.
321- pub ( crate ) struct Admitted < ' a , C > {
348+ /// Only [`walk_with`] can build one, so a dispatch that skipped the breaker does not compile. `token`
349+ /// is whatever the plane's admission handed back: on the MCP/A2A planes that is the bare probe epoch
350+ /// from [`LaneRuntime::try_admit_breaker`]; on the model plane it is the [`crate::store::Admit`]
351+ /// carrying the probe epoch AND the concurrency permit that plane also needs. Both are the SAME
352+ /// breaker FSM and the SAME single-flight probe — the token differs only by whether a permit rides
353+ /// along (see `store::in_memory::availability`: `try_admit` is `try_admit_breaker` plus a permit
354+ /// acquisition, nothing else). The caller releases the probe OWNER-CHECKED via
355+ /// `release_probe_owned_in` after recording an outcome.
356+ pub ( crate ) struct Admitted < ' a , C , T = u64 > {
322357 candidate : & ' a C ,
323358 position : usize ,
324- probe_epoch : u64 ,
359+ token : T ,
325360}
326361
327362/// HAND-WRITTEN so the bound lands on [`Candidate`] and NOT on `Debug`. A derive would make every
328363/// plane's candidate type `Debug` or lose `expect`/`unwrap` on the result — which is a second thing a
329364/// third plane has to write, and the acceptance test for this seam is that there is no second thing.
330365/// It prints the plane's own [`Candidate::name`], which is all an operator wants from it anyway.
331- impl < C : Candidate > std:: fmt:: Debug for Admitted < ' _ , C > {
366+ impl < C : Candidate , T > std:: fmt:: Debug for Admitted < ' _ , C , T > {
332367 fn fmt ( & self , f : & mut std:: fmt:: Formatter < ' _ > ) -> std:: fmt:: Result {
333368 f. debug_struct ( "Admitted" )
334369 . field ( "candidate" , & self . candidate . name ( ) )
335370 . field ( "position" , & self . position )
336- . field ( "probe_epoch" , & self . probe_epoch )
337371 . finish ( )
338372 }
339373}
340374
341- impl < ' a , C : Candidate > Admitted < ' a , C > {
375+ impl < ' a , C : Candidate , T > Admitted < ' a , C , T > {
342376 /// The candidate this request may be sent to.
343377 pub ( crate ) fn candidate ( & self ) -> & ' a C {
344378 self . candidate
@@ -347,10 +381,18 @@ impl<'a, C: Candidate> Admitted<'a, C> {
347381 pub ( crate ) fn position ( & self ) -> usize {
348382 self . position
349383 }
384+ /// The plane's admission token — probe epoch (MCP/A2A) or `Admit` (the model plane's probe epoch
385+ /// plus its concurrency permit). Consumed by value: an admission is used once.
386+ pub ( crate ) fn into_token ( self ) -> T {
387+ self . token
388+ }
389+ }
390+
391+ impl < ' a , C : Candidate > Admitted < ' a , C , u64 > {
350392 /// The single-flight probe owner token, for the owner-checked release after the outcome is
351- /// recorded.
393+ /// recorded. The MCP/A2A spelling of [`Admitted::into_token`].
352394 pub ( crate ) fn probe_epoch ( & self ) -> u64 {
353- self . probe_epoch
395+ self . token
354396 }
355397}
356398
@@ -377,10 +419,72 @@ pub(crate) struct Attempt<'r> {
377419 pub ( crate ) operation : & ' r str ,
378420}
379421
422+ /// WHERE THE NEXT CANDIDATE COMES FROM — the one thing [`walk_with`] does NOT own.
423+ ///
424+ /// The loop is one; the ORDER is a plane's. The MCP and A2A planes read `members:` in the order the
425+ /// operator wrote it ([`InOrder`]). The model plane's order is SMOOTH WEIGHTED ROUND ROBIN, which is
426+ /// STATEFUL — it cannot be handed over as a precomputed list because each pick mutates the SWRR
427+ /// accumulator and a refused pick must re-run the selection over the remaining set. So the order is
428+ /// asked for one candidate at a time, and `refused` reports what the walk did with the previous
429+ /// answer, which is exactly what a generator needs to advance.
430+ ///
431+ /// This is the same division [`Candidate`] draws: core owns the mechanism, a plane supplies the one
432+ /// value only it can compute. Weighting, routing policy, session affinity and drain are ORDER — they
433+ /// decide who is asked first, never who is allowed. Nothing an implementation of this trait returns
434+ /// can admit a candidate the breaker refused, because it does not perform admission at all.
435+ pub ( crate ) trait Order {
436+ /// The next position in `members` to CONSIDER, or `None` when this order is exhausted.
437+ ///
438+ /// `refused` is the position the previous call yielded and the walk then failed to admit
439+ /// (`None` on the first call, and on every call after a successful admission — which never
440+ /// happens, because the walk returns).
441+ fn next ( & mut self , refused : Option < usize > ) -> Option < usize > ;
442+ }
443+
444+ /// THE OPERATOR'S ORDER: `members:` as written, skipping anything already `tried`.
445+ ///
446+ /// The MCP and A2A order, and the one the seam's own [`walk`] uses. `members[0]` is the primary and
447+ /// the rest are its declared twins, so "first admissible in declaration order" is the whole policy —
448+ /// there is no weighting to apply because two deployments of ONE image are not a load-balancing
449+ /// decision, they are a same-or-nothing choice.
450+ pub ( crate ) struct InOrder < ' a > {
451+ tried : & ' a [ usize ] ,
452+ cursor : usize ,
453+ len : usize ,
454+ }
455+
456+ impl < ' a > InOrder < ' a > {
457+ pub ( crate ) fn new ( tried : & ' a [ usize ] , len : usize ) -> Self {
458+ Self {
459+ tried,
460+ cursor : 0 ,
461+ len,
462+ }
463+ }
464+ }
465+
466+ impl Order for InOrder < ' _ > {
467+ fn next ( & mut self , _refused : Option < usize > ) -> Option < usize > {
468+ // A refused position is never revisited because the cursor only moves forward — the walk's
469+ // own `refused` bookkeeping is redundant here, which is why it is ignored rather than
470+ // re-checked.
471+ while self . cursor < self . len {
472+ let position = self . cursor ;
473+ self . cursor += 1 ;
474+ if !self . tried . contains ( & position) {
475+ return Some ( position) ;
476+ }
477+ }
478+ None
479+ }
480+ }
481+
380482/// THE WALK: pick the next candidate in `members` that this request may be sent to, and admit it
381483/// through THE circuit breaker.
382484///
383- /// There is exactly ONE selection loop for these planes and this is it.
485+ /// There is exactly ONE selection loop and this is it. Every plane reaches it: MCP and A2A through
486+ /// [`walk`], the model plane through `proxy::select::pick_among`, which supplies an SWRR [`Order`]
487+ /// and the permit-carrying admission and owns NO loop of its own.
384488///
385489/// The order of the checks is the contract, because the first failure is what the operator is told:
386490///
@@ -390,16 +494,33 @@ pub(crate) struct Attempt<'r> {
390494/// early on purpose: it is a statement about the REQUEST, and no upstream's health can change it.
391495/// 3. **Do the pins agree?** Every hop after the primary must present the primary's exact
392496/// `interchange_key`. Checked before admission so a mismatched candidate is never given a probe.
393- /// 4. **Will the breaker have it?** [`LaneRuntime::try_admit_breaker`] — the same call the model
394- /// plane's queue dispatch makes, against the same `(pool, lane)` cell — walked in order until one
395- /// admits.
396- pub ( crate ) fn walk < ' a , C : Candidate > (
397- store : & dyn LaneRuntime ,
497+ /// 4. **Will the breaker have it?** The plane's `admit` — [`LaneRuntime::try_admit_breaker`] on the
498+ /// MCP/A2A planes, [`crate::store::LaneRuntime::try_admit`] (the same breaker plus a concurrency
499+ /// permit) on the model plane — walked in `order` until one admits.
500+ ///
501+ /// `admit` is a hook and not a hardcoded call for ONE reason and it is not extensibility: the model
502+ /// plane must acquire its concurrency permit and win the breaker probe ATOMICALLY (see `try_admit`'s
503+ /// capacity-peek-before-probe-CAS argument — splitting them wedged a tripped-and-saturated lane
504+ /// forever). It is the SAME FSM, the SAME cell and the SAME single-flight probe either way; only the
505+ /// permit rides along. A plane cannot use this hook to admit a candidate the breaker refused,
506+ /// because every implementation of it is one of those two store methods and there are no others.
507+ ///
508+ /// `passed_over` is filled with `(position, why)` for EVERY candidate this walk could not admit, in
509+ /// the order it considered them — on the success path as well as the refusal path, because a plane
510+ /// that reroutes still owes its operator the reasons the earlier members were skipped. It is an
511+ /// out-parameter rather than a field of the two results precisely because BOTH results need it: the
512+ /// model plane's `on_exhausted` disposition reads it to decide 503-vs-queue-vs-spill, and it needs
513+ /// the reasons whether the walk ended in a dispatch or in a shed. The walk CLEARS it first, so a
514+ /// caller reusing one buffer across failover hops reports this hop's exhaustion, never a stale one.
515+ pub ( crate ) fn walk_with < ' a , C : Candidate , T > (
398516 pool : & str ,
399517 members : & ' a [ C ] ,
400518 attempt : & Attempt < ' _ > ,
401- now : u64 ,
402- ) -> Result < Admitted < ' a , C > , Refusal > {
519+ order : & mut dyn Order ,
520+ passed_over : & mut Vec < ( usize , Unavailable ) > ,
521+ admit : & mut dyn FnMut ( usize , & ' a C ) -> Result < T , Unavailable > ,
522+ ) -> Result < Admitted < ' a , C , T > , Refusal > {
523+ passed_over. clear ( ) ;
403524 // 1. Nothing to select from.
404525 let Some ( primary) = members. first ( ) else {
405526 return Err ( Refusal :: Empty {
@@ -419,11 +540,14 @@ pub(crate) fn walk<'a, C: Candidate>(
419540 } ) ;
420541 }
421542
422- let mut refused: Vec < ( String , Unavailable ) > = Vec :: new ( ) ;
423- for ( position, member) in members. iter ( ) . enumerate ( ) {
424- if attempt. tried . contains ( & position) {
425- continue ;
426- }
543+ let mut last_refused: Option < usize > = None ;
544+ while let Some ( position) = order. next ( last_refused) {
545+ let Some ( member) = members. get ( position) else {
546+ // An order that names a position outside the pool has no candidate to admit. Treat it as
547+ // the end of that order rather than panicking on a slice index: the walk's job is to
548+ // refuse safely, never to trust an index.
549+ break ;
550+ } ;
427551 // 3. THE PIN CHECK. The primary defines the pool's identity; every other member has to prove
428552 // it is the same deployment. `None` never matches — not even another `None`, because two
429553 // registrations that have each approved nothing are two unknowns, not one fact.
@@ -442,27 +566,66 @@ pub(crate) fn walk<'a, C: Candidate>(
442566 } ) ;
443567 }
444568 }
445- // 4. THE ONE BREAKER. Not a second admission path, not a copy of one: this is
446- // `LaneRuntime::try_admit_breaker`, the same method `proxy::engine::walk` calls on the
447- // model plane, against the same per-(pool, lane) cell.
448- match store. try_admit_breaker ( pool, member. lane ( ) , now) {
449- Ok ( probe_epoch) => {
569+ // 4. THE ONE BREAKER. Not a second admission path, not a copy of one: `try_admit_breaker` and
570+ // `try_admit` are the same FSM over the same per-(pool, lane) cell.
571+ match admit ( position, member) {
572+ Ok ( token) => {
450573 return Ok ( Admitted {
451574 candidate : member,
452575 position,
453- probe_epoch ,
576+ token ,
454577 } )
455578 }
456- Err ( why) => refused. push ( ( member. name ( ) . to_string ( ) , why) ) ,
579+ Err ( why) => {
580+ passed_over. push ( ( position, why) ) ;
581+ last_refused = Some ( position) ;
582+ }
457583 }
458584 }
459585
460586 Err ( Refusal :: NoneAdmissible {
461587 pool : pool. to_string ( ) ,
462- tried : refused,
588+ tried : passed_over
589+ . iter ( )
590+ . map ( |( position, why) | {
591+ (
592+ members
593+ . get ( * position)
594+ . map ( |m| m. name ( ) . to_string ( ) )
595+ . unwrap_or_default ( ) ,
596+ * why,
597+ )
598+ } )
599+ . collect ( ) ,
463600 } )
464601}
465602
603+ /// THE SEAM'S SPELLING OF [`walk_with`]: the operator's `members:` order, admitted breaker-only.
604+ ///
605+ /// The MCP and A2A call sites' entry point. It adds NO selection logic — it names the two things
606+ /// those planes supply ([`InOrder`] and [`LaneRuntime::try_admit_breaker`]) and hands them to the one
607+ /// loop.
608+ pub ( crate ) fn walk < ' a , C : Candidate > (
609+ store : & dyn LaneRuntime ,
610+ pool : & str ,
611+ members : & ' a [ C ] ,
612+ attempt : & Attempt < ' _ > ,
613+ now : u64 ,
614+ ) -> Result < Admitted < ' a , C > , Refusal > {
615+ let mut order = InOrder :: new ( attempt. tried , members. len ( ) ) ;
616+ // These planes render their refusal from `Refusal` itself (which already carries every reason by
617+ // name), so the positional buffer is local and dropped here.
618+ let mut passed_over = Vec :: new ( ) ;
619+ walk_with (
620+ pool,
621+ members,
622+ attempt,
623+ & mut order,
624+ & mut passed_over,
625+ & mut |_position, member| store. try_admit_breaker ( pool, member. lane ( ) , now) ,
626+ )
627+ }
628+
466629/// RECORD WHAT THE UPSTREAM DID, through the ONE classifier and onto the ONE breaker cell.
467630///
468631/// [`crate::breaker::classify`] is protocol-agnostic already — it consumes a `CanonicalSignal` a
0 commit comments