Skip to content

Commit 5136dcc

Browse files
committed
feat(core): one selection loop for llm, mcp and a2a
R-I: "Unify now — R-B should be true on selection too in 1.6.0." It is. There were two candidate-selection loops and the ledger claimed one. `failover::walk_with` is now the only one; `proxy::select::pick_among` owns no loop and calls it. WHICH LOOP SURVIVED AND WHY, from the code: The seam survived. The model plane's extras turned out not to be selection at all. Weighting, routing policy and session affinity are ORDER — they decide who is asked first, never who is allowed — so they became a `failover::Order` implementation on the model plane, and nothing they return can admit a candidate the breaker refused because they perform no admission. Queueing and `on_exhausted` are what that plane does AFTER selection finds nothing; `engine::walk`'s own header already said the queue wait "lives HERE in on_exhausted dispatch, never inside `pick_among` — selection stays non-blocking". They did not move. The one real difference is the concurrency permit, and it is one field: `try_admit` is `try_admit_breaker` plus a permit acquisition over the same cell, the same verdict decoder and the same single-flight probe CAS, so it rides in as the plane's ADMISSION. The other direction was rejected on evidence, not taste: `pick_among` is welded to `App`, `WeightedLane` and lane indices into `app.lanes`, which MCP servers and A2A agents are not members of (they run on `PlaneBreakers`' own runtime); and it has neither the pin check nor the repeat-safety rule, so both would have had to be bolted on or lost on the two planes that need them most. WHAT THE LOOP OWNS, identically for all three planes: is there a candidate set, is this hop a repeat and is that allowed, do the pins agree, will the breaker admit, and what the refusal is. What a plane owns: a `Candidate`, a pin, an `Order`, an admission — and no loop. TWO PLANE FACTS NOW STATED INSTEAD OF ASSUMED. The model plane's `interchange_key` is the pool name, because a model endpoint has no digest to verify — `pools:` members are interchangeable because the operator wrote them in one pool. The check is not skipped for this plane; it runs against the only pin this plane has. And the model plane declares `Repeatable::Yes`: a completion is a read, so an after-dispatch hop has always been allowed here, while MCP/A2A hand the same rule `Repeatable::No` unless the operator names the operation. The rule is one; the answers differ because the operations do. THE LLM SUITE PASSES UNEDITED. Not one existing test changed. The sticky fast path's quirk — a refused sticky is NOT locally excluded, so SWRR may attempt the same lane a second time, which `handle_queue` dedups and documents — is preserved exactly, in the model plane's `Order` where it is one visible branch. PROVING MUTATIONS. (1) `if position != 0 { break }` in `failover::walk_with`: `failover-reroute` × llm, × mcp-client and × a2a-client all go red together, 4,513 green — three planes, one loop, one mutation. (2) `if refused.is_some() { return None }` in `SwrrOrder::next` (model plane only): the new saturation test goes red, MCP/A2A stay green, and so does the cell's OLD pin — which is why it is re-pointed. Gate exit 0, 41/41 across 11 build configurations. busbar-core lib 4,540 (+2). Equality ledger unchanged at 11 missing: the three cells were already proven, they are now proven on ONE mechanism, and the definition says so.
1 parent 89cd49e commit 5136dcc

6 files changed

Lines changed: 728 additions & 163 deletions

File tree

crates/busbar-core/src/failover/mod.rs

Lines changed: 201 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,12 @@
2121
//! `(pool, lane)` — is good and is not rewritten here. This module is the seam that lets a plane
2222
//! reach it.
2323
//!
24-
//! **What core owns:** what a candidate SET is, the ORDER candidates are walked in, the
25-
//! interchangeability CHECK, the retry-safety RULE, the admission (through the one breaker), and the
26-
//! refusal it produces. **What a plane owns:** what a CANDIDATE is ([`Candidate`]) and what makes two
27-
//! of them interchangeable (the pin it hands back from
28-
//! [`Candidate::interchange_key`]). [`crate::egress_auth::gate`] is the precedent this copies rather
24+
//! **What core owns:** what a candidate SET is, the LOOP that walks it, the interchangeability
25+
//! CHECK, the retry-safety RULE, the admission (through the one breaker), and the refusal it
26+
//! produces. **What a plane owns:** what a CANDIDATE is ([`Candidate`]), what makes two of them
27+
//! interchangeable (the pin it hands back from [`Candidate::interchange_key`]), the ORDER they are
28+
//! offered in ([`Order`]) and which admission primitive its dispatch needs — and nothing else.
29+
//! [`crate::egress_auth::gate`] is the precedent this copies rather
2930
//! than a new idea: a plane supplies a grant kind and keeps its refusal wording; it does not keep its
3031
//! own decision. [`crate::audit`] is the nearer one still: core owns the mechanism, a stream supplies
3132
//! one record type.
@@ -85,6 +86,36 @@
8586
//! for a plane busbar does not have and shows it selects, admits, trips and reroutes with NO second
8687
//! breaker, NO second walk and NO error type written for it.
8788
89+
// ## ONE SELECTION LOOP, ON ALL THREE PLANES (owner ruling R-I: "Unify now — R-B should be true on
90+
// ## selection too in 1.6.0")
91+
//
92+
// [`walk_with`] is that loop, and it is the only one. R-B was already true of the BREAKER — one FSM,
93+
// one `breaker::classify`, one disposition pipeline, no plane-local state machine — and this is what
94+
// made it true of SELECTION as well. Until it landed the model plane had its OWN admission loop in
95+
// `proxy::select::pick_among`, and this module's own definition of the capability said "the one
96+
// selection loop" while two existed. Both are now the same function.
97+
//
98+
// WHY THIS DIRECTION. The model plane's loop carried SWRR weighting, routing policy, session
99+
// affinity, queueing and `on_exhausted`, and the natural fear was that folding it in would drag all
100+
// of that into core. It did not, because those are not selection:
101+
//
102+
// * WEIGHTING / ROUTING POLICY / AFFINITY are ORDER — they decide who is ASKED first, never who is
103+
// ALLOWED. They are now an [`Order`] implementation on the model plane, and nothing they return
104+
// can admit a candidate the breaker refused, because they perform no admission at all.
105+
// * QUEUEING and `on_exhausted` (503 / `fallback_pool` / `least_bad` / `queue`) are what that plane
106+
// does AFTER the loop finds nothing. `proxy::engine::walk`'s own header already said so: the
107+
// queue wait "lives HERE in on_exhausted dispatch, never inside `pick_among` — selection stays
108+
// non-blocking". They never moved and they were never a second selection.
109+
// * The CONCURRENCY PERMIT is the one real difference, and it is a difference of one field:
110+
// `try_admit` is `try_admit_breaker` plus a permit acquisition, over the same cell, the same
111+
// verdict decoder and the same single-flight probe CAS. It rides in as the plane's ADMISSION.
112+
//
113+
// The opposite direction — making `pick_among` the one loop and moving MCP and A2A onto it — was
114+
// rejected on the code: it is welded to `App`, to `WeightedLane` and to lane indices into
115+
// `app.lanes`, which MCP servers and A2A agents are not members of (they run on `PlaneBreakers`'
116+
// own runtime); and it has neither the pin check nor the repeat-safety rule, so those two safety
117+
// rules would have had to be bolted onto it or lost on the two planes that need them most.
118+
//
88119
// MOUNTED. The reroute-parity unit (owner ruling R-B: "llm mcp a2a are identical") consumes this
89120
// seam on both non-LLM planes: `mcp::reroute` walks a `tool_pools:` candidate set before every
90121
// `tools/call` leg, and `a2a::receive` walks an `agent_pools:` set at submission admission. Both
@@ -312,33 +343,36 @@ impl std::fmt::Display for Refusal {
312343
}
313344
}
314345

315-
/// A CANDIDATE THAT MAY BE DISPATCHED TO, and the probe it now owns.
346+
/// A CANDIDATE THAT MAY BE DISPATCHED TO, and the admission token it now owns.
316347
///
317-
/// Only [`walk`] can build one, so a dispatch that skipped the breaker does not compile. `probe_epoch`
318-
/// is the owner token [`LaneRuntime::try_admit_breaker`] handed back — the caller releases it
319-
/// OWNER-CHECKED via `release_probe_owned_in` after recording an outcome, exactly the discipline the
320-
/// model plane's queue dispatch uses.
321-
pub(crate) struct Admitted<'a, C> {
348+
/// Only [`walk_with`] can build one, so a dispatch that skipped the breaker does not compile. `token`
349+
/// is whatever the plane's admission handed back: on the MCP/A2A planes that is the bare probe epoch
350+
/// from [`LaneRuntime::try_admit_breaker`]; on the model plane it is the [`crate::store::Admit`]
351+
/// carrying the probe epoch AND the concurrency permit that plane also needs. Both are the SAME
352+
/// breaker FSM and the SAME single-flight probe — the token differs only by whether a permit rides
353+
/// along (see `store::in_memory::availability`: `try_admit` is `try_admit_breaker` plus a permit
354+
/// acquisition, nothing else). The caller releases the probe OWNER-CHECKED via
355+
/// `release_probe_owned_in` after recording an outcome.
356+
pub(crate) struct Admitted<'a, C, T = u64> {
322357
candidate: &'a C,
323358
position: usize,
324-
probe_epoch: u64,
359+
token: T,
325360
}
326361

327362
/// HAND-WRITTEN so the bound lands on [`Candidate`] and NOT on `Debug`. A derive would make every
328363
/// plane's candidate type `Debug` or lose `expect`/`unwrap` on the result — which is a second thing a
329364
/// third plane has to write, and the acceptance test for this seam is that there is no second thing.
330365
/// It prints the plane's own [`Candidate::name`], which is all an operator wants from it anyway.
331-
impl<C: Candidate> std::fmt::Debug for Admitted<'_, C> {
366+
impl<C: Candidate, T> std::fmt::Debug for Admitted<'_, C, T> {
332367
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
333368
f.debug_struct("Admitted")
334369
.field("candidate", &self.candidate.name())
335370
.field("position", &self.position)
336-
.field("probe_epoch", &self.probe_epoch)
337371
.finish()
338372
}
339373
}
340374

341-
impl<'a, C: Candidate> Admitted<'a, C> {
375+
impl<'a, C: Candidate, T> Admitted<'a, C, T> {
342376
/// The candidate this request may be sent to.
343377
pub(crate) fn candidate(&self) -> &'a C {
344378
self.candidate
@@ -347,10 +381,18 @@ impl<'a, C: Candidate> Admitted<'a, C> {
347381
pub(crate) fn position(&self) -> usize {
348382
self.position
349383
}
384+
/// The plane's admission token — probe epoch (MCP/A2A) or `Admit` (the model plane's probe epoch
385+
/// plus its concurrency permit). Consumed by value: an admission is used once.
386+
pub(crate) fn into_token(self) -> T {
387+
self.token
388+
}
389+
}
390+
391+
impl<'a, C: Candidate> Admitted<'a, C, u64> {
350392
/// The single-flight probe owner token, for the owner-checked release after the outcome is
351-
/// recorded.
393+
/// recorded. The MCP/A2A spelling of [`Admitted::into_token`].
352394
pub(crate) fn probe_epoch(&self) -> u64 {
353-
self.probe_epoch
395+
self.token
354396
}
355397
}
356398

@@ -377,10 +419,72 @@ pub(crate) struct Attempt<'r> {
377419
pub(crate) operation: &'r str,
378420
}
379421

422+
/// WHERE THE NEXT CANDIDATE COMES FROM — the one thing [`walk_with`] does NOT own.
423+
///
424+
/// The loop is one; the ORDER is a plane's. The MCP and A2A planes read `members:` in the order the
425+
/// operator wrote it ([`InOrder`]). The model plane's order is SMOOTH WEIGHTED ROUND ROBIN, which is
426+
/// STATEFUL — it cannot be handed over as a precomputed list because each pick mutates the SWRR
427+
/// accumulator and a refused pick must re-run the selection over the remaining set. So the order is
428+
/// asked for one candidate at a time, and `refused` reports what the walk did with the previous
429+
/// answer, which is exactly what a generator needs to advance.
430+
///
431+
/// This is the same division [`Candidate`] draws: core owns the mechanism, a plane supplies the one
432+
/// value only it can compute. Weighting, routing policy, session affinity and drain are ORDER — they
433+
/// decide who is asked first, never who is allowed. Nothing an implementation of this trait returns
434+
/// can admit a candidate the breaker refused, because it does not perform admission at all.
435+
pub(crate) trait Order {
436+
/// The next position in `members` to CONSIDER, or `None` when this order is exhausted.
437+
///
438+
/// `refused` is the position the previous call yielded and the walk then failed to admit
439+
/// (`None` on the first call, and on every call after a successful admission — which never
440+
/// happens, because the walk returns).
441+
fn next(&mut self, refused: Option<usize>) -> Option<usize>;
442+
}
443+
444+
/// THE OPERATOR'S ORDER: `members:` as written, skipping anything already `tried`.
445+
///
446+
/// The MCP and A2A order, and the one the seam's own [`walk`] uses. `members[0]` is the primary and
447+
/// the rest are its declared twins, so "first admissible in declaration order" is the whole policy —
448+
/// there is no weighting to apply because two deployments of ONE image are not a load-balancing
449+
/// decision, they are a same-or-nothing choice.
450+
pub(crate) struct InOrder<'a> {
451+
tried: &'a [usize],
452+
cursor: usize,
453+
len: usize,
454+
}
455+
456+
impl<'a> InOrder<'a> {
457+
pub(crate) fn new(tried: &'a [usize], len: usize) -> Self {
458+
Self {
459+
tried,
460+
cursor: 0,
461+
len,
462+
}
463+
}
464+
}
465+
466+
impl Order for InOrder<'_> {
467+
fn next(&mut self, _refused: Option<usize>) -> Option<usize> {
468+
// A refused position is never revisited because the cursor only moves forward — the walk's
469+
// own `refused` bookkeeping is redundant here, which is why it is ignored rather than
470+
// re-checked.
471+
while self.cursor < self.len {
472+
let position = self.cursor;
473+
self.cursor += 1;
474+
if !self.tried.contains(&position) {
475+
return Some(position);
476+
}
477+
}
478+
None
479+
}
480+
}
481+
380482
/// THE WALK: pick the next candidate in `members` that this request may be sent to, and admit it
381483
/// through THE circuit breaker.
382484
///
383-
/// There is exactly ONE selection loop for these planes and this is it.
485+
/// There is exactly ONE selection loop and this is it. Every plane reaches it: MCP and A2A through
486+
/// [`walk`], the model plane through `proxy::select::pick_among`, which supplies an SWRR [`Order`]
487+
/// and the permit-carrying admission and owns NO loop of its own.
384488
///
385489
/// The order of the checks is the contract, because the first failure is what the operator is told:
386490
///
@@ -390,16 +494,33 @@ pub(crate) struct Attempt<'r> {
390494
/// early on purpose: it is a statement about the REQUEST, and no upstream's health can change it.
391495
/// 3. **Do the pins agree?** Every hop after the primary must present the primary's exact
392496
/// `interchange_key`. Checked before admission so a mismatched candidate is never given a probe.
393-
/// 4. **Will the breaker have it?** [`LaneRuntime::try_admit_breaker`] — the same call the model
394-
/// plane's queue dispatch makes, against the same `(pool, lane)` cell — walked in order until one
395-
/// admits.
396-
pub(crate) fn walk<'a, C: Candidate>(
397-
store: &dyn LaneRuntime,
497+
/// 4. **Will the breaker have it?** The plane's `admit` — [`LaneRuntime::try_admit_breaker`] on the
498+
/// MCP/A2A planes, [`crate::store::LaneRuntime::try_admit`] (the same breaker plus a concurrency
499+
/// permit) on the model plane — walked in `order` until one admits.
500+
///
501+
/// `admit` is a hook and not a hardcoded call for ONE reason and it is not extensibility: the model
502+
/// plane must acquire its concurrency permit and win the breaker probe ATOMICALLY (see `try_admit`'s
503+
/// capacity-peek-before-probe-CAS argument — splitting them wedged a tripped-and-saturated lane
504+
/// forever). It is the SAME FSM, the SAME cell and the SAME single-flight probe either way; only the
505+
/// permit rides along. A plane cannot use this hook to admit a candidate the breaker refused,
506+
/// because every implementation of it is one of those two store methods and there are no others.
507+
///
508+
/// `passed_over` is filled with `(position, why)` for EVERY candidate this walk could not admit, in
509+
/// the order it considered them — on the success path as well as the refusal path, because a plane
510+
/// that reroutes still owes its operator the reasons the earlier members were skipped. It is an
511+
/// out-parameter rather than a field of the two results precisely because BOTH results need it: the
512+
/// model plane's `on_exhausted` disposition reads it to decide 503-vs-queue-vs-spill, and it needs
513+
/// the reasons whether the walk ended in a dispatch or in a shed. The walk CLEARS it first, so a
514+
/// caller reusing one buffer across failover hops reports this hop's exhaustion, never a stale one.
515+
pub(crate) fn walk_with<'a, C: Candidate, T>(
398516
pool: &str,
399517
members: &'a [C],
400518
attempt: &Attempt<'_>,
401-
now: u64,
402-
) -> Result<Admitted<'a, C>, Refusal> {
519+
order: &mut dyn Order,
520+
passed_over: &mut Vec<(usize, Unavailable)>,
521+
admit: &mut dyn FnMut(usize, &'a C) -> Result<T, Unavailable>,
522+
) -> Result<Admitted<'a, C, T>, Refusal> {
523+
passed_over.clear();
403524
// 1. Nothing to select from.
404525
let Some(primary) = members.first() else {
405526
return Err(Refusal::Empty {
@@ -419,11 +540,14 @@ pub(crate) fn walk<'a, C: Candidate>(
419540
});
420541
}
421542

422-
let mut refused: Vec<(String, Unavailable)> = Vec::new();
423-
for (position, member) in members.iter().enumerate() {
424-
if attempt.tried.contains(&position) {
425-
continue;
426-
}
543+
let mut last_refused: Option<usize> = None;
544+
while let Some(position) = order.next(last_refused) {
545+
let Some(member) = members.get(position) else {
546+
// An order that names a position outside the pool has no candidate to admit. Treat it as
547+
// the end of that order rather than panicking on a slice index: the walk's job is to
548+
// refuse safely, never to trust an index.
549+
break;
550+
};
427551
// 3. THE PIN CHECK. The primary defines the pool's identity; every other member has to prove
428552
// it is the same deployment. `None` never matches — not even another `None`, because two
429553
// registrations that have each approved nothing are two unknowns, not one fact.
@@ -442,27 +566,66 @@ pub(crate) fn walk<'a, C: Candidate>(
442566
});
443567
}
444568
}
445-
// 4. THE ONE BREAKER. Not a second admission path, not a copy of one: this is
446-
// `LaneRuntime::try_admit_breaker`, the same method `proxy::engine::walk` calls on the
447-
// model plane, against the same per-(pool, lane) cell.
448-
match store.try_admit_breaker(pool, member.lane(), now) {
449-
Ok(probe_epoch) => {
569+
// 4. THE ONE BREAKER. Not a second admission path, not a copy of one: `try_admit_breaker` and
570+
// `try_admit` are the same FSM over the same per-(pool, lane) cell.
571+
match admit(position, member) {
572+
Ok(token) => {
450573
return Ok(Admitted {
451574
candidate: member,
452575
position,
453-
probe_epoch,
576+
token,
454577
})
455578
}
456-
Err(why) => refused.push((member.name().to_string(), why)),
579+
Err(why) => {
580+
passed_over.push((position, why));
581+
last_refused = Some(position);
582+
}
457583
}
458584
}
459585

460586
Err(Refusal::NoneAdmissible {
461587
pool: pool.to_string(),
462-
tried: refused,
588+
tried: passed_over
589+
.iter()
590+
.map(|(position, why)| {
591+
(
592+
members
593+
.get(*position)
594+
.map(|m| m.name().to_string())
595+
.unwrap_or_default(),
596+
*why,
597+
)
598+
})
599+
.collect(),
463600
})
464601
}
465602

603+
/// THE SEAM'S SPELLING OF [`walk_with`]: the operator's `members:` order, admitted breaker-only.
604+
///
605+
/// The MCP and A2A call sites' entry point. It adds NO selection logic — it names the two things
606+
/// those planes supply ([`InOrder`] and [`LaneRuntime::try_admit_breaker`]) and hands them to the one
607+
/// loop.
608+
pub(crate) fn walk<'a, C: Candidate>(
609+
store: &dyn LaneRuntime,
610+
pool: &str,
611+
members: &'a [C],
612+
attempt: &Attempt<'_>,
613+
now: u64,
614+
) -> Result<Admitted<'a, C>, Refusal> {
615+
let mut order = InOrder::new(attempt.tried, members.len());
616+
// These planes render their refusal from `Refusal` itself (which already carries every reason by
617+
// name), so the positional buffer is local and dropped here.
618+
let mut passed_over = Vec::new();
619+
walk_with(
620+
pool,
621+
members,
622+
attempt,
623+
&mut order,
624+
&mut passed_over,
625+
&mut |_position, member| store.try_admit_breaker(pool, member.lane(), now),
626+
)
627+
}
628+
466629
/// RECORD WHAT THE UPSTREAM DID, through the ONE classifier and onto the ONE breaker cell.
467630
///
468631
/// [`crate::breaker::classify`] is protocol-agnostic already — it consumes a `CanonicalSignal` a

crates/busbar-core/src/proxy/engine/walk.rs

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,12 @@
1+
//! ON_EXHAUSTED DISPOSITION — what the model plane does AFTER the one selection loop finds nowhere
2+
//! to send a request. This file is NOT a selection loop and, since the one-loop unification (owner
3+
//! ruling R-I), nothing here selects: `handle_fallback_pool` re-enters `pick_among` — which is the
4+
//! model plane's [`crate::failover::walk_with`] call site — for the spillover pool, `handle_queue`
5+
//! waits for a permit and then re-asks the SAME `try_admit_breaker` every plane asks, and
6+
//! `handle_least_bad` is the ONE documented breaker bypass in the tree (a last-resort degraded route
7+
//! that owns no probe and says so). The candidate ordering, the pin check, the repeat-safety rule
8+
//! and the admission all live in core, identically for llm, mcp and a2a.
9+
110
use super::*;
211
// See `engine::mod`'s identical import for why this is a bare, unqualified import rather than a
312
// `crate::observability::HOTPATH_LEVEL` path spelled out at the instrument site: `level = <path>`

crates/busbar-core/src/proxy/mod.rs

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,10 @@ mod forward_once_pool_cell_tests;
209209
#[path = "tests/ordered_walk_tests.rs"]
210210
mod ordered_walk_tests;
211211

212+
#[cfg(test)]
213+
#[path = "tests/reroute_pool_tests.rs"]
214+
mod reroute_pool_tests;
215+
212216
#[cfg(test)]
213217
#[path = "tests/lane_availability_proptest.rs"]
214218
mod lane_availability_proptest;

0 commit comments

Comments
 (0)