Skip to content

Commit 3d32405

Browse files
committed
refactor(flowcontrol): scrub shard vocabulary and rewrite docs to single-processor model
Post-llm-d#889 the flow control layer is single-processor, but the package/type docs and many comments/log strings/test names still described the old sharded supervisor-worker-pool design (and referenced deleted types like RegistryShard/ShardProcessor). Rename the vestigial 'shard' vocabulary to match reality (sp->p receiver, log/error strings, comments, test names), rewrite the doc.go files and stale type docs to the single-processor model, make doc.go the single source of truth for the controller package architecture, correct the wrong distributeRequest and managedQueue.Add docstrings, and drop the one-use flowComponents wrapper. No behavior change. Refs: llm-d#889 Signed-off-by: Luke Van Drie <lukevandrie@google.com>
1 parent 8bef5f0 commit 3d32405

15 files changed

Lines changed: 213 additions & 229 deletions

File tree

pkg/epp/flowcontrol/controller/controller.go

Lines changed: 12 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,6 @@ See the License for the specific language governing permissions and
1414
limitations under the License.
1515
*/
1616

17-
// Package controller contains the implementation of the FlowController engine.
18-
//
19-
// The FlowController is the central processing engine of the Flow Control layer. It is a high-throughput
20-
// component responsible for managing the lifecycle of all incoming requests. It achieves this by acting as a stateless
21-
// supervisor that orchestrates a stateful worker (Processor).
2217
package controller
2318

2419
import (
@@ -46,7 +41,7 @@ type registryClient interface {
4641
contracts.FlowRegistryDataPlane
4742
}
4843

49-
// processor is the minimal internal interface that the FlowController requires from its workers.
44+
// processor is the minimal internal interface that the FlowController requires from its worker.
5045
type processor interface {
5146
Run(ctx context.Context)
5247
Submit(item *internal.FlowItem) error
@@ -70,8 +65,8 @@ type processorFactory func(
7065
var _ processor = &internal.Processor{}
7166

7267
// FlowController is the central, high-throughput engine of the Flow Control layer.
73-
// It is designed as a stateless distributor that orchestrates a stateful worker (Processor), following a
74-
// supervisor-worker pattern.
68+
// It is the request-facing front end that hands each request to a single stateful worker (the Processor) and blocks
69+
// until the request reaches a terminal outcome.
7570
//
7671
// Request Lifecycle Management:
7772
//
@@ -332,7 +327,7 @@ func (fc *FlowController) withConnectionWithFallback(
332327
})
333328
}
334329

335-
// tryDistribution handles a single attempt to select a shard and submit a request.
330+
// tryDistribution handles a single attempt to submit a request to the processor.
336331
// It uses the provided `conn` to access the registry data plane.
337332
// If this function returns an error, it guarantees that the provided `item` has been finalized.
338333
func (fc *FlowController) tryDistribution(
@@ -424,21 +419,19 @@ func (fc *FlowController) createRequestContext(
424419
return reqCtx, cancel, enqueueTime
425420
}
426421

427-
// distributeRequest implements a flow-aware, two-phase "Join-Shortest-Queue-by-Bytes" (JSQ-Bytes) distribution strategy
428-
// with graceful backpressure. It attempts to submit an item to the best-ranked candidate from the provided list.
422+
// distributeRequest submits an item to the processor with graceful backpressure.
429423
//
430-
// The algorithm operates as follows:
431-
// 1. Phase 1 (Non-blocking Fast Failover): It iterates through the ranked candidates and attempts a non-blocking
432-
// submission. The first successful submission wins.
433-
// 2. Phase 2 (Blocking Fallback): If all non-blocking attempts fail, it performs a single blocking submission to the
434-
// least-loaded candidate, providing backpressure.
424+
// It operates in two phases:
425+
// 1. Non-blocking submit: a fast Submit that succeeds immediately if the processor's enqueue channel has capacity.
426+
// 2. Blocking fallback: if the processor is busy, a single blocking SubmitOrBlock that applies backpressure until the
427+
// item is accepted, the context expires, or the processor shuts down.
435428
//
436429
// The provided context (ctx) is used for the blocking submission phase (SubmitOrBlock).
437430
//
438431
// Ownership Contract:
439-
// - Returns nil: Success. Ownership transferred to Processor.
440-
// - Returns error: Failure (Context expiry, shutdown,, etc.).
441-
// Ownership retained by Controller. The Controller MUST finalize the item.
432+
// - Returns nil: Success. Ownership transferred to the Processor.
433+
// - Returns error: Failure (context expiry, shutdown, etc.).
434+
// Ownership retained by the Controller, which MUST finalize the item.
442435
func (fc *FlowController) distributeRequest(
443436
ctx context.Context,
444437
item *internal.FlowItem,

pkg/epp/flowcontrol/controller/doc.go

Lines changed: 16 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -18,30 +18,29 @@ limitations under the License.
1818
//
1919
// # Overview
2020
//
21-
// The FlowController is the central processing engine of the Flow Control layer. It acts as a stateless supervisor that
22-
// orchestrates a pool of stateful workers (internal.ShardProcessor), managing the lifecycle of all incoming requests
23-
// from initial submission to a terminal outcome (dispatch, rejection, or eviction).
21+
// The FlowController is the central processing engine of the Flow Control layer. It manages the lifecycle of all
22+
// incoming requests from initial submission to a terminal outcome (dispatch, rejection, or eviction).
2423
//
25-
// # Architecture: Supervisor-Worker Pattern
24+
// # Architecture
2625
//
27-
// This package implements a supervisor-worker pattern to achieve high throughput and dynamic scalability.
26+
// The engine is split into two cooperating roles:
2827
//
29-
// - The FlowController (Supervisor): The public-facing API of the system. Its primary responsibilities are to execute
30-
// a distribution algorithm to select the optimal worker for a new request and to manage the lifecycle of the worker
31-
// pool, ensuring it stays synchronized with the underlying shard topology defined by the contracts.FlowRegistry.
32-
// - The internal.ShardProcessor (Worker): A stateful, single-goroutine actor responsible for the entire lifecycle of
33-
// requests on a single shard. The supervisor manages a pool of these workers, one for each contracts.RegistryShard.
28+
// - The FlowController: The public-facing API of the system. Each call to EnqueueAndWait runs on its own (caller's)
29+
// goroutine: it acquires a flow lease from the contracts.FlowRegistry, hands the request to the Processor, and
30+
// blocks until the request is finalized. It owns asynchronous finalization driven by the request Context
31+
// (TTL/cancellation) and queue occupancy metrics.
32+
// - The internal.Processor (Worker): A single, stateful, single-goroutine actor that owns the request data plane. It
33+
// runs the dispatch loop, performs capacity checks, sweeps externally finalized items, and finalizes requests
34+
// synchronously (dispatch, capacity rejection, shutdown).
3435
//
3536
// # Concurrency Model
3637
//
37-
// The FlowController is designed to be highly concurrent and thread-safe. It acts primarily as a stateless distributor.
38+
// The FlowController is designed to be highly concurrent and thread-safe. This rests on two properties:
3839
//
39-
// - EnqueueAndWait: Can be called concurrently by many goroutines.
40-
// - Worker Management: Uses a sync.Map (workers) for concurrent access and lazy initialization of workers.
41-
// - Supervision: A single background goroutine (run) manages the worker pool lifecycle (garbage collection).
42-
//
43-
// It achieves high throughput by minimizing shared state and relying on the internal ShardProcessors to handle state
44-
// mutations serially (using an actor model).
40+
// - EnqueueAndWait: Can be called concurrently by many goroutines, each handing its item to the single Processor over
41+
// a buffered channel.
42+
// - Single-Writer Actor: Routing all state mutations through the Processor's single Run goroutine makes complex
43+
// transactions (such as capacity checks) inherently atomic without coarse-grained locks.
4544
//
4645
// # Request Lifecycle and Ownership
4746
//

pkg/epp/flowcontrol/controller/internal/doc.go

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -22,15 +22,15 @@ limitations under the License.
2222
// # Design Philosophy: The Single-Writer Actor Model
2323
//
2424
// The concurrency model for this package is built around a single-writer, channel-based actor pattern, as implemented
25-
// in the ShardProcessor. All state-mutating operations for a given shard (primarily enqueuing new requests) are
26-
// funneled through a single Run goroutine.
25+
// in the Processor. All state-mutating operations (primarily enqueuing new requests) are funneled through a single Run
26+
// goroutine.
2727
//
28-
// This design makes complex, multi-step transactions (like a hierarchical capacity check against both a shard's total
28+
// This design makes complex, multi-step transactions (like a hierarchical capacity check against both the global total
2929
// limit and a priority band's limit) inherently atomic without locks. This avoids the performance bottleneck of a
30-
// coarse, shard-wide lock and allows the top-level Controller to remain decoupled and highly concurrent.
30+
// coarse, global lock and allows the top-level Controller to remain decoupled and highly concurrent.
3131
//
3232
// # Key Components
3333
//
34-
// - ShardProcessor: The implementation of the worker actor. Manages the lifecycle of requests for a single shard.
34+
// - Processor: The implementation of the worker actor. Manages the lifecycle of all queued requests.
3535
// - FlowItem: The internal representation of a request, managing its state and synchronization across goroutines.
3636
package internal

0 commit comments

Comments
 (0)