fix(flowcontrol): apply effective defaults for DefaultRequestTTL and per-band MaxRequests

[LukeAVanDrie]

What type of PR is this?

/kind bug

What this PR does / why we need it:

DefaultRequestTTL and per-band MaxRequests had no effective defaults: validate() only rejects negative TTLs, and the band MaxRequests doc comment references a defaultPriorityBandMaxRequests constant that does not exist. A default configuration could queue requests with no time or count bound until the 1 GB per-band byte limit.

Changes:

• defaultRequestTTL defaults to 60s when unset. The API field is a *metav1.Duration, so unset and explicit zero are distinguishable: an explicit "0s" keeps disconnect-only behavior and is documented as such. The 60s value is a queue-wait budget. The well-lit guides configure no gateway request timeout, so this is the only bound on queue wait in those deployments. Where a shorter gateway or client deadline exists, context cancellation evicts the request first. This repo's chart defaults the route timeout to 300s, so with default config the TTL fires first; a request queued for 60s has already lost most of its time-to-first-token budget, and operators who prefer longer waits can raise defaultRequestTTL.
• Per-band maxRequests defaults to 5000 when unset, applied in applyDefaults alongside the existing MaxBytes handling. Band-level 0 is treated as unset, matching how maxBytes: 0 is handled today. The two defaults are consistent: under a dispatch halt, occupancy self-limits at arrival rate times TTL, so the count cap only binds above roughly 83 req/s per band.
• Global limits stay unlimited by default. Band count is bounded by the distinct priorities defined in InferenceObjectives, so per-band caps bound total memory exposure.
• Doc fixes: the FlowGCTimeout default (5 minutes; the comment claimed 1 hour), the DefaultNegativePriorityBand comment claiming maxBytes: "0" drops traffic immediately (band-level 0 has never meant zero capacity), explicit zero-semantics notes on all four limit fields, and the InitialEffectiveTTL adapter docstring with a TODO for per-request and per-band TTL scopes ([Flow Control] Allow setting different default maxBytes configs for priorities >=0 and for < 0 #1090).

Tests: defaults asserted at the controller, registry (including the auto-provisioned priority-0 band and both dynamic templates), and loader layers, with nil vs explicit-zero round-trips for both fields. The flowcontrol tree passes under -race.

Design discussion and the recorded 60s rationale are in the plan comment on #2095.

Which issue(s) this PR fixes:

Fixes #2095

Part of #1187.

Release note:

Flow control queueing is now bounded by default: `defaultRequestTTL` defaults to 60s and per-band `maxRequests` defaults to 5000 when unset. For existing flow control users this is a behavior change: an explicit `defaultRequestTTL: "0s"` disables the TTL (requests then wait until client disconnect), and a band `maxRequests`/`maxBytes` of 0 is treated as unset and receives the default. To approximate the legacy fast-fail behavior for sheddable traffic, configure a small `defaultNegativePriorityBand.maxRequests`. Scale-to-zero deployments should raise `defaultRequestTTL` above the expected cold-start time.

[LukeAVanDrie]

There is one consequence of the 60s TLL default worth deciding. The queue also serves as the scale-from-zero waiting room (with an empty pool, requests queue instead of failing). The default TTL cuts that room to 60s. Cold starts may take O(minutes), and the chart's route timeout would have carried them, so a scale-from-zero deployment on pure defaults now sheds everything during cold start (after 60s).

Options:

1. Ship 60s and document it (this PR). The shed is visible (503 with a TTLExpired reason header) and the fix is one config line; sizing guidance goes in [Flow Control] [Graduation Blocker] Enable the flowControl feature gate by default #2104.
2. Default to ~300s to match the chart. Under sustained saturation a FIFO queue with a TTL converges to a standing queue at the TTL (every served request also waits that long), so 300s means holding five minutes of doomed work, and the TTL stops being a useful queue-wait budget.
3. Pause or extend the TTL while the pool is empty, so it measures "the pool is up and you still couldn't dispatch."

[3] is probably the right end state, but it's new mechanism.

This is the biggest user-facing decision for enablement by default, so we want to make sure we get it right.

[LukeAVanDrie]

cc: @ahg-g, @RishabhSaini

[ahg-g]

Requests can come with a timeout in their context, right? if so, what happens if this is lower than the timeout set by the client?

[LukeAVanDrie]

The two deadlines compose with minimum-wins semantics.

We derive the queue context via context.WithDeadlineCause(ctx, enqueueTime.Add(TTL), ErrTTLExpired), where ctx is the inbound request context.

• On client deadline sooner than the TTL, the parent's deadline wins and fires first. Crucially, the ErrTTLExpired cause only attaches if the TTL's own deadline is the one that expires; when the parent fires, the cause is the parent's cancellation. So it flows down the QueueOutcomeEvictedContextCancelled path and is reported as client disconnect/timeout, not as a TTL shed. Attribution stays honest in metrics and status mapping.
• On client deadline later (or absent), the 60s TTL fires first with the ErrTTLExpired cause → EvictedTTL → the TTLExpired dropped-reason response. The client's larger budget is deliberately cut at 60s of queue wait. The remaining budget is preserved for generation. I.e., TTL is "queue-wait budget" not "client-deadline proxy".

In short, if the client sets a higher timeout, we still enforce maximum of 60s queue-wait budget by default.

[LukeAVanDrie]

Long-term I would like to be able to express TTL at the per-priority (InfObj) and request level, but that is not in scope for this change as it would be a new transport mechanism.