[Tracking] v0.9.0 Roadmap

[shimib]

Tracking issue for the v0.9.0 release. Milestone: https://github.com/llm-d/llm-d-async/milestone/1

Supersedes the stale roadmap in #1.

🎯 Epic: Pipeline 2.0 — Per-Pool Topology & Unified Gating

The backbone of the release. Foundation is complete — the remaining items are leaf gating features.

#205 Per-pool Worker Topology         (foundation: Pool struct, PoolDispatch map)
 ├─ #221 Model Override               (per-Pool model_name_override)
 ├─ #206 Unified Gating Interface     (fail-fast / retry / continue + Release callbacks)
 │   ├─ #207 Formalized Message Labels & Propagation
 │   └─ #208 Pool-Level Admission Control   (chain of gates per Pool)
 │       └─ #209 Tier-priority merging & advanced capacity gates
 ├─ #141 DispatchGates -> inference targets  (shared budget across queues -> same EP)

• [Feature]: Per-pool Worker Topology #205 Per-pool Worker Topology — foundation
• [Feature]: Unified Gating Interface #206 Unified Gating Interface — (pairs with [Feature]: Formalize Extension Points #216)
• [Feature]: Formalized Message Labels & Propagation #207 Formalized Message Labels & Propagation
• [Feature]: Pool-Level Admission Control #208 Pool-Level Admission Control
• [Feature]: Tier-priority merging and advanced capacity gates #209 Tier-priority merging & advanced capacity gates
• [Feature]: DispatchGates should be assigned to inference targets #141 DispatchGates assigned to inference targets — depends on [Feature]: Unified Gating Interface #206 (low-priority; may be doc-only — document the 1:1 queue↔gate limitation)
• [Feature]: Model Override #221 Model Override — depends on [Feature]: Per-pool Worker Topology #205

🔭 Observability

• [Feature]: Metrics #217 Metrics — core suite delivered (backlog feat: add queue depth, in-flight, and broker backlog metrics #244, residence time feat: add queue residence time metric #254, dispatch budget + pool worker limit feat(metrics): add dispatch budget and pool worker-limit gauges (#217) #290, gate decisions feat(metrics): add gate-decision breakdown counter (#217) #291, saturation value + threshold feat(metrics): expose gate metric value + threshold gauges (#217) #339). Closed; two nice-to-haves split to backlog: token throughput [Metrics] Token Throughput counter (input + output tokens processed) #344, deadline proximity [Metrics] Deadline Proximity histogram (time-to-deadline of queued items) #345.
• [Feature]: readiness probe for pub/sub #246 Readiness probe for Pub/Sub

🧩 Extensibility & Packaging

• [Feature]: Formalize Extension Points #216 Formalize extension points (Gates / Merge Policies as plugins)
• [Enhancement]: Split lightweight client packages into Go sub-modules (api + producer) for dependency isolation #107 Split lightweight client packages into Go sub-modules (api + producer)

🧹 Housekeeping

• [Feature]: Change container name and chart name to llm-d-async #179 Rename container & chart async-processor → llm-d-async — feat!: rename Helm chart async-processor → llm-d-async (#179) #340 (also renamed the binary + OTEL service name). Guides follow-up: docs(guides): point async guide at the renamed chart (charts/llm-d-async) llm-d#2099 (draft, gated on the next release publishing charts/llm-d-async).
• Add PubSub queue impl to e2e tests #83 Add Pub/Sub impl to e2e tests (emulator) — complements [Feature]: readiness probe for pub/sub #246
• Publish Helm Chart Script Improvement #87 Helm publish script: printf over echo for token piping
• Typos detected by nightly scan (1 found) #202 Typo nightly-scan fix

🛡️ Reliability, correctness & CI (landed this cycle)

• Gate capacity reservation leaks on retry → inFlight ratchets up → queue wedges permanently #311 Gate reservation leak on retry → queue wedges permanently — fix: release gate reservation on retry to prevent queue wedge (#311) #334 (+ regression test)
• Concurrency-mode redis-quota over-admits when the counter TTL expires mid-flight #335 redis-quota over-admits on counter-TTL expiry — fix: refresh redis-quota concurrency TTL to prevent over-admission (#335) #336 (minimal refresh-on-activity; a per-holder-lease redesign remains a follow-up)
• [Feature]: Add benchmark for queue clearing and saturation #309 Benchmark for queue clearing & saturation — Add E2E reddis sorted set queue clearing benchmark #321
• [Feature]: Produce nightly AP image #315 Produce main/nightly AP image (:main + chart v0) — Refactor CI container image and chart to use main and v0 tags #330
• CI stability: flaky TestMultipleTenantsIsolation — test(producer): fix flaky TestMultipleTenantsIsolation #337; pin Helm in pre-commit — ci: pin Helm in pre-commit to a stable 3.x (fix flaky unit-test step) #338

Remaining for v0.9.0

The epic foundation, observability probe, extensibility/packaging, and housekeeping (rename) are done, and the reliability/correctness bugs are fixed. Open:

• Gating features: [Feature]: Model Override #221 (Model Override), [Feature]: DispatchGates should be assigned to inference targets #141 (inference-target budget — possibly doc-only).
• Test infra: Add PubSub queue impl to e2e tests #83 (Pub/Sub e2e via emulator).

Suggested sequencing

1. Build the epic foundation ([Feature]: Per-pool Worker Topology #205 → [Feature]: Unified Gating Interface #206/[Feature]: Formalized Message Labels & Propagation #207, [Feature]: Formalize Extension Points #216) ✅ done.
2. Remaining gating features: [Feature]: Model Override #221, [Feature]: DispatchGates should be assigned to inference targets #141.
3. Observability: Add PubSub queue impl to e2e tests #83 (with the emulator).

🗃️ Backlog (deferred out of v0.9.0)

• [Feature]: Support per-queue dispatch gates for ephemeral Redis (redis-pubsub) #76 Per-queue dispatch gates for ephemeral Redis (redis-pubsub) — parity with sortedset; moved out of this milestone.
• [Metrics] Token Throughput counter (input + output tokens processed) #344 Token Throughput counter — metrics nice-to-have split from [Feature]: Metrics #217.
• [Metrics] Deadline Proximity histogram (time-to-deadline of queued items) #345 Deadline Proximity histogram — metrics nice-to-have split from [Feature]: Metrics #217 (redis-sortedset only).

Note: Safety/reliability fixes #131 (message loss on crash) and #63 (producer metadata attack) are intentionally not part of this milestone and are tracked separately.

[shimib]

Status (2026-07-23)

Refreshed the checklist. The Pipeline 2.0 epic foundation is complete (#205–#209, #216), the pub/sub readiness probe (#246) and client-submodule split (#107) are in, and the chart/container rename (#179) landed via #340. All milestone reliability/correctness bugs are fixed: reservation-leak wedge (#311 → #334), quota-TTL over-admission (#335 → #336), plus benchmark (#309), main/nightly image (#315), and CI-stability fixes (#337, #338).

Remaining for v0.9.0 (5):

• Gating features — [Feature]: Model Override #221 (Model Override), [Feature]: Support per-queue dispatch gates for ephemeral Redis (redis-pubsub) #76 (redis-pubsub per-queue gates), [Feature]: DispatchGates should be assigned to inference targets #141 (inference-target budget; possibly doc-only per the earlier note).
• Test infra — Add PubSub queue impl to e2e tests #83 (Pub/Sub e2e via emulator).
• [Feature]: Metrics #217 metric fast-follows — deadline proximity (histogram) + token throughput (counter); the saturation-value gauge landed in feat(metrics): expose gate metric value + threshold gauges (#217) #339.

Follow-up (not blocking this milestone): the guides chart-path sweep (llm-d/llm-d#2099, draft) flips once the next release publishes charts/llm-d-async; and a per-holder-lease redesign for the quota gate (sibling of #335).