List view

• Multimodal Request Support

  **Target** next release (v0.9.0). **Description:** Extend latency prediction to multimodal requests (image and video inputs), where prefill cost is driven by more than text token count. Today the predictor's features assume text-only requests (`input_token_length`, `num_request_running`, `prefix_cache_score`, etc.). Image and video inputs add prefill work that text token count alone doesn't capture, so TTFT prediction will degrade on multimodal traffic without new features. Scope (workstreams / candidate features): - **Image inputs** — number of images per request, image dimensions / pixel count, and the resulting vision-token count after the processor expands them - **Video inputs** — frame count, resolution, and clip duration / sampling rate - **Feature engineering** — derived signals analogous to the text path (e.g. total multimodal tokens, per-modality density) so the model can learn non-linear prefill cost - **Pipeline wiring** — thread the new fields through request schema, feature prep, training, prediction, and metrics in both servers (the same surfaces touched by the `prefill_density` change) - **Validation** — actual-vs-predicted TTFT on real multimodal benchmark traffic, with and without the new features Each feature should land with the testing bar we're standardizing: per-feature actual-vs-predicted evidence on a named workload, and regression coverage once the CI epic lands.

  Due by August 1, 2026
  0% complete0 open0 closed

• CI/CD Testing Automation

  **Due date:** _(target: next release, v0.9.0 — set date when scheduled)_ **Description:** Target: next release (v0.9.0). Stand up real CI gates beyond lint + dry container build. Today a PR only gates on `ruff` and a no-push Dockerfile build — no tests run. This milestone tracks the epic to fix that: - **#11** — source directory restructuring (prediction/, training/, common/, test/ subpackages) — prerequisite for the rest - **#12** — Epic: automated functional and regression tests in CI - **#13** — reusable build-image workflow (push PR-scoped images so test jobs have something to pull) - **#14** — functional test enablement in CI (start containers, run fast test subset, block merge on failure) - **#15** — gated regression tests (quantile coverage, monotonicity, e2e) Dependency order: #11 → #13 → #14 → #15.

  Due by August 1, 2026
  0% complete0 open0 closed