Skip to content

feat: release and resume inference runtime - #56

Open
0z5a wants to merge 5 commits into
ViperEkura:mainfrom
0z5a:codex/inference-release-resume
Open

feat: release and resume inference runtime#56
0z5a wants to merge 5 commits into
ViperEkura:mainfrom
0z5a:codex/inference-release-resume

Conversation

@0z5a

@0z5a 0z5a commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a reversible lifecycle for scheduler-owned inference runtime state
  • release KV pages, task caches, executor workspaces, and graph state while retaining model weights and policy version
  • reconstruct through resume() and restart an async loop only when it was previously running
  • expose lifecycle through InferenceEngine, RolloutGenerator, and RolloutRunner, including rollout tensor-cache cleanup
  • reject release for caller-owned caches instead of silently discarding external state

NVIDIA L20 benchmark

AstrAI 1B, BF16, batch 4, prompt 128, four greedy decode tokens, five cycles per cell:

Max context Runtime footprint Reclaimed Reclaimed % Release median Resume median Greedy parity
2,048 200.97 MiB 192.85 MiB 95.96% 105.62 ms 2.01 ms 5/5
8,192 777.14 MiB 769.01 MiB 98.95% 90.16 ms 5.09 ms 5/5
32,768 3,081.81 MiB 3,073.68 MiB 99.74% 89.27 ms 5.10 ms 5/5

Implementation commit: 5900c786322b522162af0bef0674464806f1628a.

A separate three-L20 stability soak ran 100 cycles per context with the tiny deterministic preset:

Max context Cycles Reclaimed Release median / p99 Resume median / p99 Greedy parity
2,048 100 0.572 MiB 72.201 / 88.869 ms 0.410 / 0.553 ms 100/100
8,192 100 2.239 MiB 74.530 / 100.495 ms 0.581 / 6.293 ms 100/100
32,768 100 8.907 MiB 68.257 / 93.089 ms 0.441 / 0.557 ms 100/100

All 300/300 post-resume greedy outputs matched. The tiny-preset footprint is stability evidence and is not directly comparable to the AstrAI-1B memory table.

Validation

  • local: Ruff format/import checks passed; 628 passed, 103 skipped
  • NVIDIA L20 focused scheduler/engine/rollout suite: 71 passed
  • 100-cycle result and five-cycle raw per-sample JSON are checked in under docs/benchmarks/
  • InfraSWE 811bc775: comparison cell valid; 53 Draft/system-path tests passed
  • diagnostic ProjectFit 92.29/100, BenchmarkTrust 97.40/100

InfraSWE scoring is diagnostic and unsealed. CUDA graphs were disabled to isolate scheduler-owned state. The source-mounted runtime used established extension fallbacks, so this PR makes a lifecycle/allocation claim rather than a kernel-throughput claim.

0z5a added 5 commits September 2, 2026 21:42
Add a reversible scheduler lifecycle that drops scheduler-owned KV storage, decode workspace, and CUDA graph state while retaining shared model weights and policy version. Expose the lifecycle through inference and rollout APIs, clear cached rollout tensors, and add single-GPU memory/parity benchmarking.
Capture five release/resume cycles for the AstrAI 1B BF16 preset at 2K, 8K, and 32K context bounds, including raw memory, latency, and greedy-parity data from the implementation commit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant