Skip to content

Commit 2c64d31

Browse files
katopzkatopz
committed
release: promote develop to main (katgpt-core v0.2.0)
2 parents 1727e57 + 1a2d11b commit 2c64d31

20 files changed

Lines changed: 4092 additions & 7058 deletions

‎.benchmarks/334_sudoku_speculate_perf.md‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
# Bench 334: Sudoku Speculative-Solve Perf — Arto Inkala (Hardest)
2+
3+
Pure perf characterization (no GOAT gate — this is a perf-truth bench, not a
4+
primitive-promotion gate). Answers: **how fast can we solve the hardest Sudoku
5+
the speculate way, and can it beat backtracking?**
6+
7+
## Setup
8+
9+
- **Puzzle**: Arto Inkala "World's Hardest" — 21 clues, 60 empty cells.
10+
- **Bench**: `benches/sudoku_speculate_bench.rs` (`harness = false`, `std::time::Instant`,
11+
median-of-31 × batch-of-16, 3 warmup — matches `cucg_bench.rs` convention).
12+
- **Drafter**: uniform marginals over digits 1–9 (worst case, zero signal — the
13+
pruner supplies all constraint information).
14+
- **Pruner**: path-aware `SudokuPruner` (100% valid branches, cross-depth conflict
15+
checking).
16+
- **Run**: `cargo bench --bench sudoku_speculate_bench --features sudoku`
17+
18+
## Three Modes
19+
20+
| Mode | What it measures |
21+
|------|------------------|
22+
| 1. `backtrack` | Canonical `Sudoku9x9::solve()` — ground-truth complete solver (naive: try 1-9 in order, no MRV). |
23+
| 4. `solve_fast` | `Sudoku9x9::solve_fast()` — MRV cell selection + bitmask candidates + naked-singles constraint propagation. **The faster way.** |
24+
| 2. `speculate_iterative` | Iterative DDTree + greedy path commit + backtrack fallback. The realistic "speculative decoding" pattern. |
25+
| 3. `build_one_tree` | Raw DDTree primitive throughput — nodes/µs for one 8-deep build. |
26+
27+
## Results (macOS, release, 2026-06-27)
28+
29+
### Mode 1 — backtrack baseline
30+
31+
| Metric | Value |
32+
|--------|-------|
33+
| solved | ✅ true |
34+
| steps | 49,559 |
35+
| median time | **2.417 ms/solve** |
36+
| per-step | 0.05 µs |
37+
38+
Matches the docs baseline (49,559 steps) — bench is correct.
39+
40+
### Mode 4 — solve_fast (MRV + constraint propagation) ← THE FASTER WAY
41+
42+
| Metric | Value |
43+
|--------|-------|
44+
| solved | ✅ true |
45+
| steps | **1,851** (vs backtrack 49,559 → **27× fewer**) |
46+
| median time | **367 µs/solve** |
47+
| per-step | 0.20 µs |
48+
| speedup | **6.6× faster** wall time |
49+
50+
Pure modelless: MRV cell ordering + bitmask candidate tracking + naked-singles
51+
constraint propagation. No training, no gradient descent — just deterministic
52+
rules. This is the honest answer to "is there a faster way": **yes, and it was
53+
hiding in plain sight** — the original `solve()` was a deliberately naive
54+
proof-of-concept for the streaming/hull-attention demo, not a fast solver.
55+
56+
### Mode 2 — speculate_iterative (DDTree + greedy commit + fallback)
57+
58+
| lookahead | budget | solved | spec_commits | fallback_steps | tree_nodes | time |
59+
|-----------|--------|--------|--------------|----------------|------------|------|
60+
| 4 | 32 | ❌ false | 13 | 3 | 85 | 11.02 µs |
61+
| 8 | 64 | ❌ false | 14 | 18 | 145 | 18.33 µs |
62+
| 8 | 128 | ❌ false | 12 | 237 | 224 | 31.40 µs |
63+
| 16→8 | 256 | ❌ false | 7 | 4 | 259 | 28.05 µs |
64+
65+
Every config falls back to backtracking (solved=false means the speculation
66+
hit a dead-end and the fallback ran — but the bench reports pre-fallback
67+
spec_commits and the fallback step count). The speculate phase itself is
68+
microseconds-fast, but it never solves Inkala: uniform marginals have no
69+
signal, so greedy commits paint into corners within ~7–14 cells, then revert
70+
+ backtrack.
71+
72+
### Mode 3 — DDTree primitive throughput (lookahead=8)
73+
74+
| budget | nodes_built | time | nodes/µs |
75+
|--------|-------------|------|----------|
76+
| 64 | 64 | 6.97 µs | 9.2 |
77+
| 256 | 256 | 25.24 µs | 10.1 |
78+
| 1,024 | 1,024 | 106.81 µs | 9.6 |
79+
| 4,096 | 2,678 | 262.75 µs | 10.2 |
80+
| 16,384 | 2,678 | 270.12 µs | 9.9 |
81+
82+
Steady-state ~10 nodes/µs (10 M nodes/sec). Tree saturates at 2,678 nodes —
83+
that's the full 8-deep pruned search space for Inkala's first 8 empties
84+
(9⁸ raw = 43M, pruned to 2,678 = 16,000× reduction by the path-aware pruner).
85+
86+
## Key Finding — Architectural Ceiling
87+
88+
**`TreeNode.parent_path: u128` packs 16-bit tokens → hard max lookahead = 8
89+
(128/16).** The DDTree speculate primitive is a **token-level speculative-
90+
decoding kernel**, NOT a full-puzzle search. A 60-empty Sudoku **cannot be
91+
solved in one tree** — it physically cannot fit in the u128 path encoding.
92+
93+
The coupled limit: `TreeBuilder::parent_tokens_buf` is sized to
94+
`config.draft_lookahead + 1` (= 9 by default). Exploring depth ≥ 9 panics with
95+
`range end index 10 out of range for slice of length 9`. This is why
96+
`Config::draft()` ships with `draft_lookahead: 8`.
97+
98+
## Verdict — Can speculate beat backtrack on hardest Sudoku?
99+
100+
**No, not with the current infrastructure + uniform drafter.** Two reasons:
101+
102+
1. **No signal**: With uniform marginals, the drafter contributes zero
103+
information — every digit it proposes is already constraint-valid via the
104+
pruner. Speculation at best matches backtrack; at worst it pays tree-build
105+
overhead (~10–30 µs/round) before falling back.
106+
107+
2. **8-deep ceiling**: Even with a perfect drafter, the u128 layout caps
108+
lookahead at 8. Solving 60 cells requires ≥8 speculate rounds, each
109+
paying the primitive cost, with dead-end reverts between them.
110+
111+
**Break-even**: speculate wins only when
112+
`acceptance_rate × commits_per_round × per_commit_savings > tree_build_overhead`.
113+
With `p_accept = 1/9` (uniform) on Inkala, the LHS ≈ 1 × 8 × 0.05µs = 0.4µs
114+
<< RHS ≈ 10–30µs. Never holds.
115+
116+
## What Would Make Speculate Win
117+
118+
Per the modelless-first mandate (AGENTS.md), before deferring to riir-train:
119+
120+
1. **MRV cell ordering** (modelless): reorder empties by minimum-remaining-
121+
values so the drafter proposes forced moves first. This is a deterministic
122+
drafter improvement, no training needed. Could push `p_accept` from 1/9
123+
toward 1/1 on forced cells.
124+
2. **Constraint propagation as the drafter** (modelless): use naked/hidden
125+
singles as the draft signal — any cell with 1 valid digit is committed
126+
without speculation. Pure deterministic rules engine.
127+
3. **Trained digit priors** (→ riir-train, deferred): a real draft model that
128+
proposes the RIGHT digit, not just a valid one. Out of scope for modelless.
129+
130+
Options 1 and 2 are modelless and should be tried first per §3.5 of the
131+
research skill. Filing as an optimization candidate (see `issues/`).
132+
133+
## Files
134+
135+
- `benches/sudoku_speculate_bench.rs` — the bench (380 LOC).
136+
- `Cargo.toml` — `[[bench]]` entry (required-features = `["sudoku"]`, harness = false).
137+
138+
## TL;DR
139+
140+
Hardest Sudoku (Inkala) solves in **2.417 ms** via naive backtrack (49,559
141+
steps), or **367 µs** via `solve_fast` (1,851 steps, **6.6× faster**) — a
142+
pure modelless MRV + constraint-propagation solver.
143+
144+
The speculate way **cannot beat either** with the current DDTree infra:
145+
(a) uniform marginals give the drafter zero signal, and (b)
146+
`TreeNode.parent_path: u128` hard-caps lookahead at 8, so a 60-cell puzzle can
147+
never be solved in one tree. The DDTree primitive runs at ~10 M nodes/sec —
148+
fast for token-level speculative decoding, but the wrong tool for full-puzzle
149+
search.
150+
151+
**Lesson**: before asking "can speculate beat backtrack", check whether the
152+
baseline solver itself is optimal. The 27× step reduction from MRV+CP dwarfs
153+
anything speculation could deliver on top of a naive backtracker.

‎.github/workflows/release-plz.yml‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
name: Release-plz
2+
3+
# Two trigger modes:
4+
# 1. push to main/develop → automatic (normal git-flow cadence)
5+
# 2. workflow_dispatch → manual via `./scripts/release.sh` or the Actions tab
6+
on:
7+
push:
8+
branches:
9+
- main
10+
- develop
11+
workflow_dispatch:
12+
inputs:
13+
command:
14+
description: "release-plz command to run"
15+
required: true
16+
default: "release-pr"
17+
type: choice
18+
options:
19+
- release-pr
20+
- release
21+
22+
jobs:
23+
# ──────────────────────────────────────────────────────────────────────
24+
# Publish + tag + GitHub Release. Fires on:
25+
# - push to main (normal release cadence)
26+
# - manual `release` dispatch
27+
# ──────────────────────────────────────────────────────────────────────
28+
release-plz-release:
29+
name: Release-plz release (publish)
30+
if: >-
31+
(github.event_name == 'push' && github.ref == 'refs/heads/main') ||
32+
(github.event_name == 'workflow_dispatch' && inputs.command == 'release')
33+
runs-on: ubuntu-latest
34+
permissions:
35+
contents: write # push tags, create GitHub releases
36+
pull-requests: read # detect the release PR and its commits
37+
steps:
38+
- name: Checkout repository
39+
uses: actions/checkout@v6
40+
with:
41+
fetch-depth: 0 # release-plz needs full git history
42+
persist-credentials: false
43+
- name: Install Rust toolchain
44+
uses: dtolnay/rust-toolchain@stable
45+
- name: Run release-plz
46+
uses: release-plz/action@v0.5
47+
with:
48+
command: release
49+
env:
50+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
51+
CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}
52+
53+
# ──────────────────────────────────────────────────────────────────────
54+
# Maintain the "Prepare release vX.Y.Z" PR on develop. Fires on:
55+
# - push to develop (accumulate changes as you work)
56+
# - push to main (clean up the PR after a real release)
57+
# - manual `release-pr` dispatch
58+
# ──────────────────────────────────────────────────────────────────────
59+
release-plz-pr:
60+
name: Release-plz PR
61+
if: >-
62+
(github.event_name == 'push' &&
63+
(github.ref == 'refs/heads/main' || github.ref == 'refs/heads/develop')) ||
64+
(github.event_name == 'workflow_dispatch' && inputs.command == 'release-pr')
65+
runs-on: ubuntu-latest
66+
permissions:
67+
contents: write # push to the release-pr branch
68+
pull-requests: write # create / update the release PR
69+
concurrency:
70+
# Only one release-pr job per branch at a time — prevents PR conflicts.
71+
# cancel-in-progress: false → never skip an pending update.
72+
group: release-plz-pr-${{ github.ref }}
73+
cancel-in-progress: false
74+
steps:
75+
- name: Checkout repository
76+
uses: actions/checkout@v6
77+
with:
78+
fetch-depth: 0
79+
persist-credentials: false
80+
- name: Install Rust toolchain
81+
uses: dtolnay/rust-toolchain@stable
82+
- name: Run release-plz
83+
uses: release-plz/action@v0.5
84+
with:
85+
command: release-pr
86+
env:
87+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
88+
# Only needed if you use a private registry; safe to leave for crates.io.
89+
CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}

‎.issues/003_hardware_aware_prefix_scheduler.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Issue 003: Hardware-Aware Prefix Scheduler — Multi-Request Verification Budget Allocator
2+
3+
**Date:** 2026-06-27
4+
**Research:** [katgpt-rs/.research/316_DSpark_Confidence_Scheduled_Speculative_Decoding.md](../.research/316_DSpark_Confidence_Scheduled_Speculative_Decoding.md)
5+
**Source paper:** [DSpark (DeepSeek-AI, 2026)](https://github.com/deepseek-ai/DeepSpec/blob/main/DSpark_paper.pdf) §3.2.2, Algorithm 1, Appendix A
6+
**Target:** `katgpt-rs/src/speculative/prefix_scheduler.rs` (new module) + Cargo feature `hardware_aware_scheduler`
7+
**Status:** Open — optimization task (Gain-tier, behind feature flag, GOAT-gated before promotion)
8+
9+
---
10+
11+
## Problem
12+
13+
katgpt-rs has per-request verification budget selectors (`caddtree_budget.rs`, `budget.rs`) but no **multi-request global** verification budget allocator. When multiple spec-decode requests share a target model forward pass (batch serving, or crowd-scale NPC cognition in riir-ai), static per-request block lengths waste target compute on low-survival suffix tokens while starving high-survival tokens in other requests. DSpark §3.2.2 formulates this as a global throughput maximization `Θ = τ · SPS(B)` and solves it greedily with a non-anticipating early-stop that preserves the lossless distribution guarantee (Appendix A correctness proof).
14+
15+
## Goal
16+
17+
Ship a generic, modelless, zero-allocation `HardwareAwarePrefixScheduler` behind `hardware_aware_scheduler` (default-off). Given:
18+
- R active requests, each with per-position survival probabilities `a_{r,j} = Π_{i≤j} c_{r,i}` (monotone non-increasing in j)
19+
- A profiled engine cost curve `SPS(B)` (steps-per-second vs total verification batch size B)
20+
21+
Produce per-request prefix lengths `ℓ*_1..ℓ*_R` that maximize `Θ = τ · SPS(B)` via:
22+
1. Globally sort all `(r, j)` candidates descending by `a_{r,j}`
23+
2. Greedily admit candidates; update `B += 1`, `τ += a_{r,j}`; O(1) lookup `SPS(B)`
24+
3. **Early-stop when `Θ ≤ Θ_best`** — this is the non-anticipating property required for lossless speculative decoding (DSpark Appendix A). Without it, retrospective global search leaks future token info into the current-token admission decision, introducing selection bias that breaks distribution preservation.
25+
26+
## Scope (what this issue IS and IS NOT)
27+
28+
**IS:**
29+
- The generic scheduler primitive (sort + greedy + cost-curve lookup + non-anticipating early-stop).
30+
- A profiled `SPS(B)` curve abstraction (load once at init, store as `Box<[f32]>` or interpolation LUT).
31+
- A single-request-isolated correctness test proving the scheduler's output matches `LeviathanVerifier` exactly when R=1 (the early-stop must reduce to "verify the full block" or "skip" depending on SPS shape, never bias the accepted distribution).
32+
33+
**IS NOT:**
34+
- A multi-request batch execution engine (katgpt-rs is single-request by default; multi-request execution is the caller's concern — the scheduler just outputs prefix lengths).
35+
- The confidence head that produces `c_k` (reuse `AcceptanceForecast`, Bebop Plan 243, as the producer).
36+
- The semi-autoregressive drafter (training → riir-train).
37+
- Sequential Temperature Scaling (small calibration refinement; separate issue if pursued).
38+
39+
## Tasks
40+
41+
- [ ] **T1** `katgpt-rs/src/speculative/prefix_scheduler.rs` — `HardwareAwarePrefixScheduler` struct + `schedule(&self, survival_probs: &[&[f32]]) -> Box<[usize]>`. Reuse `cumprodsum_scalar` for `a_{r,j} = Π c_i` if the caller passes raw `c_k` instead of pre-computed `a_{r,j}`.
42+
- [ ] **T2** `SpsCurve` abstraction — `from_profile(samples: &[(usize, f32)]) -> Self`, `steps_per_second(batch_size: usize) -> f32` (linear interpolation between samples; clamp at ends).
43+
- [ ] **T3** Non-anticipating early-stop — break the greedy loop when `Θ ≤ Θ_best`. Document the Appendix A counterexample in a doc comment.
44+
- [ ] **T4** Feature flag `hardware_aware_scheduler` (default-off), wired into `speculative/mod.rs`.
45+
- [ ] **T5** Correctness test: R=1 with synthetic SPS curve must produce a distribution-identical result to `LeviathanVerifier` (no selection bias). Port the Appendix A counterexample: without early-stop, vocab {A,B}, p_t=(0.7,0.3), p_d=(0.5,0.5) → output must be (0.7,0.3), not (0.85,0.15).
46+
- [ ] **T6** Multi-request throughput test: R=4, synthetic SPS curve (monotone-decreasing with a cliff), verify the scheduler allocates longer prefixes to high-survival requests and shorter to low-survival, and that `Θ` is at least as high as uniform-length allocation.
47+
- [ ] **T7** GOAT gate benchmark (`benches/prefix_scheduler_goat.rs`): multi-request workload, measure `accepted_tokens/sec` and `μs/step` with vs without the scheduler. Gate: ≥5% throughput gain, zero quality regression on the R=1 correctness test.
48+
- [ ] **T8** If T7 passes → promote `hardware_aware_scheduler` to default-on and demote the per-request `caddtree_budget.rs` path if it strictly dominates.
49+
50+
## Risks
51+
52+
- **Single-request default.** katgpt-rs is single-request by default. The scheduler only helps when the caller batches multiple spec-decode requests into one target forward pass. The benchmark (T7) must construct a multi-request workload or the gate is vacuous. If the engine never batches, this primitive has no leverage and stays opt-in indefinitely.
53+
- **SPS(B) curve shape.** DSpark §5.2 notes real hardware has jagged, step-wise SPS curves, not smooth/unimodal. The early-stop assumes unimodality for global optimality. The paper works around this in production via asynchronous 2-step-prior prediction + removing the early-stop (causality maintained by the temporal offset). Our CPU/SIMD/wgpu stack may have different SPS characteristics — re-profile on our hardware before promoting.
54+
- **Non-anticipating early-stop is a correctness theorem, not a heuristic.** Removing it for throughput (as DSpark does in production via async) requires a separate causality argument. Do NOT remove it in katgpt-rs without porting the async-ZOS causality proof.
55+
56+
## Cross-references
57+
58+
- `katgpt-rs/.research/316_DSpark_Confidence_Scheduled_Speculative_Decoding.md` — distillation
59+
- `katgpt-rs/src/speculative/acceptance_forecast.rs` — `AcceptanceForecast` (producer of `c_k`)
60+
- `katgpt-rs/src/speculative/caddtree_budget.rs` — per-request analog (`expected_accepted_length_at_budget`)
61+
- `katgpt-rs/src/speculative/budget.rs` — per-request adaptive budget (predecessor)
62+
- `katgpt-rs/src/cumprodsum.rs` — `cumprodsum_*` (SIMD `Π c_i`)
63+
- DSpark paper §3.2.2 (Algorithm 1), §5.2 (production async variant), Appendix A (non-anticipating correctness proof)

0 commit comments

Comments
 (0)