|
| 1 | +# Reproducing and extending RecurQuant |
| 2 | + |
| 3 | +This guide separates three different activities: |
| 4 | + |
| 5 | +1. **Use the cache** to run the supported Qwen3.5 checkpoint. This checks that |
| 6 | + the integration works and reports resident recurrent-state bytes; it does |
| 7 | + not reproduce a quality result. |
| 8 | +2. **Reproduce development evidence** on the 90-task MBPP validation split. |
| 9 | + This repeats an already inspected development experiment. |
| 10 | +3. **Run or replicate confirmation** on the 500-task MBPP test split. The |
| 11 | + original project run is "untouched" only because its policy, manifest, gates, |
| 12 | + and code were frozen before outcomes were computed. Once outcomes are |
| 13 | + public, another run is a replication, not a new untouched confirmation. |
| 14 | + |
| 15 | +None of these workflows establishes generated-code correctness, lower latency, |
| 16 | +lower whole-model or peak memory, cross-model generality, novelty, or a |
| 17 | +breakthrough. |
| 18 | + |
| 19 | +## Frozen reference |
| 20 | + |
| 21 | +The v0.2 numerical evidence is scoped to: |
| 22 | + |
| 23 | +- Python `3.11.15`; |
| 24 | +- PyTorch `2.11.0+cu128`, CUDA runtime `12.8`, and NVIDIA driver `592.15`; |
| 25 | +- Transformers `5.14.1` and Datasets `4.8.5`; |
| 26 | +- `Qwen/Qwen3.5-0.8B-Base` at revision |
| 27 | + `dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68`; |
| 28 | +- `google-research-datasets/mbpp`, configuration `full`, at revision |
| 29 | + `4bb6404fdc6cacfda99d4ac4205087b89d32030c`; and |
| 30 | +- batch one, eager attention, BF16 model weights on CUDA, evaluation mode, and |
| 31 | + teacher-forced reference-code tokens. |
| 32 | + |
| 33 | +The accepted run used an NVIDIA GeForce RTX 5070 Laptop GPU with 8 GB VRAM. |
| 34 | +The 90-task development matrix took 4,198.6 seconds, about 70 minutes. The |
| 35 | +500-task confirmation matrix is a long, multi-hour run. CPU execution uses |
| 36 | +FP32 model weights and may be substantially slower; it is supported by the |
| 37 | +scripts but is not the environment behind the published numerical evidence. |
| 38 | +Treat a different software or hardware environment as a replication rather |
| 39 | +than a byte-for-byte repeat. |
| 40 | + |
| 41 | +Create the normal contributor environment from the repository root: |
| 42 | + |
| 43 | +```powershell |
| 44 | +uv venv --python 3.11 .venv |
| 45 | +uv pip install --python .venv\Scripts\python.exe -e ".[dev,eval]" |
| 46 | +.venv\Scripts\python.exe -m pytest |
| 47 | +.venv\Scripts\python.exe -m ruff check . |
| 48 | +``` |
| 49 | + |
| 50 | +The package pins Transformers `5.14.1`; the other exact reference versions |
| 51 | +above are provenance pins, not the full range declared by `pyproject.toml`. |
| 52 | +Record the versions actually resolved for a replication. On Linux or macOS, |
| 53 | +replace `.venv\Scripts\python.exe` with `.venv/bin/python` below. |
| 54 | + |
| 55 | +The scripts download the pinned model and dataset by default. Add |
| 56 | +`--local-files-only` only when the model and tokenizer are already cached. It |
| 57 | +does not apply to the streamed MBPP dataset loader. |
| 58 | + |
| 59 | +## 1. Use the packed cache |
| 60 | + |
| 61 | +The installed command runs the pinned checkpoint with the frozen v0.2 layout: |
| 62 | +model layer 0 at INT8 and the other 17 recurrent layers at INT4, group size 128, |
| 63 | +FP16 scales, and nearest rounding. |
| 64 | + |
| 65 | +```powershell |
| 66 | +.venv\Scripts\python.exe -m recurquant.cli qwen35 ` |
| 67 | + --device cuda ` |
| 68 | + --max-new-tokens 32 ` |
| 69 | + --prompt "Explain recurrent-state quantization in two sentences." |
| 70 | +``` |
| 71 | + |
| 72 | +Use `--device auto` to select CUDA when available and CPU otherwise. The output |
| 73 | +includes resident packed bytes, the full-precision recurrent-state equivalent, |
| 74 | +the largest single materialized recurrent state, and the resident-state |
| 75 | +compression ratio. That largest-state field is not a peak-workspace or peak |
| 76 | +CUDA-memory measurement. |
| 77 | + |
| 78 | +For application code, use `create_qwen35_v02_mixed_cache(model)`. Use |
| 79 | +`create_qwen35_packed_cache(...)` with `layer_specs` only for an explicitly |
| 80 | +custom policy. The model must be in evaluation mode, loaded with eager |
| 81 | +attention on one materialized device, and every forward must run inside |
| 82 | +`torch.inference_mode()` or `torch.no_grad()`. See |
| 83 | +[`compatibility.md`](compatibility.md) for unsupported generation and runtime |
| 84 | +modes. |
| 85 | + |
| 86 | +## 2. Reproduce the development evidence |
| 87 | + |
| 88 | +The committed calibration and prepared manifest are sufficient to repeat the |
| 89 | +frozen 90-task method matrix: |
| 90 | + |
| 91 | +```powershell |
| 92 | +.venv\Scripts\python.exe scripts\evaluate_mbpp.py ` |
| 93 | + --phase development ` |
| 94 | + --calibration-artifact evidence\mbpp-v02-calibration.json ` |
| 95 | + --prepared-manifest evidence\mbpp-v02-development-manifest.json ` |
| 96 | + --device cuda ` |
| 97 | + --checkpoint artifacts\replication-development.checkpoint.json ` |
| 98 | + --output artifacts\replication-development.json |
| 99 | +``` |
| 100 | + |
| 101 | +To regenerate each input instead of trusting the committed copies, run the |
| 102 | +full calibration and then prepare an outcome-free token manifest: |
| 103 | + |
| 104 | +```powershell |
| 105 | +.venv\Scripts\python.exe scripts\calibrate_mbpp_layers.py ` |
| 106 | + --device cuda ` |
| 107 | + --output artifacts\replication-calibration.json |
| 108 | +
|
| 109 | +.venv\Scripts\python.exe scripts\evaluate_mbpp.py ` |
| 110 | + --phase development ` |
| 111 | + --calibration-artifact artifacts\replication-calibration.json ` |
| 112 | + --manifest-only ` |
| 113 | + --device cuda ` |
| 114 | + --output artifacts\replication-development-manifest.json |
| 115 | +``` |
| 116 | + |
| 117 | +Then substitute those two artifact paths in the first command. A full |
| 118 | +development or confirmation run requires `--prepared-manifest`. Do not use |
| 119 | +`--limit`, change `--group-size 128` or `--bootstrap-samples 10000`, or add |
| 120 | +`--skip-qdq-preflight` when claiming a protocol-eligible replication. The |
| 121 | +evaluator rejects those changes for a full public run. |
| 122 | + |
| 123 | +## 3. Replicate the confirmation phase |
| 124 | + |
| 125 | +The explicit lock is an acknowledgement token, not a credential. It prevents |
| 126 | +accidental access to the test split. Run this only after freezing your own |
| 127 | +policy and analysis plan: |
| 128 | + |
| 129 | +```powershell |
| 130 | +.venv\Scripts\python.exe scripts\evaluate_mbpp.py ` |
| 131 | + --phase confirmation ` |
| 132 | + --calibration-artifact evidence\mbpp-v02-calibration.json ` |
| 133 | + --prepared-manifest evidence\mbpp-v02-confirmation-manifest.json ` |
| 134 | + --confirmation-lock recurquant:unlock-mbpp-confirmation:rq-v0.2 ` |
| 135 | + --device cuda ` |
| 136 | + --checkpoint artifacts\replication-confirmation.checkpoint.json ` |
| 137 | + --output artifacts\replication-confirmation.json |
| 138 | +``` |
| 139 | + |
| 140 | +At the commit represented by this guide, the committed confirmation file is a |
| 141 | +prepared manifest with `outcomes_computed: false`; it is not a result artifact. |
| 142 | +Do not inspect partial candidate metrics, retune on test tasks, or rerun an |
| 143 | +unfavourable outcome. A rerun is valid only for a documented infrastructure or |
| 144 | +evaluator failure defined without reference to which method looked better. |
| 145 | + |
| 146 | +## Manifests, checkpoints, and hashes |
| 147 | + |
| 148 | +Prepared manifests record the dataset rows and tokenization alongside the |
| 149 | +model, dataset, calibration, candidate plan, and repository commit before a |
| 150 | +long run. The evaluator recomputes and compares the phase, model revision, |
| 151 | +dataset-manifest hash, token-manifest hash, and calibration-evidence hash. |
| 152 | +Checkpoints are written atomically after each task. Their run signature binds |
| 153 | +the phase, model revision, current repository commit, calibration evidence |
| 154 | +hash, prepared-manifest hash, token-manifest hash, group size, and current |
| 155 | +candidate plan. Resume by repeating the exact command. A checkpoint from |
| 156 | +changed code or settings is rejected instead of silently reused. |
| 157 | + |
| 158 | +The committed integrity anchors are: |
| 159 | + |
| 160 | +| Artifact | Tasks | File SHA256 | Canonical evidence SHA256 | |
| 161 | +|---|---:|---|---| |
| 162 | +| `evidence/mbpp-v02-calibration.json` | 128 | `d3d2f9acf6113ad455cce78d1b957a265c6675236845564d855c7cb537267125` | `7aa8227dd0b19bb7494963c0b590c8ec53cee29d3b696ccd4087c71a5ac461ee` | |
| 163 | +| `evidence/mbpp-v02-development-manifest.json` | 90 | `7d51f732f9d0147c485d3bd3214e9bfcd16132de07df342f0c6b9696160aac3f` | `2b13dfb1799472b2fd0006cb87cea60c67a7af37825a87f1686a6fe3d8e38d7c` | |
| 164 | +| `evidence/mbpp-v02-development.json` | 90 | `5980fd58aa0933ad97deb896d4901fcd37350c4a57d8a80022ab218aaf77e727` | `301c52e194bbd23059a0040a8e94aeac97dc33de1100f13edbf17dc877755488` | |
| 165 | +| `evidence/mbpp-v02-confirmation-manifest.json` | 500 | `c6a7d0db6ef7577a66ac19fbbc0be166279488f6a6be432b364bd9eb6833f7b0` | `21a6d18c6a0887b1499d156a3d610d4bfafdd59d3557713485b62038e263b96a` | |
| 166 | + |
| 167 | +Their recorded source commits are respectively |
| 168 | +`cc35f4396ef4dd475908d8f96e05fe9c559f13be`, |
| 169 | +`3a3c4a2a11c0822f6c456a74327127f294ce67e1`, |
| 170 | +`20a5ea95a8ed692600ee1645d2913f3a4b8a6795`, and |
| 171 | +`44d75a2776fa36441e17cc688965c9825c4c1a1c`. |
| 172 | + |
| 173 | +Cross-machine invariant manifest hashes are: |
| 174 | + |
| 175 | +| Phase | Dataset manifest SHA256 | Token manifest SHA256 | |
| 176 | +|---|---|---| |
| 177 | +| Calibration | `129698d01bdf7f08989878ff7c980230456095c5b2cae42a010f567ecc49dc1a` | not produced by the calibration script | |
| 178 | +| Development | `8fed3da0aae864f4e30c70ad70b0269f759d3592dccb9ab87f24444fa24d65dc` | `3c19f37f3c35cb22f17e66a5438bb9968b214e0aff64067930bc3a7124f63f5c` | |
| 179 | +| Confirmation | `060aaff7117dc47af6c01253a912f34b6956241c336bbc7216e73bca8624d2d4` | `199a8836489af9bd0af3fec027e85d57df356bd9919492b24015de51d143f525` | |
| 180 | + |
| 181 | +`canonical_evidence_sha256` is SHA256 over `artifact["evidence"]` serialized |
| 182 | +with sorted keys, two-space indentation, standard JSON ASCII escaping, |
| 183 | +`allow_nan=False`, and one trailing newline. It protects the evidence payload |
| 184 | +but includes environment and command provenance, so a legitimate replication |
| 185 | +on another commit or machine need not match the accepted canonical hash. The |
| 186 | +dataset and token manifest hashes should match when the frozen inputs and |
| 187 | +tokenizer do. |
| 188 | + |
| 189 | +## Extend with a new policy |
| 190 | + |
| 191 | +Do not edit the committed v0.2 evidence files or reinterpret its confirmation. |
| 192 | +Start with a separate diagnostic artifact. The existing smoke script can test |
| 193 | +one or more promoted recurrent layers without touching MBPP: |
| 194 | + |
| 195 | +```powershell |
| 196 | +.venv\Scripts\python.exe scripts\run_qwen35_smoke.py ` |
| 197 | + --cache-mode packed ` |
| 198 | + --low-bits 4 ` |
| 199 | + --high-bits 8 ` |
| 200 | + --upgrade-layers 6 ` |
| 201 | + --prompt-profile code ` |
| 202 | + --device cuda ` |
| 203 | + --output artifacts\policy-layer6-diagnostic.json |
| 204 | +``` |
| 205 | + |
| 206 | +Use `--sensitivity-sweep` instead of `--upgrade-layers` to evaluate every Gated |
| 207 | +DeltaNet layer on one of the script's synthetic prompt profiles. These traces |
| 208 | +are diagnostics, not public-task or confirmation evidence. |
| 209 | + |
| 210 | +If a policy remains promising: |
| 211 | + |
| 212 | +1. add it through `layer_specs` and add focused unit/integration tests; |
| 213 | +2. write a new versioned protocol with its own calibration, development, and |
| 214 | + confirmation partitions or dataset; |
| 215 | +3. freeze new candidate definitions and outcome-free manifests before scoring; |
| 216 | +4. write artifacts to new versioned filenames rather than replacing anything |
| 217 | + under `evidence/mbpp-v02-*`; and |
| 218 | +5. report failed gates and negative controls with successful results. |
| 219 | + |
| 220 | +Changing `candidate_definitions()` in `scripts/evaluate_mbpp.py` creates a new |
| 221 | +experiment; it does not extend the frozen v0.2 claim. Follow |
| 222 | +[`CONTRIBUTING.md`](../CONTRIBUTING.md), keep the pull request to one testable |
| 223 | +change, and include the exact command, environment, artifact, metric definition, |
| 224 | +and claim boundary. Read the complete frozen design in |
| 225 | +[`PUBLIC_EVAL_PROTOCOL_V02.md`](../research/PUBLIC_EVAL_PROTOCOL_V02.md) and the |
| 226 | +current restrictions in [`CLAIM_BOUNDARY.md`](../research/CLAIM_BOUNDARY.md). |
0 commit comments