Shorthand for Thought

This repository contains the public code for Shorthand for Thought: Compressing LLM Reasoning with Learned Supertokens. The package name is reastok because the code builds reasoning supertokens on top of a base tokenizer.

The repo is intentionally self-contained: no private workspace paths are required. Large inputs such as merge TSVs, Hugging Face models, pretokenized datasets, trained checkpoints, and evaluation metrics are configured by CLI flags or environment variables.

Setup

uv sync --all-groups

Most analysis commands only need the default dependencies. Training uses the optional train dependency group and should be launched through Slurm for GPU runs.

Core Pipeline

1. Count reasoning n-grams:

uv run python scripts/tokenize_ngrams.py --project openthoughts3-300k-qwen3-30b-sglang

2. Build BPE merge tables:

uv run python scripts/bpe_from_ngrams.py --project openthoughts3-300k-qwen3-30b-sglang --experiment exp5c_structural_plus

3. Prepare a self-contained tokenizer artifact:

uv run python scripts/prepare_model.py \
  --model Qwen/QwQ-32B \
  --tsv cache/bpe_exp5c_1000m.tsv \
  --num-merges 250 \
  --output-dir outputs/repro/qwq32b-exp5c-250

4. Encode with supertokens:

from reastok.merge_apply import SuperPostTokenizer

tokenizer = SuperPostTokenizer.load("outputs/repro/qwq32b-exp5c-250")
ids = tokenizer.encode("Let us reason this through carefully.")

Paper Reproduction

docs/paper_results.md maps paper tables and figures to the scripts that generate them. The main SFT workflow is under scripts/repro/ and stops after checkpoint training; evaluation is intentionally external because benchmark runners are environment-specific.

REPO_ROOT=$PWD scripts/repro/repro_submit_sft.sh

To summarize evaluation results from any public runner that writes metrics.json files:

uv run python scripts/summarize_eval_metrics.py --logs-dir /path/to/eval/logs

All path-like defaults can be overridden with environment variables; see the top of each Slurm script for the knobs.