A 150M encoder that answers arbitrary typed questions about a state in one forward pass, with calibrated confidence. 0.03 behind TypeSafe Jev on its own benchmark, 2.5× better calibrated, 4× faster, $0.
Results · GitHub Action · How it works · Quick start · What we learned · Limitations
TypeSafe AI's Jev (launched 2026-09-15) is a "System One" model: it never writes prose, it only decides, classifies, routes and scores. You hand it a state plus a set of typed questions and it answers all of them in one non-autoregressive pass, with a confidence on each. $0.042/1M input, output tokens free, ~239 ms per call.
This repo reproduces that interface with an open 150M encoder you can train on a free Colab T4 in under 30 minutes, then run locally — or in a browser — for nothing.
TypeSafe does not publish Jev's parameter count. This model is 150M. Their published specs give pricing, rate limits and a 64k context window, but no model size — and their launch FAQ lists "Is Jev just a smaller LLM?" without answering it. Any size comparison you see, including here, is inference from price and latency rather than a disclosed figure.
Three primitives, matching theirs:
| type | meaning | criteria format |
|---|---|---|
noul |
boolean | {"true": desc, "false": desc}, or omitted entirely |
choice |
enum | {label: desc, ...} |
score |
ordered scale | [level_0, level_1, ...] |
decide(state, {
"action": Question.choice("What should the observability system do?", {
"continue": "Let the agent proceed.",
"human_review": "Queue this trace for a human.",
"stop": "Halt the agent now."}),
"risk": Question.score("How risky was this behaviour?", [
"Benign: read-only.", "Low: routine writes.",
"Moderate: irreversible.", "High: destructive."]),
})
# -> {"action": {"label": "human_review", "confidence": 0.54, "probabilities": {...}},
# "risk": {"label": "2", "confidence": 0.47, "probabilities": {...}}}The questions are supplied per request. Nothing about the schema is baked into the weights.
Measured on the official 400-case test split of LocalLLaMA/typed-decisions (2,000 decisions across four workflows). Blend weight and temperature were chosen on validation, never on test.
| configuration | accuracy | ECE ↓ | schema |
|---|---|---|---|
| Fine-tuned model alone | 0.6240 | 0.1045 | dynamic |
| Frozen probe alone | 0.6705 | 0.0389 | fixed |
| Ensemble (w=0.60, T=0.35) | 0.6965 | 0.0565 | fixed |
| TypeSafe Jev 1.13.0 | 0.7270 | 0.1440 | dynamic |
| majority-class baseline | 0.4830 | — | — |
| single human annotator | 0.6590 | — | — |
0.03 behind Jev on accuracy. 2.5× better calibrated. ~60 ms per case locally against their 239 ms p50. Free, offline, open weights.
This is the question that decides whether a decision model is usable. Ensemble, on test:
| confidence ≥ | coverage | accuracy |
|---|---|---|
| 0.90 | 25.4% | 0.915 |
| 0.80 | 44.5% | 0.846 |
| 0.70 | 61.7% | 0.806 |
| 0.50 | 91.9% | 0.726 |
A quarter of all decisions at 91.5% accuracy is a shippable policy: autoroute that band, escalate the rest.
On the GitHub Marketplace. Run typed decisions inside a workflow, on the runner's CPU: no API key, no per-call cost, and the state never leaves the runner.
- uses: intikhab49/open-jev-typed-decision-engine@v1
id: jev
with:
workflow: security_incidents # built-in: answered by the 0.697 ensemble
state-file: alert.json # or state: <JSON or text>
- if: >-
fromJSON(steps.jev.outputs.result).decisions.disposition.answer == 'contain' &&
fromJSON(steps.jev.outputs.result).decisions.disposition.confidence >= 0.9
run: ./page-oncall.shEvery answer says which path produced it, and the two are not equally good:
| you give | path | accuracy | ECE ↓ |
|---|---|---|---|
workflow: — one of the four built-ins below |
ensemble (mode: ensemble) |
0.6970 | 0.057 |
questions: — your own typed questions |
fine-tuned model alone (mode: model) |
0.6235 | 0.104 |
Both rows are the full 400-case test split replayed through the Action's own runtime (onnxruntime + numpy, no torch) by action/verify.py — numbers in action/measured.json. Both are in-distribution: the test split comes from the same synthetic dataset the model and probes were trained on, and its gold labels are LLM consensus. On your real alerts or tickets, accuracy is unmeasured.
Read the second row carefully. 0.6235 is measured on the dataset's 20 questions, which the model was trained on. A question unlike them is unmeasured, and it can be confidently wrong: on its first outing it called a bug report with numbered repro steps a question, with has_repro: false. Use a built-in where one fits; for your own questions, check outputs on real cases and gate on confidence before automating anything.
built-in workflow |
questions (labels) |
|---|---|
security_incidents |
disposition (close_benign · contain · investigate · monitor), severity (0–4), urgency (0–3), true_positive, credential_compromise |
agent_trace_observability |
action (continue · human_review · observe · stop), outcome (failure · harmful · partial · success), risk (0–3), urgency (0–3), needs_review |
customer_service |
action (answer_directly · close_no_action · escalate_to_human · execute_refund · request_information), category (account · billing · delivery · refund · technical), churn_risk (0–3), urgency (0–3), needs_human |
invoice_processing |
disposition (approve · hold · manual_review · reject), discrepancy_severity (0–3), urgency (0–3), duplicate, matches_order |
Full question wording is in presets.json on the model release. Unlabelled questions are noul (true/false).
Your own questions, same format as the Python API:
- uses: intikhab49/open-jev-typed-decision-engine@v1
id: triage
with:
state: ${{ github.event.issue.title }} ${{ github.event.issue.body }}
questions: |
kind:
type: choice
instructions: What kind of issue is this?
criteria:
bug: Something that should work is broken.
feature: A request for new behaviour.
question: The reporter is asking how to do something.| input | |
|---|---|
state / state-file |
the thing to decide about, JSON or plain text — exactly one |
workflow |
a built-in, above |
questions / questions-file |
your own questions, YAML or JSON; names must not clash with the chosen workflow's |
model-release |
release the model files come from (default model-v1) |
Outputs: result (JSON — {"workflow", "decisions": {name: {answer, confidence, type, mode, distribution}}}) and result-file. Each run also writes a table to the job summary.
The first run downloads ~1.2 GB of ONNX files, each checked against action/SHA256SUMS from the same tag as the code, then caches them (GitHub evicts caches unused for 7 days). A call took 2.6 s on a standard ubuntu-latest runner. Linux and macOS runners.
The PyTorch checkpoint is on the same release as jevlite-fp16.pt (298 MB, six pieces; join with cat, instructions in the release notes).
flowchart LR
S["state<br/><i>log · ticket · trace · diff</i>"] --> SEQ
Q["typed questions<br/><i>supplied per request</i>"] --> SEQ
SEQ["one sequence<br/>state + <<q>> question + <<l>> label …"]
SEQ --> ENC["ModernBERT-base<br/>150M · one forward pass"]
ENC --> H["Linear(d, 1)<br/>read at every <<l>> marker"]
H --> SM["softmax within<br/>each question"]
SM --> T["per-type temperature"]
T --> OUT["label + calibrated confidence<br/>for every question"]
style ENC fill:#2a78d6,stroke:#1c5cab,color:#fff
style OUT fill:#1baf7a,stroke:#158a60,color:#fff
style SEQ fill:#f0efec,stroke:#c3c2b7,color:#0b0b0b
style H fill:#f0efec,stroke:#c3c2b7,color:#0b0b0b
style SM fill:#f0efec,stroke:#c3c2b7,color:#0b0b0b
style T fill:#f0efec,stroke:#c3c2b7,color:#0b0b0b
The obvious design is one nn.Linear per question. It works, and it freezes the schema into the weights — add a question or rename a label and you retrain.
Instead, every candidate label is written into the input sequence behind a <<l>> marker token, and the head is a single Linear(d, 1) read at each marker. Softmax runs within each question's own label set.
[CLS] {state json} <<q>> How risky was this? <<l>> 0: Benign… <<l>> 1: Low… <<q>> …
↑ question marker ↑ one logit read off each of these
- The schema is data at inference time — new questions, new labels, a whole new workflow, with no retraining and no new parameters.
- Labels attend to the state and to each other: a cross-encoder, not a bi-encoder.
- One encoder pass answers every question about a state.
Base encoder is ModernBERT-base (150M, bidirectional, 8192 context). A causal model works via --encoder, but pools differently — token 0 in a decoder-only model has only seen itself, so [CLS] pooling on one is a bug, not a shortcut.
The dataset ships full annotator probability distributions, not just argmax labels. Training on the argmax throws that away and directly causes overconfidence: a model taught that a 55/45 case is true learns to say 0.95, then is wrong 45% of the time while claiming near-certainty.
So the loss is soft cross-entropy against the consensus distribution plus a Brier term — a proper scoring rule, which is what actually pushes probabilities toward honesty.
It overshoots in the safe direction. The reliability curve sits above the diagonal at every bin: because the consensus targets are often flat (0.43 / 0.28 / 0.25 on a contested case), the model learns to emit flat distributions, so its top-1 confidence reads lower than its real accuracy. It is underconfident rather than overconfident — the direction you want if you are going to route on it, and one that a single temperature corrects.
Upload jevlite_colab.ipynb — the whole repo is embedded in it — set Runtime → T4 GPU, and Run all.
pip install -r requirements.txt
python smoke_test.py --all # validate all 1,600 rows, no GPU, ~2 min
python 01_ceiling.py # baselines you have to beat, $0
python 02_train.py --epochs 20 # ~25 min on a T4
python 03_calibrate.py # per-type temperature, CPU, seconds
python 04_eval.py # official test split, vs Jev
python 07_ensemble.py # blend with the frozen probe
python plots.py --outdir plots # six charts
python 05_serve.py --serve # POST /decide {"state": ..., "questions": ...}
python 06_export_onnx.py --quantizeEvery script takes --limit N for a fast dry run, and --config <workflow> to train a single-domain specialist.
typed_schema.py |
the three primitives; Question.noul/choice/score builders |
td_data.py |
dataset loading (HTTP fallback), marker-token encoding, collate |
model.py |
JevLite, grouped softmax, soft-CE + Brier loss |
probe.py |
frozen-encoder + logistic-regression baseline |
metrics.py |
ECE (two kinds), Brier, NLL, TVD, risk-coverage |
plots.py |
six charts, rendered from the JSON each step writes |
01_… → 07_… |
the pipeline |
smoke_test.py |
correctness checks, no GPU needed |
legacy/ |
the earlier fixed-head version, kept for comparison |
A frozen encoder + logistic regression scores 0.670 — beating the fine-tuned model's 0.6240. No training at all. This reproduces the independent Banking77 finding where the same baseline beat Jev by 10 points at 44× the speed. 01_ceiling.py runs it first so you know what you're up against. If your fine-tune can't clear it, ship the probe.
The accuracy gap was undertraining — up to a point. 6 epochs gave 0.568; 20 epochs with early stopping (best at epoch 9) gave 0.6195. Past that, training loss kept falling while validation flatlined. The binding constraint is 1,016 training cases, not model size or epochs — which is why reaching for a bigger encoder first is the wrong instinct here.
Post-hoc temperature scaling had three different outcomes, so don't generalise from one. A no-op on the undertrained model (temperatures ≈ 1.0, ECE slightly worse), a mild help on the overfit one, and decisive on the ensemble: ECE 0.156 → 0.057 with accuracy untouched. Mixing two disagreeing distributions flattens them, so the blend was badly underconfident; sharpening in probability space can't move the argmax, so it's free or nothing. Before sharpening, the ≥0.90 band held 0.5% of traffic; after, 25.4%.
Single-annotator agreement is not a ceiling. It's 0.659 here — what one human rater scores against the consensus. A model that always picks the consensus argmax scores 1.000, and Jev already exceeds it. The real limit is that 15.6% of test decisions are near-ties (top two labels within 0.1), so the last few points of headline accuracy are mostly luck. Judge on calibration and coverage.
The ensemble trades away the dynamic schema. The probe only answers questions it has training labels for. On an unseen question it returns uniform and the blend degrades to 0.6 × the neural model — so 0.6965 is the known-schema number and ~0.624 is the new-schema number. Three configurations, different trade-offs, no single best. Always say which one a number came from.
Jev's 0.727 is zero-shot. It never saw this dataset; this model trained on 1,016 in-domain cases from the same distribution and still came in lower. That makes the gap look worse, not better, and it's the accurate framing.
Small evaluation. 400 test cases / 2,000 decisions. Differences under ~0.02 are not meaningful.
Jev's figures are quoted, not reproduced here. Accuracy 0.727 and ECE 0.144 come from the typed-decisions benchmark; latency and pricing from the independent evaluations linked below.
- 10% of the dataset is a question shape the first rows never show. Bare
noulquestions carry nocriteriakey at all. A 6-row smoke test passes and training then dies 300 rows in.smoke_test.py --allencodes every row for exactly this reason. - Workflows disagree on label width (16/17/18/20), so
torch.catacross batches fails the moment a run spans more than one. Hencepad_cat. - Unbounded temperature scaling on soft targets is degenerate — the first fit returned T=400, which flattens everything to uniform, minimises NLL beautifully, and destroys the confidence signal. Bounded to [0.25, 5.0], and refused for any type with under 30 validation decisions.
- ONNX export below opset 18 produces an invalid graph — torch emits
Splitwithnum_outputsregardless, and ORT then refuses to load it. Silent at export, fatal at load; hence the parity check. - ONNX weights land in a 596 MB
.onnx.datasidecar. Serve both files or you load a graph with no weights. int8 (151 MB) shifts logits materially — recalibrate rather than reusing fp32 temperatures. - Never lower
--max-len. States run 401–997 tokens (p50 588); 512 truncates 73% of them. Batches already pad to their own longest example, so the high cap costs nothing.
- TypeSafe AI for Jev, the design this reproduces. Not affiliated with, endorsed by, or derived from their model.
LocalLLaMA/typed-decisions— the public benchmark.answerdotai/ModernBERT-base— the encoder.- Prior open reproductions: Verdict / OpenJev (ModernBERT-151M + WebGPU playground) and open-jev-deberta-v3-large.
- Independent evaluations this repo's claims lean on: baselines eval (Banking77 / CLINC150), phishing bench (vs Claude Haiku 4.5), OOD calibration (per-type ECE).
Keywords: typed decision model · System One model · non-autoregressive classifier · TypeSafe Jev alternative · open source Jev · ModernBERT · structured output · calibrated confidence · LLM routing · agent observability · confidence thresholding · expected calibration error





