Skip to content

About

Open reproduction of TypeSafe Jev: a 150M typed decision engine (noul/choice/score in one non-autoregressive pass, calibrated confidence). 0.697 vs Jev's 0.727, 2.5x better calibrated, 4x faster, free. Trains on a Colab T4 in 30 min.

Topics

Resources

Stars

50 stars

Watchers

1 watching

Forks

Use this GitHub action with your project
Add this Action to an existing workflow or create a new one
View on Marketplace

Latest commit

 

History

8 Commits

Folders and files

Repository files navigation

Open Jev - a typed decision engine you can train for free. 0.697 accuracy against TypeSafe Jev's 0.727, 2.5x better calibrated, 4x faster, zero cost.

Open Jev — a typed decision engine you can train for free

A 150M encoder that answers arbitrary typed questions about a state in one forward pass, with calibrated confidence. 0.03 behind TypeSafe Jev on its own benchmark, 2.5× better calibrated, 4× faster, $0.

License Python Colab Benchmark Marketplace

Results · GitHub Action · How it works · Quick start · What we learned · Limitations


What this is

TypeSafe AI's Jev (launched 2026-09-15) is a "System One" model: it never writes prose, it only decides, classifies, routes and scores. You hand it a state plus a set of typed questions and it answers all of them in one non-autoregressive pass, with a confidence on each. $0.042/1M input, output tokens free, ~239 ms per call.

This repo reproduces that interface with an open 150M encoder you can train on a free Colab T4 in under 30 minutes, then run locally — or in a browser — for nothing.

TypeSafe does not publish Jev's parameter count. This model is 150M. Their published specs give pricing, rate limits and a 64k context window, but no model size — and their launch FAQ lists "Is Jev just a smaller LLM?" without answering it. Any size comparison you see, including here, is inference from price and latency rather than a disclosed figure.

Three primitives, matching theirs:

type meaning criteria format
noul boolean {"true": desc, "false": desc}, or omitted entirely
choice enum {label: desc, ...}
score ordered scale [level_0, level_1, ...]
decide(state, {
    "action": Question.choice("What should the observability system do?", {
        "continue": "Let the agent proceed.",
        "human_review": "Queue this trace for a human.",
        "stop": "Halt the agent now."}),
    "risk": Question.score("How risky was this behaviour?", [
        "Benign: read-only.", "Low: routine writes.",
        "Moderate: irreversible.", "High: destructive."]),
})
# -> {"action": {"label": "human_review", "confidence": 0.54, "probabilities": {...}},
#     "risk":   {"label": "2",            "confidence": 0.47, "probabilities": {...}}}

The questions are supplied per request. Nothing about the schema is baked into the weights.


Results

Measured on the official 400-case test split of LocalLLaMA/typed-decisions (2,000 decisions across four workflows). Blend weight and temperature were chosen on validation, never on test.

configuration accuracy ECE ↓ schema
Fine-tuned model alone 0.6240 0.1045 dynamic
Frozen probe alone 0.6705 0.0389 fixed
Ensemble (w=0.60, T=0.35) 0.6965 0.0565 fixed
TypeSafe Jev 1.13.0 0.7270 0.1440 dynamic
majority-class baseline 0.4830 — —
single human annotator 0.6590 — —

0.03 behind Jev on accuracy. 2.5× better calibrated. ~60 ms per case locally against their 239 ms p50. Free, offline, open weights.

Bar chart of accuracy on the typed-decisions test split: majority class 0.483, fine-tuned model 0.620, single annotator 0.659, frozen probe 0.670, this repo's ensemble 0.697, against a dashed reference line at TypeSafe Jev's 0.727.

Can you route on the confidence?

This is the question that decides whether a decision model is usable. Ensemble, on test:

confidence ≥ coverage accuracy
0.90 25.4% 0.915
0.80 44.5% 0.846
0.70 61.7% 0.806
0.50 91.9% 0.726

A quarter of all decisions at 91.5% accuracy is a shippable policy: autoroute that band, escalate the rest.

Line chart of accuracy against coverage as the confidence threshold rises, climbing from 0.73 at full coverage to 0.92 on the most confident quarter of decisions.


GitHub Action

On the GitHub Marketplace. Run typed decisions inside a workflow, on the runner's CPU: no API key, no per-call cost, and the state never leaves the runner.

- uses: intikhab49/open-jev-typed-decision-engine@v1
  id: jev
  with:
    workflow: security_incidents      # built-in: answered by the 0.697 ensemble
    state-file: alert.json            # or state: <JSON or text>

- if: >-
    fromJSON(steps.jev.outputs.result).decisions.disposition.answer == 'contain' &&
    fromJSON(steps.jev.outputs.result).decisions.disposition.confidence >= 0.9
  run: ./page-oncall.sh

Every answer says which path produced it, and the two are not equally good:

you give path accuracy ECE ↓
workflow: — one of the four built-ins below ensemble (mode: ensemble) 0.6970 0.057
questions: — your own typed questions fine-tuned model alone (mode: model) 0.6235 0.104

Both rows are the full 400-case test split replayed through the Action's own runtime (onnxruntime + numpy, no torch) by action/verify.py — numbers in action/measured.json. Both are in-distribution: the test split comes from the same synthetic dataset the model and probes were trained on, and its gold labels are LLM consensus. On your real alerts or tickets, accuracy is unmeasured.

Read the second row carefully. 0.6235 is measured on the dataset's 20 questions, which the model was trained on. A question unlike them is unmeasured, and it can be confidently wrong: on its first outing it called a bug report with numbered repro steps a question, with has_repro: false. Use a built-in where one fits; for your own questions, check outputs on real cases and gate on confidence before automating anything.

built-in workflow questions (labels)
security_incidents disposition (close_benign · contain · investigate · monitor), severity (0–4), urgency (0–3), true_positive, credential_compromise
agent_trace_observability action (continue · human_review · observe · stop), outcome (failure · harmful · partial · success), risk (0–3), urgency (0–3), needs_review
customer_service action (answer_directly · close_no_action · escalate_to_human · execute_refund · request_information), category (account · billing · delivery · refund · technical), churn_risk (0–3), urgency (0–3), needs_human
invoice_processing disposition (approve · hold · manual_review · reject), discrepancy_severity (0–3), urgency (0–3), duplicate, matches_order

Full question wording is in presets.json on the model release. Unlabelled questions are noul (true/false).

Your own questions, same format as the Python API:

- uses: intikhab49/open-jev-typed-decision-engine@v1
  id: triage
  with:
    state: ${{ github.event.issue.title }} ${{ github.event.issue.body }}
    questions: |
      kind:
        type: choice
        instructions: What kind of issue is this?
        criteria:
          bug: Something that should work is broken.
          feature: A request for new behaviour.
          question: The reporter is asking how to do something.
input
state / state-file the thing to decide about, JSON or plain text — exactly one
workflow a built-in, above
questions / questions-file your own questions, YAML or JSON; names must not clash with the chosen workflow's
model-release release the model files come from (default model-v1)

Outputs: result (JSON — {"workflow", "decisions": {name: {answer, confidence, type, mode, distribution}}}) and result-file. Each run also writes a table to the job summary.

The first run downloads ~1.2 GB of ONNX files, each checked against action/SHA256SUMS from the same tag as the code, then caches them (GitHub evicts caches unused for 7 days). A call took 2.6 s on a standard ubuntu-latest runner. Linux and macOS runners.

The PyTorch checkpoint is on the same release as jevlite-fp16.pt (298 MB, six pieces; join with cat, instructions in the release notes).


How it works

flowchart LR
    S["state<br/><i>log · ticket · trace · diff</i>"] --> SEQ
    Q["typed questions<br/><i>supplied per request</i>"] --> SEQ
    SEQ["one sequence<br/>state + &lt;&lt;q&gt;&gt; question + &lt;&lt;l&gt;&gt; label …"]
    SEQ --> ENC["ModernBERT-base<br/>150M · one forward pass"]
    ENC --> H["Linear(d, 1)<br/>read at every &lt;&lt;l&gt;&gt; marker"]
    H --> SM["softmax within<br/>each question"]
    SM --> T["per-type temperature"]
    T --> OUT["label + calibrated confidence<br/>for every question"]

    style ENC fill:#2a78d6,stroke:#1c5cab,color:#fff
    style OUT fill:#1baf7a,stroke:#158a60,color:#fff
    style SEQ fill:#f0efec,stroke:#c3c2b7,color:#0b0b0b
    style H fill:#f0efec,stroke:#c3c2b7,color:#0b0b0b
    style SM fill:#f0efec,stroke:#c3c2b7,color:#0b0b0b
    style T fill:#f0efec,stroke:#c3c2b7,color:#0b0b0b
Loading

The label-embedding head

The obvious design is one nn.Linear per question. It works, and it freezes the schema into the weights — add a question or rename a label and you retrain.

Instead, every candidate label is written into the input sequence behind a <<l>> marker token, and the head is a single Linear(d, 1) read at each marker. Softmax runs within each question's own label set.

[CLS] {state json} <<q>> How risky was this? <<l>> 0: Benign… <<l>> 1: Low… <<q>> …
                   ↑ question marker         ↑ one logit read off each of these
  • The schema is data at inference time — new questions, new labels, a whole new workflow, with no retraining and no new parameters.
  • Labels attend to the state and to each other: a cross-encoder, not a bi-encoder.
  • One encoder pass answers every question about a state.

Base encoder is ModernBERT-base (150M, bidirectional, 8192 context). A causal model works via --encoder, but pools differently — token 0 in a decoder-only model has only seen itself, so [CLS] pooling on one is a bug, not a shortcut.

Training on distributions, not labels

The dataset ships full annotator probability distributions, not just argmax labels. Training on the argmax throws that away and directly causes overconfidence: a model taught that a 55/45 case is true learns to say 0.95, then is wrong 45% of the time while claiming near-certainty.

So the loss is soft cross-entropy against the consensus distribution plus a Brier term — a proper scoring rule, which is what actually pushes probabilities toward honesty.

It overshoots in the safe direction. The reliability curve sits above the diagonal at every bin: because the consensus targets are often flat (0.43 / 0.28 / 0.25 on a contested case), the model learns to emit flat distributions, so its top-1 confidence reads lower than its real accuracy. It is underconfident rather than overconfident — the direction you want if you are going to route on it, and one that a single temperature corrects.

Reliability diagram: predicted confidence against observed accuracy, with every bin sitting above the diagonal, showing the model is underconfident across the whole range. Before and after per-type temperature scaling are almost identical.


Quick start

Colab (free T4, ~30 min, no API key)

Upload jevlite_colab.ipynb — the whole repo is embedded in it — set Runtime → T4 GPU, and Run all.

Local

pip install -r requirements.txt

python smoke_test.py --all      # validate all 1,600 rows, no GPU, ~2 min
python 01_ceiling.py            # baselines you have to beat, $0
python 02_train.py --epochs 20  # ~25 min on a T4
python 03_calibrate.py          # per-type temperature, CPU, seconds
python 04_eval.py               # official test split, vs Jev
python 07_ensemble.py           # blend with the frozen probe
python plots.py --outdir plots  # six charts
python 05_serve.py --serve      # POST /decide {"state": ..., "questions": ...}
python 06_export_onnx.py --quantize

Every script takes --limit N for a fast dry run, and --config <workflow> to train a single-domain specialist.

Files

typed_schema.py the three primitives; Question.noul/choice/score builders
td_data.py dataset loading (HTTP fallback), marker-token encoding, collate
model.py JevLite, grouped softmax, soft-CE + Brier loss
probe.py frozen-encoder + logistic-regression baseline
metrics.py ECE (two kinds), Brier, NLL, TVD, risk-coverage
plots.py six charts, rendered from the JSON each step writes
01_… → 07_… the pipeline
smoke_test.py correctness checks, no GPU needed
legacy/ the earlier fixed-head version, kept for comparison

What the runs actually proved

A frozen encoder + logistic regression scores 0.670 — beating the fine-tuned model's 0.6240. No training at all. This reproduces the independent Banking77 finding where the same baseline beat Jev by 10 points at 44× the speed. 01_ceiling.py runs it first so you know what you're up against. If your fine-tune can't clear it, ship the probe.

The accuracy gap was undertraining — up to a point. 6 epochs gave 0.568; 20 epochs with early stopping (best at epoch 9) gave 0.6195. Past that, training loss kept falling while validation flatlined. The binding constraint is 1,016 training cases, not model size or epochs — which is why reaching for a bigger encoder first is the wrong instinct here.

Two panels: training loss falling steadily from 1.40 to 0.93 across 15 epochs, and validation accuracy rising to a best of 0.653 at epoch 9 then flattening - the signature of memorisation rather than learning.

Left: validation accuracy against blend weight, peaking at w=0.60. Right: test accuracy per configuration - model 0.624, probe 0.670, ensemble 0.697, sharpened ensemble 0.697 at ECE 0.057 - against Jev's 0.727.

Post-hoc temperature scaling had three different outcomes, so don't generalise from one. A no-op on the undertrained model (temperatures ≈ 1.0, ECE slightly worse), a mild help on the overfit one, and decisive on the ensemble: ECE 0.156 → 0.057 with accuracy untouched. Mixing two disagreeing distributions flattens them, so the blend was badly underconfident; sharpening in probability space can't move the argmax, so it's free or nothing. Before sharpening, the ≥0.90 band held 0.5% of traffic; after, 25.4%.

Accuracy and calibration error split by question type: noul 0.737, choice 0.630, score 0.524, with all three types calibrated well below Jev's 0.144 reference line.

Single-annotator agreement is not a ceiling. It's 0.659 here — what one human rater scores against the consensus. A model that always picks the consensus argmax scores 1.000, and Jev already exceeds it. The real limit is that 15.6% of test decisions are near-ties (top two labels within 0.1), so the last few points of headline accuracy are mostly luck. Judge on calibration and coverage.


Limitations

The ensemble trades away the dynamic schema. The probe only answers questions it has training labels for. On an unseen question it returns uniform and the blend degrades to 0.6 × the neural model — so 0.6965 is the known-schema number and ~0.624 is the new-schema number. Three configurations, different trade-offs, no single best. Always say which one a number came from.

Jev's 0.727 is zero-shot. It never saw this dataset; this model trained on 1,016 in-domain cases from the same distribution and still came in lower. That makes the gap look worse, not better, and it's the accurate framing.

Small evaluation. 400 test cases / 2,000 decisions. Differences under ~0.02 are not meaningful.

Jev's figures are quoted, not reproduced here. Accuracy 0.727 and ECE 0.144 come from the typed-decisions benchmark; latency and pricing from the independent evaluations linked below.


Traps worth knowing if you fork this

  • 10% of the dataset is a question shape the first rows never show. Bare noul questions carry no criteria key at all. A 6-row smoke test passes and training then dies 300 rows in. smoke_test.py --all encodes every row for exactly this reason.
  • Workflows disagree on label width (16/17/18/20), so torch.cat across batches fails the moment a run spans more than one. Hence pad_cat.
  • Unbounded temperature scaling on soft targets is degenerate — the first fit returned T=400, which flattens everything to uniform, minimises NLL beautifully, and destroys the confidence signal. Bounded to [0.25, 5.0], and refused for any type with under 30 validation decisions.
  • ONNX export below opset 18 produces an invalid graph — torch emits Split with num_outputs regardless, and ORT then refuses to load it. Silent at export, fatal at load; hence the parity check.
  • ONNX weights land in a 596 MB .onnx.data sidecar. Serve both files or you load a graph with no weights. int8 (151 MB) shifts logits materially — recalibrate rather than reusing fp32 temperatures.
  • Never lower --max-len. States run 401–997 tokens (p50 588); 512 truncates 73% of them. Batches already pad to their own longest example, so the high cap costs nothing.

Credits

License

Apache-2.0.


Keywords: typed decision model · System One model · non-autoregressive classifier · TypeSafe Jev alternative · open source Jev · ModernBERT · structured output · calibrated confidence · LLM routing · agent observability · confidence thresholding · expected calibration error

About

Open reproduction of TypeSafe Jev: a 150M typed decision engine (noul/choice/score in one non-autoregressive pass, calibrated confidence). 0.697 vs Jev's 0.727, 2.5x better calibrated, 4x faster, free. Trains on a Colab T4 in 30 min.

Topics

Resources

Stars

50 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages