Skip to content

Latest commit

 

History

History
37 lines (32 loc) · 2.11 KB

drafts.md

File metadata and controls

37 lines (32 loc) · 2.11 KB

Historical demo post drafts

These drafts were kept in LowRank's old content plan. Their dates were planned dates; publication has not been verified.

Testing the scoring code (planned October 26, 2026)

My eval said Claude Haiku scored 72.8%. The real number was 76.1%.

Nothing was wrong with Haiku. My own scoring code cost it 3.3 points. The two big bugs:

  1. I lowercased every answer before comparing. One label in the dataset has a capital letter: "Refund_not_showing_up". Haiku got it right 35 times out of 40 and was marked wrong every time.
  2. I capped answers at 20 tokens. Claude's tokenizer needs more than that for long labels, so "balance_not_updated_after_cheque_or_cash_deposit" was cut in half, 61 times.

What gave it away: four categories at exactly 0%. A real model is rarely perfectly wrong. A broken scorer often is.

What I do now before trusting any number: → Read 20–30 raw outputs, not just the score → Test the scorer on hand-made cases, including the awkward labels (thanks Vijay Kumar for putting it well) → Treat any 0% or 100% category as a bug until proven otherwise

Most "the model is bad" findings I've seen in evals turn out to be scoring bugs. Has that happened to you?

Hidden reasoning tokens (planned October 28, 2026)

Claude Haiku returned an empty answer. Not a wrong label: nothing at all.

The message was ambiguous: "How do I locate my card?" Haiku, by default, reasons before it answers. It spent 128 tokens thinking, hit my output limit, and the response came back with no text.

I raised the limit to 200. Still empty. It kept thinking.

With reasoning turned off, it answered in 7 tokens: card_arrival.

Two lessons:

  1. Reasoning defaults can silently break classifiers. An empty answer looks like a model failure, but it's a configuration problem.
  2. They cost money. Hidden reasoning tokens are billed as output tokens. For a one-word label, that can mean paying for 10–20× more output than you need.

If you run a high-volume classifier on a reasoning-capable model, check what it's doing by default. It may be thinking on every request.