Fix schemas that check less than specified, and test that they don't by DZakh · Pull Request #185 · open-circle/schema-benchmarks

DZakh · 2026-07-26T15:33:33Z

Three libraries were benchmarked against schemas that checked less than specified, and nothing caught it: errorData breaks many constraints at once, so a library that misses one is still reported as invalid.

Benchmark corrections

ata-validator — dropped the parsing benchmark. validate(data) returns { valid, data, errors } where data is the input by reference, typed unknown, so it validates without producing a value — the same reason ajv has no parsing entry. Its only value-producing API, validateAndParse, takes a JSON string. It was ranking Add benchmarks for typia/compile #1 in parsing at ~5× the runner-up.
ata-validator — format: "url" never ran (no such format in the library, and unknown formats are silently ignored); now "uri". Validation moved from isValidObject to validate(...).valid, because instanceof comes from @ata-project/keywords, which only patches validate — the boolean API never checked created was a Date.
typia — tags constrained the array to 1–30 items instead of each tag to 1–30 characters, so no per-tag length was checked.
typebox — in Type.Script, with { minimum: 1, maximum: 100 } on number | null applied to neither member, so discount: 0 passed. Now parenthesised around number.

Consistency

stars and quantity had a minimum of 1 in a third of the libraries and 0 in the rest, so they weren't all validating the same schema. Now 0 everywhere — the majority spelling, and a product with none left or a zero-star rating is data the schema should accept. Touches arktype, decoders, typebox, typia, valibot and effect.

Tests

Two datasets in schemas/src/data.ts, asserted against every validation, parsing and standard schema config:

constraintViolations — 23 copies of successData each breaking exactly one constraint, all of which must be rejected
constraintBoundaries — 17 copies sitting exactly on a bound, all of which must be accepted, so a schema stricter than specified fails too

Values are chosen so coercion can't rescue them ("abc", not "252"), which keeps joi and yup honest without special-casing them.

Deviations a library provably cannot express live in constraintGaps, with a reason, and are asserted to still deviate — so an upstream fix retires the entry instead of hiding in it. One entry today: @ata-project/keywords doesn't recurse into arrays, so ata-validator can't check created is a Date inside images/ratings in any of its APIs.

3734 tests pass; typecheck and lint are clean.

Results

Download sizes are regenerated (raw bytes unchanged — "1" and "0" are the same length — gzip shifts a few bytes). Runtime results in bench/bench.json need a full re-run: I removed only the rows whose config no longer exists, and deliberately didn't re-measure, since numbers from my machine shouldn't be mixed with the committed set.

`schema.validate(data)` returns `{ valid, data, errors }` where `data` is the input by reference, typed `unknown` (`withKeywords()` returns `Validator`, so the inferred type is erased). It validates and reports errors without producing a value, which is what the parsing category measures - the same reason `ajv` has no parsing benchmark. Its only value-producing API, `validateAndParse`, takes a JSON string. Also switch the url field to `format: "uri"`. ata-validator has no "url" format, and unknown formats are silently ignored, so the constraint never ran in either benchmark while every other library validated it. Stale ata-validator rows are dropped from the committed parsing results; the remaining numbers are regenerated by CI.

Silently skipped checks are how a library ends up winning a benchmark it isn't comparable in, and nothing caught them: `errorData` breaks many constraints at once, so one unchecked constraint is still reported as invalid. Add `constraintViolations` - copies of `successData` that each break exactly one constraint - and assert every validation, parsing and standard schema config rejects all of them. Values avoid the disagreements between libraries (coercion, and minimums some set to 0 and others to 1) so a failure always means a missing check. Deviations a library provably cannot express go in `constraintGaps` with a reason, and are asserted to still deviate, so an upstream fix retires the entry instead of hiding in it. That found two schemas checking something other than what was specified: - typia constrained `tags` to 1-30 *items* rather than each tag to 1-30 characters, so no per-tag length was checked - typebox's `Type.Script` schemas attached `with { minimum: 1, maximum: 100 }` to `number | null`, where it applies to neither member, so `discount: 0` passed Both now match the other libraries. Also switch ata-validator's validation benchmark from `isValidObject` to `validate(...).valid`: the `instanceof` keyword comes from @ata-project/keywords, which only patches `validate`, so the boolean API never checked `created` was a Date. Its remaining gap - that keyword doesn't recurse into arrays - is documented in `constraintGaps`. Download sizes are updated for the changed schemas. Stale ata-validator validation rows are dropped from the committed results, since they measured a call the config no longer makes.

Two thirds of the libraries specified `stars` as 0-5 and `quantity` as 0-10, the rest as 1-5 and 1-10, so they weren't validating the same schema. 0 matches the majority - and a product with none left, or a rating of zero stars, is data the schema should accept. Updated arktype, decoders, typebox (object and script), typia and valibot for both fields, and effect (v3 and beta) for `quantity`. Add `constraintBoundaries` to lock it in: copies of `successData` sitting exactly on a bound, which every config must accept. A schema stricter than specified now fails the suite the same way a schema missing a check does. This is also what makes the violation set stricter - it no longer has to dodge the disagreement, so `stars: 0` and `quantity: 0` are boundaries rather than untestable. Download sizes are regenerated for the changed schemas (raw bytes are identical, gzip shifts by a few bytes).

netlify · 2026-07-26T15:33:39Z

✅ Deploy Preview for schemabenchmarks ready!

Name	Link
🔨 Latest commit	`7452179`
🔍 Latest deploy log	https://app.netlify.com/projects/schemabenchmarks/deploys/6a6629df00d3210008061768
😎 Deploy Preview	https://deploy-preview-185--schemabenchmarks.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 21fa6b2189

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-26T15:42:26Z

Regenerate results for the corrected benchmark configurations

The committed runtime data no longer matches the benchmark definitions consumed by the website: both ata-validator validation rows are deleted even though schemas/libraries/ata-validator/benchmarks.ts still defines validation using the new schema.validate(data).valid implementation, while retained ata-validator standard and typia/typebox rows still contain timings measured before their schemas gained additional checks. Publishing this file therefore omits an active validator and compares stale measurements from weaker schemas; rerun the runtime benchmarks after these changes rather than only removing the former parsing rows.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-26T15:42:26Z

+  "ratings: text too long": variant((data) => (data.ratings[0]!.text = "a".repeat(1001))),
+  "ratings: nested image malformed url": variant(
+    (data) => (data.ratings[0]!.images[0]!.url = "nope"),
+  ),
+};


Add cases for every constraint the suite claims to cover

The new exhaustive check omits many independently implemented constraints, including brand/description minimum lengths, image id/title/size, the ratings array type, rating id/title/minimum text length, and several nested-image fields. If any library drops one of those checks, every case here still passes even though that library benchmarks less work—the exact regression this matrix is intended to prevent—so each omitted field and bound needs its own single-violation or boundary variant.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-26T15:42:26Z

+    // ata-validator has no "url" format - an unknown format is silently ignored,
+    // so use "uri" to keep the check equivalent to the other libraries
+    url: t.string({ format: "uri" }),


Reject malformed URLs with a real URL check

When an image URL has a scheme prefix but is otherwise malformed, such as https: not a url, this schema still accepts it: ata-validator 1.2.1 implements its built-in uri format as only ^[a-zA-Z][a-zA-Z0-9+.-]*:. The new "nope" fixture does not expose this because it lacks a scheme, while the other URL implementations reject the malformed value, so ata-validator remains benchmarked against a weaker constraint; use a custom format/pattern with the intended URL semantics or record this as a genuine gap.

Useful? React with 👍 / 👎.

EskiMojo14 · 2026-07-26T15:57:29Z

this has some good catches, thank you! please make sure to follow the contributing guide wrt git etiquette going forward - i'll rebase it this time, as i have a few tweaks i want to add

EskiMojo14

it looks like i don't have permission to push to the branch, so these are my initial review thoughts - please also make sure to rebase and get rid of the merge commit :)

EskiMojo14 · 2026-07-26T16:07:51Z

+function variant(mutate: (data: ProductData) => void): unknown {
+  const data = structuredClone(successData);
+  mutate(data);
+  return data;
+}


instead of a full structuredClone, use mutative - we already have it as a dependency for website, just move it to the catalog and add it to schemas

Suggested change

function variant(mutate: (data: ProductData) => void): unknown {

const data = structuredClone(successData);

mutate(data);

return data;

}

function variant(mutate: (data: Draft<ProductData>) => void): unknown {

return create(successData, mutate)

}

EskiMojo14 · 2026-07-26T16:08:40Z

+ * which cast it to `"1"` by design.
+ */
+export const constraintViolations: Record<string, unknown> = {
+  "id: not a number": variant((data) => ((data as { id: unknown }).id = "abc")),


switch each of these so they don't implicitly return

Suggested change

"id: not a number": variant((data) => ((data as { id: unknown }).id = "abc")),

"id: not a number": variant((data) => {

(data as { id: unknown }).id = "abc"

})

EskiMojo14 · 2026-07-26T16:09:29Z

+ * constraint, its test fails and the entry (plus the workaround in its schema) should be removed.
+ */
+const constraintGaps: Record<string, Record<string, string>> = {
+  "ata-validator": {


if ata can't check Date and url properly, we may have to exclude it from benchmarking until it does - have you opened issues with them?

@mertcanaltin what do you think?

thanks for catches, this is right, I will solve this week

at once, I will publish 1.2.2 version for this bug, can you use this version?

nowly continue test, at once publish https://github.com/ata-core/ata-validator/actions/runs/30214793286/job/89826964330

release: https://github.com/ata-core/ata-validator/releases/tag/v1.2.2

publish action: https://github.com/ata-core/ata-validator/actions/runs/30215414858

I said this week, but it got resolved early sorry

main branch's ata-validator version has been updated (#189), so this PR should get it once rebased

DZakh · 2026-07-26T17:49:18Z

I'll think about what to do with ata and fix it during the week 🙏

I usually squash commits and don't really care about commit hygiene.

claude added 3 commits July 26, 2026 14:20

Merge branch 'main' into claude/ata-validator-parsing-cum6at

7452179

chatgpt-codex-connector Bot reviewed Jul 26, 2026

View reviewed changes

EskiMojo14 requested changes Jul 26, 2026

View reviewed changes

DZakh marked this pull request as draft July 26, 2026 17:50

Uh oh!

Conversation

DZakh commented Jul 26, 2026

Benchmark corrections

Consistency

Tests

Results

Uh oh!

netlify Bot commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for schemabenchmarks ready!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 26, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 26, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 26, 2026

Choose a reason for hiding this comment

Uh oh!

EskiMojo14 commented Jul 26, 2026

Uh oh!

EskiMojo14 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mertcanaltin Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DZakh commented Jul 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

netlify Bot commented Jul 26, 2026 •

edited

Loading

mertcanaltin Jul 26, 2026 •

edited

Loading