Add a JSON Schema benchmark case - #186
Conversation
✅ Deploy Preview for schemabenchmarks ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ab4e4becc7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ab4e4be to
fddebb6
Compare
|
my initial thoughts on this are that we should either call it Standard JSON Schema and only test schemas that actually implement the standard (no workarounds using custom adapters etc), or just test JSON Schema conversion without the requirement for the standard the average user is not monkey patching their schemas to add Standard JSON Schema support, so we shouldn't either this is the same reason we don't include TypeBox in our Standard Schema benchmarking - they provide an example of how you'd get it to work, but no official API |
|
Yeah makes sense - I think general JSON Schema section with to/from API would be the best. And standard schema should be an additional column there. We can also add test suite compatability check TypeBox author suggests 👍 |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 149bcf5630
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
0a35432 to
b8154ab
Compare
DZakh
changed the title
Benchmarks generating a JSON schema from a schema, through whichever API
each library provides for it, on a new /json-schema page.
The subject is a small schema with a codec, so the input and output types
differ and both are worth generating:
z.object({
id: z.number(),
name: z.string(),
price: z.codec(z.string(), z.number(), { decode: Number, encode: String }),
})
Every target (draft 2020-12, draft 07, OpenAPI 3.0) and direction is
attempted. A library is expected to throw for anything it can't convert, so
those combinations are recorded as unsupported - with the library's own
reason - and shown in a support matrix instead of being benchmarked.
Thirteen libraries can produce a JSON schema, and two columns say how:
- source: where the schema comes from - converted at runtime, generated at
build time (typia), or nothing to convert because the library's schemas are
JSON Schema already (ajv, ata-validator, typebox). The last two are
constants, so they're reported as 0 rather than timed - benchmarking a
property read would measure the harness.
- standard JSON schema: whether the interface is implemented natively, behind
an opt in, by a separate package, or not at all. A library claiming support
has to provide the schema exposing `~standard.jsonSchema`, and it's asserted
to convert exactly what the library's own API converts, so the column can't
drift from reality.
What each library generates is checked two ways: the schema has to accept the
data of its own direction and reject the other's, and it has to match one of
the schemas in test/accepted-json-schemas.ts. Libraries disagree on how to
describe the same type, so rather than normalising, each accepted shape is
listed with the libraries that generate it - a new one fails the test until
it's looked at. `pnpm run gen:json-schemas` regenerates that list from what
the libraries currently produce, so the diff is what gets reviewed.
Adds joi-to-json and zod-to-json-schema, alongside @valibot/to-json-schema.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WLojRJg5z3PaekFfJSRvst
b8154ab to
c2b7ea1
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a07507d6b6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (direction === "output") { | ||
| throw new Error("No JSON schema can be generated for the output type"); | ||
| } | ||
| return parse(jsonSchemaSubject, getJsonSchemaMode(target)) as object; |
There was a problem hiding this comment.
With the Joi 18.2.3 build checked into this revision, this branch benchmarks the separate joi-to-json package and consequently records Standard JSON Schema support as none, rejects the native draft-2020-12 target, and omits output results. However, schemas/libraries/joi/download_compiled/unminified.js:539-542 exposes native schema["~standard"].jsonSchema.input and .output methods. Fresh evidence in this revision is the newly added parse(...) configuration here, so use Joi's native interface and a subject that exercises its input/output conversion instead; otherwise the support matrix and timings describe the adapter rather than Joi.
Useful? React with 👍 / 👎.
|
@EskiMojo14 I think this is good for now.
But these can be done separately. And I'm out of Claude this week 😰 https://x.com/dzakh_dev/status/2082024245417173175 |
3 participants
Benchmarks generating a JSON schema from a schema, through whichever API each library provides for it, on a new
/json-schemapage.Standard JSON Schema is a column here, not the case itself — most libraries that can emit a JSON schema don't implement the interface, and leaving them out would have made the page look far emptier than reality.
What's measured
{ id: number, name: string, price: string ↔ number }inputdescribespriceas a string,outputas a numberDate— it has no JSON schema representation, andsuryandeffectrefuse to convert it at alldraft-2020-12,draft-07,openapi-3.0Which libraries, and how
Thirteen libraries can produce a JSON schema. Two columns say how, both validated enums rather than free text:
Source — where the schema comes from:
arktype,effect,effect@beta,joi,sury,valibot,zod,zod/mini,zod/v3typiaajv,ata-validator,typeboxThe last four aren't timed — reading a property or a build-time constant would measure the harness, not the library — so they report
0, and the source column says why.Standard JSON Schema — how
~standard.jsonSchemais provided:zod,arktypesury, viaS.enableStandardJSONSchema()valibot, via@valibot/to-json-schemaA library claiming any support has to hand over the schema exposing the interface, and a test asserts it converts exactly what the library's own API converts — so the column can't quietly drift from reality.
New dependencies:
joi-to-jsonandzod-to-json-schema, alongside@valibot/to-json-schema.Page