Tests for bundling

[jdesrosiers]

A while ago I released an implementation of the official bundling process and included a set of tests in JSON that could be adopted here if there is interest.

Here's an example:

{
  "description": "One external reference",
  "schema": {
    "$id": "https://bundler.hyperjump.io/main",
    "$schema": "https://json-schema.org/draft/2020-12/schema",

    "type": "object",
    "properties": {
      "foo": { "$ref": "/string" }
    }
  },
  "externalSchemas": [
    {
      "$id": "https://bundler.hyperjump.io/string",
      "$schema": "https://json-schema.org/draft/2020-12/schema",

      "type": "string"
    }
  ],
  "bundledSchema": {
    "$id": "https://bundler.hyperjump.io/main",
    "$schema": "https://json-schema.org/draft/2020-12/schema",

    "type": "object",
    "properties": {
      "foo": { "$ref": "/string" }
    },

    "$defs": {
      "https://bundler.hyperjump.io/string": {
        "$id": "https://bundler.hyperjump.io/string",

        "type": "string"
      }
    }
  }
}

Notes/Comments/Rationale:

• I think this structure is pretty self explanatory, but please ask if anything isn't clear.
• In hindsight, it might make more sense for "externalSchemas" to be an object rather than an array. The keys would be retrieval URIs and the values would be schemas. This would allow for tests where the retrieval URI and the $id don't match, or where the $id is relative, or where there is no $id.
• Some of my tests might be a bit implementation specific, so I'm not sure the should all be here. For example, I have an option that allows the bundler to skip certain schemas that you want to remain external. That's not necessarily a feature everyone needs to support.

[Relequestual]

In theory, I don't think the bundling section actaully adds any requirements, but lays them out clearly in relation to processing requirements and the handling of "schema resources".

Would any such tests be testing something not already covered by these? https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/main/tests/draft2020-12/id.json

[jdesrosiers]

@Relequestual Have a closer look at that example I posted. It's not testing the validation behavior of a bundled schema. It's testing the bundling process itself. Given a "schema" and a set of available "externalSchemas", the "bundledSchema" is the expected result of the bundling process.

In theory, I don't think the bundling section actaully adds any requirements, but lays them out clearly in relation to processing requirements and the handling of "schema resources".

100% agree.

Would any such tests be testing something not already covered by these? https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/main/tests/draft2020-12/id.json

No. The tests I proposed adding wouldn't cover validation at all, just the bundling process. Whether an implementation can validate instances using the bundled schema is a separate concern covered by the normal test suite that you link to.

[Relequestual]

Have a closer look at that example I posted. It's not testing the validation behavior of a bundled schema. It's testing the bundling process itself.

Yeah... I don't know how I missed that. Looks like I hardly read it.
(I blame sleep deprevation.)

The tests I proposed adding wouldn't cover validation at all, just the bundling process. Whether an implementation can validate instances using the bundled schema is a separate concern covered by the normal test suite that you link to.

OK. Great.
Worth remembering that the keys in $defs can be anything unique and don't have to be the schema resource's $id. Might already have that in mind, IDK.

[gregsdennis]

FWIW, mine results in this:

{
  "$id": "https://bundler.hyperjump.io/main(bundled)",
  "$defs": {
    "0708a427dd": {
      "$id": "https://bundler.hyperjump.io/main",
      "$schema": "https://json-schema.org/draft/2020-12/schema",
      "type": "object",
      "properties": {
        "foo": {
          "$ref": "/string"
        }
      }
    },
    "7082beaf59": {
      "$id": "https://bundler.hyperjump.io/string",
      "$schema": "https://json-schema.org/draft/2020-12/schema",
      "type": "string"
    }
  },
  "$ref": "https://bundler.hyperjump.io/main"
}

This creates a wholly new schema that encapsulates the two inputs.

The original post has a problem in that it maintains the original $id value, meaning that the original schema and the bundled schema have the same $id, which is forbidden since they're technically different schemas.

[jdesrosiers]

Worth remembering that the keys in $defs can be anything unique and don't have to be the schema resource's $id.

Yep. I have an option that let's users change the naming strategy to use UUIDs. I actually don't have tests for that because it's non-deterministic. I figured that people would probably at least have an option to support the recommended naming strategy, so they could run the tests in that mode. There are going to be lots of implementation specific modes people could invent.

FWIW, mine results in this:

Yeah, there are multiple valid ways to construct a bundle. I support two different modes plus a few options that could further effect the result. I said earlier that these tests don't validate anything, but maybe they should. The one sure way to know if a bundle is correct is if it produces the same output as its unbundled equivalent. If the tests were designed using this principle, it wouldn't matter what strategies or options an implementation chooses to use.

[jdesrosiers]

The original post has a problem in that it maintains the original $id value, meaning that the original schema and the bundled schema have the same $id, which is forbidden since they're technically different schemas.

I disagree. The way I see it, $ids identify a schema resource, which doesn't include embedded schemas. So whether the schema includes embedded schemas or not, it's still the same schema resource. In any case, there's no reason for someone to load all the constituent parts and the bundle at the same time, so the unbundled version would never conflict with the bundled version. So, I don't see a problem.

[Relequestual]

I'm actually not sure you CAN change the $id value in this situation. You may have references in the root schema resource which are relative URIs. If you're changing the root resource's $id value, those relative references may no longer work.

Maybe we need to add somethint to the specification to make this clearer.

[gregsdennis]

$ids identify a schema resource, which doesn't include embedded schemas. - @jdesrosiers

But a schema resource does include keys.

In your example, you've added $defs, which is changing the schema. You therefore have a different schema.

If your schema already had $defs, then you're adding to it, which is also changing the schema, and you again have a different schema.

You can't give it the same $id because that would be attempting to identify two schemas with a single $id.

Yes, you get the same validation outcome, but two schemas with different keywords or different keyword contents are different schemas. { "type": "integer" } is a different schema from { "type": "integer", "$defs": { "foo": { "$id": "foo-obj" } }. They're semantically equivalent, but they are not the same thing.

In any case, there's no reason for someone to load all the constituent parts and the bundle at the same time, so the unbundled version would never conflict with the bundled version.

You can't make assumption about what our users will do.

If I want to bundle all of the 2020-12 meta-schemas into a single schema, I can't register that under https://json-schema.org/draft/2020-12/schema. It would overwrite (or fail to overwrite) the actual meta-schema. Anyone who then wants to get the meta-schema would get the bundled one instead, which is incorrect.

The bundled schema needs a new $id.

I'm actually not sure you CAN change the $id value in this situation. You may have references in the root schema resource which are relative URIs. - @Relequestual

Look at my example. All of the $ids and $refs stay the same, and all of the relative references still work.

The only way this might break is if you had references that were just pointer fragments, and the implementation didn't handle changing the base URI right.

For example:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://example.com/main",
  "type": "object",
  "properties": {
    "foo": {
      "$ref": "/string"
    },
    "nested": { "$ref": "#" }
  }
}

For /properties/nested, if the implementation isn't changing the base URI when it bundles this schema (the way I do it), then the # may resolve to the bundled root. This would affect any bundled schema.

But the problem is a bug in how the implementation manages the base URI while evaluating, not how bundling works.

[jdesrosiers]

In your example, you've added $defs, which is changing the schema. You therefore have a different schema. [...] They're semantically equivalent, but they are not the same thing.

That's a fair point. I agree. They are equivalent, but not entirely the same.

If I want to bundle all of the 2020-12 meta-schemas into a single schema, I can't register that under https://json-schema.org/draft/2020-12/schema. It would overwrite (or fail to overwrite) the actual meta-schema. Anyone who then wants to get the meta-schema would get the bundled one instead, which is incorrect.

That sounds like the desired behavior to me. If those schemas already exist, loading the bundle should either fail or overwrite the existing schemas depending on how the implementation handles duplicate identifiers.

[gregsdennis]

That sounds like the desired behavior to me.

If I ask for X, I should get X. This is expected behavior.

Getting X-with-all-of-its-references is unexpected and surprising behavior.

[Relequestual]

...

The bundled schema needs a new $id.

I'm actually not sure you CAN change the $id value in this situation. You may have references in the root schema resource which are relative URIs. - @Relequestual

Look at my example. All of the $ids and $refs stay the same, and all of the relative references still work.

I can see that. I hadn't conisdered bundling the previous root entity too to maintain the given $id identifier.

The only way this might break is if you had references that were just pointer fragments, and the implementation didn't handle changing the base URI right.

For example:
...
But the problem is a bug in how the implementation manages the base URI while evaluating, not how bundling works.

Agreed. But this isn't what I meant when I said "You may have references in the root schema resource which are relative URIs." It may have been helpful if I had provided an example, but I didn't consider you bundling the primary resource too. I don't think this is an unreasonable assumption...

The bundling process for creating a Compound Schema Document is defined as taking references (such as "$ref") to an external Schema Resource and embedding the referenced Schema Resources within the referring document. Bundling SHOULD be done in such a way that all URIs (used for referencing) in the base document and any referenced/embedded documents do not require altering. - https://json-schema.org/draft/2020-12/json-schema-core#section-9.3.1-1

The bundling process, you bundle the referenced resources, not the primary one too. (Although, I actually think your solution is quite smart and possibly even preferable.)

Consider, if I were to take your example and modify it slightly...

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://example.com/main",
  "type": "object",
  "properties": {
    "foo": {
      "$ref": "/string"
    },
    "nested": { "$ref": "https://example.com/main" }
  }
}

Bundling without also embedding the root resource would break the internal reference. Although you could argue that the URI was already registered by the unbundled version. (Although an implementation which only supports referencing URIs defined in the same document, might not support such a registry.)

As I said, I think your solution @gregsdennis to also bundling the root resource is smart and possibly preferable, however I think it does fall foul of the following...

A Bundled Schema Resource MUST NOT be bundled by replacing the schema object from which it was referenced, or by wrapping the Schema Resource in other applicator keywords. - https://json-schema.org/draft/2020-12/json-schema-core#section-9.3.1-5

We COULD change that, and even promote your approach to bundling, but I think that would be a spec change. I am leaning towards being in favour of this.

However, I make note of the following, and some considerations...

In order to produce identical output, references in the containing schema document to the previously external Schema Resources MUST NOT be changed, and now resolve to a schema using the "$id" of an embedded Schema Resource. Such identical output includes validation evaluation and URIs or paths used in resulting annotations or errors. - https://json-schema.org/draft/2020-12/json-schema-core#section-9.3.1-6

The intent, at least from me, was to be able to use the bundled and unbundled version of a schema interchangibly. Maybe only in memory, maybe as some sort of compile time assistance. Reason doesn't matter. A user or implementation should not care or even know any difference between the two.

When processing a bundle (Compound Schema Document), you have to (I think) start with a fresh schema registry. When initially processing and indexing a schema and its references, you'll come across the other schema resources which will later be bundled. When later processing a bundled version of the schema, now with embedded schema resources, you'll encounter the same URIs. They are of course identical. Do you validate that they are identical, or else raise an error? Maybe you do, which negates this point.

---

Re...

That sounds like the desired behavior to me.

If I ask for X, I should get X. This is expected behavior.
Getting X-with-all-of-its-references is unexpected and surprising behavior.

I refer back to my earlier mentioned intent, that users should be unaware of a bundled vs non-bundled version of any given schema resource. It should be transparant. (Much like URI normalization or encoding... multiple physical strings resolve to logically the same thing.)

I think this was one of the reasons for choosing a specific and distinct name for source a document.

To be clear, by expressing intent, I'm not trying to lock down discussion or direction of discussion. I think we've potentially found an intersting and different than intended approach which might be preferable. (Assuming we aren't hung up on the "identical output" clause: https://json-schema.org/draft/2020-12/json-schema-core#section-9.3.1-6 - I am not.)

[jdesrosiers]

The intent, at least from me, was to be able to use the bundled and unbundled version of a schema interchangibly.

That was my intent as well. I think it's likely to have a development environment where the schemas are unbundled for ease of development and debugging, but bundled in production. I think guaranteeing that the behavior is identical in both environments is a property people will appreciate.

Greg's approach can still be used and produce identical output, but it depends on how you use that bundle. If you validate an instance directly against the bundled schema, the evaluation path will be different because of the added $ref to point to the main schema. But, if you just use the bundle only as a way of loading multiple schemas at once, you can validate directly against the main schema. Here's what it would look like in my implementation.

// Loads schemas /main(bundled), /main, and /string
addSchema({
  "$id": "https://bundler.hyperjump.io/main(bundled)",
  ...
});

// Produces identical output
const output = validate("https://bundler.hyperjump.io/main", { "foo", 42 });

// Produces different output
const output = validate("https://bundler.hyperjump.io/main(bundled)", { "foo", 42 });

The first will produce identical output to if you loaded the schemas individually, while the second would differ in evaluation path.