🧹 Clarification: usage of non-canonical identifiers in `$schema`

[karenetheridge]

What is unclear?

Is it legal to use a non-canonical identifier in a $schema keyword to refer to a metaschema?

e.g. if I have this document:

{
  "$id": "https://example.com/some_document",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$defs": {
    "my_embedded_metaschema": {
      "$id": "https://example.com/my_embedded_metaschema",
      "$vocabulary": {
        ...
      }
    },
    ...
  }
}

..and use the embedded metaschema in another document, can I reference it using a non-canonical identifier?

{
  "$id": "https://example.com/another_document",
  "$schema": "https://example.com/some_document#/$defs/my_embedded_metaschema",
  ...
}

I would think this is foolish, and might break some implementations (if using the $schema value directly as a lookup in a list of known metaschemas), but it doesn't seem to be strictly disallowed.

edit: I found https://json-schema.org/draft/2020-12/json-schema-core#section-8.1.2-6:

"The "$vocabulary" keyword SHOULD be used in the root schema of any schema document intended for use as a meta-schema. It MUST NOT appear in subschemas."

However, I don't think the $vocabulary keyword is strictly required in a metaschema (or perhaps this is the point that needs clarification?) -- as some metaschemas can be simply a combination of other schemas via an allOf.

Proposal

I can't see a good reason to allow this, so I would recommend making it explictly prohibited.

Do you think this work might require an [Architectural Decision Record (ADR)]? (significant or noteworthy)

Yes

[gregsdennis]

We've removed the allowance of fragments in $id. It makes sense to disallow them in $schema as well if we haven't already.

My implementation would reject this saying that it doesn't recognize that meta-schema.

[gregsdennis]

Section 8.1.1:

The value of this keyword MUST be a URI [RFC3986] (containing a scheme) and this URI MUST be normalized.

Normalization is discussed in RFC 3986 section 6.2.1 (multiple subsections, so I won't quote them), which discusses URI equivalence. The gist is that normalization covers casing, character encoding, path adjustments, trailing slashes, etc. I didn't see anything about fragments during a brief scan. Normalization is not defined outside of equivalence.

I'd say that https://example.com/my_embedded_metaschema is not equivalent to https://example.com/some_document#/$defs/my_embedded_metaschema by these rules because the URIs themselves are not equivalent, even though they resolve to the same data.

[karenetheridge]

Other interesting gaps:

• nothing specifies that a metaschema need have a $schema keyword (this seems acceptable, although not recommended, as we can infer the metaschema from the implementation's default spec version, or from the parent schema if it is not at the document root)

• nothing specifies that a metaschema need be a schema resource (i.e. have an $id keyword) -- this means we can even refer into the middle of a properties or allOf keyword and use that as a metaschema - bizarre, but legal, and I see no reason to allow this

I'd say that https://example.com/my_embedded_metaschema is not equivalent to https://example.com/some_document#/$defs/my_embedded_metaschema by these rules because the URIs themselves are not equivalent, even though they resolve to the same data.

If this was the $ref keyword, that statement would be false, as a schema can be identified with more than one uri, but we do not explicitly state if this is the case for $schema as well, or if its $id is an explicit part of the metaschema (and therefore using a non-canonical uri to reference it would not mean the same thing). We should be explicit about this -- the easiest thing to do is to say that the $schema keyword may only use the canonical uri for a resource.

[gregsdennis]

If this was the $ref keyword, that statement would be false

$ref is different because it's attempting to resolve a schema, and it can do so using fragments. $schema is identifying only, not resolving, so I don't think the comparison exactly applies.

a schema can be identified with more than one uri

Pedantically, it can't be identified with a URI that has a fragment (anymore) because we removed fragments from $id. It can be resolved with one, but not identified.

But, yes, this needs to be cleaned up.

[karenetheridge]

a schema can be identified with more than one uri

Pedantically, it can't be identified with a URI that has a fragment (anymore) because we removed fragments from $id. It can be resolved with one, but not identified.

I mean that it can be referred to by more than one uri (each of which could have a fragment), not identified by itself as more than one uri (with the $id or $anchor keyword).

Right now, we can't say "the metaschema can only be referred to (via the $schema keyword) using the identifier that the metaschema self-identifies as (via the $id keyword) because we do not require that a metaschema have an $id keyword.

Summarizing, I think we should explicitly say this:

• a metaschema (a schema resource that another schema may wish to reference in a $schema keyword) MUST have an $id keyword
• the uri value of a $schema keyword MUST be the canonical uri for that schema resource (this will disallow the use of anchors or json pointers to refer to the metaschema)
• a metaschema need not have a $vocabulary keyword (no change from today)
• a metaschema need not have a $schema keyword (no change from today)
• a $vocabulary keyword may appear elsewhere in a document than at the document root (this is a change from today - see Core §8.1.2-6, which forces this keyword to be at the document root), as this enables schema bundling
• a metaschema MAY exist as a schema contained by another schema (i.e. not at the document root), as this enables bundling (no change from today)
• a schema MUST NOT define its metaschema to be a schema that is contained within itself (as this causes problems with parsing, as a containing schema cannot be parsed without first parsing the document that contains it, and the parsing semantics are defined by that metaschema)
• a document should not (must not?) contain a schema whose metaschema is defined in the same document (e.g. siblings -- same problem with parsing)

[gregsdennis]

a metaschema need not have a $schema keyword (no change from today)

This one concerns me. Surely a meta-schema needs to trace back to one of the meta-schemas we publish. At the very least, it needs to know under what rules to process the schema... unless you want to bring back a previous discussion we (JSON Schema) have had (somewhere) about the separation between processing semantics and dialect. (I believe that discussion was never resolved, leaving things as-is.)

a $vocabulary keyword may appear elsewhere in a document than at the document root...

I would add that it still needs to be at the resource root, not buried somewhere.

[karenetheridge]

This one concerns me. Surely a meta-schema needs to trace back to one of the meta-schemas we publish.

In this case the implementation would have to fall back to a stated default (likely the latest draft that it supports), or accept an out of band value passed in, so we should at the very least give a very strong recommendation to always include the $schema keyword for the sake of interoperability.

I would add that it still needs to be at the resource root, not buried somewhere.

Yes absolutely, we would need to clarify the $vocabulary keyword needs to be at the root of the metaschema, rather than somewhere in the middle (not even at a resource root lower down).

[jdesrosiers]

Is it legal to use a non-canonical identifier in a $schema keyword to refer to a metaschema?

The semantics for non-canonical identifiers is effectively implementation defined. Therefore, I think you can't practically use a non-canonical identifier for $schema despite it not being forbidden to do so. An implementation might do something with it, but it's not clear what it might do. So, I think it's clear that this is not something implementations should support, but it's not clear what an implementation should do if it does support it. It's probably better to simply not allow it.

However, I don't think that necessarily means we should disallow fragments in $schema. There's one use case I can think of that could be pretty nice to have. It would be nice to be able to use https://json-schema.org/draft/2020-12/schema#strict that just uses the 2020-12 dialect, but uses unevaluatedProperties: false.

{
  "$schema": "https://json-schema.org/draft/2020-12",
  "$id": "https://json-schema.org/draft/2020-12",
  "$vocabulary": {...},

  "$ref": "#/$defs/meta-schema",

  "$defs": {
    "meta-schema": { ... },
    "strict": {
      "$anchor": "strict",
      "$ref": "#/$defs/meta-schema",
      "unevaluateProperties": false
    }
  }
}

{
  "$schema": "https://json-schema.org/draft/2020-12#strict",
  ...
}

I think this is entirely consistent with the spec today, although I'd be surprise if any implementations supported it. (Mine doesn't. It strips and ignores fragments.)

It should work because despite it pointing to a subschema, the implementation should be looking at the root of the schema resource for $vocabulary to determine the dialect. Then it can use the subschema for evaluation. Based on the current spec, I think this should technically be required behavior. I think that's probably a good thing and we should add tests to ensure it gets implemented.

$schema does two distinct things. It identifies the dialect (semantics) and identifies a schema that describes the schema (structure). Because $vocabulary is always found at the root, that effectively means that the fragment is meaningless in dialect identification. It can safely be ignored. But, for the purpose of identifying a schema that describes the schema, the fragment does have meaning. Unlike dialect determination, evaluation can start in a subschema.

[jdesrosiers]

My thoughts on Karen's list

• a metaschema (a schema resource that another schema may wish to reference in a $schema keyword) MUST have an $id keyword

I don't think this is necessary, although I wouldn't opposed to "SHOULD".

• the uri value of a $schema keyword MUST be the canonical uri for that schema resource (this will disallow the use of anchors or json pointers to refer to the metaschema)

I feel like this is just repeating what we already say about non-canonical URIs, but it might be helpful to add a quick note and a link. However, as I pointed out in the previous comment, this does not disallow all fragments because some use of fragments are canonical.

• a $vocabulary keyword may appear elsewhere in a document than at the document root (this is a change from today - see Core §8.1.2-6, which forces this keyword to be at the document root), as this enables schema bundling

I believe the intention has always been that $vocabulary should appear in the root of any schema resource (not schema document) intended to be used as a meta-schema. It definitely shouldn't appear anywhere other than a schema resource root (which the spec doesn't actually say). I think this is just sloppy wording in the spec.

[notEthan]

We've removed the allowance of fragments in $id. It makes sense to disallow them in $schema as well if we haven't already.

(Previously raised in #1059 and #1292)

Given that

• $schema is used to select among different specifications of JSON Schema
• published specifications do use a fragment in $schema (albeit an empty fragment), up to draft 7 (they never should have, but no undoing that now)

I don't think $schema can be forbidden to include at least an empty fragment, or to use that non-canonical form.

Changing rules for handling $schema in one specification is questionable, when $schema's nature is across various specifications. Having rules for $schema that are effective across all published versions past and foreseeable future seems desirable, but I'm not sure how that would really work.

Back to fragments: my own implementation resolves $schema exactly the same as it resolves $ref, and fragments are unproblematic - the empty fragment past meta-schemas have used is resolved as an empty pointer. Non-empty pointers resolve as well, and anchors work too.

I would be opposed to forbidding all fragments, as that doesn't work for past specifications. Forbidding non-empty fragments is probably good, though fragment resolution is workable (works for me anyway) it's not something people can expect to be interoperable.

---

@karenetheridge

Summarizing, I think we should explicitly say this:

• a metaschema (a schema resource that another schema may wish to reference in a $schema keyword) MUST have an $id keyword

• a metaschema need not have a $schema keyword (no change from today)

These two seem subtly contradictory. The first supposes that $schema is the only expected way a schema references its metaschema. The second supposes a metaschema (presumably described by a metaschema other than itself) may have some other way to to reference its metaschema, something implementation-defined.

I think a meta-schema SHOULD have an $id; indeed it must if other schemas are to reference it by $schema and $schema is forbidden from using a (non-empty) fragment. But as implementations may have a mechanism other than $schema to indicate a schema's meta-schema, I think it's only a SHOULD that a meta-schema has an $id, and might also be a SHOULD that a meta-schema uses $schema.

---

(I go a bit off into the weeds below, feel free to ignore...)

---

@jdesrosiers

There's one use case I can think of that could be pretty nice to have. It would be nice to be able to use https://json-schema.org/draft/2020-12/schema#strict that just uses the 2020-12 dialect, but uses unevaluatedProperties: false.

I tried putting this together and couldn't get it to work out because the /$defs/strict schema doesn't have "$dynamicAnchor": "meta" so any "$dynamicRef": "#meta" goes to the non-strict metaschema. Adding "$dynamicRef": "#meta" to strict doesn't work because it's in the same resource as the non-strict meta-schema, and has to be for the strict anchor to resolve. It might be possible but seems unnecessarily convoluted compared to just putting an $id on the strict meta-schema.

---

@karenetheridge

a schema MUST NOT define its metaschema to be a schema that is contained within itself (as this causes problems with parsing, as a containing schema cannot be parsed without first parsing the document that contains it, and the parsing semantics are defined by that metaschema)

I don't suggest this should be a thing JSON Schema enables, but just note as a curiosity, Google's API discovery document has a configuration in which one document (defining the API discovery service) is described by a schema within that document (at /schemas/RestDescription) and that schema is described by a meta-schema at /schemas/JsonSchema (which describes itself as well, and other schemas in the document). My tooling does implement support for this strange configuration - bootstrapping it is tricky (though bootstrapping 2020-12's multi-document configuration, where every piece of the meta-schema describes itself and all the other pieces, is substantially trickier).

[jdesrosiers]

I tried putting this together and couldn't get it to work out

Good point! We'd need to give it an $id for it to work. In which case, we might as well just make it https://json-schema.org/draft/2020-12/schema-strict instead of https://json-schema.org/draft/2020-12/schema#strict.

However, I think the point still stands. Just because this example didn't make sense, doesn't mean there isn't some other example that does. The question is, are we sure we want to close that door? I can't think of a use right now, but I also don't see any reason to forbid it.

My tooling does implement support for this strange configuration

The problem with schema documents including their meta-schema is that you can't find embedded schemas until you know the dialect and you can't know the dialect until you find embedded schemas. You can't process the embedded schemas before you know the dialect because the dialect determines whether you need to look for id or $id in definitions or $defs. It can be done in a lot of cases using deduction, but that's complicated to implement and not something we want to require.

It sounds like the Google example wasn't a schema that contains it's meta-schema, but a JSON document that separately contains a schema and it's meta-schema. If I understand that correct, I don't think that's the same thing that Karen is talking about.

[gregsdennis]

you can't find embedded schemas until you know the dialect

Over all of the drafts (at least since draft 4, use of earlier specs is a rare find), $schema hasn't really changed. I think at this point it's safe to say we understand what $schema does.

This gets into the "we can't define what $schema does in a JSON document" problem a little, but supposing we could define that as "a processor that wants to validate an instance can identify the applicable schema from the $schema property", it follows that a schema could define its own meta-schema, even if that meta-schema is self-contained.

My point is that I don't think that not knowing the dialect because you don't know how to interpret $schema (or to interpret it at all) is a valid argument. We know what $schema means because it hasn't ever changed, and we have no plans to change it.