Add canonical `$id` fields to every Beacon v2 JSON/YAML schema

[M-casado]

Context
• Follow-up to the discussion in PR #232 (“consistent relative framework $ref in endpoints”).
• Root-cause analysis in Issue #206 shows that missing $id values break Biovalidator/AJV resolution.
• A working prototype exists in M-casado#1 (where I injected $ids automatically).

---

Problem statement

Beacon v2 schemas currently mix absolute and relative $refs without defining canonical $ids for each schema document.
JSON-Schema Draft 2020-12 (§8.2.1.1) recommends that every root schema provides an absolute $id, and validators such as AJV treat this $id as the base-URI for resolving further $refs.
Lack of $id values leads to:

• Broken resolution chains when model ↔ framework schemas reference each other.
• Inability to rely on Biovalidator, AJV and other off-the-shelf tooling.
• Hidden coupling to the main branch through hard-coded raw-GitHub URLs.

Proposal

1. Inject a canonical $id into every JSON and YAML schema under

   • framework/json/**, framework/src/**
   • models/json/**, models/src/**

2. Canonical URI format

   • Use https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/<branch>/<path-to-file>
   • MUST be main on release and the current branch for PR artefacts.
   • Fragments (#...) are not included in the $id itself.

3. Tooling

   • Adapt the idAddition.zip script (see PR Addition of "$id" to model and framework schemas M-casado/beacon-v2#1).
   • (Optional, but honestly... very recommended) Add a GitHub Actions workflow that:
     • Fails if any schema lacks $id
     • Fails on duplicate or non-absolute $ids
     • Checks if the 'repo owner' (e.g., ga4gh-beacon), the 'branch' (e.g., main), and the path to file match the expected. For example, to avoid that there's a fork or another branch, with the URI pointing to a different branch.

Expected benefits

• Full compatibility with Biovalidator, AJV ≥ 8.12, Swagger-UI and other tooling.
• Alignment with GA4GH standards such as VRS that already publish $idd schemas (see example: https://w3id.org/ga4gh/schema/vrs/2.0.0/json/Allele).
• Avoidance of branch-mixing bugs and 404s caused by raw-GitHub absolute $refs.
• Clearer provenance and easier schema versioning.

---

References

• PR consistent relative framework $ref in endpoints #232 (discussion on relative $refs and the need for $id)
• Issue Error when resolving schemas through Biovalidator #206 (Biovalidator report and deep dive)
• PR Addition of "$id" to model and framework schemas M-casado/beacon-v2#1 (working PoC + script)
• JSON-Schema Draft 2020-12, §8.2.1.1
• AJV docs: “$ref is resolved using the schema $id as base URI”

[redmitry]

HI all,

Using $id may complicate thing even more... In JSON Schema, el $id changes the context, so all relative $ref(s) must be resolved in this context. The thigs may be much more complex if we refer to inner parts of some schema.

IMHO we would better use w3id.org proxy to have something like:

"$id": "https://w3id.org/ga4gh/schema/beacon/2.0.0/json/framework/..." and use absolute references everywhere (to the w3id.org).

This pattern is already user in GA4GH VRS:

{
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "$id": "https://w3id.org/ga4gh/schema/vrs/2.0.0/json/Allele",
}

This way:

"$ref": "/ga4gh/schema/vrs/2.0.0/json/Expression" resolves to "https://w3id.org/ga4gh/schema/vrs/2.0.0/json/Expression" which points (redirects) to:

https://raw.githubusercontent.com/ga4gh/vrs/2.0.0/schema/vrs/json/Expression

Dmitry

[M-casado]

Hi @redmitry, I'm not following the argument against the use of $ids:

...all relative $ref(s) must be resolved in this context

That's indeed the idea: that if a relative reference is used, it resolves as a unit. Of course, if you're referencing an external schema that is not the one in $id, one would use an absolute reference.

The thigs may be much more complex if we refer to inner parts of some schema

The use of $id on a source JSON Schema does not affect at all any $ref that remains to be absolute. In other words, if you add the $id to a JSON schema, and within that same file you do an absolute $ref to the inside property of another JSON Schema, that works seamlessly.

If you add the $id to the target JSON Schema, it doesn't affect it either, because the validator fetches the nested property through the given URI+JSON-Path, not based on the inside $id. The only thing that would be modified is that a validator would store in cache the schema with the given $id. But as long as the one giving the $id uses the URI to retrieve the schema, it would work seamlessly as well, and... it would be a mistake not to, following JSON Schema standards.

"$id": "https://w3id.org/ga4gh/schema/beacon/2.0.0/json/framework/..." and use absolute references everywhere (to the w3id.org).

I don't see the added value of having an intermediate step, redirecting to w3id first, and then that redirecting to the source of the JSON Schema through another URI. In my opinion you would be simply obfuscating the URI of the JSON Schema that you create, and relying on an extra (unnecessary) step. You can do the same with GitHub raw... URIs, which give directly the source JSON Schema.

Either that, or I'm missing something. If so, do question my comment.

Furthermore, the best argument in favour of $ids being added without breaking anything is that I already did. And it works, without anything breaking down. If you check the PR in my fork, referenced above, I added programmatically the $id of every JSON Schema in this repo.

[M-casado]

Regarding the automation of the $id, $ref and such, see the script modify-ids.py that I recently made for that purpose within the FEGA repo.

It can easily be used in any other repo, including beacon-v2, giving the appropriate positional arguments. I'll soon implement the script through GitHub workflows so that it's used automatically when pointers need to change.

For example, changing all the pointers from main to v2.0.1 both in the data and schemas directories would be as simple as the following:

python3 scripts/py/modify-ids.py data schemas --branch main v2.0.1 -vv -w

Do bear in mind the idiosyncrasies of the script, which is built for the FEGA purpose, and thus has regex that match raw github URLs (e.g., https://raw.githubusercontent.com/M-casado/fega-metadata-schema/main/schemas/FEGA.cohort.json).