Error when resolving schemas through Biovalidator

[M-casado]

Summary

As of elixir-europe/biovalidator#87, Biovalidator (in my fork) now accepts Draft 2020. I was looking forward to integrating Beacon v2 model JSON Schemas, when I encountered an issue regarding a broken reference:

https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/common/beaconCommonComponents.json

I'm writing all these details here with the aim to get some insight from you guys as the Beacon maintainers, because I'm pulling my hair off not being able to find the broken reference.

Details

So, in a nutshell, Biovalidator fetches references ($ref) through the net, and then compiles them to use them as the schema. It's done in an iterative way until all are compiled, and then it validates the data with them.

When I use any reference to Beacon v2 JSON Schemas in the schema I feed to Biovalidator, and it starts compiling the internal schema references (mostly relative) in your model, it encounters a broken reference to beaconCommonComponents.json that points not to framework/json/common/beaconCommonComponents.json path, where I did find it in main, but to common/beaconCommonComponents.json, which doesn't exist.

My guess is that somewhere in the (../common/..) JSON Schemas there's a relative reference to beaconCommonComponents.json that must be resolving against its own (../common/..) JSON Path, ending up with the broken reference beaconCommonComponents.json. Either that, or Biovalidator is hallucinating, which would surprise me a lot, but could be a possibility.

Now, onto what I tried and didn't work:

• I tried to increase (a lot) the verbosity of Biovalidator logs, to figure out what JSON file had this broken reference. I unfortunately couldn't pinpoint it still.
• I tried myself forking the Beacon v2 model, and redacting chunks to identify which is the file with the broken, but it seems to be anything that references the common JSON schemas, which then in turn reference a bunch of other stuff. Regardless of whether it's used or not for a specific validation, if there's a $ref, it'll be compiled. And all objects reference the common in some way.
• I checked all beaconCommonComponents.json references, relative or absolute, within the repository, and at glance I still couldn't pinpoint which one was resolving against common instead of framework.

Do you recall any possible outdated reference to ../common/beaconCommonComponents.json? Perhaps in an old commit or somewhere else that might be picked up? When you validate Beacon metadata through the JSON Schemas, do you compile all the $ref prior validation? If so, does your schema compiler not complain of any broken reference?

How to reproduce

• Install Biovalidator from my fork:

git clone git@github.com:M-casado/biovalidator.git
git checkout dev
npm install
node src/biovalidator

• Open a different terminal.
• Create a test JSON file to validate:

cat > beacon_test.json
{
  "schema": {
      "$id": "Beacon test",
      "$ref": "https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/cohorts/defaultSchema.json"
  },
  "data": {}
}

• Trigger validation:

curl --data @beacon_test.json -H "Content-Type: application/json" -X POST "http://localhost:3020/validate" | jq

Logs

Validation log:

$ curl --data @beacon_test.json -H "Content-Type: application/json" -X POST "http://localhost:3020/validate" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   427  100   229  100   198  66299  57324 --:--:-- --:--:-- --:--:--  138k
{
  "error": "Failed to compile schema: {\"error\":\"Failed to resolve $ref: https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/common/beaconCommonComponents.json, status: undefined\"}"
}

Biovalidator local server logs:

$ node src/biovalidator
2025-03-17T11:59:19.133Z [info] Custom keywords successfully added. Number of custom keywords: 5
2025-03-17T11:59:19.182Z [info] ---------------------------------------------
2025-03-17T11:59:19.183Z [info] ------------ ELIXIR biovalidator ------------
2025-03-17T11:59:19.183Z [info] ---------------------------------------------
2025-03-17T11:59:19.184Z [info] Started server on port 3020 with base URL /
2025-03-17T11:59:19.184Z [info] Server available at http://localhost:3020/
2025-03-17T11:59:19.184Z [info] PID file is available at .../biovalidator/server.pid
2025-03-17T11:59:19.184Z [info] Writing logs to: .../biovalidator/logs/
2025-03-17T11:59:22.477Z [info] New validation request: Malformed data. Please provide both 'schema' and 'data' in request body.
2025-03-17T11:59:30.879Z [info] New validation request: Malformed data. Please provide both 'schema' and 'data' in request body.
2025-03-17T11:59:52.660Z [info] Saving compiled schema in cache, $id: Beacon test
2025-03-17T11:59:52.870Z [info] Returning referenced schema from network : https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/cohorts/defaultSchema.json
2025-03-17T11:59:52.871Z [debug] Skipping meta-schema fetch: https://json-schema.org/draft/2020-12/schema
2025-03-17T11:59:53.148Z [info] Returning referenced schema from network : https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/framework/json/common/ontologyTerm.json
2025-03-17T11:59:53.465Z [info] Returning referenced schema from network : https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/framework/json/common/beaconCommonComponents.json
unknown format "datetime" ignored in schema at path "#/properties/eventDate"
unknown format "datetime" ignored in schema at path "#/properties/eventDate"
unknown format "datetime" ignored in schema at path "#/properties/eventTimeline/properties/end"
unknown format "datetime" ignored in schema at path "#/properties/eventTimeline/properties/end"
unknown format "datetime" ignored in schema at path "#/properties/eventTimeline/properties/start"
unknown format "datetime" ignored in schema at path "#/properties/eventTimeline/properties/start"
2025-03-17T11:59:53.799Z [info] Returning referenced schema from network : https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/common/ageRange.json
2025-03-17T11:59:54.115Z [info] Returning referenced schema from network : https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/common/age.json
2025-03-17T11:59:54.433Z [info] Returning referenced schema from network : https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/common/disease.json
2025-03-17T11:59:54.720Z [info] Returning referenced schema from network : https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/common/timeElement.json
2025-03-17T11:59:55.051Z [info] Returning referenced schema from network : https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/common/gestationalAge.json
2025-03-17T11:59:55.337Z [info] Returning referenced schema from network : https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/common/commonDefinitions.json
2025-03-17T11:59:55.651Z [info] Returning referenced schema from network : https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/common/timeInterval.json
2025-03-17T11:59:55.962Z [error] Failed to retrieve referenced schema: https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/common/beaconCommonComponents.json, {"message":"Request failed with status code 404","name":"AxiosError","stack":"AxiosError: Request failed with status code 404\n    at settle (.../biovalidator/node_modules/axios/dist/node/axios.cjs:2031:12)\n    at IncomingMessage.handleStreamEnd (.../biovalidator/node_modules/axios/dist/node/axios.cjs:3148:11)\n    at IncomingMessage.emit (node:events:529:35)\n    at endReadableNT (node:internal/streams/readable:1400:12)\n    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)\n    at Axios.request (.../biovalidator/node_modules/axios/dist/node/axios.cjs:4258:41)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)","config":{"transitional":{"silentJSONParsing":true,"forcedJSONParsing":true,"clarifyTimeoutError":false},"adapter":["xhr","http","fetch"],"transformRequest":[null],"transformResponse":[null],"timeout":0,"xsrfCookieName":"XSRF-TOKEN","xsrfHeaderName":"X-XSRF-TOKEN","maxContentLength":-1,"maxBodyLength":-1,"env":{},"headers":{"Accept":"application/json, text/plain, */*","User-Agent":"axios/1.8.3","Accept-Encoding":"gzip, compress, deflate, br"},"method":"get","url":"https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/common/beaconCommonComponents.json","responseType":"json","allowAbsoluteUrls":true},"code":"ERR_BAD_REQUEST","status":404}
2025-03-17T11:59:55.964Z [error] Failed to compile schema: {"error":"Failed to resolve $ref: https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/common/beaconCommonComponents.json, status: undefined"}
2025-03-17T11:59:55.965Z [error] An error occurred while running the validation: {"error":"Failed to compile schema: {\"error\":\"Failed to resolve $ref: https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/common/beaconCommonComponents.json, status: undefined\"}"}
2025-03-17T11:59:55.968Z [error] New validation request: Server failed to process data: {"error":"Failed to compile schema: {\"error\":\"Failed to resolve $ref: https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/common/beaconCommonComponents.json, status: undefined\"}"}

[costero-e]

Hi @M-casado,
Thanks for the report. I have my own validator for beacon v2 schemas and I have checked everything and it doesn't raise this error nor we have this weird reference Beacon v2 JSON Schemas, hence I don't think there's a problem with the beacon v2 schemas references, in this case.
What might happen (but I don't know for sure, I haven't checked Biovalidator code and it's just an idea I came up with as this sometimes happens with relative paths), is that Biovalidator is creating itself this reference from the root path declared and then adding up the relative paths that we have in our schemas, so it's mixing up a bit the initial reference path with the relative paths that appear along the files.
If you can double check this is what's happening, maybe it helps you finding the origin of this problem. But maybe I'm wrong eh, I don't know 100%. Anyways, thanks for keeping up the work with beacon v2 schemas, I hope this helps you and you can find the solution.

[gsfk]

I had a look at this briefly, and it's similar to issues I've had before when relying on the raw.githubusercontent.com references.

The local references in the schema are correct, and the raw.githubusercontent.com references are probably correct as well, but the schemas ultimately bottom out to local references so you have to keep track of where you are in the schema.

For example the cohorts spec in your example has this reference for an ontology term

"$ref": "https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/framework/json/common/ontologyTerm.json"

This resolves correctly, but contains a relative reference:

"id": { "$ref": "./beaconCommonComponents.json#/$defs/CURIE" },

So I wonder if your resolver is able to keep track of the current path in the schema after a double-hop like this.

[redmitry]

So I wonder if your resolver is able to keep track of the current path in the schema after a double-hop like this.

Two years ago I found that RUST JSON validator was unable to follow such links, have no idea what is a situation now.

[M-casado]

Thanks for checking it @costero-e, @gsfk & @redmitry

I'll continue testing it every now and then, but first if anything I'll focus on increasing the logs of Biovalidator to see if I can pinpoint where exactly this comes from. I did increase dramatically the logs in the past and still couldn't exactly find it.

Regarding the use of raw.githubusercontent.com, I have tested these raw references both currently (e.g., here) and in the past (e.g., here, although outdated), where I fed URI $ids to Biovalidator, and the tool was seamlessly fetching the raw files and resolving references as it should. I have yet to explore referencing multi-repository schema constraints besides mine and Beacon v2, to see if other commonly used repositories have the same issue.

Normally Biovalidator resolves relative JSON Paths against the $id of the schema. So "$ref": "./beaconCommonComponents.json#/$defs/CURIE" would commonly be properly handled. If I remember correctly during my testing with Beacon v2 model, the logs showed that it was properly fetching schemas in sequence, but when it reached the common definitions, one of the references was broken and the error was raised. I wonder if it could be a lack of explicit $ids in the Beacon schemas.

[M-casado]

After testing thoroughly, I found what the issue was. I'll detail it below, with a set of insights and recommendations as well to be applied (or not) to the Beacon source repo.

Details of the issue

In short, the issue was the lack of $ids identifying the root of each Beacon-v2 JSON Schema. These $ids are used by multiple resolving services, including AJV (A JSON Schema validator), to trace back relative paths. AJV is, like in many other cases, the JSON validator used by Biovalidator.

Without the $id, Biovalidator was iterating through the schemas while compiling them, assuming their retrieval paths, relative paths, and $ids and storing them in cache. When schemas from model/ were referencing framework/ back and forth, the schemas' root URIs were being mixed, and it ended up with framework/ schema beaconCommonComponents.json being cached as part of the models/ set. Thus, there are no broken references within the schema to a non-existing https://raw.githubusercontent.com/ga4gh-beacon/beacon-v2/main/models/json/beacon-v2-default-model/common/beaconCommonComponents.json. This fits with your intuitions @costero-e, @gsfk & @redmitry - AJV is indeed able to keep track of the schema's URIs, and properly resolve relative paths, but can only do so much when the $id of a schema is not provided.

For this obscure reason, the issue was so difficult to figure out: it's a mixture between what AJV expected based on the recommended specification, and what Beacon v2 has, resulting in a sort of mishandled root schema identifiers. As per the JSON Schema definition RFC Section 8.2.1.1, $id is not required, but heavily recommended.

How to resolve

In order to resolve the issue, I forked the Beacon repo, and added the $ids myself in a branch (biovalidator-test). See details on what changes I made, and how I automated them, in this PR.

Testing

We can test that the issue is resolved with the content of the biovalidator-test branch and a mock JSON file.

# Set up Biovalidator
git clone git@github.com:M-casado/biovalidator.git
cd biovalidator
git checkout improve-logger
npm install
node src/biovalidator

# Store in a variable a test JSON, containing mock 'data' and a 'schema' pointing towards the Beacon schemas (in my fork) with $ids
read -r -d '' json <<'EOF'
{
  "schema": {
    "$ref": "https://raw.githubusercontent.com/M-casado/beacon-v2/biovalidator-test/models/json/beacon-v2-default-model/cohorts/defaultSchema.json"
  },
  "data": {}
}
EOF

# Call the local Biovalidator server with the mock JSON
curl -s -H 'Content-Type: application/json' -d "$json" http://localhost:3020/validate | jq

The result of the call is, as expected (since data is simply {}), that the validation is not passed. That's precisely the result we're looking for, meaning the schemas were compiled and used appropriately:

[
  {
    "dataPath": "/id",
    "errors": [
      "must have required property 'id'"
    ]
  },
  {
    "dataPath": "/name",
    "errors": [
      "must have required property 'name'"
    ]
  },
  {
    "dataPath": "/cohortType",
    "errors": [
      "must have required property 'cohortType'"
    ]
  }
]

Additional notes

There are many ways in which Beacon-v2 could easily add $id to schemas, enabling their use through Biovalidator (and its handy API integration), and improving the overall value of Beacon-v2 following the JSON Schema specification.

I was going to create a PR with these $ids to the main Beacon source repo, but the thing is that since the identifiers are supposed to be absolute URIs, they include as well the branches. Therefore, if I give an $id to a JSON Schema in the develop branch while the URI points to main, it'll be incorrect. I've faced this challenge in the past, and there are some ways to circumvent it and make use of GitHub raw content for $ids:

• Use of bespoke scripts (e.g., what I did in the EGA-schemas repo, or the idAddition.py from this Beacon PR). These scripts can be coupled with GH workflows, so that every path is absolute when identifying schemas (closer to the specification recommendation).
• Use relative paths in the $id, and depend on the "retrieval URI". Not recommended as per the specification, but doable.

I might add that I found multiple times absolute paths being used along relative paths in the models/ and framework/ schema sets. In my opinion, this can easily lead to hard to debug issues, since the absolute paths (132 out of 135) you use are pointing all to the main branch. If the files you're pointing to change in any way within another branch, the mix of versions is going to be a hell to untangle. To note the mixture:

# Checking the number of absolute paths in the project
$ grep -RInE --include='*.json' '"\$ref"[[:space:]]*:[[:space:]]*"(\/|[A-Za-z]:[\\/]|https?://)' . | wc -l
135

# Checking the number of relative paths in the project 
$ grep -RIn --include='*.json' -P '"\$ref"\s*:\s*"\K(?!https?://|#)[^"]+' . | wc -l
246

[M-casado]

I might add that I found multiple times absolute paths being used along relative paths in the models/ and framework/ schema sets. In my opinion, this can easily lead to hard to debug issues

Speaking of the devil, I just encountered an issue for this matter. I was validating stuff against my fork of Beacon-v2, which has $ids, but there were some absolute references that were pointing to your main repo, which does not have $ids. As you can imagine, this petty discrepancies take time to narrow down, albeit little time to resolve.

All this to say... this is indeed a problem, and I'm sure it will eventually bite you too. More arguments advocating for a proper reference format.

M-casado@aacb6e3