schema: add `stripping.line`

[tiborsimko]

Current behaviour

For the LHCb ntuples, it is desirable to have a stripping line information in the metadata, for example:

    "stripping": {
      "line": "StrippingLb2LcPiLc2PKPiBeauty2CharmLine",
      "stream": "BHADRON",
      "version": "stripping21r1"
    },

See cernopendata/opendata.cern.ch#3720

This was previously uploaded to QA and led to https://opendata-qa.cern.ch/record/93949/export/json i.e. the "stripping" field was created with "stream" and "version", and the unknown "line" property was ignored.

Now, when doing local development, after the latest update to registry.cern.ch/cernopendata/cernopendata-portal:0.13.0, the file is explicitly refused from being uploaded:

$ docker exec -i -t opendatacernch-web-1 cernopendata fixtures records --mode insert-or-replace -f /content/data/records/lhcb-ntuples-hackathon.json
...
  File "/opt/invenio/var/instance/python/lib/python3.9/site-packages/opensearchpy/connection/base.py", line 315, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
opensearchpy.exceptions.RequestError: RequestError(400, 'strict_dynamic_mapping_exception', 'mapping set to strict, dynamic introduction of [line] within [stripping] is not allowed')

Desired behaviour

Let's enrich the cernopendata-portal's JSON Schema and OpenSearch mappings in order to allow "line" in the "stripping" field.

We would then need to bump up the registry.cern.ch/cernopendata/cernopendata-portal version and to update the opendata.cern.ch docker compose override accordingly.

[psaiz]

Thanks for the ticket.
Just to be clear, at the moment there are two places where we have to define what we store:

• The json schema mentioned in this ticket. Note that the current schemas do not have "additionalProperties": false, which means that they allow anything that the user sends
• The opensearch mappings. The error displayed in the ticket is from here.

We could address only the second one (and that would solve this ticket), or, even better, we could enable the additionalProperties and then changes in the records structure would be already detected on the pipelines that check the schema of the records

I would prefer the second option, even if it has the risk that there might be some records in the system that will fail the validation if we enable the strict properties.

What do you think?

[tiborsimko]

I like introducing strict checks if we consider the schema to be "stable" by now. We can amend opendata.cern.ch CI checks to catch any troubles early. And we can pass through all the fixtures to make sure that the schema is respected...

[psaiz]

I've been playing with this. We have three type of entries: records, docs, glossary-terms

• Glossary-terms. Around 1k. Most of them validate. There are 2 that fail because the field accelerator is not there.
• docs: there are at least two terms missing : slug and body.format. I'll add them to the schema
• records: they are missing at least the field files. I'll leave the validation running to detect other fields

[psaiz]

Ok, not too many changes. And there are only three records that do not validate. See the attched pull request