Reconsider dependency on `sssom_schema`?

[gouttegd]

SSSOM-Py currently depends on sssom_schema, the Python package provided by the mapping-commons/sssom repository (the repository holding the SSSOM spec and documentation.

That package provides 3 things:

• the sssom_schema.datamodel.sssom_schema.py Python module, containing the Python data classes representing the SSSOM data model (the MappingSet and Mapping classes and the associated enums);
• the sssom_schema.yaml LinkML schema, which is used by SSSOM-Py to get some details about the data model that are not contained within the generated data classes;
• the sssom_schema.context.jsonld file, which is used by SSSOM-Py for JSONLD-related stuff.

It’s not immediately clear to me what are the benefits of getting those files from a released version of sssom_schema, compared to embedding them directly into this SSSOM-Py repository. And there is at least one very clear drawback, which is that SSSOM-Py cannot start supporting any new thing from the SSSOM schema until we’ve had a release of sssom_schema on PyPI. This is what forced us to start making “pre-releases” of sssom_schema, which apparently are now causing problems with uv because uv has seemingly strong opinions about dependencies that are in pre-release stage.

So I would like to suggest that we should consider dropping the dependency on sssom_schema, and instead start bundling “vendored” copies of the aforementioned three files, copied directly from the mapping-commons/sssom repository. We would then be able to update those files whenever we feel like it, without having to go through a PyPI release of sssom_schema.

(This in turn would raise the question of “what is the point of the sssom_schema package”, to which my instinctive answer would be “none”.)

[cthoyt]

I would agree with the idea to vendor these three files in the short- and medium term and remove the dependency from sssom-py on sssom-schema.

(This in turn would raise the question of “what is the point of the sssom_schema package”, to which my instinctive answer would be “none”.)

Also agreed. In the long term, I suggest we consider merging sssom-py and sssom-schema, or just deleting sssom-schema

[gouttegd]

I suggest we consider merging sssom-py and sssom-schema

Disagree. The SSSOM spec and documentation should remain separate from the SSSOM-Py implementation.

[cthoyt]

sure, not to conflate that the https://github.com/mapping-commons/sssom repository contains the spec, docs, and the python package sssom-schema. I was just thinking that github.com/mapping-commons/sssom could not have the python code in it.

[matentzn]

Before considering this (which I am open to, we use vendoring in other projects) I would like to know how we can track exactly which version we have vendored in (short of file hash and search). Are we adding a commit hash somewhere to track which exact version of SSSOM brach was used?

[gouttegd]

Not fully convinced that we’d need to “track exactly which version we have vendored in”, but if we want/need to we could simply add a comment with either the commit hash or the output of git describe (slightly more human-readable than a “pure” hash).

The comment can either be inserted manually, or we could amend the Makefile on the sssom-spec side to automatically inject it whenever the sssom_schema.py file is generated.

[matentzn]

Ok, I am fine with that then. If we have a way to track the version I agree to vendoring for this project here - I would not do this for all projects I have to say. My main problem is this: sssom is a beast for depndencies, and we force quite a beefy import on folks that my ONLY want sssom_schema. However, given that I dont think there is currently any user of just sssom_schema, I am ok with vendoring, by a margin of 51% to 49% against.

[gouttegd]

My main problem is this: sssom is a beast for depndencies, and we force quite a beefy import on folks that my ONLY want sssom_schema

When you say sssom_schema, do you mean: (a) the LinkML schema (sssom_schema.yaml) or (b) the Python module containing the LinkML-derived Python data classes corresponding to the schema (sssom_schema.py)?

If (a), I’d say that people who needs the SSSOM LinkML schema should not get it from a Python package anyway, because that’s not what a Python package is for (and that’s also not what PyPI is for). They should get it from this repository, in a completely Python-independent way.

If (b), it’s unclear to me why everyone would want to get our generated classes (which would, among other things, force upon them the namespace that we chose to use) rather than running the generator themselves and produce their own classes.

[matentzn]

I am thinking (b) so you can do (pseudocode)

import Mapping from sssom_schema.util

def x():
    m = Mapping(...)

Does this not make sense?

[gouttegd]

In my opinion? No, it does not really make sense. :)

If you need classes representing the SSSOM data model in your project, I think it would be much better for you to generate your own classes rather than use pre-generated classes from a third-party (even if that third-party happens to be the authors of the original SSSOM schema).

This way:

(1) you control everything about how the classes are generated (e.g.: Pydantic or data classes? with or without CLASSVAR infos? with class and slot URIs or model URIs? etc.);

(2) you do not run the risk of your code breaking because you updated all your dependencies and it so happens that the updated data classes introduced a breaking change. Granted, in the specific case of SSSOM this should never happen because we promise that the model will always remain backwards compatible (at least until we reach SSSOM 2.x), but in the general case this can very well happen. If you generate your own classes, then you decide if and when to re-generate them from a newer version of the schema.

[gouttegd]

By the way, this is why in the opening comment I wrote the following:

(This in turn would raise the question of “what is the point of the sssom_schema package”, to which my instinctive answer would be “none”.)

[matentzn]

Hmm.. I see. I never though of it that way. I sort of hat this in mind as an interface class between different libraries, so someone can construct a MappingSet() in one library and use it in another..

import MappingSet from sssom_schema.util
import process1 from lib1
import process2 from lib2

.....
MappingSet: ms = process1()
process2(ms)
.....

Am I wrong here (could very well be).

[gouttegd]

Yeah, theoretically it would be possible, but in practice I highly doubt this would be usable. The idea that simply having the same dataclasses would be enough for two independent libraries to interoperate seems like wishful thinking to me.

We ensure interoperability at the file level (a file created by library 1 can be read by library 2), this I believe is all we should hope/aim for.

[matentzn]

Let me think this over. I am not entirely convinced yet but I see your point.