What version(s) of Substrait are/should be supported?

[ingomueller-net]

This project seems to upgrade to the latest version of Substrait at a pretty regular basis. This is nice for users who want to use the latest version but also makes it difficult to find and use the version of substrait-python for a specific version of Substrait.

For example, substrait-mlir uses Substrait v0.42.1 and substrait-python v0.12.1 for testing some end-to-end Python use cases. Now, we can't easily upgrade to pyarrow v16 or higher because those versions require substrait-python v0.15 or higher (due to how the generated proto files are packaged, AFAIU), which includes a different version of Substrait than what we use.

Note that Substrait regularly introduces breaking changes, in particular, to the text format, so it is not generally possible to use the protobuf definitions of a newer version.

To remedy the situation, I can imagine three things:

1. Add a table to the README and/or PyPI that shows the correspondance between the version(s) of substrait-python and Substrait. This would make it easier to find the right version of substrait-python. In some situations, that may be all that is necessary. Currently, the only way I know how to do this is to install several versions of substrait-python and print substrait.__substrait_version__, which isn't ideal.
2. We might want to investigate whether it isn't possible to package several versions of Substrait inside of substrait-python, for example, under substrait.v0_74_0 and substrait.v0_42_1, possibly with the latest version living in substrait as well.
3. Another possibility would be to separate the packaging of the generated proto files from the other functionality of this repository. substrait-mlir and, AFAIU, pyarrow only need the former. Such a package could either aim to be "released once" such that, for every version of Substrait, there would be a release of that package that would never need to be updated (ideally with the same versioning scheme as Substrait itself), or use the multi-version scheme from the previous point.

[tokoko]

Another possibility would be to separate the packaging of the generated proto files from the other functionality of this repository. substrait-mlir and, AFAIU, pyarrow only need the former. Such a package could either aim to be "released once" such that, for every version of Substrait, there would be a release of that package that would never need to be updated (ideally with the same versioning scheme as Substrait itself), or use the multi-version scheme from the previous point.

This seems a lot cleaner than option 2. For me the only concern with it would be that other language implementations don't do the same, so I'm not sure why python should be the odd one out. I also think that if the goal is for the new package (let's call it substrait-proto for example) to match upstream substrait versions, it makes more sense for the build/publish steps to be part of substrait rather than substrait-python.

This is probably something that should be discussed in the main repo, but I can see a setup where a new substrait release means publishing a set of proto-generated barebones "libraries" in all main different languages. The language repos would then depend on those libraries and add some additional utilities on top.

[ingomueller-net]

Actually, there is already substrait-io/substrait-protobuf, which seems to correspond quite exactly to what you describe. Maybe we should indeed implement proto packaging logic for Python there and use (and thus contribute to) that approach?

Someone should probably also ping the developers of the repositories of the other languages to do the same... 😉

[tokoko]

We can do that. Having a separate repo for it seems very much a go-inspired decision 😄 because you need git as a destination for artifacts. For other languages, doing the releases from the main repo sounds better. With the current approach one would be forced to essentially do a release twice. I don't really mind either approach, I'm not the one doing releases. Probably best to continue the discussion on slack.. I'll post a message there