Add Extension Registry

[tokoko]

I'm thinking of building an Extension Registry in rust similar to the one in substrait-python. The registry will have three primary roles:

1. It will generate and hand out unique anchor numbers.
2. It will check that the intended signature of the function call is covered by an impl in the extensions.
3. If a suitable impl is found, it will infer a return type for the function call.

The biggest challenge in rewriting the code in rust is that the python registry depends on antlr to handle return type inference. Since there's no antlr rust target available, we will either have to forgo return type inference altogether or just for those cases where complicated inference logic is required (such as this).

[Blizzara]

Just fwiw, we've written something similar in DF, in case that's any use as example: https://github.com/apache/datafusion/blob/c5df6eee106df1076e033e15bfa3118a1da8886d/datafusion/substrait/src/extensions.rs#L31

That said DF doesn't (thankfully 😄) validate the signatures so can't help you there.

[tokoko]

yup, I'm familiar with the one implemented in DF. The main motivation for moving ExtensionRegistry from python to rust is to eventually supplant Extensions struct in DF with the one implemented here to solve this. I thought it would be easier to start from scratch here first and then see how best to utilize it from DF later.

[vbarua]

I think it makes sense to have this functionality in substrait-rs and then have it be used in DataFusion. That would also allowed other Rust-based systems to leverage it. IMO the purpose of the language specific libraries is to provide this kind of functionality.

Since there's no antlr rust target available,

This has also come up for us internally when thinking about the test parser. I don't think it would be unreasonable to declare other grammar, and use the ANTLR one as the reference.

python registry depends on antlr to handle return type inference.

Does the Python registry actually support computing the return types that have computations embedded in them like the example you linked?

[tokoko]

This has also come up for us internally when thinking about the test parser. I don't think it would be unreasonable to declare other grammar, and use the ANTLR one as the reference.

that might be necessary. btw, it seems like substrait-validator is using antlr-rust, which is both unofficial and unmaintained at this point, but maybe it does the job...

Does the Python registry actually support computing the return types that have computations embedded in them like the example you linked?

yes, impl is here and the decimal example is actually tested in ci.

[vbarua]

it seems like substrait-validator is using antlr-rust, which is both unofficial and unmaintained at this point, but maybe it does the job...

That both seems reasonable in an "If it ain't broke don't fix it kind of way", but also a little sus given it hasn't been updated in 3 years. It would probably be good to migrate to a better supported alternative, but that's not necessarily work you need to do.

yes, impl is here and the decimal example is actually tested in ci.

TIL that something can actually process those, which is neat.

[benbellick]

Hi! Just wanted to flag one small thing here, which is that there is an ongoing URI -> URN migration. I'm not sure the speed required to get this out the door, but the implementation will almost certainly be easier post-migration rather than during.

This is because doing the extension registry during the migration means managing references to both URI and URN.