Is your feature request related to a problem? Please describe

Currently Arrow Flight Stream based SPIs introduced in #16691 make use of generics. Since plugins have a separate classloaders, so its not possible to share BufferAllocators and VectorSchemaRoots from arrow-memory to be shared between plugins thus making SPIs not usable.

Describe the solution you'd like

@reta has an idea to make use of extension plugin classloader for such purposes #17580. Still evaluating this one.

Related component

Search:Performance

Describe alternatives you've considered

One way is to shade all the dependencies and include them in server classpath. That's the solution I tried to implement here but its not a clean solution and can make classloader bulky and dirty -

arrow-memory shaded deps - https://github.com/rishabhmaurya/OpenSearch/blob/flight-producer/libs/arrow-memory-shaded/build.gradle
flight-core shaded deps - https://github.com/rishabhmaurya/OpenSearch/blob/flight-producer/libs/flight-core-shaded/build.gradle

Additional context

No response