Extensible support for other libstreamers3.so implementations

[pwschuurman]

Motivation

The RunAI Model Streamer currently supports a shared library model for loading libstreamers3. This interface is defined here. The current implementation of libstreamers3.so uses the aws-sdk-cpp library, and S3 compliant APIs (eg: S3, GCS, Minio) are supported.

In order to support more features of other object storage providers (eg: client configuration, performance, authentication), it would be ideal for compatible libstreamers3 shared libraries to be dynamically invoked based on the user specified endpoint. For example, if the user specifies a GCS URL prefixed with gs://, the Run AI Model Streamer could prefer to use a GCS specific libstreamers3 library. If not available, Run AI Model Streamer could fallback to the current libstreamers3 implementation (leveraging the S3 compliant authentication and API offered by Google Cloud Storage).

Design Requirements

• Allow endpoint prefixes to automatically select the preferred shared library to call. This can be done dynamically when identifying an appropriate libstreamers3 interface (eg: case switch in s3_wrapper.cc.
  • The user should also be able to override the underlying library implementation, through an environment variable.
• The libstreamers3 interface should be packaged in an optional PIP package. Today, this is done with the optional package runai-model-streamer-s3. Additional implementations can be made available in separate packages (eg: runai-model-streamer-gcs).

Additional Considerations

• With the exception of the S3Credentials resource, there aren't any S3 specific details in the cpp/common or cpp/streamer. In order to disambiguate naming, S3 specific references (eg: interface definitions, c++ namespaces) could be renamed to ObjectStorage in these source directories, to reduce code ambiguity.
• With the shared library fallback logic, user authentication may not work as expected, depending on the underlying PIP dependencies that are installed. For example, if the user was expecting to use a GCS specific package (runai-model-streamer-gcs) that included support for Workload Identity Federation on GCP, but only had the runai-model-streamer-s3 package installed, model streaming would fail. The S3 library requires the user to input GCS HMAC credentials. So merely transferring configuration across installation environments with different optional dependencies could lead to unexpected incompatibility. The user troubleshooting experience should at least be considered if supporting this type of behavior.
• With more libstreamers3 implementations, and potential interface changes, ABI versioning becomes a consideration. The libstreamers3 interface must be compatible with the upstream libstreamer.so library that is calling into it. Additionally, if either libstreamers3 or libstreamer introduce forwards/backwards incompatible changes, minimum supported versions should be included and managed as part of the PyPI distribution. In order to get ahead of this problem, ABI versions and PIP dependencies should be introduced as part of this feature.

[noa-neria]

Thanks @pwschuurman we absolutely love this proposal!
Leveraging the plug-in architecture for different storage backends would be perfect 💯

You are right that fallback to libstreamers3.so could lead to potential issues. Also, the GCS implementation should be the best choice for GCS backend. How about the following suggestion:

• No fallback for vendor specific endpoint - If a GCS endpoint is passed and libstreamergcs.so is not found the Streamer should return an appropriate error message (similar to the current error message if libstreamers3.so was not found, and stating the expected package)
• Overriding the underlying library implementation, through an environment variable, is always respected - the Streamer returns an error only if the desired library was not installed
• Compatible storage fallback to libstreamers3.so, unless the environment variable specifies otherwise

Setting the desired object store backend with environment variable is also valuable for S3 compatible storage (e.g. for supporting HTTP2) and it would be great to offer more implementations. Off course, in this case the user should pass the corresponding GCS environment variables. It would be great to have this in the documentation.

We should definitely generalize S3 to ObjectStorage and keep vendor specific data in plugins.

We very much agree on the need for ABI versioning for compatibility and clarity.

Since the Streamer is aware of the different implementations and the implementation is selected according to the endpoint, it is OK to pass vendor specific information if needed (e.g. for creating the client such as S3Credentials). However, using native environment variables or configuration files in the plugin could be more simple. The reason that we introduced S3Credentials is limitations of the AWS S3 CPP SDK in comparison to boto3

[pwschuurman]

You are right that fallback to libstreamers3.so could lead to potential issues. Also, the GCS implementation should be the best choice for GCS backend. How about the following suggestion:

• No fallback for vendor specific endpoint - If a GCS endpoint is passed and libstreamergcs.so is not found the Streamer should return an appropriate error message (similar to the current error message if libstreamers3.so was not found, and stating the expected package)
• Overriding the underlying library implementation, through an environment variable, is always respected - the Streamer returns an error only if the desired library was not installed
• Compatible storage fallback to libstreamers3.so, unless the environment variable specifies otherwise

Great suggestion. I think this provides intuitive behavior for end users.

We very much agree on the need for ABI versioning for compatibility and clarity.

I think the simplest approach would be to version the shared libraries along the same lines as the PyPI packages (this is implicitly done today, without versions, as .so files are embedded within PyPI packages). This works well for a monolithic release, where all PyPI packages will always be shipped together (eg: new version of runai-model-streamer is published at the same time as runai-model-streamer-s3 and runai-model-streamer-gcs, with a minor version dependency).

If there is a possibility of version skew between the PyPI packages (eg: independent releases, or separate release ownership), libstreamers3 interface should be versioned separately, and versions updated appropriately (eg: major version update implies an ABI breaking change). This will require dependent packages (eg: runai-model-streamer, runai-model-streamer-s3) to adhere to a libstreamers3 ABI.

I would lean towards the status quo of coupled PyPI package releases for now, and reconsider if the need arises (releases delayed due to coupling, increased maintenance burden of supporting additional dependencies and release packages).