Questions regarding to the object-store - deltalake interface

[thhapke]

We have an object_store implementation for our objectstore API Hana Datalake Filesystem (HDLFS) for testing datafusion and implementing a couple of rust based solutions. When the requirement came up to have a Python binding for using deltalake and HDLFS, we added the integration coding to a delta-rs repo clone. When considering to add HDLFS to deltalake concerns were raised about the project-architecture:

Why is this project mixing object_store implementations with the deltalake code? This comes mostly in addition to the arrow-rs-object_store implemenation. There seems to be no need once your object_store implementation is matching the delta-rs requirments. At least we had no need to do this but just add the registering code to the python/lib.rs:

// module name need to match project name
fn _internal(m: &Bound<'_, PyModule>) -> PyResult<()> {
    use crate::error::{CommitFailedError, DeltaError, SchemaMismatchError, TableNotFoundError};

    // Register hdlfs handlers
    use hdlfs;
    hdlfs::hdl_store_factory::register_handlers();

When we tried to cross-compile the code for windows we ran into aws-issues and could not exclude the "aws"-feature from the project compilation. The virtual project setup was just to complex for me.

Many thanks for your clarifcations or input

[ion-elgreco]

We have an object_store implementation for our objectstore API Hana Datalake Filesystem (HDLFS) for testing datafusion and implementing a couple of rust based solutions. When the requirement came up to have a Python binding for using deltalake and HDLFS, we added the integration coding to a delta-rs repo clone. When considering to add HDLFS to deltalake concerns were raised about the project-architecture:

Why is this project mixing object_store implementations with the deltalake code? This comes mostly in addition to the arrow-rs-object_store implemenation. There seems to be no need once your object_store implementation is matching the delta-rs requirments. At least we had no need to do this but just add the registering code to the python/lib.rs:

// module name need to match project name
fn _internal(m: &Bound<'_, PyModule>) -> PyResult<()> {
    use crate::error::{CommitFailedError, DeltaError, SchemaMismatchError, TableNotFoundError};

    // Register hdlfs handlers
    use hdlfs;
    hdlfs::hdl_store_factory::register_handlers();

When we tried to cross-compile the code for windows we ran into aws-issues and could not exclude the "aws"-feature from the project compilation. The virtual project setup was just to complex for me.

Many thanks for your clarifcations or input

I don't really get the question. We need object_store handlers for python deltalake to be registered globally otherwise there is no integration with the stores