Skip to content

Add class methods wrapper #331

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Mar 12, 2025
Merged

Add class methods wrapper #331

merged 18 commits into from
Mar 12, 2025

Conversation

kylebarron
Copy link
Member

@kylebarron kylebarron commented Mar 7, 2025

This PR allows all functional methods, like get, head, list, to also be called as methods on each store class.

I want obstore to define an "object storage interface spec" that allows applications to depend on an obstore-like interface, but which allows middlewares in between. So for example, metrics (#105) or caching (#247) could be implemented externally on the Python side. And a library depending on an obstore like interface wouldn't need to know anything about middlewares, as long as the input conforms to the correct protocol.

I started exploring these object storage interface spec ideas in #330. I think the API I'm considering is just a collection of protocols. E.g.

class Head(Protocol):
    def head(self) -> HeadResult: ...

class HeadAsync(Protocol):
    async def head(self) -> HeadResult: ...

etc. Then a downstream application could pick and choose which APIs they require by doing

class RequiredObspecMethods(Head, Get, List, Protocol):
    pass

which would union the required methods.

But this obspec idea requires a class-based approach, with named functions that can match the given protocol methods.

Hence, this requires revisiting #160, to consider exposing methods on the Python based classes. I think revisiting this is worthwhile.

Change list

  • Allow subclassing of raw Rust-exported python classes, like S3Store
  • Ensure that the raw functional API imports from _store, so that the functional API supports both the non subclassed and subclassed variants.
  • Convert the store-local from_url, i.e. S3Store.from_url to respect subclasses. That is, use cls() to construct the class instance.
    • Also change the typing to return -> typing.Self instead of the concrete class itself.
  • Add a top-level from_url method in Python, so that obstore.store.from_url can construct the subclassed versions of each class.
    • Add a private parse_scheme helper function exported from Rust to make sure we parse the URLs in the same way that object_store does.
  • Add class methods to docs
  • Update the register_store_module function so that the user of pyo3-object_store can decide whether to register .store or ._store
  • Update tests to use this new class-based API

There are a bunch of moving pieces here:

  • We define a mixin, _ObjectStoreMixin, which defines the method wrappers. So these methods are defined in terms of the pure functions.
  • Because these mixins are defined on the Python side, we don't increase the API surface exported to other Rust libraries (via pyo3-object_store).
  • We subclass the underlying Rust classes instead of wrapping. This means that these same classes can be passed in to the existing functional API (so it's backwards compatible).
  • Pickling still works because the __module__ attribute points to the top-level Python class, not the core Rust class (does cloudpickle work too?, hopefully)

Closes #160

cc @ion-elgreco

@kylebarron
Copy link
Member Author

I'm not sure why, but the exported classes don't seem to be the correct (subclassed) ones

@kylebarron kylebarron merged commit a4f915a into main Mar 12, 2025
4 checks passed
@kylebarron kylebarron deleted the kyle/class-methods branch March 12, 2025 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Document design decision for functional API
1 participant