Skip to content

[FEAT] Create obstore store in fsspec on demand #198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 58 commits into from
Feb 28, 2025
Merged

[FEAT] Create obstore store in fsspec on demand #198

Show file tree
Hide file tree
Changes from 39 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
909b5b0
feat: split bucket from path + construct store
machichima Feb 3, 2025
29464a7
feat: remove store + add protocol + apply to all methods
machichima Feb 4, 2025
a0d9e1d
feat: inherit from AsyncFsspecStore to specify protocol
machichima Feb 4, 2025
6614906
fix: correctly split protocol if exists in path
machichima Feb 6, 2025
75c738e
feat: use urlparse to extract protocol
machichima Feb 7, 2025
2209839
Merge branch 'main' into obstore-instance-in-fsspec
kylebarron Feb 7, 2025
46c6b59
update typing
kylebarron Feb 7, 2025
9ab35e1
fix: unbounded error
machichima Feb 8, 2025
cb80495
fix: remove redundant import
machichima Feb 8, 2025
b6a3d3a
feat: add register() to register AsyncFsspecStore for provided protocol
machichima Feb 8, 2025
68cdff9
feat: add validation for protocol in register()
machichima Feb 8, 2025
fa5b539
test: for register()
machichima Feb 8, 2025
b704779
feat: add async parameter for register()
machichima Feb 8, 2025
61deac4
test: test async store created by register()
machichima Feb 8, 2025
4bc1599
feat: add http(s) into protocol_with_bucket list
machichima Feb 8, 2025
4dc9143
feat: ls return path with bucket name
machichima Feb 9, 2025
fb607d0
feat: enable re-register same protocol
machichima Feb 9, 2025
b74948a
test: update pytest fixture to use register()
machichima Feb 9, 2025
f6ba27c
test: update test with new path format
machichima Feb 9, 2025
30250cf
fix: mkdocs build error
machichima Feb 9, 2025
4a8e6fc
Merge branch 'main' into obstore-instance-in-fsspec
machichima Feb 13, 2025
27a0ac7
fix: error when merging
machichima Feb 13, 2025
d2d0235
build: add some ruff ignore
machichima Feb 13, 2025
1f97703
fix: ruff error
machichima Feb 13, 2025
b002afb
build: add cachetools dependencies
machichima Feb 13, 2025
9088104
Merge branch 'main' into obstore-instance-in-fsspec
machichima Feb 13, 2025
897beb0
better scoping of lints
kylebarron Feb 13, 2025
0726999
lint
kylebarron Feb 13, 2025
5b87c46
fix: update lru_cache + clean class attribute
machichima Feb 14, 2025
79a03f7
Merge branch 'main' into obstore-instance-in-fsspec
machichima Feb 14, 2025
dc4215d
fix some bugs when using get/put/cp/info/ls
machichima Feb 15, 2025
f59152b
Merge branch 'main' into obstore-instance-in-fsspec
machichima Feb 15, 2025
4896ba3
fix: declare lru_cache in __init__
machichima Feb 16, 2025
c9378b8
fix: make AsyncFsspecStore cachable
machichima Feb 16, 2025
549a4ac
test: for cache constructed store and filesystem obj
machichima Feb 16, 2025
a93fe2e
build: remove dependencies
machichima Feb 16, 2025
a54a3fe
fix: prevent send folder path to cat_file
machichima Feb 19, 2025
c804a18
fix: enable cp folders
machichima Feb 19, 2025
6c2c513
lint
machichima Feb 19, 2025
c6392f2
fix: clobber=False to prevent re-register and cause memory leak
machichima Feb 23, 2025
347e63e
test: clean up after each test to prevent memory leak
machichima Feb 23, 2025
69bbed6
Merge branch 'main' into obstore-instance-in-fsspec
machichima Feb 23, 2025
9e423f5
Simplify protocol registration
kylebarron Feb 24, 2025
096845c
fix+test: register check types
machichima Feb 25, 2025
5ea2ba8
small edits
kylebarron Feb 26, 2025
12560f4
fix+test: update conftest
machichima Feb 26, 2025
6e200e9
style: format
machichima Feb 26, 2025
4a653b3
feat: enable setting protocol in __init__
machichima Feb 26, 2025
f5147bc
docs: update example in docstring
machichima Feb 26, 2025
45c4020
fix: better split path way
machichima Feb 26, 2025
4e1f4e7
fix: split path for protocol with no bucket properly
machichima Feb 27, 2025
a212996
test: for split path
machichima Feb 27, 2025
278cc38
refactor: take out _fill_bucket_name function
machichima Feb 27, 2025
92a70f5
refactor: take out runtime type check for register
machichima Feb 27, 2025
969dbff
test: remove test register invalid type
machichima Feb 27, 2025
6c409c8
Merge branch 'main' into obstore-instance-in-fsspec
kylebarron Feb 28, 2025
e443668
Switch to checking if protocol does not require bucket
kylebarron Feb 28, 2025
4949348
Warn on unknown protocol
kylebarron Feb 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading