Sketch to support (private) data-proxy dataset #61
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
'Absolute IRI confused with prefix.'
datalad-next now does wholistic parameter validation.
This change is merely adding the ability to recognize and
process non-public dataset data-proxy URLs.
However, it is not enough to support such datasets, because
the underlying `fairgraph` query to get a dataset's file listing
returns no results.
The query is essentially this
```py
batch = omcore.File.list(
self.client,
file_repository=dvr,
size=chunk_size,
from_index=cur_index)
```
and for the dataset referenced in
#58 it returns an empty
list with
- a properly authenticated `client`
- `dvr`: `FileRepository(name='buckets/d-07ab1665-73b0-40c5-800e-557bc319109d', iri=IRI(https://data-proxy.ebrains.eu/api/v1/buckets/d-07ab1665-73b0-40c5-800e-557bc319109d)...`
- `chunk_size`: 10000
- `cur_index`: 0
With the same requesting account, I can browser-visit
https://data-proxy.ebrains.eu/datasets/07ab1665-73b0-40c5-800e-557bc319109d
and see a file listing.
Previously we went via the KG to get a per-dataset file listing. This is extremely slow (#52). With this change, we query the data-proxy directly, if a data proxy bucket is detected to be a datasets's file repository. This speeds up operations by about 30x.
Otherwise the processing stops with a HTTP400 response.
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #61 +/- ##
==========================================
- Coverage 94.86% 90.47% -4.39%
==========================================
Files 8 8
Lines 253 273 +20
==========================================
+ Hits 240 247 +7
- Misses 13 26 +13
☔ View full report in Codecov by Sentry. |
We need to start with the `dataset/` API endpoint, so get that URL to the point where requests need to be made.
mih
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change started out by merely adding the ability to recognize and process non-public dataset data-proxy URLs.
However, it was not enough to support such datasets, because the underlying
fairgraphquery to get a dataset's file listing returns no results.The query is essentially this
and for the dataset referenced in
#58 it returns an empty list with
clientdvr:FileRepository(name='buckets/d-07ab1665-73b0-40c5-800e-557bc319109d', iri=IRI(https://data-proxy.ebrains.eu/api/v1/buckets/d-07ab1665-73b0-40c5-800e-557bc319109d)...chunk_size: 10000cur_index: 0With the same requesting account, I can browser-visit https://data-proxy.ebrains.eu/datasets/07ab1665-73b0-40c5-800e-557bc319109d and see a file listing.
To mitigate this issue, the PR now queries the data proxy API directly -- leading to noticeable speed-ups.
This PR prepares for addressing #58, but does not achieve it, due to outstanding authentication issues.