Authentication special remote

[effigies]

What would you like to see added?

It should be possible for us to provide authenticated access to restricted data according to the following logic:

• Receive GET $KEY request
• Load user token
• Send $KEY request to authentication server
• Receive access URL with authorization token (or URL to encrypted file and a decryption key)
• Fetch $KEY from out-of-band URL, decrypt if needed

Alternatives

This is an alternative to the special remote accessing an API that authenticates and returns the data over the same connection.

Do you have any interest in helping implement the feature?

Yes

Additional information / screenshots

No response

[effigies]

An alternative method (via @yarikoptic) would be to add web URLs like, for example, https://api.openneuro.org/files/ds00WXYZ/path/to/restricted/file. That API checks the user's token, mints a short-lived (30m) download URL and returns a 302 redirect. That requires no custom special remote, just the authentication bit to make sure that user tokens are passed.

I believe the additional bit needed to make this work on the public bucket is a tag to prevent download without the short-lived URLs, but that's filtering Yarik's words through my treasured ignorance of S3 mechanics.

[yarikoptic]

That requires no custom special remote, just the authentication bit to make sure that user tokens are passed.

it would require use of the already existing in stock datalad, datalad special external remote so it would manage storing/using credentials. The dandiset on which you should be able to try is https://dandiarchive.org/dandiset/000344?pos=1 which exists as https://github.com/dandisets/000344/ on github and to which you should have access @effigies .

Try to annex get any file from under it - how would it go? (doesn't work for me ATM due to some creds request UI fiasco :-/ ;) but should work)

I believe the additional bit needed to make this work on the public bucket is a tag to prevent download without the short-lived URLs, but that's filtering Yarik's words through my treasured ignorance of S3 mechanics.

I believe this is relevant part of the design explained in docs: https://github.com/dandi/dandi-archive/blob/master/doc/design/embargo-redesign.md?plain=1#L23 with implementation in TF: https://github.com/dandi/dandi-infrastructure/blob/0e31d6d3530d4b2c24170ab8e34a7ba19a23f405/terraform/modules/dandiset_bucket/main.tf#L250 with an example on adding the tag for upload: https://github.com/dandi/dandi-archive/blob/HEAD/dandiapi/api/manifests.py#L83