Is there a way to provide auth for an HTTPStore?

[chuckwondo]

In perusing the code, there appears to be no way to use an HTTPStore when auth is required, but perhaps I'm missing something, so I'm asking here before opening an enhancement request. Is there a way to do so that I'm not seeing?

[kylebarron]

No there isn't. What are you looking to do? Obstore doesn't intend to be a generic HTTP client and so it doesn't expose all the custom http headers that you might want to use for auth.

Is there a webdav-like store that you're trying to work with that's behind authentication?

[chuckwondo]

Perhaps the subtlety of HTTP usage should be more explicitly documented, particularly since obstore provides an HTTPStore and it is possible to call obstore.store.from_url with an HTTP URL and a credential provider and obtain an HTTPStore without issue, but unbeknownst to the user, the credential provider will not be used.

Thus, I'm unclear on this point:

As done in all current credential providers, you should use a Python HTTP client like requests or aiohttp for making arbitrary HTTP requests inside a credential provider's implementation.

I understand how to do this for the S3 case that I want to handle. I have created a credential provider to make the underlying HTTP calls necessary for authentication, which I can pass through, like so, when I have an S3 URL in hand:

cp = MyS3CredentialProvider(...)  # makes HTTP auth requests
url = "s3://..."
store = obstore.store.from_url(url, credential_provider=cp)
result = obstore.get(store, "", options={"range": [0, 100]})
assert len(result.bytes()) == 100
# assertion holds

However, it's not clear to me how to do similar with an HTTP URL in hand, because there is no way to make use of a credential provider with an HTTP URL.

While I can supply a credential provider with an HTTP URL, obstore does not use it for HTTP URLs:

cp = MyHTTPCredentialProvider(...)  # makes HTTP auth requests
url = "https://..."  # the HTTPS URL alternative to the S3 URL above
store = obstore.store.from_url(url, credential_provider=cp)
result = obstore.get(store, "", options={"range": [0, 100]})
assert len(result.bytes()) == 100
# 401 Unauthorized: HTTP Basic: Access denied
# (because cp is NOT invoked)

Is there a way to make use of a credential provider with an HTTP URL that I'm missing, or are you saying that I shouldn't be using obstore at all with HTTP URLs when I know I need to supply creds?

[kylebarron]

the credential provider will not be used

You should get a type error from your type checker already. Perhaps we should enforce a runtime error.

I shouldn't be using obstore at all with HTTP URLs when I know I need to supply creds

Correct. It's not possible to pass credentials to arbitrary HTTP URLs with obstore (you can hack around it with default_headers if you really wanted to, but I recommend against it). The HTTPStore is only for WebDAV "buckets". If you're hitting an HTTP API, as you are for authentication purposes, you should not be using the HTTPStore.

For example if you look at the PlanetaryComputerCredentialProviders, they use requests or aiohttp to perform the generic HTTP calls in their implementation.

[chuckwondo]

the credential provider will not be used

You should get a type error from your type checker already. Perhaps we should enforce a runtime error.

Unfortunately, as far as I'm aware, Python's type system does not support any type of annotation of obstore.store.from_url such that a type checker can determine whether the URL string represents an S3 URL or an HTTP URL, and thus cannot tell whether or not it should complain about also being given a credential provider. This is perhaps contributing to my confusion.

I shouldn't be using obstore at all with HTTP URLs when I know I need to supply creds

Correct. It's not possible to pass credentials to arbitrary HTTP URLs with obstore (you can hack around it with default_headers if you really wanted to, but I recommend against it). The HTTPStore is only for WebDAV "buckets". If you're hitting an HTTP API, as you are for authentication purposes, you should not be using the HTTPStore.

For example if you look at the PlanetaryComputerCredentialProviders, they use requests or aiohttp to perform the generic HTTP calls in their implementation.

Sure, I know how to use such libs for auth access to HTTP URLs.

I was trying to use obstore for authenticated access to both S3 and HTTP URLs, rather than having to know which type of URL I have and then either use obstore (for my S3 URLs) or directly use an HTTP library (such as requests) with my HTTP URLs.

Is that what you're saying is the case?

[kylebarron]

rather than having to know which type of URL I have

Are we talking about credential providers still? When would you have S3 URLs in a credential provider?

But yes, for arbitrary HTTP APIs that do not represent a webdav store, you should be using something else, like requests.

(fwiw this is the idea of obspec, so downstream libraries can have a generic protocol and then users can choose between the obstore implementation or a generic HTTP client implementation)

[chuckwondo]

I think we're not quite on the same page. I'm not talking about using S3 URLs in a cred provider. When I'm talking about S3 vs. HTTP URLs, I'm talking about "top level" URLs that I can pass to obstore.store.from_url.

But I think it is now crystal clear that for HTTP APIs requiring auth, I must simply make direct use of some other library, using obstore only for my S3 URLs.

Thanks for clearing that up Kyle!