Issue with encoding when using with other services than elasticsearch

[ulf5]

When using AWSRequestsAuth with e.g. execute-api service, then there are issues with urlencoding of spaces. The requests lib uses quote_plus and this causes issues like this:

The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The canonical string for this request should have been

When a space is encoded as + instead of %20.

What do you think of a optional parameter that allows specifying of the quote-function? The elasticsearch lib handles this by itself, but it would be nice for other services.

[DavidMuller]

Thanks for the report @ulf5.

The elasticsearch lib handles this by itself, but it would be nice for other services.

Yes, as you've observed we previously removed query string quoting in #13 particularly because the Elasticsearch client libraries already do their own quoting.

aws-requests-auth/aws_requests_auth/aws_auth.py

Lines 202 to 220 in 10d4d0b

	def get_canonical_querystring(cls, r):
	"""
	Create the canonical query string. According to AWS, by the
	end of this function our query string values must
	be URL-encoded (space=%20) and the parameters must be sorted
	by name.
	This method assumes that the query params in `r` are *already*
	url encoded. If they are not url encoded by the time they make
	it to this function, AWS may complain that the signature for your
	request is incorrect.
	It appears elasticsearc-py url encodes query paramaters on its own:
	https://github.com/elastic/elasticsearch-py/blob/5dfd6985e5d32ea353d2b37d01c2521b2089ac2b/elasticsearch/connection/http_requests.py#L64
	If you are using a different client than elasticsearch-py, it
	will be your responsibility to urleconde your query params before
	this method is called.
	"""

^ that PR also left a comment indicating that other users of this library would be responsible for their own query string quoting.

Is it possible to do the url quoting on the url you are passing into requests in the first place (e.g. mimic what the Elasticsearch clients do already)?

[ulf5]

Thank you for the response!

It is possible to quote the url beforehand. But this kind of removes the convenience of using the requests library. And since it seems rather rare to have requirements that disallow + for encoding spaces, would it not make sense if this could be done here? Since it is a requirement set by AWS.

[DavidMuller]

Thanks for getting back

I could see the convenience of putting the optional quoting in this package, I agree. That said, at this time I would probably not accept a PR that added optional query param quoting (this library is postured around ES and intention is to keep it as focused on request signing (vs e.g. url quoting) as possible).

Writing down some alternatives I thought of while googling about this:

• Based on this stack overflow answer it seems possible to (at the last moment) convert a "params dictionary" into a params string with the desired quoting %20 vs + quoting:

import urllib
import requests

params_dict = {'key1': 'value 1', 'key2': 'value  2'}
params_str = urllib.parse.urlencode(payload, quote_via=urllib.parse.quote)
requests.get('http://example.com', params=params_str)

---

• Pretty hacky...but it would be probably be possible to "chain" together AWSRequestsAuth with another auth callable to handle the url quoting issue in a way that feels seamless...

import requests

aws_auth = AWSRequestsAuth(required_args='etc.')

class CustomURLParamEncodingAWSAuth(requests.auth.AuthBase):
    def __call__(self, r):
        # custom url param quote handling on r
        r = aws_auth(r)
        return r

requests.get('http://example.com', auth=CustomURLParamEncodingAWSAuth())

---

• Another request signing library to give a shot might be aws4auth

[ulf5]

I was unaware that requests did not encode the params when it is passed as a string. Yes, I agree that I could do that.

Anyway, I will keep using this library because the project is mainly using ES, but I stumbled onto this issue when using the same auth for a simple get request to an API gateway (one out of many in integration tests) and lost a lot of time figuring out exactly what was special about it. No matter if this is remedied or not, I hope that this issue can be found by someone experiencing the same issue and can hopefully resolve it quicker.

Edit: Also since this is a special requirement on the query string set by AWS, it makes an otherwise valid request fail. It was my thinking that it could be handled nicely in the package made for communication with AWS. But the philosophy of doing one thing and that thing being signing does make sense as well.

[pghole]

@DavidMuller, @ulf5 Which version of requests you are using. I am using 2.25.0 and while debugging the get_canonical_querystring function is do see some %20 in my query for space. I am passing the parameters in the requests as follows
requests.get("http://xyz.com/abc?uploadId=12&partNumber=124") --> Is this passing params as strings

Do I still need to encode it again ?

Can you guys suggest me the changes I should make to have this s3 compatible. I am performing multipart uploads ? - The functions I am concerned about are the get_canonical_querystring() and get_canonical_path(). https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html

@DavidMuller Do you suggest using a unquote() first and then use quote() since I am not sure whether requests library
is encoding that or not ?

The apis I am making requests to are - ?uploads, ?partNumber=121&uploadId=uploadid

[DavidMuller]

Can you guys suggest me the changes I should make to have this s3 compatible

Hi @pghole - this library has turned out to be relatively limited - it seems to work best with the Elasticsearch service.

I'd suggest giving another request signing library that explicitly supports s3 a try. For example: aws4auth.

[pghole]

@DavidMuller Thanks. This library works very well for my s3 use-case but I was verifying the signing process with aws documentation for my understanding. Using requests is already encoding the query params. I think changes in the https://github.com/akarnani/aws-requests-auth/blob/master/aws_requests_auth/aws_auth.py#L196-L202 can take of both situations.Any reason why this code is not there in the library now ?

[ScottHull]

I've found that having https:// in the aws_host parameter for AWSRequestsAuth can cause this same issue.