s3 client. list_bucket_metrics_configurations() does not return documented response

[jonathanwcrane]

Hi there. So I'm doing (approximately)

session = boto3.Session(profile_name=opts_dict['profile'],region_name=opts_dict['region'])
s3_client = session.client('s3')
res = s3_client.list_buckets()
for b in res['Buckets']:
	bn = b['Name']
	print("---------------------------------------")
	print(bn)
	res = s3_client.list_bucket_metrics_configurations(Bucket=bn)
	print(res)

and for all 14 buckets in my account, I get a response that looks like this:

{'ResponseMetadata': {'RequestId': '047D2D7A74F939C6', 'HostId': 'ka0BBTFI7G1mIMZutxKIyAxpgVkDA0viv3yajzwi09valCoLXdwK9hZxqlJhKSVvIfgl7yU01jA=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'ka0BBTFI7G1mIMZutxKIyAxpgVkDA0viv3yajzwi09valCoLXdwK9hZxqlJhKSVvIfgl7yU01jA=', 'x-amz-request-id': '047D2D7A74F939C6', 'date': 'Thu, 09 Nov 2017 14:33:44 GMT', 'content-length': '185', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'IsTruncated': False}

I'm expecting an element called 'MetricsConfigurationList' in that dict, per this documentation: http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.list_bucket_metrics_configurations but it's not there.

Is this an error in the documentation, boto3 itself, or in my understanding of how list_bucket_metrics_configurations() is supposed to work?

Edit:

boto3 version: 1.4.4
botocore version: 1.7.43
OS: Sierra 10.12.6

[joguSD]

It's possible that the version of boto3/botocore you have doesn't know about this field. Try updating boto3 and see if the field is present. If updating doesn't work providing debug logs by calling boto3.set_stream_logger('') would be helpful.

[jonathanwcrane]

Alright I upgraded to 1.4.7 (the latest available via pip). Can you confirm that's as new as I need to be to eliminate that angle?

It's still happening, so I'll try to set up debug logs. I've never done that before so if you can point me in a direction to figure out how to do that efficiently I appreciate it.

Also I may not have time to figure this out before next week, so I would appreciate it if you could keep this open for a while. Thanks!

[jonathanwcrane]

OK that was easier than I thought. I limited the script to only look at one bucket and here's the output:

2017-11-09 16:53:34,538 botocore.hooks [DEBUG] Event before-parameter-build.s3.ListBucketMetricsConfigurations: calling handler <function validate_bucket_name at 0x105f24c80>
2017-11-09 16:53:34,538 botocore.hooks [DEBUG] Event before-parameter-build.s3.ListBucketMetricsConfigurations: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x1064817f0>>
2017-11-09 16:53:34,539 botocore.hooks [DEBUG] Event before-parameter-build.s3.ListBucketMetricsConfigurations: calling handler <function generate_idempotent_uuid at 0x105f248c8>
2017-11-09 16:53:34,539 botocore.hooks [DEBUG] Event before-call.s3.ListBucketMetricsConfigurations: calling handler <function add_expect_header at 0x105f271e0>
2017-11-09 16:53:34,539 botocore.hooks [DEBUG] Event before-call.s3.ListBucketMetricsConfigurations: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x1064817f0>>
2017-11-09 16:53:34,539 botocore.endpoint [DEBUG] Making request for OperationModel(name=ListBucketMetricsConfigurations) (verify_ssl=True) with params: {'url_path': '/cf-templates-1hydczjiawe2y-us-east-1?metrics', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.4.7 Python/3.6.0 Darwin/16.7.0 Botocore/1.7.43'}, 'body': b'', 'url': 'https://s3.amazonaws.com/cf-templates-1hydczjiawe2y-us-east-1?metrics', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x10645fe48>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'cf-templates-1hydczjiawe2y-us-east-1'}}}
2017-11-09 16:53:34,539 botocore.hooks [DEBUG] Event request-created.s3.ListBucketMetricsConfigurations: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x10645fdd8>>
2017-11-09 16:53:34,539 botocore.hooks [DEBUG] Event choose-signer.s3.ListBucketMetricsConfigurations: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x10643dcf8>>
2017-11-09 16:53:34,539 botocore.hooks [DEBUG] Event choose-signer.s3.ListBucketMetricsConfigurations: calling handler <function set_operation_specific_signer at 0x105f247b8>
2017-11-09 16:53:34,539 botocore.hooks [DEBUG] Event before-sign.s3.ListBucketMetricsConfigurations: calling handler <function fix_s3_host at 0x105ddf048>
2017-11-09 16:53:34,540 botocore.auth [DEBUG] Calculating signature using v4 auth.
2017-11-09 16:53:34,540 botocore.auth [DEBUG] CanonicalRequest:
GET
/cf-templates-1hydczjiawe2y-us-east-1
metrics=
host:s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20171109T215334Z
x-amz-security-token:FQoDYXdzEMf//////////wEaDL2/CMHTmnzTm79qfiKRAhD+adVhWUwI4WyNNAqI97A7pO7Jq18gY0fjW6SKJMUDQ7WTpWr7XMQMtSedEBv3u1ss1sxzThE+xVO6eIqxePVDFxDOVhOG7DVsaPghtjQ5Qb1EcbCPYt4FR9fdeikylnpfsaCmqYD/t1eeQuhv/YcK33JbOw+2AsA2Y367haUguTPt2K3ZMXsL2RllOFYJ+pkIZhsWfb5kPx/VCY8ZanlNd1pDakDtCBn9D4yC1v+0c1fVp8RD1gXIicYUv6v7XDPJox/JgtOrGV9mXQ/GfgOivAUCZWUhwj7/efa/WwAqrOIQH/p4Xp+tRFzeOrVjKu/aZQ8HYzYb8F1voExzgb6EqGDpe9zHOAqfTnyr2pbFHyi9mJPQBQ==

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2017-11-09 16:53:34,540 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20171109T215334Z
20171109/us-east-1/s3/aws4_request
0f53e532973e0c4123359e470e7402fe29f18d6600fa1fc911cb0a7caf2f7532
2017-11-09 16:53:34,540 botocore.auth [DEBUG] Signature:
32038ef9df9c8b27d70d3cf126f14494ea7fd267e498bbac1d1f889087c718be
2017-11-09 16:53:34,540 botocore.endpoint [DEBUG] Sending http request: <PreparedRequest [GET]>
2017-11-09 16:53:34,620 botocore.vendored.requests.packages.urllib3.connectionpool [DEBUG] "GET /cf-templates-1hydczjiawe2y-us-east-1?metrics HTTP/1.1" 200 185
2017-11-09 16:53:34,621 botocore.parsers [DEBUG] Response headers: {'x-amz-id-2': 'qyyZOBrreM97Verp8Ck8E/5Xxewfq6n2gm/hnutZ0PXkOAJHjTXQrAGwqwglii+CW7oKkd9fP2E=', 'x-amz-request-id': '010C793D7742CE6F', 'date': 'Thu, 09 Nov 2017 21:53:39 GMT', 'content-length': '185', 'server': 'AmazonS3'}
2017-11-09 16:53:34,621 botocore.parsers [DEBUG] Response body:
b'<?xml version="1.0" encoding="UTF-8"?><ListMetricsConfigurationsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><IsTruncated>false</IsTruncated></ListMetricsConfigurationsResult>'
2017-11-09 16:53:34,621 botocore.hooks [DEBUG] Event needs-retry.s3.ListBucketMetricsConfigurations: calling handler <botocore.retryhandler.RetryHandler object at 0x106481320>
2017-11-09 16:53:34,621 botocore.retryhandler [DEBUG] No retry needed.
2017-11-09 16:53:34,621 botocore.hooks [DEBUG] Event needs-retry.s3.ListBucketMetricsConfigurations: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x1064817f0>>
{'ResponseMetadata': {'RequestId': '010C793D7742CE6F', 'HostId': 'qyyZOBrreM97Verp8Ck8E/5Xxewfq6n2gm/hnutZ0PXkOAJHjTXQrAGwqwglii+CW7oKkd9fP2E=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'qyyZOBrreM97Verp8Ck8E/5Xxewfq6n2gm/hnutZ0PXkOAJHjTXQrAGwqwglii+CW7oKkd9fP2E=', 'x-amz-request-id': '010C793D7742CE6F', 'date': 'Thu, 09 Nov 2017 21:53:39 GMT', 'content-length': '185', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'IsTruncated': False}

Please let me know what other information I can provide to help you help me.

Thanks!

[jonathanwcrane]

Any update on this status, or workarounds?

[joguSD]

Looking at the debug logs, no data is coming back from the service. Do any of these buckets actually have metrics configurations? Can you get this data from any other tool?

[jonathanwcrane]

Thanks joguSd;

Well if this is expected behavior, the documentation should reflect that.

Secondly, another tool? Isn't boto3 the flagship SDK for AWS? I figure if the functionality exists and is expressed well anywhere, it should be boto3. So I don't know--what other tools are there that you know of that could do something similar? My eventual goal is to turn on response metrics for all buckets, but I figured I should see what the current setting is before I change anything.

j

[joguSD]

By other tools I meant the AWS S3 Console or other official AWS SDKs (we have SDKs for many languages such as ruby, go, etc). I was misunderstanding your issue, I thought you had a metrics configuration setup and were expecting the data to be there. If there is no metrics configuration, that field being omitted from the response is the expected behavior. Sorry for any confusion.

[jonathanwcrane]

Glad we're on the same page. Here are the action items I see from here:

• Update the documentation here: http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.list_bucket_metrics_configurations to specify the following:
  • There is also a "ResponseMetadata" element in the response
  • If there is no metrics configuration on the bucket, the "MetricsConfigurationList" element is omitted from the response.

Edit: I tried to use the "edit on GitHub" button on the ReadTheDocs page but it gives me a 404.

[jamesls]

The issue here is that this applies in general to all responses. If the key isn't present in the xml response we don't add the key to the returned dictionary. We should probably document this in a single location, but then people wouldn't find it if they go to the docs for a single API method.

Labeling this as documentation.

[jonathanwcrane]

We should probably document this in a single location, but then people wouldn't find it if they go to the docs for a single API method.

You can kind of work around that but adding a "the usual caveats apply" kind of link to the documentation of each function. Then you a) have it in one place and b) at least have a pointer to that location from every place where people would conceivably find it useful.

It's more automatable than having to templatize a statement like "the {{X}} element may be omitted from the response if there is no data on {{X}} available."