marshal dynamodb results directly to JSON

[jonsmirl]

When writing lambda functions that query dynamodb I often simply send this data back to the lamdba caller as a JSON result. I would find it useful to have marshalling functions that go directly from the dynamodb result to json (in addition to the existing ones). This would optionally let me skip a lot of pointless conversions and memory allocations building the intermediate go structure which is promptly destroyed one line later in my code. Note, I'm not bringing back just a single item, usually it is an array of several thousand items.

[jasdel]

Thanks for posting this feature request, @jonsmirl. Are you looking for the DynamoDB API calls to return a JSON string of the result directly, instead of the AttributeValues that are normally returned?Or are you looking for a utility that allows you to convert AttributeValue collections to the DynamoDB JSON representation?

[jonsmirl]

Does it work like this?

http request goes off to DynamoDB and the results come back as JSON.
Go then turns that JSON into a structure like this:
{
"ConsumedCapacity": null,
"Count": 23,
"Items": [
{
"thing": {
"B": null,
"BOOL": null,
"BS": null,
"L": null,
"M": null,
"N": null,
"NS": null,
"NULL": null,
"S": "70b2d28ecf7e01dd3a4ba88da3ddde44",
"SS": null
},
I marshal it with a structure definition and it turns in to:
error := dynamodbattribute.UnmarshalListOfMaps(result.Items, &items)
[
{
"timestamp": 1519510162585,
"type": "clip",
"thing": "70b2d28ecf7e01dd3a4ba88da3ddde44"
},
{
Which I then turn back into json with this:
res, _ := json.Marshal(items)

---

Two optimizations seem possible

1. Leave the data in JSON when it comes back from dynamodb. Let me get access to it without doing any transforms to it. The idea here is to make it easy to just send the result set on out to the caller.

2. Leave the data in JSON until dynamodbattribute.UnmarshalListOfMaps(result.json, &items) is called. ie avoid that first expanded representation and make the Unmarshal functions directly work on json.

Also, parallel optimizations can be applied to writing data to DynamoDB. My incoming data is already in a JSON string, it would be great if I could simply pass that string into DynamoDB unchanged.

My Go code is used to validate the user and to ensure they don't read/write from unauthorized places (I use CognitoIDs). The Go code doesn't touch the data in either direction.

On the plus side I am converting these functions from Javascript (where they run 500-1000ms) over to Go which so far is running in the 200-300ms range. Getting rid of pointless shuffling of the data by Go might reduce my run-time down to 100ms.

[warrenstephens]

A function that would convert AttributeValue collections directly to a valid JSON string would be very handy.

The JSON strings can be passed to all manner of library functions and other things -- without the need to pass thru Go specific struct definitions.

My current need is simply to pass the retrieved data via a .JSON file -- the Go code never uses the content of the object at all.

[jasdel]

Thanks for the update @jonsmirl and @warrenstephens. Since the AttributeValue type is used within as a field within a JSON document, there is not a clean way to inject the AttributeValue as a JSON string/byte slice.

Since we have a good understanding of the use case we can use this issue to track design proposals for this feature. There are a few ways I think this could be implemented with varying levels of "completeness"

Ideas:

1.) Add custom member to AttributeValue

Adding a custom member, (e.g. Raw) to the AttributeValue that can contain a JSON byte slice. This new member would always be used if not-nil. Serializing the AttributeValue to the request would write the Raw member instead of serializing the AttributeValue's other members.

Reading a response from DynamoDB, the Raw member would contain the response's raw value for the AttributeValue.

This would be limited to AttributeValue type only, does not cover map[string]AttributeValue, nor []AttributeValue. Requires the user to split a JSON document's map/list into separate AttributeValues.

2.) Add Additional byte slice API input/output parameters

For every API input/output that contains an AttributeValue parameter the SDK's generator would also generate a new Raw<ParamName> parameter that contains a raw JSON byte slice for the JSON document to be sent.

The second option seems to be more of a sledge hammer to the problem, and it would be better to find a cleaner, more concise solution to this issue.

[tarndt]

Any updates on this issue? Approach #1 would greatly increase the performance of some applications I maintain.

[neoramos]

I'm also interested on the resolution of this enhancement. thank you.

[codeherk]

Any updates on this? I'm interested as well.

[marc-ostrow]

I think a better example would be that json.Unmarshal of DynamoDB JSON would result in a map[string]types.AttributeValue because then one can use attributevalue.UnmarshalMap will get you your go type which can be json.Marshaled to normal JSON.

I had to port dynamodbattribute from v1 SDK so I could get my go type from a DynamoDB Streams Event

[vudh1]

Hi all, I have discussed with the team, and we are planning to implement this. However, there is no timeline for this yet, I will comment if we have any update.

[ryan-ju]

There is already a function

aws-sdk-go-v2/service/dynamodb/serializers.go

Line 3374 in 98519e6

func awsAwsjson10_serializeDocumentExpressionAttributeValueMap(v map[string]types.AttributeValue, value smithyjson.Value) error {

that does almost what we need.

This can be copied into a public function. This should only take a couple of days to implement.