Support DynamoDb

[bartelink]

In the spirit of #62, this ticket tracks the considerations for implementing a storage adapter for Amazon DynamoDb; It absolutely does not represent a roadmapped task or anything that's an organic need relevant to a current usage scenario.

The scheme used by Equinox.Cosmos seems to map to the [new DynamoDb transactional API) (https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/transaction-apis.html)

Not 100% about how best to manage OCC - the client Position tracks the etag in the Cosmos impl, and a detailed read and research of guarantees provided is obviously warranted. One thing that does seem clear is that it It seems the client token idempotency is definitely not relevant. Wild guess: Maintaining expectedVersion either inside Tip or as metadata within it and then doing the writes conditional on that should work.

One key difference is the 400K document size upper limit, which means:

• Unfolds can't practically be maintained directly in Tip - e.g. if you ever need to support a transition in a snapshot schema in a blue/green deploy scenario, that halves the 400K. This means that an equivalent of Cosmos: Provide ability to separate storage of snapshots #61 would need to be implemented
• Likely one should explore re-introducing the [Tip isa Batch] semantics removed in Remove Tip isa Batch semantics #58 as that is in general an efficient scheme in terms of reducing document counts, contention and Request Capacity Init consumption Should follow the Equinox.CosmosStore v3 impl, which implements Tip isa Batch semantics
• The 400K may make Provide practical default Event size limit checks at storage level #31 more relevant than it has proven for CosmosStore - client side mechanisms about managing how one is faring wrt the limit will be big consideration given its 3MB vs 400K - the former is effectively unlimited for most reasonable sets of events and/or unfolds one might be maintaining.

[bartelink]

NOTE: It's important to note, before embarking on an implementation, that it is believed that DynamoDb does not facilitate full replays of all events in the same manner as the CosmosDb Change Feed does -- the key implication being that doing a full re-projection of all existing events will not be possible as it is for CosmosDb (which means one would need to find an alternate way to enumerate streams and do the initial projection, prior to letting a change feed processor work off events as they are being written)

[CumpsD]

@bartelink just out of curiosity, when you say the key implication being that doing a full re-projection of all existing events will not be possible as it is for CosmosDb, why is that? What is DynamoDb missing exactly?

[bartelink]

@CumpsD First off, most of the above is based on a few hours of scanning docs over the course of a few days so I can definitely be wrong.

I was trying to suggest that there may be a gap between:

• what one assumes possible on CosmosDb: you can re-traverse (either to denormalize or e.g. emit to Kafka etc) all the events in order on a live system, knowing you have not missed an event (the CFP scheme presents a way to have multiple aggregate-cursors across each of the physical partitions (even as they split), which can be trivially rewound and reprocessed)
• what my scanning and some informal discussions with other suggested for DynamoDb: there does not seem to be a clear way to select out all the documents in order (either fed out via Kinesis or via some querying mechanism) such that one can do the same: know you've traversed every event and be able to continue to maintain that link (the way I was reading it is that the notification scheme seems to emit as the updates happen, but as a black box thing)

Its very possible I'm wrong and I'd be delighted to be so - I've done lots of googling and not found anything that definitely says no; it's more me saying "I have seen nothing that suggests there's a way to effect a re-traverse" to blow away a denormalised/projected form and rebuild.

[CumpsD]

Thanks for your feedback @bartelink.

I'm starting a new pet project and was considering using DynamoDb as the ES. Didn't expect such a crucial thing to be missing though, I'll double check too.

Too bad there is no EventSource SaaS offering, that would be nice :)

[bartelink]

Yes, I suspect more folks than just me have asked about such a thing, but here we are ;)
Would love to see it implemented; even if I personally don't do any aws stuff atm, it'd be great to see how well it can be made to run
I continue to believe the schema in Equinox.Cosmos represents a very good way to manage events (but also optimistic-concurrency-driven syncing of any state, even with a single document and no actual events as such) in a Document DB - would be great to have my confirmation bias independently validated!

[bartelink]

FYI @CumpsD Slightly necroposting but
#321 is nearly out the door - I ended up implementing a changefeed too (there's a Propulsion.DynamoStore and a Propulsion.DynamoStore.Lambda)
... And there is an ESDB Cloud now

[CumpsD]

Come a far way 👍🏻 :)

[bartelink]

(Beta nugets have been available for a few days)