Skip to content

Cosmos: Provide ability to separate storage of snapshots #61

Description

@bartelink

The unfolds mechanism, as documented in #50 provides the ability to efficiently stash relevant data facilitating reading of state based on a point read.

However, this is can be suboptimal in the cases of larger snapshot states:-

  • the data gets compressed and travels each time
  • writing/updating occasions read and write costs within the sync stored procedure each time
  • competing writers pay all but the update case in the case of a re-run of the same transaction
  • readers not interested in/unable to process a specific snapshot schema still have to pay the price to transport it
  • each update goes to the changefeed triggering more read costs
  • the size and cost of the snapshots is harder to separate out
  • snapshots contribute to partition size limits

Thus it makes sense to do some, or all of the following in the case where there is a projector in play and there is an aux collection (in the case of multiple writers one might even consider making an aux collection even if you don't have a projector):

  • maintain the snapshots in the aux collection, where one is not paying for changefeed re-reads of the same data
  • reduce the write frequency for snapshots (when the write happens should be deterministic for competing writers)
  • implement ability for stored proc to maintain events since snapshot in tip when not updating snapshot
  • write to a nominated collection (typically aux), with a deterministic guid based on the case / event type tag of the union as a point write
  • refer to the snapshot in the unfolds; automate loading of the referenced snapshot if picked by isOrigin
  • expose the ability to customize the snapshot loading process (but batteries should be included)
    • (perhaps do a sample showing parallel reads of the Snapshots and the Tip?)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions