Skip to content

Cosmos: Provide ability to separate storage of snapshots #61

@bartelink

Description

@bartelink

The unfolds mechanism, as documented in #50 provides the ability to efficiently stash relevant data facilitating reading of state based on a point read.

However, this is can be suboptimal in the cases of larger snapshot states:-

  • the data gets compressed and travels each time
  • writing/updating occasions read and write costs within the sync stored procedure each time
  • competing writers pay all but the update case in the case of a re-run of the same transaction
  • readers not interested in/unable to process a specific snapshot schema still have to pay the price to transport it
  • each update goes to the changefeed triggering more read costs
  • the size and cost of the snapshots is harder to separate out
  • snapshots contribute to partition size limits

Thus it makes sense to do some, or all of the following in the case where there is a projector in play and there is an aux collection (in the case of multiple writers one might even consider making an aux collection even if you don't have a projector):

  • maintain the snapshots in the aux collection, where one is not paying for changefeed re-reads of the same data
  • reduce the write frequency for snapshots (when the write happens should be deterministic for competing writers)
  • implement ability for stored proc to maintain events since snapshot in tip when not updating snapshot
  • write to a nominated collection (typically aux), with a deterministic guid based on the case / event type tag of the union as a point write
  • refer to the snapshot in the unfolds; automate loading of the referenced snapshot if picked by isOrigin
  • expose the ability to customize the snapshot loading process (but batteries should be included)
    • (perhaps do a sample showing parallel reads of the Snapshots and the Tip?)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions