Upload staging bundle to fixed path in DO storage

[lukevalenta]

• 714c7a0
  Add support for multipart uploads to DO storage backend

  The KV API for DO storage limits values to 2MB, which we could exceed
  when sequencing moderately-sized batches (1000 entries of size 2kB
  each). Add support for splitting these writes across multiple DO storage
  values to avoid this limit.

• b92b8bc

  Upload staging bundle to fixed path in DO storage

  This commit includes two major changes:

  • Write staging bundle to a fixed path, fixes Write staging bundle to a fixed path #34.

    This means that we don't need to clean up old staging bundles from
    object storage. Insert the current tree size and hash into the staging
    bundle for good measure, but I don't believe this is necessary.

    As an added bonus, this cuts out one R2 write/sec (replaced w/ a write
    to DO storage, which should be faster).

  • Write staging bundle to DO storage instead of R2

    The reasons not
    to use a fixed path in sunlight are to avoid zombie writes/rollbacks,
    and against multiple sequencers writing at the same time. With the
    strong consistency guarantees provided by Durable Objects, these
    failure modes are not applicable.

    Zombie writes: The DO storage relay service requires that writes are
    acknowledged by a least three of five "follower" machines, which helps
    to ensure that all confirmed writes are correctly persisted in the
    proper order.

    Multiple sequencers: Durable Objects also guarantees that only a
    single 'scheduled' task can run at a time, so we don't have to worry
    about multiple sequencers.

  Also reword and simplify the storage 'break' modes in the tests.