Data durability and consistency with datascript and datahike

[arkRedM]

I will position this close to the problem statement of real-time collaboration than anything else.

Setting up context, the first part of rtc that you are seeing in athens-backend and app is a way to stream transactions. We used

1. datahike as single write
2. datascript as client db
3. ws to stream txns

But the current one won't cut it. More specifically the data flow is not enough to ensure data durability and consistency.

I'll talk about it in the loom

1. Durability - https://www.loom.com/share/2a73776b2c42489b9e4df80aaa94acfe
2. Consistency - https://www.loom.com/share/8613cc912cb44d7482125b1f0caee3a3

PR backend: https://github.com/athensresearch/athens-backend/pull/2
PR athens app: #1075

Let me know if I couldn't get the ideas out correctly or if something feels fragmented.

[tangjeff0]

Just to make I understand the loom videos correctly, the new changes are that the client db must now wait until the server acknowledges the previous transaction before sending new transactions. This is sort of like an ACK pattern a la TCP?

[arkRedM]

Not just ack(that can be only be done with exact copy of data -- like I talked in loom). here we are making aware of final data state from backend with updated remote-ids.

[tangjeff0]

This (v2) is more responsive than v1! Did some stress testing with the newest commits on backend (https://github.com/athensresearch/athens-backend/pull/2/commits/260e54477f9eee4e2e6d8dbb13c6b3312acbf141) and frontend (ec83342).

I know frontend and backend latest updates are both drafts, but there are still sync issues. Demo: https://www.loom.com/share/3f5aa016d830472da966a40e4f94dab0

I remember @neotyk said a single transactor was the way to go. Doesn't seem like we are doing that right now. Any reason to not try that for now? Will be slower but consistency seems more important for an MVP.

[arkRedM]

single transactor

backend is still a single transactor. for the client-side we are using the second db to sync and add txns irrespective of curr user view state. Tangling view, data and sync was not great and lead to all weird things. Maybe we can ditch db-with-remote. Need to think more.

[arkRedM]

but there are still sync issues

First one(bug-1) was interesting. Not sure how that happened.
Rest I'm actively looking into.

[tangjeff0]

backend is still a single transactor. for the client-side we are using the second db to sync and add txns irrespective of curr user view state. Tangling view, data and sync was not great and lead to all weird things. Maybe we can ditch db-with-remote. Need to think more.

I think this was the motivation for making the client-side db read-only. Anyway to test that out?

[tangjeff0]

Setup a remote EC2 instance. Anyone can connect with the following credentials:

address: ec2-3-16-89-123.us-east-2.compute.amazonaws.com
token: x

A less powerful server (and further networking) seems to make it easier for clients/db to get out of sync.

[tangjeff0]

Synced up with @neotyk. Notes from our call.

Problems

• fix shape of data: tx-id, user-id in different places fix(rtc): shape of data - tx-id, user-id in different places #1141
• redo construction of context-dependent tx-data ~> fix(rtc): common events #1140
  • define pure verbs for blocks, pages, links, orders, strings, children, e.g. indent, split-block, write-string
  • centralize all side effects to the very end at the edge as a CLJC library
  • the CLJC library manages all context
• with number 2 done, we can separate frontend and backend so local or remote dbs resolution is easy

Other Notes

the problem is that each tx expects the current state of datascript to be correct
this is why we had to debounce enter
alex: removed debounce and "Locked" the current block

how do we construct tx-data?
much of the time, tx-data looks at surrounding context -> siblings, parents, kids, target, origin
is it possible to "pub" messages that do not look at surrounding context, and then the single writer db determines final datascript state?

is it possible to send context-independent, declarative transactions? then let backend resolve "side effects"

"side effects"

• order (enter, indent)
• block/string (join)
• walk-string
  • determine if a page or reference exists or not
    • "" -> "[[test]]" if test page doesn't exist yet, create page and reference
    • "" -> "[[test]]" if test page exists, create reference
    • "[[test]]" -> "" delete reference
    • "[[test]]" -> "" and it's only the reference, delete reference and delete page
    • "((block ref))" create reference if it doesn't exist yet

[tangjeff0]

If each client can correct tx-data, how do we know they would correct them and eventually arrive at the same state?

[neotyk]

• In a situation where:

  • Page has a list of 10 blocks
  • 2 clients are connected
  • both clients in the same time create new second block (by pressing Enter at the end of 1st block)
    • tx from client to server approach:
      • Each client adds new block and reorders 9 blocks + updates parent
      • Each client applies these transaction to their best known state of DB
      • Both clients send tx log to backend (:db/ids are managed)
      • Whichever clients data arrives first get's applied to backend DB
      • Now interesting things happen when second client's tx log arrives
        • We'll create another second block
        • reordering of remaining 9 blocks should be the same
        • parent update from tx log does not include knowledge of changes made by 1st client
        • We'll end up with 1st clients 2nd block orphaned
    • verbs/events to server approach:
      • Both clients send an event to server :create-new-block-event
        • This event includes knowledge of last seen transaction by this client
      • Server receives 1st client's event
        • resolves :create-new-block-event in context of only true DB and transacts
        • tx-log get's broadcasted to connected clients
      • Server receives 2nd client's event
        • Compares last-seen-tx by client to real last-tx and rejects the request because out of sync, communicates what is global last-tx
      • 2nd client receives rejection with last-tx
        • Client compares it's last-seen-tx
          • if already up to date sends same event again
            • (because broadcast was received before rejection)
          • If not up to date, register tx-wait-event that when awaited tx arrives get's fired and send the event/verb again

• I don't see a solution to provide consistency in the 1st scenario where we share tx-log from client to server, what did I miss?

• Maybe I'm missing something in my approach.

[arkRedM]

First of all two approaches are not different in terms of actions vs txns being sent. Actions or txns the end result is the same(maybe some data format but nothing out of normal). The only change that is happening here is rejecting a txn based on last-tx-id.

In scenario described the most expected solution is both new blocks get added without losing any data. It makes more logical sense as both are independent blocks of a parent. I am not sure locking/rejecting such a txn even makes sense. There is actually no conflict in data layer at all and no implicit reason to block the txn.

Similarly with same block data change and locking. Last one should always win

[neotyk]

If we both modify same block, and your tx got transacted 1st, I would expect my tx to be rejected, and not overwrite your changes.

[arkRedM]

Why reject when there is no conflict?

[arkRedM]

server: 45.79.158.23:3010
token: x

Running with most minimal setup(need some testing and rtc sync testing here -- server in NJ -- expect delays not bugs). Please record loom/some info that can help debug issues. This will help productionize RTC faster
Thanks.

Note: please check front end from dev https://github.com/athensresearch/athens/pull/1075/files

[tangjeff0]

@arkRedM could you add your thoughts to the newest frontend and backend PR? I saw you removed datsync, not sure if there are other larger changes being made https://github.com/athensresearch/athens-backend/pull/2/files. Communication very helpful once there are multiple people thinking and working on something.

[arkRedM]

Datsync was not adding much value sticking around in the backend. I just picked fundamentals and some code from datsync to manage how txns are handled and removed the dep.

Regarding front-end datsync still manages most of remote id stuff so keeping it there until I find a cleaner abstraction. But it's non-blocking in the current set.

Other major changes we have been discussing in the backend channel and this thread.