Listening to updates from frontend client

[Ontopic]

It would be nice to create a websocket that receives a stream of updates from updates in the admin panel. I'm not sure what performance implications this has though and I'm sure others might have as well in the future, so I thought worth a discussion.

I for now see a few options;

1. Using a hook from Payload (beforerRead, afterUpdate, afterCreate?), that as a side-effect writes the raw "payload" to an "in-memory database / quick storage", a socket server (can be node, Golang or whatever) that's basically just listens to updates on postgres / sqlite / casandra / redis db and broadcasts them
2. Directly listen to the MongoDB updates through the watch functionality, funnelled through one endpoint (so one listener for Mongo)

Pretty sure there are more ways to go about this. Interested in hearing what you guys see as the most future-proof 🙃

[DanRibbens]

I was talking to @jmikrut about this yesterday actually. There are a few use-cases that need this kind of communication, though we are still concepting and a ways out from building these.

1. Collection Actions - I would like to have a flow for adding actions directly on collections that adds a place in the UI To call a method and the UI may be able to display the status of those actions in realtime to have more detail than just http allows.
2. Shared edit UX - consider two editors are trying to update the same collection item, I think we'll need websockets to give that interactive (think google docs) experience.
3. Serving content that actually requires realtime streaming of data (no idea how this works and there would need to be some demand for it first).

I like your architectural ideas, we haven't done this exploration yet. We're a ways out from having either of those, but these will likely be in the roadmap when we can turn our attention to that again!

[Ontopic]

Listening to the MongoDB container from one "broadcasting server" and keeping sockets from there works surprisingly well quite quickly. Only for simple collections or raw data of course, though. I'll update here what I explore, would love to hear if you guys have any updates as well (y)

[Ontopic]

Even for a complex field like the Rich-Text field, since it's based on Slate, I think even sending just diffs could be a possibility. For quick testing https://github.com/Jack-Works/async-call-rpc is a nice start to get running quickly without diving into the WebsSocket protocol ;)

[DanRibbens]

That looks promising. In addition to diffs, the multiple editor feature would need to broadcast changes to user field input focus. That would be simple enough to model out. Some other things to plan are how we want to have the workflow of saving changes and updating the live/published version. The current workflow where one editor working on updating a document and then clicking "save" will not be sufficient for multiple editors.

We will report back when we have a go at this one, no telling when that will be just yet.

[Ontopic]

I was on the most rudimental setup possible able to live-edit the same document today with someone. A million miles from production-ready, but at least good enough proof-of-concept for me

Linkdump: https://github.com/docket-hq/slate-sheikah#readme https://stackoverflow.com/questions/60183962/in-slate-js-editor-applyoperation-is-not-applying-split-node-operations-corr

[BrettBedarf]

Any update on this?

For shared edit UI, are there any unexpected technical hurdles to creating a custom rich text editor component that handles all of the websockets etc within? Is payload writing content updates to the db live or only after a user saves the post?

[jmikrut]

Hey @BrettBedarf — right now, updates to documents are saved to the database in two ways:

1. by saving the post
2. by configuring autosave - and then Payload will autosave documents as they are edited

We have discussed multi-user editing in the past, and building our recent Autosave feature was the first step to achieving that vision. But, true multi-user editing will be a future enhancement and we have not started working on that feature specifically. Is it an important feature to you?

[endlessuniverse]

Interesting info for offline-first and server sync with Dexie.js:

https://dexie.org/docs/dexie-react-hooks/useLiveQuery()
https://stackoverflow.com/questions/72344460/dexie-live-query-and-service-worker
https://github.com/dikshanain100/DexieIndexDB-SW-SYNC

DexieIndexDB-SW-SYNC

This is a project downloaded from Frontend :: https://github.com/AldoHub/Offline-Online-Sync-Client/blob/f8ed457cd199137f5e206aee66a3cf22f326c061/src/app/todo.service.ts#L145 Backend :: https://github.com/AldoHub/Offline-Online-Sync-API

Video Link :: https://www.youtube.com/watch?v=lrzRGyBeWpQ&t=1033s

It uses Dexie Lib for pushing data in IndexDB when application goes offline. Mongo DB is used for backend which gets updated as soon as application gets internet connection again. Once data is updated to MongoDb from IndexDB, it is deleted from indexdb from browser. The project also uses SW for offline functionality. Structure of project has been modifies as per Axis project i.e using Internal Http Client for centralized http call.

SW - For serving application offline Index DB - For storing data offline on browser Dexie DB - Library to store data in Index DB

[endlessuniverse]

https://graphback.dev/docs/datasync/intro/

The @graphback/datasync package consists of the Data Synchronization Schema plugin and compatible data sources, provides out of the box Data Synchronization strategies for GraphQL clients with offline functionality e.g. Offix.

NOTE
Currently this plugin only supports MongoDB data sources, with support for other kinds of data sources coming in a future release.

https://offix.dev/docs/introduction

Offix Datastore is a fully capable client side store for JavaScript apps with real-time synchronization over GraphQL. Offix Datastore saves all your data locally and enables a GraphQL replication mechanism to your generic GraphQL server, this way, your data is available with or without an internet connection. All changes made locally, when offline, are queued until an internet connection is available.

Offline-first
Offix Datastore allows you to build "Offline-first" apps with ease. User data is always available locally and the local database is synchronized with server as soon as the user is online.

GraphQL CRUD compatible
Offix Datastore supports synchronization with GraphQL CRUD compatible servers by default. If your server doesn't support GraphQLCrud, you can build your own synchronization engine.

Browser and mobile support
Offix Datastore works on all devices so you can build seamless offline experiences on any device. In the browser, Offix Datastore uses IndexedDB or WebSQL to persist data locally. On mobile, Offix Datastore uses SQLite.

React Support
Offix Datastore provides hooks to help React devs build offline apps quickly.