Conversation
This commit removes the datastore plugin threadpool executor that was serializing all database reads / writes inside a single thread. Instead it is replaced with per-thread SQL connection objects with generation tracking (for noting when we need to reopen the database due to config file upload). Reads are MT-safe, writes are serialized behind a threading lock primarily for the purposes of enterpise HA and cross-node database consistency.
themylogin
left a comment
There was a problem hiding this comment.
Was there any performance bottleneck somewhere or is there any other reason to do this?
themylogin
left a comment
There was a problem hiding this comment.
If we want to achieve concurrent read and write, PRAGMA journal_mode=WAL; must also be enabled.
I haven't robustly checked for performance issues, but I'm concerned that a slow writer (if we're having to replicate changes) can block reads significantly. I'll enable WAL. I was also thinking of plumbing in asyncio taskgroups to concurrently gather render context for etc file generation and testing. It's kind of pointless though if entire datastore layer is single-threaded. |
|
Sorry, but I am not sure the problem even exists. Our database is 1MB tops, it fits entirely into RAM, how can there be slow readers or writers? |
This commit removes the datastore plugin threadpool executor that was serializing all database reads / writes inside a single thread. Instead it is replaced with per-thread SQL connection objects with generation tracking (for noting when we need to reopen the database due to config file upload). Reads are MT-safe, writes are serialized behind a threading lock primarily for the purposes of enterpise HA and cross-node database consistency.