SMP server PostgreSQL storage - feature branch (merge, dont squash) #1467
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* smp server: queue store typeclass * parameterize JournalMsgStore * typeclass for queue store * postgres WIP * compiles, passes tests * remove StoreType * split migrations * progress * addQueueRec * reduce type spaghetti * remove addQueue from typeclass definition * getQueue * test postgres storage in SMP server * fix schema * comment * import queues to postgresql * import queues to postgresql * log * fix test * counts * ci: test smp server with postgres backend (#1463) * ci: test smp server with postgres backend * postgres service * attempt * attempt * empty * empty * PGHOST attempt * PGHOST + softlink attempt * only softlink attempt * working attempt (PGHOST) * remove env var * empty * do not start server without DB schema, do not import when schema exists * export database * enable all tests, disable two tests * option for migration confirmation * comments --------- Co-authored-by: spaced4ndy <[email protected]>
* agent, smp server: PostgreSQL connection pool * simplify, create all connections before start * rename * remove imports
* refactor journal store * update postgres store methods to update STM cache only after DB update * todos * drop Co-authored-by: spaced4ndy <[email protected]> * fix GHC 8.10.7 --------- Co-authored-by: spaced4ndy <[email protected]>
…ted, combine tables (#1471) * smp server: expire messages in postgres database * tty * fail if nothing updated in db * remove old deleted queues * index * fix tests
…ngs on server start (#1478) * smp server: check queue storage settings on server start * fix incorrect postgres migration not throwing exception
spaced4ndy
reviewed
Mar 12, 2025
* tests: run agent tests with PostgreSQL SMP servers * agent tests with postgres database * enable tests * fix store log tests * fix test
…t loading and compacting, for debugging during migration) (#1480) * smp server: optionally maintain store log with postgres storage (without loading and compacting, for debugging during migration) * refactor * remove comment
…tate consistency (#1481) * smp server: mask database store operations from async exception for state consistency * remove unused, comments
…pirations (#1483) * smp server: expire only active queues * version * do not cache all queues while processing expirations * refactor * foldWithOptions_ * version * use shared lock when expiring all queues * use TMVar * comment * rename * remove fold options * do not create locks in the Map for temporarily loaded queues * fix * revert version
…equire postgres library in the main binary (#1482) * smp server: split postgres support to a separate executable, to not require postgres library in the main binary * comments * enable server_postgres flag by default, add CPP option to test * refactor * change default for server_postgres to False * diff
…1484) * smp server: CLI option to compact store log with PostgreSQL database * version * fix test
…urrency and error handling (#1487) * smp server: use COPY to import store log to postgres db * compact queues when importing to postgres * mempty * version * handle errors while expiring, mask async exceptions while getting queue * whitespace * version
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
remove lock from map on queue deletion?store cache misses?