[WIP] using db-pool library to create a pool of databases#5846
[WIP] using db-pool library to create a pool of databases#5846momentary-lapse wants to merge 87 commits intoLemmyNet:mainfrom
Conversation
momentary-lapse
requested review from
Nutomic,
SleeplessOne1917,
dessalines,
dullbananas and
phiresky
as code owners
| .await; | ||
|
|
||
| // TODO make compatible with ActualDbPool | ||
| db_pool.pull_immutable().await |
There was a problem hiding this comment.
I created this WIP PR to share the progress and the issue I'm stuck with currently. The crate I use operates with its own structure wrapping connection pools: code
And we have our own ActualDbPool. They are kinda same, but it's not obvious for me how to correctly convert one to another.
I had an idea to make ActualDbPool a enum with two possible values: RegularPool and ReusablePool, but stuck on trying to adapt stuff like LemmyContext, which also requires pool struct to be clone-able (and ReusablePool is not). And it seems a lot of changes to the main codebase for purely test changes.
Do you folks have any ideas how to manage that? Or should I stick to the initial plan without using this library?
There was a problem hiding this comment.
Our ActualDbPool is just a type alias for deadpool Pool<AsyncPgConnection>.
Their crate should be able to work with deadpool pools, but I'm not familiar with how to plug that into their crate... you'll have to ask them.
There was a problem hiding this comment.
Yeah, I see. I returned to this issue today after a week of a break. I'm in contact with the db-pool author and they're helping to understand a lot of moments and really willing to collaborate, so i think we'll make this work.
I'd like to clarify one moment: do we want build_db_pool_for_tests to return still ActualDbPool? db-pool has its own wrapper ReusableConnectionPool which works like a deadpool Pool, but a bit different and needs adaptation. And it might be easier to adapt tests for working with ReusableConnectionPool than converting ReusableConnectionPool to ActualDbPool
There was a problem hiding this comment.
The return type of build_db_pool_for_tests may be changed. Also, a DbPool variant may be added if needed.
| #[derive(Clone)] | ||
| pub struct LemmyContext { | ||
| pool: ActualDbPool, | ||
| pool: ContextPool, |
There was a problem hiding this comment.
This is the moment which currently blocks me, and i think it's better to consult with you again. LemmyContext structure must be cloneable, therefore all the fields, therefore the pool. But unfortunately, reusable pool from db-pool crate is not, and i don't have access to its fields to implement the trait here.
But before asking db-pool developer, i'd like to be sure we really need this pool cloning stuff, especially for the tests. Cloning the pool seems a bit strange to me, but i may miss something. I'm looking at the code now, but maybe you folks already have some insights on this
There was a problem hiding this comment.
Wrap it in Arc for now.
|
Update: I'm working on the topic; cannot devote much time for it, but it slowly going forward, and i keep the code in the branch up-to-date. I connected |
This reverts commit eba169d.
|
The tests are all passing locally for me (although there are many marked as ignore). Makes me wonder why its failing only in CI. From what I can see both .woodpecker.yml and test.sh are unchanged, so they should run tests in the exact same way. The error shown in CI is "migrations must be managed using lemmy_server instead of diesel CLI". Maybe its because diesel-cli is somehow used by the db-pool crate. I would try to deleted |
| let pool = &mut pool.into(); | ||
| let pool_arc = data.pool(); | ||
| let pool_ref = &***pool_arc; | ||
| let pool = &mut pool_ref.into(); |
Yeah, I temporarily ignored the tests which fail on my local machine. But some are still failing on a pipeline, probably because concurrency related issues. Trying to find out why the tests are not running in parallel, these might be related issues. I did some parallel tests on db-pool library in isolation, and it works perfectly. So the problem is most definitely in pool configuration for lemmy. I now (at least this week) have capacity to work on the issue more consistently, so I hope my investigations will bear some results |
There must be some mis-merge from main somewhere. |
|
Okay, for now i simply ignored cleanup errors in db-pool, and tests are passing now. Tried to disable There's a small speed improvement comparing to main build (~4 min). But i suspect that's because one heavy test is still ignored ( |
How many tests are currently running at the same time? Not sure where that is defined, but try to change the number higher or lower and see if it helps.
It might be possible to pass the port number as parameter to make them parallel. But there are only a few tests in that crate so it can be changed later in another PR. |
I hardcoded the pool size for tests to 30 for now. But both pool sizes (restricted and privileged) don't seem to actually affect the test speed at all.
Honestly, I tried to set up each test to its own port (changed it here and here) and run them together, but they started failing and hanging. I spent some time, but then switched to other stuff. |
4 participants
Addresses: #4979