Parallel testing with a shared server and code coverage

[jtwaleson]

Hi all,

I started seriously using the test framework a couple of weeks ago but was a bit annoyed that it got slow. I noticed all tests were using serial. You're restoring a clean database with every test and tests can only run one at a time, while I have 16 cores on my machine. This is great for consistency, but not for speed.

I wanted to see if I could run my test suite parallel with a shared_server that only gets initialized once and here is the outcome. As a bonus, I also got code coverage to work (see below).

static SHARED_TEST_SERVER: tokio::sync::OnceCell<BootResult> = tokio::sync::OnceCell::const_new();

/// Initialize the shared test server if it hasn't been initialized yet.
pub async fn get_shared_test_server() -> (TestServer, &'static AppContext) {
    let boot = SHARED_TEST_SERVER
        .get_or_init(|| async {
            // This will only run once, so we can use block_on here
            let _ = env_logger::builder().is_test(true).try_init();
            let config = App::load_config(&Environment::Test).await.unwrap();
            let boot_result = App::boot(boot::StartMode::ServerOnly, &Environment::Test, config)
                .await
                .unwrap();
            reset::<Migrator>(&boot_result.app_context.db)
                .await
                .unwrap();
            // if you need any initializers to run manually, add them here
            // now seed the test data
            let path = std::path::Path::new("src/fixtures");
            App::seed(&boot_result.app_context, path).await.unwrap();
            boot_result
        })
        .await;
    let routes = boot.router.clone().unwrap();
    let request_config: RequestConfig = RequestConfig::default();
    let server = TestServer::new_with_config(
        routes.into_make_service_with_connect_info::<SocketAddr>(),
        request_config,
    )
    .expect("Failed to create test server");

    (server, &boot.app_context)
}

#[allow(clippy::future_not_send)]
pub async fn shared_request<H: Hooks, F, Fut>(callback: F)
where
    F: FnOnce(TestServer, &'static AppContext) -> Fut,
    Fut: std::future::Future<Output = ()>,
{
    let (server, ctx) = get_shared_test_server().await;
    callback(server, ctx).await;
}

Now you can run tests like this:

use serial_test::parallel;


#[tokio_shared_rt::test(flavor = "multi_thread", worker_threads = 20)]
#[parallel]
async fn test_tagging_system() {
    configure_insta!();

    shared_request::<App, _, _>(|request, ctx| async move {
        let (token, _user_id, _email) = create_user(&request, &ctx).await;
    }).await;
}

Note that you will need the tokio-shared-rt crate as a dev depency: cargo add --dev tokio-shared-rt, as otherwise the test framework will shut down the tokio runtime after a test, and it might still be in use by other tests.

Now all of my tests run in parallel and within 40 seconds instead of 70s. Also, as my app is supposed to be a high traffic multi-tenant environment (single db), the tests also help catch cross-tenant bugs. If another test modified my data, the asserts or snapshots will show it.

Whenever I need a serial test, I tag it with serial and these run after (or before) all of the parallel tests.

As an extra, I've also started using cargo-llvm-cov and now get beautiful code coverage output like this:

That actually took a bit of time to set up correctly, as I'm also using cranelift and the wild linker for fast compilation times.

It took a couple of days of tweaking but I'm now super happy with my setup!

[kaplanelad]

Thanks @jtwaleson for sharing!

In Loco, we also have the option to run tests with async builds. You can check it out in the docs here.

Have you tried it?

• If not: I’m curious how you run tests in parallel using the same DB, where each test truncates tables and inserts different seed data. We handle this scenario as well, and I agree—it can make tests run much faster.
• If you have tried it and it didn’t work: I’d really love to understand your issue and see how we could incorporate your approach into the codebase.

[jtwaleson]

I have not tried it, I looked at the functionality and realized it would run in parallel, but would still create/migrate/seed a database per test (which is expensive overhead, in my case, where I'd like to basically run hundreds of small integration tests).

So what I do now instead: I created a small library with helper functions (create a user, create an org, create a project). For each test case I create a new user/org/project programmatically and perform my tests within that context.

I make little use of the seed data from the fixtures, but I am only initializing one server and running the migrations once.

Just to emphasize, the server is now creating many users, orgs, projects concurrently, which helps me catch tenancy-breaking bugs :)

Not sure yet if I would want to include this as a standard feature in Loco, as it's pretty opinionated and has drawbacks. Looking forward to see what the community thinks.