Looking for advice on multi-crate diesel application design

[isosphere]

Backend

PostgreSQL

Diesel version

2.3.5

Diesel Features

"postgres", "numeric", "chrono", "network-address", "r2d2", "uuid"

Operating System Version

Debian Testing

Third party libraries

No response

What do you want to do?

I am building a modular application framework.

Common features used across many applications are provided by one crate. This crate includes diesel migrations for tables appropriate for these common features; i.e.: authentication, email verification, etc.

This module is then used by an application that needs that functionality, but it builds upon it with other arbitrary features which themselves have their own diesel migrations.

I am looking for advice on how I should structure this to make diesel migrations relatively painless to manage and run: i.e. centrally managed at the application level.

My current thinking is that I will want to use embedded migrations, possibly with some custom migration harness in a build script. I'm thinking that filesystem-based migrations will be tricky without keeping dependency migrations at hardcoded paths, though I might be able to get around that via the OUT_DIR build script variable. Perhaps I could clobber-copy over the migrations and then run them.

Compile time error

No response

What code do you already have?

I have separate migrations folders, but I have to run the migrations manually and separately.

Additional details

#559 (comment) might have a good starting point

Checklist

• I have already looked over the issue tracker and the discussion forum for similar possible closed issues.

[weiznich]

I think the most convenient method is to just use embed_migrations!() and bundle the migrations with your library. This still requires downstream users to:

• Call a function to run migrations (some sort of setup function)
• Possibly ignore the tables in their diesel configuration

Running different sets of migrations from different places is fine and won't cause problems. Diesel mostly checks that all migrations are applied, so there is no need to run them all at once or even in "order" as long as they don't depend on each other. If there is a dependency, you either need to make sure that you run these migrations first or run tham at least at once so that diesel sorts them for you according to the version. For the second approach a custom MigrationSource can be helpful to combine several sources into one. As matter of fact: That would also be something we would gratefully accept as addition to diesel_migrations itself, as it is really useful for others as well.

[isosphere]

Thank you for the reply!

I'll give the second approach a shot - there will be dependencies.

[isosphere]

My first step has gone completely painlessly, though I anticipate that I might have some trouble with reverting and redoing migrations in future. I might have been overthinking the problem, though.

I made a setup.rs binary with the following code:

use diesel::prelude::*;
use diesel_migrations::MigrationHarness;
use common::server::get_database_url; 

use auth::backend::MIGRATIONS as AUTH_MIGRATIONS;
use app::backend::MIGRATIONS;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let db_url = get_database_url();
    let mut connection = PgConnection::establish(&db_url).expect("Unable to establish a databse connection");

    connection.run_pending_migrations(AUTH_MIGRATIONS).expect("Authentication migrations have failed.");
    connection.run_pending_migrations(MIGRATIONS).expect("Application migrations have failed.");

    println!("Migrations state updated.");

    Ok(())
}

This is working fine, and I don't currently have any understanding of what a custom implementation of a MigrationSource would look like and what benefits it would afford me above this basic duct-taping, so I'll stick with it until it causes me some trouble.

[weiznich]

That approach will work as long as you don't have a situation where AUTH_MIGRATIONS contains a migration dependent on another migration in MIGRATIONS and vice versa. If that happens this obviously won't work anymore as you then need to sort migrations by versions to apply them in order. If you are sure that this won't happen for now this is a reasonable solution.

[isosphere]

That shouldn't happen for my case; the dependency structure is: crate-common -> crate-auth -> crate-app, with each building upon the prior crates.

I'll mark this as the answer then, thank you for your input!

[weiznich]

For reference: I also opened #4946 which includes beside other things such a combining source that would allow to mix and match migrations that depend on each other as long as the migration versions are sorted correctly.