Reducing migration friction during development

[colinhacks]

When developing an application and rapidly iterating on the schema,

It may be worth noting that a Cloud-first development process eliminates a lot of this friction. Once an initial development instance is up and running, we could have a DX that feels

1. the process of creating and applying many migrations can get burdensome.
2. It's not currently possible to clean up a messy migration history without blowing away the database.
3. Users are experiencing issues after forgetting to re-run code generation or forgetting to apply migrations, which may signal some unnecessary complexity/friction in the workflow.

Below are a number of non-mutually-exclusive proposals for alleviating this.

1 Headless EdgeDB

Provide a way to run EdgeDB or a partial version of it in a "headless" way. Code generation commands can spin up a "headless" instance on demand instead of connecting to a project-linked instance.

Altneratively, this could be a standalone Python package that parses SDL into an introspectable form, no backend SQL required.

Regarding code generation

Once something like this is implemented, all code generation tools should be modified to take advantage of the "headless mode". In this scenario, the schema file(s) would act as the source of truth for the codegen tool.

To prepare users for this and minimize transition difficult, we should consider modifying all codegen tools to verify that the database is in sync with the latest schema files. This can be achieved by checking the result of start migration to.

2 Applying schema changes without creating migration files

edgedb migration apply --dev-mode

Alternatives:

• --unstaged
• --schema-only

This creates and applies a migration in one step, considering the following:

1. Extra migrations in the database are okay, new one is created on top
2. No prompts (like migration create --yolo below)
3. No file for migration is created

If revision history in the files and database diverge, the command does the following:

1. Updates database to the latest schema without touching files
2. Creates a temporary database
3. Applies file history to the temporary database
4. Creates a new migration to the current schema
5. Rebases history of the orignal database to match temporary database using ALTER MIGRATION (see below)
6. Drops temporary database

Note: (4) can return empty diff, this means rebase is simpler.

3 Migration squashing

The following command is used to "switch from" dev mode and provide committable
migration:

edgedb migration create --squash

Aternatives:

• --from-dev-mode

This mode fails if schema in the database != schema in the files.

By default migration create will ask interactively whether you want to squash if it detects, that schema matches but there are multiple extra revisions in the database.

Internally this works in the following steps:

1. Check for consistency: START MIGRATION TO ..., and verify diff is zero
2. Create a temporary database and apply all migrations in files.
3. Do START MIGRATION TO and do regular (preferably interactive) migration process, storing the migration file
4. Drop temporary database
5. Do ALTER MIGRATION .. on the original database

New EdgeQL syntax:

ALTER MIGRATION <name> ONTO <parent-name> {
    CREATE MIGRATION <name1> ONTO <parent1-name> "{"
        [ SET message := <message> ; ]
        <subcommand> ; [...]
    "}"
    [...]
"}" ;

This works along the lines of the following:

1. Recreates schema up to a <parent-name> from migrations recorded in the database
2. Applies all the migrations in the statement body
3. Applies all the migrations recorded in the database after <name>
4. Verifies that computed schema matches current database schema
5. Rewrites in-database history

Note: this operation never changes schema and data

Generally this allows full rebase process for migrations, but I'd reserve this only for migration apply --dev-mode and migration create --squash in 3.0.

4 Branching

This proposal involves introducing a concept of a branch. Branches are a schema-only concept. Creating a branch does not copy any data.

In practice, each "branch" is a new database with a UUID as the name.

This requires some notion of a "checked out branch". By default the checkout out branch/database is default. This is tracked in edgedb.toml.

[edgedb]
server_version = 2.1
current_branch = "default"

To check out a new branch:

$ edgedb migration checkout newbranch

This will:

• Create a new database with a random UUID as the name and applies all migrations from default against it
• Create a new directory dbschema/migrations/newbranch containing metadata.toml file to track the parent revision:

  [metadata]
  db_name = "<database uuid>" 
  parent = "<revision hash>"

• Update edgedb.toml to set current_branch to "newbranch"

From this point on, all created migration files are written to dbschema/migrations/newbranch.

Once the user is satisfied with their changes, they can squash them back into default.

$ edgedb migration squash default

All migration files in dbschema/migrations/newbranch are squashed into a single migration file and written to schema/migrations.

• This squash could simply concatenate together all the commands in the various newbranch migration files, or it could be more clever (e.g. if a property goes from optional to required then back to optional, it can drop the
• If default has changes

Then dbschema/migrations/newbranch/metadata.toml is updated to indicate that the branch is "fully merged".

parent = "<revision hash>"
merged_into = "<revision hash>"

This is roughly analogous to merge --squash conceptually. Once a branch is merged back into its parent, it shouldn't be possible to add more migrations in that branch. I don't think an equivalent of rebase is practical or useful?

5 "Live mode"

Basically "live mode" is edgedb migration apply --dev-mode with watch. But there are nuances:

1. Fixtures (Seeding/fixtures/migration hooks #4288) should also be updated
2. Fixtures are not idempotent and reflect latest schema, so sometimes they can't be applied. So edgedb database wipe must be done before.
3. Codegens should also be updated for some languages

So to make it more more convenient another meta command should be added:

edgedb watch

A way to synchronize the database with the latest version of the schema files without generating a migration file. Likely will involve clearing the database contents and possibly auto-running a seed script/fixture afterwards.

Alternative names:

• edgedb live
• edgedb monitor

6 migration create --yolo

Per geldata/gel-cli#796.

Changelog:

1. At (2): migration push -> migration apply --dev-mode (it should not store file, and keep number of commands at minimum)
2. At (3): squashing section is fully rewriten
3. What was in live mode (4) is basically moved to (2) and (3). And "live mode" section now contains watch command

[1st1]

Altneratively, this could be a standalone Python package that parses SDL into an introspectable form, no backend SQL required.

FWIW I experimented with creating a standalone light-weight self-contained executable that does exactly that. It's doable. cc @elprans

[tailhook]

Can we just run edgedb-server --schema-only --listen-fd, or edgedb-server --language-server, or similar? EdgeDB is currently "portable" so we can use its distro and not manage packages of any other executable. (Is it what @colinhacks, regards as Headless)?

If we want "self-contained executable" we can do it for edgedb-server. Or does the presense of postgres stops us from doing it?

[tailhook]

I think the biggest pain in "reducing migration friction during development" is not generating migration files "headlessly". It's the fact that to run the app, migrations must be applied to the DB. So if users do it with the current workflow they will end up with tens of migrations during single feature development.

We need live schema update without producing migration files. Or with keeping track of which schema files can be safe-fully squashed (which is not possible in the general way, but we might develop some subset of it that might work for most users). Note that with squashing there is quite a big chance that something will be committed prematurely or squashed more than needed (and recovery is painful). So I'm looking forward for "no-artifact live schema update" as DX (even if internally it's done by squashing).

[colinhacks]

Added "live mode" proposal - feel free to directly edit the discussion with additional detail/considerations.

[tailhook]

internally squash all unreflected migrations into one

At a glance, I like this approach. Just not sure that migration create by default doing that is a good idea. I need to think more, and provide more feedback/editing later.

[tailhook]

Updated according to my vision. See changelog at the bottom of the discussion.

[tailhook]

Another problem with "headless" mode, is: which version of headless EdgeDB should be used for migrations/codegen? Is it specified in edgedb.toml? Is it used in the current project? What if project have not set up (i.e. either no linked instance, or it is inaccessible, or no edgedb.toml)? Do we still have to connect to the database to get the relevant version? (in this case "headless mode" makes not much sense).

[haikyuu]

My two cents on this:

• edgedb migrate --dev would create a "temporary" migration file and apply it without prompts.
• The developer will need to edgedb migration create && edgedb migrate when he's done with a feature. This is the normal flow of migration.
• Having a flow like this for development might mean adding first class support for seed files. i.e each migration would have its seed file that inserts/updates/deletes data after that migration is applied. Why? in order to have a db that's throwable and be okay with data loss
• This means that there'd be two instances: one for dev to apply temp migrations to + another one that the normal migration workflow is applied to.

I don't think generating query builder code is a pain point here. It's one command with no prompts.
Prompts, having to make decisions (especially not binary ones), and speed are the pain point. Ideally I want changes to the schema to be reflected in my db asap.

Of course this would be done behind a flag.

[tailhook]

Yes this is close to what I'm thinking of. Additional thoughts:

1. I'm not sure seed files should be connected to migrations. You can have .edgeql file(s) to seed the database and fill it with additional info as you see fit. We might help applying it (so you don't need to much bash scripting), but that should be enough.
2. There are things that is hard to make seed file for. Good example is a user session: it's hard to put it into a seed file, but if you have to log in again after every schema change it might cause too much friction.
3. I'm not sure if "two instances" (this probably means two databases in the single instance in our terminology) is good enough. I mean, we should somehow simplify it even more, because switching instance/database is not frictionless (you need to restart app with env variable for example, and also different data can be confusing to test).

But generally yes, the core of this issue is whether we want to keep data intact (or start from scratch/seed) in migrate --dev mode and how specifically it can be done.

[tailhook]

Another thing to account for is: test driven development (TDD). Currently, you can do edgedb migration create --non-interactive --allow-unsafe (or edgedb migration create --yolo after geldata/gel-cli#796), but that leaves file(s) around. We should have a migration push --no-artifacts or migrate --dev --no-watch, so that tests can be run on test instance/database without messing with files.

[elprans]

On "2 Applying schema changes without creating migration files"

I think we can and should collapse the notion of dev-mode and squash into one for simplicity. How about:

We add edgedb migrate --auto -- automatically creates a migration file before migrating (equivalent to edgedb migration create --yolo && edgedb migrate). Adding --amend updates the top migration both in the filesystem and in the database, so the "dev-mode" flow becomes edgedb migrate --auto followed by edgedb migrate --auto --amend. Once schema is satisfactory, no extra operations need to be done as a valid migration is already there.

[haikyuu]

Love this proposal. It's very familiar, feels like running a dev server

[tailhook]

This was my initial appoach. But the problem is that CLI doesn't know whether the last revision comes from git pull or it comes from previous migrate --auto. So after git pull or git commit user has to manually run edgedb migration create manually, and this has to be called when first change is done (otherwise it will not create a new file and next git migrate --auto will modify already pushed revision).

And this also means that --watch is basically unimplementable without extra interactive questions or some "magic" behind-the-scenes metadata tracking.

So the idea behind "without creating migration files" is: you modify schema in the database until satisfactory, and once file is created, revision is frozen.

[elprans]

I'm not sure I understand the problem here. The suggested --auto and --amend are context-free. --amend always updates the top migration (unless already applied).

[tailhook]

The following sequence of commands screws up migration history in your repository (if new the new migration was pulled in):

git pull
edgedb migrate --auto --amend
git commit
git push

While this:

git pull
edgedb migrate --dev-mode

Doesn't change files, so git commit does nothing until you explicitly do migration create.