Skip to content

Conversation

@davenquinn
Copy link
Member

@davenquinn davenquinn commented Jan 5, 2026

This PR reworks schema management for Macrostrat database to be based around a declarative representation of the schema. Now, the repository has a unified representation of the Macrostrat schema in the desired state, and tools to test and ensure conformance of each environment to this state. The major benefit of the declarative approach is that there is always a unified representation of the desired state of the schema, making it easier to reason about and modify.

Here, we centralize and merge schema-management tools into three major parts:

  1. Full schema provisioning - build of all schema elements in their desired state
  2. Automated schema diffs - evaluate any version of Macrostrat's database against a "target" schema, and output a plan of operations to bring the database to the desired structure.
  3. Manual migrations - fixed migration scripts that can be applied when certain database conditions are met, that do more customized operations (e.g., table renames or complex operations). Generally, we will create these only when automatic migrations cannot achieve the correct state.

We have also moved all schema management tools under a separate macrostrat schema command and moved the migrations and schema SQL files to the schema directory to improve their visibility within the repository.

The manual migrations system is inherited from the previous approach to schema management, but we expect it now to be used only for complicated migrations that can't be done easily with the automated system. The upshot of this is that it is no longer expected that migrations can fully construct the database in its desired state, and old migrations can be retired once no longer useful.

For automatic schema diffing, we evaluated PGSchema but found it a bit more restrictive than we'd like (although we did borrow the plan, review, apply workflow and terminology used by that project). For now, we use results, which is an update of migra, although there is a possibility we may need to move to a different library in the future (see djrobstep/results#7).
Several next steps remain:

  • Fully migrate the old "subsystems" approach to schema management into this more declarative system
  • Consider spinning off some aspects of the system into a shared library
  • Integrate the rockd database into the system
  • Unify handling of data migrations
  • Improve approach for loading initial data
  • Improve documentation
  • Explore the ergonomics of the system in use

@davenquinn davenquinn merged commit f2d4870 into main Jan 5, 2026
@davenquinn davenquinn changed the title Schema management rework Declarative schema management Jan 5, 2026
@davenquinn davenquinn deleted the schema-management branch January 5, 2026 03:10
This was referenced Jan 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants