Skip to content

Introduce versioned SQL schema migrations - #1522

Open
enarha wants to merge 1 commit into
tektoncd:mainfrom
enarha:introduce-sql-migration-framework
Open

enarha wants to merge 1 commit into
tektoncd:mainfrom
enarha:introduce-sql-migration-framework

Conversation

@enarha

@enarha enarha commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Replace GORM AutoMigrate on API server startup with versioned SQL migrations managed by golang-migrate and applied by a dedicated results-admin command, run as a Kubernetes Job in each release.

Schema versioning:

  • Embed the migrations in pkg/api/server/db/migrations. 001_baseline creates the established results and records schema.
  • Record the schema version and dirty flag in schema_migrations.
  • Declare compatibility with a single RequiredSchemaVersion constant. A process starts when the effective schema version (the recorded version, or one less while it is dirty) is at or above it. Migrations are additive, so several releases can share a schema, rollback needs no downgrade, and a pod restarting during a long migration still starts.
  • The API server and retention policy agent run a read-only check at startup and exit with an actionable message when it fails. There is no structural validation at startup.
  • DB_SCHEMA_REQUIRED_VERSION_OVERRIDE deliberately replaces the required version, with a warning on every start.

results-admin:

  • New binary and image with status, check [--structural], migrate-up, migrate-to and force --repaired commands, using an advisory lock and bounded connect, lock and statement timeouts.
  • migrate-up adopts a database created by GORM AutoMigrate after verifying that its tables match the baseline exactly, and fails with the list of differences otherwise. Upgrading needs no manual step.
  • --wait-for-database retries while the database is not reachable yet and re-reads the configuration before each retry.

Deployment:

  • Add the tekton-results-db-migrate Job to the release. It waits up to 10 minutes for the database and is removed 6 hours after finishing.
  • Remove DB_ENABLE_AUTO_MIGRATION; it is ignored with a warning.
  • Publish the results-admin image and wait for the Job in the e2e install script.

Tests and CI:

  • Unit tests for the embedded migrations, compatibility rule, CLI and connection retries; PostgreSQL tests (build tag e2e) for migration, adoption, drift, dirty state, force and lock timeouts.
  • New db-migrations presubmit job runs the PostgreSQL tests and rejects changes to published migration files.

Docs: add docs/database-migrations.md and update the install, external database and API configuration docs.

Assisted-by: GitHub Copilot CLI (Claude Opus 5.5) noreply@anthropic.com

Changes

/kind feature

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you review them:

  • Has Docs included if any changes are user facing
  • Has Tests included if any functionality added or changed
  • Tested your changes locally (if this is a code change)
  • Follows the commit message standard
  • Meets the Tekton contributor standards (including functionality, content, code)
  • Has a kind label. You can add a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
  • Release notes block below has been updated with any user-facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
  • Release notes contain the string "action required" if the change requires additional action from users switching to the new release

Release Notes

TODO

Replace GORM AutoMigrate on API server startup with versioned SQL
migrations managed by golang-migrate and applied by a dedicated
results-admin command, run as a Kubernetes Job in each release.

Schema versioning:
- Embed the migrations in pkg/api/server/db/migrations. 001_baseline
  creates the established results and records schema.
- Record the schema version and dirty flag in schema_migrations.
- Declare compatibility with a single RequiredSchemaVersion constant.
  A process starts when the effective schema version (the recorded
  version, or one less while it is dirty) is at or above it. Migrations
  are additive, so several releases can share a schema, rollback needs
  no downgrade, and a pod restarting during a long migration still
  starts.
- The API server and retention policy agent run a read-only check at
  startup and exit with an actionable message when it fails. There is
  no structural validation at startup.
- DB_SCHEMA_REQUIRED_VERSION_OVERRIDE deliberately replaces the
  required version, with a warning on every start.

results-admin:
- New binary and image with status, check [--structural], migrate-up,
  migrate-to and force --repaired commands, using an advisory lock and
  bounded connect, lock and statement timeouts.
- migrate-up adopts a database created by GORM AutoMigrate after
  verifying that its tables match the baseline exactly, and fails with
  the list of differences otherwise. Upgrading needs no manual step.
- --wait-for-database retries while the database is not reachable yet
  and re-reads the configuration before each retry.

Deployment:
- Add the tekton-results-db-migrate Job to the release. It waits up to
  10 minutes for the database and is removed 6 hours after finishing.
- Remove DB_ENABLE_AUTO_MIGRATION; it is ignored with a warning.
- Publish the results-admin image and wait for the Job in the e2e
  install script.

Tests and CI:
- Unit tests for the embedded migrations, compatibility rule, CLI and
  connection retries; PostgreSQL tests (build tag e2e) for migration,
  adoption, drift, dirty state, force and lock timeouts.
- New db-migrations presubmit job runs the PostgreSQL tests and rejects
  changes to published migration files.

Docs: add docs/database-migrations.md and update the install, external
database and API configuration docs.

Signed-off-by: Emil Natan <ena@redhat.com>
Assisted-by: GitHub Copilot CLI (Claude Opus 5.5) <noreply@anthropic.com>
@tekton-robot tekton-robot added kind/feature Categorizes issue or PR as related to a new feature. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Oct 26, 2026
@tekton-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please ask for approval from enarha after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 26, 2026
if v != s.Version && v != s.Version-1 {
return fmt.Errorf("version must be %d (failed migration completed by hand) or %d (its changes undone)", s.Version, s.Version-1)
}
return drv.SetVersion(int(v), false)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/feature Categorizes issue or PR as related to a new feature. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants