[BUG] Rollback of controller fails when older binary lacks DOWN files for newer schema versions

[onematchfox]

📋 Prerequisites

• I have searched the existing issues to avoid creating a duplicate
• By submitting this issue, you agree to follow our Code of Conduct
• I am using the latest version of the software
• I have tried to clear cache/cookies or used incognito mode (if ui-related)
• I can consistently reproduce this issue

🎯 Affected Service(s)

Controller Service

🚦 Impact/Severity

Blocker

🐛 Bug Description

When the kagent controller is downgraded to a previous version, the older controller fails to start if the newer version applied schema migrations that are unknown to the older version. This was observed when attempting to rollback after a failed upgrade from 0.9.0 to 0.9.3´. The .down.sql files for those migrations are embedded only in the newer container. The older container can neither roll the schema back nor start against the newer schema, resulting in a crash loop.

🔄 Steps To Reproduce

1. Deploy application v0.9.0 — embeds migrations 000001–000003. Schema reaches version 3.
2. Upgrade to v0.9.3 — embeds migrations 000001–000004. Schema reaches version 4.
3. v0.9.3 is found to be broken; downgrade back to v0.9.0.
4. v0.9.0 starts against a DB at schema version 4. Its embedded source only knows versions 1–3.

🤔 Expected Behavior

Rolling back to a previous version is a standard recovery operation (e.g. when a new release introduces a regression) and should work reliably.

📱 Actual Behavior

No response

💻 Environment

No response

🔧 CLI Bug Report

No response

🔍 Additional Context

No response

📋 Logs

📷 Screenshots

No response

🙋 Are you willing to contribute?

• I am willing to submit a PR to fix this issue

[iplay88keys]

@onematchfox Thanks for raising this issue. Can you provide more context on the failed upgrade aspect? On each crash loop, the Kagent controller applies all down migrations before crashing, but without scaling to 0 before downgrading, the migrations would just run again.

I think we will need to have a good solution in these cases, but in the meantime, you should be able to manually apply the down migrations with golang-migrate directly:

1. Install golang-migrate
2. Stop the Kagent controller:

kubectl -n kagent scale deploy/kagent-controller --replicas=0

3. Roll back the core track:

migrate \
  -source "github://kagent-dev/kagent/go/core/pkg/migrations/core#v0.9.3" \
  -database "postgres://kagent:secret@db.internal:5432/kagent?sslmode=require&x-migrations-table=schema_migrations" \
  goto 3

[onematchfox]

Can you provide more context on the failed upgrade aspect?

Sorry, this is misleading. The upgrade didn't fail as such... I upgraded to v0.9.3 successfully to test the new version but then decided to rollback to v0.9.0 and was unable to do so.

Thanks for the tip on the workaround. Will bear that in mind if this comes up again before we find an automated way of achieving this. On that, do you have any thoughts on how we might do this? It doesn't seem like golang-migrate has a good solution to this. The only thing I could really come up with was to store the down scripts in a database table when applying the migrations so that they could be retrieved and used by any version.

[iplay88keys]

Sorry, this is misleading. The upgrade didn't fail as such... I upgraded to v0.9.3 successfully to test the new version but then decided to rollback to v0.9.0 and was unable to do so.

Ah, ok. I was worried that we had broken something on the upgrade.

On that, do you have any thoughts on how we might do this? It doesn't seem like golang-migrate has a good solution to this. The only thing I could really come up with was to store the down scripts in a database table when applying the migrations so that they could be retrieved and used by any version.

That is one approach I've been considering. I was also thinking that the kagent cli could have the migrations embedded so that you can run kagent migrate down or something similar. That would of course require the newer version of the cli to be used, but could be an option.