Skip to content

upgrade-procedures.md

Latest commit

 

History

History
84 lines (55 loc) · 3.73 KB

upgrade-procedures.md

File metadata and controls

84 lines (55 loc) · 3.73 KB
Raw

Upgrade Procedures

Work in Progress -- This page is under active development. Detailed step-by-step instructions for each upgrade path will be added as they are finalized.

Topics to Cover

Upgrade Methods

  • Cosmovisor-based Upgrades -- Automated binary upgrades triggered by on-chain governance proposals. Cosmovisor monitors the chain for upgrade plans and automatically swaps the binary at the specified block height.
  • Binary Replacement Upgrades -- Manual upgrade procedure involving stopping the node, replacing the lithod binary, and restarting. Suitable for non-governance upgrades or emergency patches.
  • Governance Proposal Upgrades -- End-to-end workflow for proposing, voting on, and executing chain upgrades through the on-chain governance module.
  • Rollback Procedures -- Steps to revert to a previous binary version and chain state in the event of a failed upgrade.

Environment Promotion

All changes follow a strict promotion path through environments before reaching mainnet:

Devnet  -->  Staging  -->  Mainnet

Each environment serves as a validation gate. Changes must pass all tests, health checks, and approval requirements at each stage before promotion to the next.

Mainnet Deployment Gate

The mainnet deployment process follows a controlled, auditable pipeline:

  1. Git Tag -- A release is tagged in the source repository (e.g., v1.2.0).
  2. CI Build -- The tagged commit triggers a CI pipeline that builds, tests, and signs the release artifacts.
  3. Approval -- A designated CODEOWNER must explicitly approve the deployment.
  4. ArgoCD Sync -- Upon approval, ArgoCD synchronizes the new version to the mainnet environment.

Rollback Procedures

Application Rollback (current flow)

Note: This page previously documented an ArgoCD-based rollback. The ArgoCD/Kubernetes deployment was never wired into Makalu — production runs on EC2 + Docker Compose. Updated 2026-05-12 to match reality.

  • Automatic -- .github/workflows/deploy-simple.yaml has a rollback job that runs on failure() and restores the previous docker-compose.yaml + .env snapshot taken at deploy start. No operator intervention needed when CI itself detects deploy failure.
  • Manual -- Operators can re-run a previous green deploy by triggering deploy-simple.yaml via workflow_dispatch and pinning the tag input to the prior successful image SHA. The post-deploy SHA-verify step confirms the rollback landed.

Database Rollback

  • Point-in-Time Recovery (PITR) -- Database state can be restored to any point in time within the retention window. This is critical for recovering from data corruption or unintended state changes introduced by a faulty upgrade.

Deployment Windows

Deployments are restricted to defined maintenance windows to minimize risk and ensure adequate staffing for monitoring:

Environment Allowed Days Allowed Hours (UTC)
Devnet Any day Any time
Staging Monday -- Thursday 09:00 -- 17:00
Mainnet Tuesday -- Thursday 14:00 -- 16:00

Freeze Periods

The following freeze periods restrict mainnet deployments to reduce risk during high-impact windows:

Rule Detail
Major events No mainnet deployments during major network events, governance votes, or external market events that could amplify the impact of any issues.
Minimum interval A minimum of 48 hours must pass between successive mainnet deployments to allow adequate observation of each change.
Holiday freeze No mainnet deployments between December 20 and January 5 (inclusive) to account for reduced staffing during the holiday period.