Learn how to configure Automatic Rollback & Recovery for Kubernetes deployments using Harness Continuous Delivery.
In this hands-on tidbit, you'll start with a sample application in GitHub, connect it to Harness, configure the required Service and Environment, import a pipeline from Harness Configuration-as-Code, deploy a healthy version, intentionally introduce a deployment failure, and watch Harness automatically restore the previous successful release.
Automatic Rollback helps protect deployments when a newly released version fails to reach a healthy state.
Harness continuously monitors the Kubernetes deployment during rollout. If the deployment fails, a configured Stage Rollback failure strategy can automatically execute the rollback steps and restore the previous successful release.
In this tidbit you will:
- Connect a GitHub repository to Harness.
- Build and push a Docker image using Harness CI.
- Configure a Harness Kubernetes Service and Environment.
- Deploy the application using a Kubernetes Rolling Deployment.
- Configure Stage Rollback for automatic recovery.
- Intentionally introduce a deployment failure.
- Observe Harness detecting the failed rollout.
- Automatically restore the previous healthy release.
- Verify the recovered application and Kubernetes deployment.
You will create a two-stage CI/CD pipeline.
The Build stage:
- Clones the application repository.
- Builds the Docker image.
- Pushes the image to Docker Hub.
The Deploy stage:
- Uses the Docker image produced by the Build stage.
- Applies the Kubernetes manifests.
- Performs a Kubernetes Rolling Deployment.
- Monitors the deployment until it reaches a healthy state.
- Automatically rolls back when the deployment fails.
The deployment uses:
- Kubernetes Deployment
- Kubernetes Service
- Docker Hub Artifact
- Harness Service
- Harness Environment
- Kubernetes Infrastructure
- Stage Rollback Failure Strategy
- Kubernetes Rolling Rollback
GitHub Repository
|
v
+----------------------+
| Harness CI |
| |
| Clone → Build → Push |
+----------+-----------+
|
v
+-------------+
| Docker Hub |
| Docker Image |
+------+------+
|
v
+----------------------+
| Harness CD |
| |
| Kubernetes Rolling |
| Deployment |
+----------+-----------+
|
v
+----------------------+
| Healthy Release |
| ✅ |
+----------+-----------+
|
New Version
|
v
+----------------------+
| Deployment Failure |
| ❌ |
+----------+-----------+
|
v
+----------------------+
| Harness Detects |
| Deployment Failure |
+----------+-----------+
|
v
+----------------------+
| Stage Rollback |
+----------+-----------+
|
v
+----------------------+
| Kubernetes Rolling |
| Rollback |
+----------+-----------+
|
v
+----------------------+
| Previous Healthy |
| Release Restored ✅ |
+----------------------+
Before starting, make sure you have:
- A Harness account.
- A Harness project.
- A GitHub account and repository access.
- A GitHub connector in Harness.
- A Docker Hub account.
- A Docker Hub connector in Harness.
- A Kubernetes cluster.
- A Kubernetes connector/delegate.
- Permission to create Services, Environments, and Pipelines in the Harness project.
This example uses:
- GitHub — source code and Harness pipeline configuration.
- Docker Hub — container image registry.
- Kubernetes — deployment target.
- Harness CI/CD — build, deployment, health monitoring, and rollback.
cd-tidbits-automatic-rollback-recovery/
├── app/
│ ├── Dockerfile
│ └── index.html
│
├── k8s/
│ ├── deployment.yaml
│ ├── namespace.yaml
│ └── service.yaml
│
├── .harness/
│ └── pipeline.yaml
│
└── README.md
Contains the sample application and Dockerfile used to build the container image.
Contains the Kubernetes resources required to deploy the application.
Contains the Harness pipeline configuration as code.
The pipeline YAML is included in the repository so the complete pipeline can be imported and reproduced instead of being created manually from scratch.
Clone the repository locally:
git clone https://github.com/harness-community/cd-tidbits-automatic-rollback-recovery.git
cd cd-tidbits-automatic-rollback-recoveryYou can also fork the repository into your own GitHub account and work from your fork.
Before importing the pipeline, configure the required connectors in your Harness project.
Navigate to:
Harness
→ Project Settings
→ Connectors
Create and test the following connectors:
Used to:
- Clone the application repository.
- Read the Kubernetes manifests.
- Read the Harness pipeline configuration.
Select Test Connection and verify that the connector is healthy.
Used to:
- Build and push the Docker image.
- Retrieve the image during deployment.
Verify the connector using Test Connection.
Used by Harness to communicate with the Kubernetes cluster and execute the deployment.
Verify that the Kubernetes Delegate is running and available.
The Harness Service defines what is being deployed.
Navigate to:
Project
→ Services
→ New Service
Create a Service named:
rollback-demo
Select:
Deployment Type: Kubernetes
Configure the Kubernetes manifests to use the GitHub repository.
Set the manifest path to:
k8s
The Service should reference the Kubernetes manifests from this repository.
Inside the rollback-demo Service, configure the primary artifact source.
Use:
Artifact Type: Docker Registry
Configure the Docker Hub connector and repository used by the Build stage.
For this example, the Docker image repository is:
pes2ug19cs219/harness-rollback
If you fork this repository or use your own Docker Hub account, replace this with your own image repository.
The Environment defines where the application is deployed.
Navigate to:
Project
→ Environments
→ New Environment
Create an environment named:
dev
Add a Kubernetes infrastructure definition.
Configure it to point to your Kubernetes cluster.
Select the appropriate Kubernetes connector/delegate.
Your final structure should look similar to:
Environment
└── dev
└── Kubernetes Infrastructure
└── Kubernetes Cluster
The complete Harness pipeline is included in:
.harness/pipeline.yaml
This is the Configuration-as-Code version of the pipeline used in the demonstration.
Instead of manually rebuilding the pipeline, import this YAML into Harness.
Navigate to:
Project
→ Pipelines
→ Create Pipeline
→ Import From Git
Select your GitHub connector.
Point it to:
cd-tidbits-automatic-rollback-recovery
Select the branch:
main
Set the pipeline YAML path to:
.harness/pipeline.yaml
Import the pipeline.
After importing the pipeline, review the YAML and update any values that are specific to your Harness account.
Typical values that may need to be changed include:
organization identifier
project identifier
GitHub connector reference
Docker Hub connector reference
GitHub repository
Docker image repository
Kubernetes infrastructure
Make sure the Service referenced by the pipeline is:
rollback-demo
And the Environment is:
dev
Save the pipeline after making the required changes.
The pipeline contains two stages.
Build
|
v
Deploy
The Build stage:
Clone Repository
|
v
Build Docker Image
|
v
Push Image to Docker Hub
The Deploy stage:
Select Artifact
|
v
Kubernetes Rolling Deployment
|
v
Wait for Healthy State
The Deploy stage also contains the rollback configuration.
Open the Deploy stage.
Navigate to:
Deploy
→ Failure Strategy
Configure:
Failure Type: All Errors
Action: Stage Rollback
The stage rollback uses the Kubernetes Rolling Rollback step to restore the previous successful deployment.
The resulting flow is:
Deployment Failure
|
v
Stage Rollback
|
v
Kubernetes Rolling Rollback
|
v
Previous Release Restored
Run the pipeline for the first time.
The expected flow is:
Build
↓
Docker Image Created
↓
Image Pushed to Docker Hub
↓
Deploy
↓
Kubernetes Rolling Deployment
↓
Healthy Application
The pipeline should complete successfully.
Verify the Kubernetes pods:
kubectl get pods -n rollback-demoCheck the ReplicaSets:
kubectl get rs -n rollback-demoCheck the deployment history:
kubectl rollout history deployment rollback-demo -n rollback-demoThe application should be running successfully.
If using the local Kubernetes setup from the demo, you can access it using:
kubectl port-forward svc/rollback-demo 8080:80 -n rollback-demoThen open:
http://localhost:8080
Now we'll intentionally break the next deployment.
For example, modify the Kubernetes deployment so the container cannot start successfully.
One example is an invalid container command:
command:
- /bin/does-not-existCommit and push the change:
git add .
git commit -m "Introduce deployment failure for rollback demo"
git pushRun the Harness pipeline again.
During the deployment, Harness will:
- Build the new Docker image.
- Push the image to Docker Hub.
- Start the Kubernetes Rolling Deployment.
- Monitor the new pods.
- Wait for the deployment to reach a healthy state.
Because the new version contains an intentional failure, the new pods will not become healthy.
The deployment eventually fails.
Because the Deploy stage is configured with:
Stage Rollback
Harness automatically starts the rollback process.
The execution flow becomes:
Rolling Deployment
|
v
New Version Fails
|
v
Harness Detects Failure
|
v
Stage Rollback
|
v
Kubernetes Rolling Rollback
|
v
Previous Version Restored
No manual kubectl rollout undo command is required.
Check the pods:
kubectl get pods -n rollback-demoCheck the ReplicaSets:
kubectl get rs -n rollback-demoCheck deployment history:
kubectl rollout history deployment rollback-demo -n rollback-demoYou should see the previous healthy deployment restored.
Verify the deployment status:
kubectl rollout status deployment rollback-demo -n rollback-demoFinally, verify the application:
kubectl port-forward svc/rollback-demo 8080:80 -n rollback-demoOpen:
http://localhost:8080
The application should be available again.
The complete workflow should look like:
Build
✅
|
v
Deploy Healthy Version
✅
|
v
Introduce Bad Version
|
v
Rolling Deployment
❌
|
v
Harness Detects Failure
|
v
Stage Rollback
🔄
|
v
Previous Version Restored
✅
After completing the main demonstration, try introducing different failure scenarios.
command:
- /bin/does-not-existUse an image that does not exist or cannot be pulled.
Configure a readiness probe to check an endpoint that does not exist.
Introduce an application configuration or startup error.
For each scenario, observe how Harness handles the failed rollout and executes the configured rollback.
Verify:
.harness/pipeline.yaml
exists in the selected branch and that the GitHub connector has access to the repository.
Verify the connector references in the imported pipeline match the connectors created in your Harness project.
Verify:
- Docker Hub repository name.
- Docker Hub connector.
- Image tag.
- Repository permissions.
Check:
kubectl get pods -n rollback-demo
kubectl describe deployment rollback-demo -n rollback-demo
kubectl describe pods -n rollback-demo
kubectl get events -n rollback-demoPay particular attention to:
- Readiness probes.
- Image pull errors.
- Container startup errors.
- Kubernetes resource availability.
Verify:
- The Deploy stage has a Stage Rollback failure strategy.
- A Kubernetes Rolling Rollback step is configured.
- A previous successful deployment exists.
- Kubernetes has multiple deployment revisions.
Check:
kubectl rollout history deployment rollback-demo -n rollback-demoThis tidbit focuses specifically on Automatic Rollback & Recovery.
It does not cover:
- Blue-Green Deployments
- Canary Deployments
- Progressive Delivery
- Traffic Shifting
- Approval Gates
- GitOps
- Continuous Verification
- Production deployment strategies
The Harness pipeline used in this demonstration is available in:
.harness/pipeline.yaml
This allows you to:
- Review the complete pipeline definition.
- Import the pipeline into your own Harness project.
- Version-control pipeline changes.
- Reproduce the demonstration without manually rebuilding the pipeline.
The pipeline includes the Build stage, Deploy stage, artifact configuration, Kubernetes Rolling Deployment, and rollback configuration.
- Harness Continuous Delivery Documentation
- Harness Kubernetes CD Quickstart
- Harness Failure Strategies
- Harness Kubernetes Rolling Deployments
- Harness Kubernetes Rollback
- Kubernetes Deployments
GitHub: https://github.com/harness-community/cd-tidbits-automatic-rollback-recovery