☁️ Cloud Resource Auditor

Automated Infrastructure Cost Governance for AWS

Automatically scans your AWS environment, identifies idle and orphaned resources, and generates audit reports — before they appear on your next bill.

---

📌 The Problem

In a fast-moving DevOps environment, it's easy to lose track of cloud resources. A developer spins up an EC2 instance for testing, deletes the instance later, but forgets to delete the attached EBS volume. Over time, these "Zombie" resources accumulate silently, leading to massive, unnecessary AWS bills.

This project acts as a financial safety net — an automated pipeline that hunts down these orphaned resources on a schedule and reports them before the damage is done.

---

🏗️ Architecture Overview

┌─────────────────────────────────────────────┐
│              Local Machine                  │
│                                             │
│   ┌─────────────┐     ┌──────────────────┐  │
│   │   Jenkins   │────▶│  aws-auditor     │  │
│   │  Container  │     │  Container       │  │
│   │  :8080      │     │  (Python/Boto3)  │  │
│   └──────┬──────┘     └────────┬─────────┘  │
│          │                     │            │
│          └─────────────────────┘            │
│               Docker Socket (DooD)          │
└─────────────────────────────────────────────┘
                      │
                      ▼
             ┌────────────────┐
             │   AWS APIs     │
             │  EC2 + CW +    │
             │  EBS           │
             └────────────────┘

---

🛠️ Tech Stack

Layer	Technology
Automation Server	Jenkins (LTS)
Containerization	Docker + Docker Compose
Cloud SDK	Python 3 + Boto3
AWS Services	EC2, CloudWatch, EBS
Credential Management	Jenkins Secret Text + .env
Scheduling	Jenkins Cron (H H/8 * * *)

---

💡 Key Design Decisions

Docker-outside-of-Docker (DooD)

Instead of the heavy and complex Docker-in-Docker (DinD) approach, this project uses DooD — mounting the host's Docker socket into the Jenkins container:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock

Why DooD?

• Jenkins reuses your machine's existing Docker engine — no nested daemon
• No --privileged flag required, making it more secure
• Faster builds and lower resource overhead

Staggered Scheduling with H

The pipeline runs every 8 hours using:

triggers {
    cron('H H/8 * * *')
}

The H (Hash) token tells Jenkins to stagger the exact start time per job, preventing a thundering-herd CPU spike when multiple jobs share the same schedule.

---

🔐 Credential Security

AWS keys are never hardcoded. Two layers of protection are in place:

Local development — .env file (git-ignored):

AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=wJalr...
AWS_DEFAULT_REGION=us-east-1

Jenkins pipeline — keys stored as Secret Text credentials, masked in all logs:

environment {
    AWS_ACCESS_KEY_ID     = credentials('AWS_ID')
    AWS_SECRET_ACCESS_KEY = credentials('AWS_KEY')
}

---

🚀 Getting Started

Prerequisites

• Docker Desktop installed and running
• AWS account with appropriate IAM permissions
• Git

Required IAM Permissions

Attach a policy with the following minimum permissions to your AWS user or role:

{
  "Effect": "Allow",
  "Action": [
    "ec2:DescribeInstances",
    "ec2:DescribeVolumes",
    "ec2:DescribeRegions",
    "cloudwatch:GetMetricStatistics",
    "sts:GetCallerIdentity"
  ],
  "Resource": "*"
}

Step 1 — Clone and Configure

git clone https://github.com/yourusername/cloud-resource-auditor.git
cd cloud-resource-auditor

# Add your AWS credentials
cp .env.example .env
nano .env

Step 2 — Launch Jenkins

# Start Jenkins container
docker compose up -d

# Fetch the initial unlock password
docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword

Open http://localhost:8080, paste the password, and install suggested plugins.

Step 3 — Store AWS Credentials in Jenkins

Jenkins → Manage Jenkins → Credentials → Global → Add Credentials

Add two Secret Text entries:

ID	Value
AWS_ID	Your AWS_ACCESS_KEY_ID
AWS_KEY	Your AWS_SECRET_ACCESS_KEY

Step 4 — Create the Pipeline Job

New Item → Pipeline → Pipeline script from SCM → Git → <your repo URL>

Jenkins will automatically detect the Jenkinsfile in your repository.

---

⚙️ Pipeline Definition

pipeline {
    agent any

    triggers {
        cron('H H/8 * * *')   // Runs every 8 hours, staggered
    }

    environment {
        AWS_ACCESS_KEY_ID     = credentials('AWS_ID')
        AWS_SECRET_ACCESS_KEY = credentials('AWS_KEY')
    }

    stages {
        stage('Audit Execution') {
            steps {
                sh 'docker compose run --rm auditor-service'
            }
        }

        stage('Report Archiving') {
            steps {
                archiveArtifacts artifacts: 'reports/*.csv', fingerprint: true
            }
        }
    }

    post {
        failure {
            echo 'Audit failed — check logs for AWS connectivity or credential issues'
        }
    }
}

---

📁 Project Structure

cloud-resource-auditor/
├── docker-compose.yml      # Jenkins + auditor service definitions
├── Dockerfile              # Multistage build for the auditor image
├── Jenkinsfile             # Pipeline definition
├── audit.py                # Core auditing logic (Boto3)
├── .env.example            # Template for local credentials
├── .env                    # Your actual credentials (git-ignored)
├── .gitignore
├── reports/                # Generated audit reports (git-ignored)
└── README.md

---

📊 What Gets Audited

Resource	Condition Flagged	Why It Costs Money
EC2 Instances	Avg CPU < 2% over 7 days	Running instances billed by the hour
EBS Volumes	Status = available (unattached)	Volumes billed even when detached

Reports are exported as timestamped .csv files and archived as Jenkins build artifacts.

---

🔧 Useful Commands

# Start Jenkins
docker compose up -d

# Stop Jenkins
docker compose down

# View Jenkins logs
docker logs -f jenkins

# Run the auditor manually (outside Jenkins)
docker compose run --rm auditor-service

# Wipe Jenkins data and start fresh
docker compose down -v

---

🧱 Challenges & Learnings

• Docker socket permissions — Getting the Jenkins container to communicate with the host Docker socket required careful UID/GID alignment between the container user and the docker group on the host.
• AWS response pagination — AWS APIs return results in pages; without paginators, scans silently miss resources beyond the first 1000 results.
• CloudWatch datapoint averaging — A single datapoint is not representative of 7-day CPU behaviour; the auditor averages across all returned datapoints for accuracy.
• Cost mindset — Shifting from "make it work" to "make it save money" fundamentally changes how you evaluate infrastructure decisions.

---

💎 Benefits

• Zero host dependencies — Python and Boto3 run inside Docker; nothing installed on your machine
• Portable — Runs identically on a local laptop or an AWS EC2 build node
• Secure by default — Credentials never touch the filesystem or appear in logs
• Extensible — Add new audit checks (unused Elastic IPs, idle RDS instances, etc.) by extending audit.py

---

🤝 Contributing

Pull requests are welcome. For major changes, open an issue first to discuss what you'd like to change.

---

👨‍💻 Author

Nishant Bhardwaj — DevOps Enthusiast & Engineering Student

---

📄 License

MIT