Skip to content
BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
.github/workflows
.github/workflows
 
 
data
data
 
 
scripts
scripts
 
 
src
src
 
 
tests
tests
 
 
.gitignore
.gitignore
 
 
.python-version
.python-version
 
 
README.md
README.md
 
 
pyproject.toml
pyproject.toml
 
 
uv.lock
uv.lock
 
 

Repository files navigation

💰 Corporate Salary Benchmark

Scrape · Analyze · Visualize — Indian job market salary data at scale

Python uv Ruff GitHub Actions License

BeautifulSoup Pandas Plotly Marimo curl-cffi

Warning

Under Development — The scraping pipeline (src/, scripts/, GitHub Actions CI) is complete and production-ready. The EDA notebook and interactive dashboard are planned but not yet finished — see steps 4 & 5 below.

✨ What This Project Does

An end-to-end data pipeline that:

  1. ✅ 🔍 Discovers company slugs from AmbitionBox listing pages
  2. ✅ 💸 Scrapes per-company salary data by role, experience band, and city — bypassing JS rendering and bot protections
  3. ✅ 🧹 Merges batch CSVs from parallel GitHub Actions runs into a single clean dataset
  4. 📅 📊 Analyzes 21 EDA questions (planned — notebooks/eda.py under development)
  5. 📅 🎛️ Visualizes an interactive 4-tab dashboard (planned — notebooks/dashboard.py under development)

Portfolio highlight: The scraper reverse-engineers AmbitionBox's Nuxt.js SSR __NUXT__ IIFE payload — extracting averaged salary figures that the site intentionally hides from the HTML.

🏗️ Project Structure

salary-benchmark/
│
├── src/
│   ├── config.py               # All project-wide constants & env-var overrides
│   └── scraper/
│       ├── discover.py         # Crawl company listing pages → company slugs
│       ├── salary.py           # Scrape per-company salary tables
│       ├── parser.py           # __NUXT__ IIFE parser + HTML fallback
│       └── utils.py            # HTTP session, UA/TLS rotation, retries, backoff
│
├── scripts/
│   ├── run_discover.py         # CLI: discover all companies
│   ├── run_scrape.py           # CLI: scrape salaries in configurable batches
│   └── merge_data.py           # CLI: merge batch CSVs → salaries.csv
│
├── notebooks/
│   ├── eda.py                  # Marimo: 21 exploratory analysis questions
│   └── dashboard.py            # Marimo: interactive 4-tab salary dashboard
│
├── tests/                      # Pytest test suite
├── data/
│   ├── companies.csv           # Discovered company slugs
│   ├── raw/                    # Per-batch CSVs (git-ignored)
│   └── salaries.csv            # Final merged dataset (git-ignored)
│
├── .github/workflows/
│   └── scrape.yml              # GitHub Actions: discover / scrape / scrape-all / merge
├── pyproject.toml              # uv project config + ruff linting rules
└── .python-version             # Pinned: Python 3.12

🚀 Quick Start

Prerequisites

# Install uv (fast Python package manager)
# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

1. Clone & Install

git clone https://github.com/<your-username>/salary-benchmark.git
cd salary-benchmark
uv sync                # installs all dependencies into .venv

2. Discover Companies

Crawls AmbitionBox listing pages and saves company slugs to data/companies.csv.

# Default — metro cities, up to 500 pages
uv run python scripts/run_discover.py

# Quick test run (5 pages)
uv run python scripts/run_discover.py --max-pages 5

# Custom city list
uv run python scripts/run_discover.py --locations "mumbai,bengaluru,pune"

# No location filter (all of India)
uv run python scripts/run_discover.py --no-locations

3. Scrape Salary Data

# Scrape companies 0–9
uv run python scripts/run_scrape.py --start 0 --size 10

# Include per-city breakdown (slower, enables geographic analysis)
uv run python scripts/run_scrape.py --start 0 --size 10 --with-locations

# Filter to CS/IT/Data roles only
uv run python scripts/run_scrape.py --start 0 --size 10 --cs-only

💡 Set LOG_LEVEL=DEBUG to see detailed request logs.

4. Merge Batches

uv run python scripts/merge_data.py

5. Run EDA notebook

Caution

WIP — notebooks/eda.py is still being refined. Charts may be incomplete or missing.

uv run marimo edit notebooks/eda.py

6. Run Dashboard

Caution

WIP — notebooks/dashboard.py is still being refined. Filters and tabs are subject to change.

uv run marimo run notebooks/dashboard.py

⚙️ Configuration

All tunables are in src/config.py and can be overridden via environment variables — no code changes needed.

Environment Variable Default Description
DELAY_MIN 1.0 Min seconds between requests
DELAY_MAX 3.0 Max seconds between requests
MAX_RETRIES 5 Retry attempts per URL
RETRY_BACKOFF 2.0 Exponential backoff multiplier
REQUEST_TIMEOUT 60 HTTP request timeout (seconds)
MAX_LISTING_PAGES 500 Max company listing pages to crawl
MAX_SALARY_PAGES 50 Max salary pages per company
BATCH_SIZE 10 Default companies per batch
SCRAPER_PROXY (none) Proxy URL (HTTP/HTTPS)
DISCOVERY_LOCATIONS Metro cities Comma-separated location slugs
RATE_LIMIT_THROTTLE_THRESHOLD 12 Extra sleep kicks in after N 429s
RATE_LIMIT_EXTRA_SLEEP 8 Extra sleep duration (seconds)

🤖 GitHub Actions — Automated Batch Scraping

The scraper is designed to run on GitHub Actions, which provides clean IPs and avoids local throttling.

Go to Actions → Scrape AmbitionBox Salaries → Run workflow

Phase What it does
discover Crawl listing pages, save data/companies.csv
scrape Scrape a single batch of companies
scrape-all Compute matrix → parallel scrape all companies → auto-merge
merge Combine all data/raw/batch_*.csv → data/salaries.csv

Data is auto-committed back to the repo after each run via git-auto-commit-action.

Key inputs:

Input Default Description
batch_start 0 Starting company index
batch_size 25 Companies per batch
with_locations true Enable per-city salary breakdown
cs_only false Filter to CS/IT/Data roles only
max_pages 500 Max listing pages for discovery
all_locations false Scrape all cities, not just metros

🔬 Technical Highlights

__NUXT__ IIFE Payload Parsing

AmbitionBox uses Nuxt.js SSR — salary data is bundled into an obfuscated JavaScript self-executing function. The HTML blurs the avgCtc figure. parser.py uses brace-counting to extract the IIFE body and arguments, then resolves variable cross-references to recover the exact numeric salary values.

Anti-Scraping Stack

Technique Implementation
TLS fingerprint impersonation curl-cffi impersonating Chrome 131, Firefox 135, Edge 101
Request delays Random uniform sleep between DELAY_MIN and DELAY_MAX
Exponential backoff On 429 / 503 with jitter
Adaptive throttling Extra sleep after N rate-limit hits
Bot challenge handling Auto-follows bm-verify meta-refresh redirects
HTTP/2 stream recovery Session reset on stream errors

Parallel Scraping

scrape-all computes a dynamic GitHub Actions matrix from data/companies.csv, spawning up to 10 parallel jobs each scraping 10 companies — the full dataset scraped in a single workflow run.

📊 EDA Coverage (21 Questions)

Click to expand

Salary Structure

  1. Which roles have the highest salaries across companies?
  2. Average salary by experience level
  3. Salary growth from 0–2 to 5–8 years
  4. Salary spread per role
  5. Median salary by role
  6. Overall salary distribution

Company Comparison 7. Which companies pay the most for the same role? 8. Data Analyst salary across companies 9. Salary distribution per company 10. Top-paying companies overall 11. Salary variance within companies

Industry Analysis 12. Which industries pay the highest? 13. IT vs Non-IT salary comparison 14. Finance vs Tech salary trends 15. Industry-wise salary growth

Experience Insights 16. Salary vs experience correlation 17. Highest paying roles at entry level 18. Mid-career salary leaders 19. Senior-level salary trends

Geographic Insights 20. Which cities offer the highest salaries? 21. Salary variation by location

🎛️ Dashboard Tabs

Tab Description
Role Comparison Top 25 roles by median salary with sidebar filters
Company Heatmap Company × Role salary heatmap
Experience Curve Median / mean salary vs years of experience
Industry Ranking Industries ranked by median pay

All tabs respond to unified sidebar filters: company, role, industry, experience band, and location.

🗃️ Data Schema

Column Type Description
company_slug str URL slug
company_name str Display name
industry str Industry category
job_role str Role / designation
avg_salary float Average CTC (INR / year)
min_salary float Minimum CTC
max_salary float Maximum CTC
typical_min_salary float Typical range lower bound
typical_max_salary float Typical range upper bound
min_experience int Min years of experience
max_experience int Max years of experience
datapoints int Number of salary reports
location str City (populated with --with-locations)
salary_lakhs float Derived: avg_salary / 100 000
mid_experience float Derived: (min + max) / 2
salary_range_width float Derived: max_salary - min_salary

🧪 Running Tests

uv run pytest
# With verbose output
uv run pytest -v

# Run linting
uv run ruff check .

# Auto-fix lint issues
uv run ruff check --fix .

# Format code
uv run ruff format .

📄 License

This project is for educational and portfolio purposes only.
Salary data is sourced from AmbitionBox and belongs to them.
Please review their Terms of Service before using this tool.

About

End-to-end pipeline to scrape, clean, and analyze Indian corporate salary data from AmbitionBox — bypassing JS rendering and bot protections via TLS fingerprint impersonation, with automated batch scraping on GitHub Actions.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages