Skip to content
BranchesTags

Repository files navigation

🚗 Traffic Accident Analysis - Big Data Project

A comprehensive Big Data analysis platform for traffic accidents across major Indian cities using PySpark, Python, and Streamlit with an interactive web-based UI.

📌 Project Overview

This project provides complete analysis and visualization of traffic accidents using distributed computing with PySpark. It includes:

  • Interactive Web Dashboard - Streamlit-based UI for real-time analysis
  • Big Data Processing - PySpark for handling 1M+ row datasets
  • Comprehensive Analytics - Multiple analysis dimensions and insights
  • Professional Visualizations - Publication-quality charts and graphs
  • Executive Reports - Automated summary reports with recommendations

🛠️ Technology Stack

Backend

  • PySpark - Distributed data processing (Spark SQL, DataFrames)
  • Pandas - Data manipulation and analysis
  • NumPy - Numerical computations
  • Python 3.8+ - Core language

Frontend

  • Streamlit - Interactive web interface
  • Matplotlib - Static visualizations
  • Seaborn - Statistical data visualization

Data Format

  • CSV - Input dataset (1M+ rows, 11 columns)

📊 Dataset Structure

Column Type Description
id Integer Unique accident identifier
date String (YYYY-MM-DD) Date of accident
time String (HH:MM:SS) Time of accident
city String City where accident occurred
weather String Weather conditions
visibility String Visibility level
road_type String Type of road
vehicles_involved Integer Number of vehicles
casualties Integer Number of casualties
severity String Accident severity level
latitude Float Geographic latitude
longitude Float Geographic longitude

Dataset Size: 50,000+ records (scalable to 1M+)

📁 Project Structure

traffic-analysis-harshi/
├── app.py                          # Streamlit web application
├── traffic_accident_bigdata.py      # Standalone PySpark script
├── utils.py                         # Utility functions and helpers
├── traffic_accidents_50000.csv      # Sample dataset
├── requirements.txt                 # Python dependencies
├── README.md                        # This file
└── output_graphs/                   # Generated visualizations
    ├── 01_accidents_per_city.png
    ├── 02_accident_trends.png
    ├── 03_heatmap_hour_vs_day.png
    ├── 04_severity_distribution.png
    ├── 05_accident_clusters.png
    ├── 06_weather_accidents.png
    ├── 07_accidents_by_hour.png
    └── summary_report.txt

🚀 Installation & Setup

Prerequisites

  • Python 3.8 or higher
  • pip or conda package manager
  • 4GB+ RAM (recommended)
  • Java Runtime Environment (JRE) for Spark

Step 1: Clone/Download Project

cd traffic-analysis-harshi

Step 2: Install Dependencies

Using pip:

pip install -r requirements.txt

Using conda:

conda create -n traffic-analysis python=3.9
conda activate traffic-analysis
pip install -r requirements.txt

Step 3: Verify Installation

python -c "import pyspark; import streamlit; print('✓ All dependencies installed')"

🎯 Usage Guide

Option 1: Run Interactive Web Dashboard (Recommended)

streamlit run app.py

The application will open in your browser at http://localhost:8501

Features:

  • 🏠 Home - Project overview and introduction
  • 📈 Dashboard - Key metrics and overview
  • 🔍 Analysis - Detailed analysis by dimension
  • 📊 Visualizations - Interactive charts
  • 📋 Reports - Summary reports and insights
  • ℹ️ About - Project information

Option 2: Run Standalone PySpark Script

python traffic_accident_bigdata.py

This will:

  1. Load the CSV dataset
  2. Clean and preprocess data
  3. Perform all analyses
  4. Generate 7 visualizations
  5. Create a summary report
  6. Save all outputs to output_graphs/

Option 3: Use Utility Functions in Custom Script

from utils import initialize_spark, load_data, clean_data, analyze_data

# Initialize Spark
spark = initialize_spark()

# Load data
df = load_data(spark, 'traffic_accidents_50000.csv')

# Clean data
df_clean = clean_data(df)

# Analyze data
results = analyze_data(df_clean)

# Access results
print(results['accidents_per_city'].show(5))

📊 Analysis Capabilities

1. City-wise Analysis

  • Accidents per city (ranked)
  • Geographic distribution
  • City-specific trends

2. Temporal Analysis

  • Accidents by hour of day
  • Monthly and yearly trends
  • Peak accident hours/days
  • Seasonal patterns

3. Weather Analysis

  • Accidents by weather condition
  • Weather impact assessment
  • Visibility correlation

4. Severity Analysis

  • Accident severity distribution
  • Severity by city/weather
  • Severity trends

5. Geographic Analysis

  • Top accident-prone locations
  • Latitude/longitude clusters
  • Spatial hotspots

6. Statistical Analysis

  • Average casualties per accident
  • Average vehicles involved
  • Casualty trends
  • Vehicle involvement patterns

📈 Visualizations Generated

  1. Bar Chart - Accidents per City (Top 15)
  2. Line Graph - Monthly/Yearly Accident Trends
  3. Heatmap - Accidents by Hour vs Day of Week
  4. Pie Chart - Accident Severity Distribution
  5. Scatter Plot - Geographic Accident Clusters
  6. Bar Chart - Weather-wise Accident Distribution
  7. Line Graph - Accidents Throughout the Day

All graphs are saved as high-resolution PNG files (300 DPI) in the output_graphs/ directory.

🔧 Configuration & Optimization

Spark Configuration (in utils.py)

class Config:
    SPARK_MEMORY = "4g"              # Driver memory
    SPARK_CORES = "4"                # Number of cores
    # Partition count: 200 (for 1M+ rows)
    # Shuffle partitions: 200

Adjust for Your System

For 8GB RAM System:

SPARK_MEMORY = "6g"

For 2GB RAM System:

SPARK_MEMORY = "1g"

Performance Tips

  • Use .cache() for frequently accessed DataFrames ✓
  • Utilize lazy transformations ✓
  • Partition data appropriately ✓
  • Filter early in the pipeline ✓
  • Use columnar operations instead of UDFs ✓

📋 Key Functions

utils.py Functions

initialize_spark()              # Create optimized Spark session
load_data(spark, path)          # Load CSV into Spark DF
clean_data(df)                  # Data cleaning & preprocessing
analyze_data(df)                # Comprehensive analysis
convert_to_pandas(df)           # Spark to Pandas conversion
save_graph(filename, dir)       # Save matplotlib figures
get_data_stats(df)              # Dataset statistics
validate_data(df)               # Data validation

app.py Functions

get_spark_session()             # Cached Spark session
load_and_cache_data(path)       # Data loading with caching
main()                          # Main Streamlit app

📊 Sample Output

Dashboard Metrics

Total Accidents: 50,000
Average Casualties/Accident: 1.25
Average Vehicles/Accident: 2.10
Cities Covered: 15

Top Cities

1. BANGALORE - 8,500 accidents
2. MUMBAI - 7,200 accidents
3. DELHI - 6,800 accidents

Peak Accident Hour

🚨 Peak hour: 18:00 with 2,450 accidents

Weather Impact

⚠️ Most accidents during 'Rainy' weather with 18,500 incidents

🐛 Troubleshooting

Issue: "PySpark not installed"

Solution:

pip install pyspark

Issue: "CSV file not found"

Solution: Ensure traffic_accidents_50000.csv is in the project root directory

Issue: "Out of memory" error

Solution: Reduce SPARK_MEMORY in utils.py Config class

Issue: Streamlit app not starting

Solution:

pip install --upgrade streamlit
streamlit run app.py --logger.level=debug

Issue: Slow performance with large dataset

Solution:

  • Increase SPARK_CORES in Config
  • Reduce number of partitions if dataset is small
  • Use .coalesce() to reduce partitions

📈 Performance Benchmarks

Operation Dataset Size Time
Data Load 50K rows ~2 sec
Data Clean 50K rows ~3 sec
All Analysis 50K rows ~5 sec
Full Pipeline 50K rows ~15 sec
1M rows ~60 sec

Times vary based on system resources

🔒 Data Privacy & Security

  • All data processing happens locally
  • No data is uploaded to external servers
  • CSV files are not modified (read-only)
  • Output graphs are saved locally
  • Implement encryption if using sensitive data

📚 Learning Resources

📝 Example Use Cases

  1. Traffic Safety Analysis - Identify high-risk areas and times
  2. Urban Planning - Data-driven infrastructure improvements
  3. Insurance Analysis - Risk assessment and premium calculation
  4. Law Enforcement - Resource allocation optimization
  5. Public Awareness - Campaign targeting based on data
  6. Emergency Services - Response optimization

🤝 Contributing

To contribute to this project:

  1. Create a feature branch
  2. Make your changes
  3. Test thoroughly
  4. Submit a pull request

📄 License

This project is provided as-is for educational and research purposes.

👨‍💻 Author & Support

Project: Traffic Accident Analysis - Big Data Project Version: 1.0 Last Updated: December 2025

For questions or issues:

🎓 Educational Value

This project demonstrates:

✓ Big Data Technologies - Real-world PySpark usage ✓ Data Engineering - ETL pipeline design ✓ Data Science - Analysis and insights ✓ Web Development - Streamlit applications ✓ Software Engineering - Modular, production-ready code ✓ Data Visualization - Professional graphics ✓ Performance Optimization - Lazy evaluation, caching

📊 Next Steps

After running the project:

  1. Explore the Dashboard for key metrics
  2. Dive into Analysis tab for detailed insights
  3. Review Visualizations to understand patterns
  4. Check Reports for recommendations
  5. Modify the code to analyze your own datasets

Happy analyzing! 🚀📊

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages