🧴 Data Scraper from Google Maps

This project is a powerful automated web scraper that collects detailed information about ANY ENTITY(E.g., Dermatologists) across various cities in India using Selenium to interact with Google Maps. It is designed to run city-by-city, track scraping progress, and save data in a clean, structured format for each city.

📌 Table of Contents

• 📸 Features
• 🔧 Requirements
• 📁 Project Structure
• 🚀 Getting Started
• 🧠 How It Works
• 📄 Output Data Format
• 🧼 Data Cleaning & Quality
• 📈 Extending the Project
• ⚠️ Disclaimers & Ethics
• 📬 Support

---

📸 Features

• ✅ Scrapes dermatologist names, ratings, addresses, phone numbers, and websites from Google Maps
• 🔁 Automatically continues from where it last stopped (using progress.txt)
• 🧹 Cleans and sanitizes noisy or unwanted symbols (e.g., 🏠, 📞) from scraped data
• 📦 Saves city-wise results to CSV files (data/<city>.csv)
• 🔍 Handles dynamic content loading and lazy-loading scroll behavior
• ⚠️ Skips duplicates and handles errors gracefully
• 💻 Easily extendable for other business categories or countries

---

🔧 Requirements

Install the following before running the project:

1. Python 3.7+
2. Chrome or Edge WebDriver (make sure the version matches your installed Browser)
3. Python packages:

   pip install selenium

---

📁 Project Structure

project-root/
│
├── cities.csv           # Input list of cities to scrape (required)
├── progress.txt         # Stores index of last scraped city
├── data/                # Output directory where city-wise CSVs are saved
│   └── delhi.csv
│
└── app.py           # Main scraper script

---

🚀 Getting Started

1. Prepare cities.csv

Create a cities.csv file like this:

city_name
Delhi
Mumbai
Chennai
Bangalore
Hyderabad
Dehradun

✅ Important: Ensure the header is exactly city_name.

---

2. Setup progress.txt

Create a file progress.txt with a single number indicating the last city index scraped.

Example:

0

The scraper will read this value and start from that city.

---

3. Run the Scraper

Execute the script:

python scraper.py

If you are going to run program for the first time, double-click on setup.bat to setup up virtual environment and install dependencies. Then you get a run.bat file. Just double-click on it each time you want to run the program.

It will:

• Load the next city from cities.csv
• Scrape all dermatologist listings from Google Maps
• Save results to /data/<city>.csv
• Update progress.txt to move to the next city

---

🧠 How It Works

1. Google Maps Search
   The script constructs a URL like:
   https://www.google.co.in/maps/search/dermatologists+in+<city>

2. Scroll Detection
   It scrolls through the list of search results, detecting when no more new listings load, or when the "You've reached the end of the list." message appears.

3. Data Extraction
   For each listing:

   • Clicks the card
   • Waits for the detail panel to load
   • Extracts:
     • Name
     • Rating
     • Address
     • Phone
     • Website

4. Duplicate Handling
   Duplicate entries (by name) are skipped to avoid redundant results.

5. Output Saving
   Cleaned and structured data is saved to data/<city>.csv.

6. Progress Tracking
   The progress.txt file ensures that scraping can resume in the next run from the last unprocessed city.

---

📄 Output Data Format

Each city CSV contains:

Name	Rating	Address	Phone	Website	City
Dr. XYZ	4.7	XYZ Road, Near ABC Hospital, Delhi	+91-9999999999	www.clinicexample.in	Delhi
Derma Clinic	4.9	Some Lane, Dehradun, Uttarakhand 248001	+91-8888888888	www.dermaclinic.com	Dehradun

---

🧼 Data Cleaning & Quality

• Address and phone fields often contain non-standard characters or icon-like Unicode (e.g., \ue0c8, \ue0b0).
  ✅ These are stripped using split("\n")[-1] to extract only the relevant data.

• Duplicate names are skipped to avoid repeated entries.

---

📈 Extending the Project

You can easily adapt this scraper to work for:

• Different business categories: e.g., cardiologists, gynecologists, etc.
• Different countries or languages
• Saving JSON or databases instead of CSV

Just modify:

query_url = f"https://www.google.co.in/maps/search/<your_search_query>+in+{city}"

And update the scraping fields as necessary.

---

⚠️ Disclaimers & Ethics

• This project is for educational and research purposes only.
• Google Maps Terms of Service prohibit automated scraping. Use responsibly.
• Always ensure data privacy, do not use or share scraped data for spam or unethical purposes.

---

📬 Support

If you found this helpful or want to contribute improvements:

• Raise an issue
• Submit a pull request
• Or reach out!

---