Welfare organizations often maintain facility databases that are difficult to keep up to date — data entry is manual, records become inconsistent over time, and information is scattered across many regional websites and external data sources. This project addresses that challenge with an automated scraping and data processing pipeline designed for the welfare sector.
The pipeline collects facility information from multiple external sources — including Google Places, OpenStreetMap, and regional association websites — and applies NLP and LLM-based methods to normalize and deduplicate the results. The outcome is a clean, consolidated dataset of welfare facilities that can be used to assess data quality, enrich existing databases, or lay the groundwork for more automated data management in the future.
The pipeline is built to be adaptable and can be pointed at any welfare organization's data landscape. The implementation in this repository uses AWO Bundesverband e.V. as a worked example.
The Arbeiterwohlfahrt (AWO — Workers' Welfare Association) is one of six legally recognized umbrella organizations for independent welfare services in Germany, employing more than 140,000 full-time staff across a federally organized, politically independent, and non-denominational structure.
AWO maintains an internal facility database (Einrichtungsdatenbank) that has become outdated, error-prone, and incomplete due to reliance on manual data entry by regional associations. AWO partnered with DSSG Berlin on a volunteer feasibility study to analyze the database's current data quality and explore pathways toward automated enrichment. This pipeline was developed as part of that project and provided a solid technical foundation for the future modernization of the Einrichtungsdatenbank.
The pipeline runs in three sequential phases:
[ Scraping ] → [ Normalization ] → [ Deduplication ]
↓ ↓ ↓
01_scraping/ 02_scraped_processed/ entities_merged_*.csv
All phases are orchestrated by the scrape.py entrypoint. Cloud vs. local behavior depends on the CloudStorageManager configuration (see Getting Started).
Collects raw records from multiple external sources:
| Source | Method |
|---|---|
| Google Places API | scrape_google_places() via places_api.gmaps_scraper |
| OpenStreetMap | scrape_osm() via Overpass/OSM |
| Regional websites | scrape_regional_sites() — handles paginated, PDF, JS-dynamic pages |
| LLM contact extraction | extract_scrape_results() via scraping.agnostic_extractor |
Outputs (written to 01_scraping/): raw CSVs per source, e.g. google_places_results_combined.csv, osm_results_raw.csv, results_llm_raw.csv, and regional scraped files.
Standardizes organization names and address components across all scraped sources, producing canonical fields such as normalized_name, normalized_street, normalized_postcode, and normalized_city.
Outputs (written to 02_scraped_processed/): one normalized CSV per source, e.g. google_places_results_normalized.csv, osm_results_normalized.csv.
Detects and merges duplicate records across all normalized sources into canonical entities, preserving provenance back to original source records.
The algorithm uses:
- Within-source deduplication (exact matches on normalized fields)
- Cross-source duplicate detection via indexing blocks (address keys, name+postcode, name+city)
- Union-Find clustering; only strong (Tier A) matches are auto-merged
Outputs (written to 02_scraped_processed/):
entities_merged_canonical.csv— one row per canonical entity with canonical name, address, coordinates, and contact fieldsentities_merged_provenance.csv— one row per source record withcanonical_idmapping
- Python 3.9+
- An AWS account with an S3 bucket (for cloud/production runs)
- A Google Places API key (for the Google scraper)
- A HuggingFace API Key (for using LLMs via HuggingFace)
- A Google Cloud Platform API key (for using LLMs via Google AI Studio
git clone https://github.com/dssg-berlin/awo-welfare-data-harvester.git
cd awo-welfare-data-harvester
# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txtCopy .env.example to .env and fill in your credentials:
cp .env.example .envOpen .env and set all required values (API keys, AWS credentials, etc.).
IMPORTANT: This code expects you to have data saved on AWS S3, which is used as the default storage location for all data. In order to run locally, you need to set the AWS S3 Credential under .env.example as environment variables and define the bucket_name and region_name input parameters in the scrape.py entry script of the project:
In scrape.py, set your S3 bucket name and region:
# Initialize cloud storage
storage = CloudStorageManager(bucket_name="your-bucket-name", region_name="your-region")python scrape.pyThis runs all three phases in order. For partial runs:
- Normalization only: call
normalize_file()fromsrc/normalization_scraped_data.pydirectly. - Deduplication only: instantiate
EntityMergerfromsrc/deduplication.py, ensuring normalized CSVs already exist in02_scraped_processed/.
awo-welfare-data-harvester/
├── docs/ # Further documentation on code specificities
├── src/ # Core source code
│ ├── places_api/ # Google Places API scraper
│ ├── scraping/ # Web scraping utilities
│ └── utils/ # General utilities
├── .env.example # Environment variable template
├── .gitignore. # Ignored files
├── README.md # This file
├── requirements.txt # Python dependencies
├── scrape.py # Pipeline entrypoint
└── setup.py # Project setup
We're grateful to all contributors who have made their work visible in this repository. The following list orders the top contributors primarily by the number of commits they've made to the default branch:
This project is licensed under the MIT License. See LICENSE for details.
This project was carried out as a volunteer initiative by DSSG Berlin in partnership with AWO Bundesverband e.V.