Multi-Source Candidate Data Transformer

A robust, modular, and schema-validated talent data ingestion pipeline that transforms multiple structured (CSV, JSON) and unstructured (TXT) candidate data sources into a single, unified canonical profile.

This repository serves as the engineering solution for the Eightfold Engineering Intern (Jul-Dec 2026) assignment.

---

Architecture & Data Flow

The project separates ingestion, normalization, deduplication, projection, and validation layers to ensure a clean, maintainable structure:

1. Ingestion Engine (transformer.py): Extends custom loading adapters for CSV, JSON, and raw text formats to read inputs into a staging buffer.
2. Canonical Normalization (normalizer.py): Standardizes input attributes:
   • Phones: Formatted to E.164.
   • Countries: Normalized to ISO-3166-1 alpha-2.
   • Skills: Split and canonicalized to unique lowercase tokens.
   • Experience & Education: Structured into nested lists.
   • Provenance & Confidence: Automatically tracks data lineage (file origin) and computes a baseline trust value (0.0 - 1.0).
3. Deduplication Engine (deduplicator.py): Resolves entity identity and groups duplicates transitively based on shared contact channels (emails or phones) using a Disjoint Set Union (DSU) graph algorithm. Combines values and boosts overall confidence for cross-verification.
4. Runtime Projection & Validation (projector.py): Accepts an external projection schema config (e.g. config.json) to dynamically map, filter, toggle metadata, validate types, and execute missing value strategies (null, omit, or error) on output records.

---

Installation & Setup

1. Clone or copy this repository on your local system.
2. Install the necessary dependencies (we use lightweight and standard libraries):

   pip install phonenumbers pycountry fpdf2

---

Usage Guide

The transformer engine supports both Interactive Command-Line mode and Automated Batch Processing mode.

1. Batch Processing Mode (Recommended)

Feed input files specified inside pipeline.json, project output records using config.json, and write the structured result to a file:

python main.py -p pipeline.json -c config.json -o output.json

• -p or --pipeline: Path to pipeline config file.
• -c or --config: Path to output projection schema config.
• -o or --output: Output file path to write results.

2. Interactive CLI Mode

Run the pipeline interactively by loading files sequentially and viewing results printed on standard output:

python main.py

• Enter file names (e.g., candidates.csv, ats_blob.json) sequentially.
• Type exit to finalize ingestion and execute normalization, deduplication, and printing.

3. Generate Design Document PDF

Generate the code-free, single-page technical design PDF (HARSSITA NANDA_nandaharssita@gmail.com_Eightfold.pdf) outlining the system design:

python generate_pdf.py

---

Running Automated Tests

A comprehensive unit test suite validates deduplication transitive merging, location/links aggregation, projection path remapping, schema requirements, and missing value strategies:

Run the entire test suite using standard unittest:

python -m unittest discover

All tests run in milliseconds and verify system correctness end-to-end.