Skip to content

Latest commit

Β 

History

224 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

PROSPECTIQ

Intelligence-Driven Prospecting Platform

Search 497K+ LinkedIn profiles with semantic search and lightning-fast queries

License Python PostgreSQL FastAPI

Features β€’ Quick Start β€’ Deployment β€’ API β€’ Architecture


🎬 Live Demo

PROSPECTIQ Demo

Semantic talent search with real-time filtering and CSV export


🎯 Overview

PROSPECTIQ is a production-ready talent intelligence platform designed for GTM teams, recruiters, and data-driven professionals. Built on PostgreSQL 17 with pgvector, it delivers sub-second semantic searches across hundreds of thousands of professional profiles.

What You Get

  • πŸ” Semantic Search - Natural language queries with vector embeddings (OpenAI text-embedding-3-small)
  • ⚑ Sub-second Performance - Optimized hybrid search (80% vector + 20% lexical)
  • πŸ“Š Rich Data - 15+ fields including summaries, skills, social profiles, and contact info
  • 🎨 Modern UI - Sleek dark theme with advanced filtering and real-time results
  • πŸ“€ CSV Export - Bulk export up to 10,000 profiles for CRM integration
  • πŸš€ Production Ready - 497K+ profiles loaded and indexed

✨ Features

Current Status: PRODUCTION READY βœ…

Search & Discovery

  • Semantic search with vector embeddings
  • Full-text search (name, title, company, summary)
  • Hybrid ranking (vector + lexical)
  • Multi-filter support (location, industry, skills, experience)
  • Pagination (20/50/100 results per page)
  • Query time: 500-1000ms

Data & Export

  • 497,552 profiles indexed
  • 15+ data fields per profile
  • CSV/NDJSON export (up to 10K rows)
  • LinkedIn, Twitter, GitHub profiles
  • Professional summaries
  • Skills arrays with 30+ skills per profile

Authentication & API πŸ†•

  • User registration and login
  • JWT-based authentication (24h/30d tokens)
  • API key generation with scopes
  • Tiered rate limiting (200-1000 req/min)
  • Dashboard for key management
  • Secure password hashing (bcrypt)

Data Fields (15+ Columns)

Category Fields
Identity First Name, Last Name, Full Name
Professional Job Title, Company, Industry, Years Experience
Location Country, Region, City, Full Location
Contact Email, Phone, LinkedIn URL, Website
Social Twitter, GitHub
Details Headline, Professional Summary, Skills (array)
Metadata Quality Score, Data Completeness %, Created/Updated timestamps

πŸš€ Quick Start

Prerequisites

  • Docker Desktop (for PostgreSQL + pgvector)
  • Python 3.11+ with Poetry
  • ~21 GB disk space
  • OpenAI API key (for embeddings)

1. Clone & Setup

# Clone the repository
git clone <your-repo-url>
cd WebApplication

# Install Python dependencies
poetry install

# Copy environment template
cp .env.example .env
# Edit .env and add your OPENAI_API_KEY

2. Start Services

# Start PostgreSQL + FastAPI backend
./start_api.sh

# The database is already loaded with 497K profiles!

3. Open Web Interface

# Serve frontend (Next.js dev server)
cd frontend && bun install && bun run dev

# Then open in browser
open http://localhost:5500

4. Create Account & API Key

# Open login page
open http://localhost:5500/login

# 1. Register a new account
# 2. Login to access dashboard
# 3. Create API key with scopes (search:read, export:read, pii:read)
# 4. Copy API key (shown only once!)

5. Search & Export

Web Interface:

  1. Enter keywords: "senior software engineer", "product manager", "data scientist"
  2. Apply filters: US States, Industries, Experience range, Skills
  3. View results with full summaries and contact information
  4. Export to CSV/NDJSON for CRM integration (HubSpot, Salesforce, etc.)

API Access:

# Use your API key to search programmatically
curl -X POST http://localhost:8000/search \
  -H "X-API-Key: your-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{"query": "senior software engineer", "limit": 100}'

πŸ—οΈ Architecture

Technology Stack

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                      PROSPECTIQ STACK                       β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  Frontend   β”‚ Vanilla JavaScript + HTML5 + CSS3             β”‚
β”‚             β”‚ β†’ No framework dependencies                   β”‚
β”‚             β”‚ β†’ Modern dark theme UI                        β”‚
β”‚             β”‚ β†’ Real-time search with filters               β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  Backend    β”‚ FastAPI (Python 3.11+)                        β”‚
β”‚             β”‚ β†’ Async/await with asyncpg                    β”‚
β”‚             β”‚ β†’ Connection pooling                          β”‚
β”‚             β”‚ β†’ Pydantic data validation                    β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  Database   β”‚ PostgreSQL 17 + pgvector                      β”‚
β”‚             β”‚ β†’ HNSW vector index (1536 dimensions)         β”‚
β”‚             β”‚ β†’ GIN full-text search indexes                β”‚
β”‚             β”‚ β†’ Composite indexes for performance           β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  ML/AI      β”‚ OpenAI text-embedding-3-small                 β”‚
β”‚             β”‚ β†’ 1536-dimensional embeddings                 β”‚
β”‚             β”‚ β†’ Semantic similarity search                  β”‚
β”‚             β”‚ β†’ Batch processing for efficiency             β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

System Architecture

graph TB
    subgraph Client["πŸ–₯️ Client Layer"]
        UI[Web UI<br/>Vanilla JS]
    end

    subgraph API["⚑ API Layer"]
        FastAPI[FastAPI Server<br/>Port 8000]
        Pool[AsyncPG Pool<br/>Connection Pool]
    end

    subgraph Data["πŸ’Ύ Data Layer"]
        PG[(PostgreSQL 17<br/>497K Profiles)]
        Vector[pgvector<br/>HNSW Index]
        FTS[Full-Text Search<br/>GIN Indexes]
    end

    subgraph ML["πŸ€– ML Layer"]
        OpenAI[OpenAI API<br/>text-embedding-3-small]
    end

    UI -->|POST /search| FastAPI
    FastAPI -->|async queries| Pool
    Pool --> PG
    PG --> Vector
    PG --> FTS
    FastAPI -.->|embeddings| OpenAI

    style UI fill:#60d5ff,color:#000
    style FastAPI fill:#00ff9d,color:#000
    style PG fill:#336791,color:#fff
    style OpenAI fill:#10a37f,color:#fff
Loading

Search Flow

User Query: "senior software engineer in NYC"
    ↓
1. Generate embedding vector (1536 dims) via OpenAI API
    ↓
2. Hybrid Search (PostgreSQL):
   - Vector Search (80%): Cosine similarity using pgvector HNSW index
   - Lexical Search (20%): Full-text search on title/summary using GIN
    ↓
3. Apply Filters:
   - Location: New York, United States
   - Experience: min_years_experience, max_years_experience
   - Industry: industries[] array
   - Skills: skills[] array (AND logic)
    ↓
4. Rank & Paginate:
   - Combined score = (0.8 Γ— vector_similarity) + (0.2 Γ— ts_rank)
   - Return top 100 results with offset
    ↓
5. Response (500-1000ms):
   - results[] array with 15+ fields
   - total_count for pagination
   - filters_applied summary

🚒 Deployment Roadmap

Phase 1: Local Development βœ… CURRENT

Status: Production-ready with 497K profiles

  • Infrastructure: Docker Compose (PostgreSQL)
  • Data: 497,552 profiles loaded
  • Performance: 500-1000ms queries
  • Cost: $0 (local)
  • Best for: Development, testing, demos

Phase 2: Cloud MVP (1M Profiles) 🎯 NEXT

Target: Deploy to cloud with 1M best profiles

Option A: Railway ⭐ Recommended

# 1-command deployment
railway login
railway init
railway add postgresql
railway up

Pros:

  • One-click PostgreSQL
  • Auto-scaling
  • Easy setup
  • Built-in SSL

Cost: ~$25-50/month

Option B: Render

# render.yaml
services:
  - type: web
    name: prospectiq-api
    env: python
    buildCommand: poetry install
    startCommand: uvicorn backend.api.app:app

databases:
  - name: prospectiq-db
    plan: standard

Pros:

  • Free tier available
  • PostgreSQL included
  • Auto-deploys from git

Cost: ~$20-40/month

Option C: Fly.io

# Fly.io deployment
fly launch
fly postgres create
fly deploy

Pros:

  • Global edge network
  • Free allowance
  • PostgreSQL clusters

Cost: ~$15-30/month

Phase 2 Checklist:

  • Extract 1M best profiles (use scripts/prepare_1m_dataset.py)
  • Add authentication (JWT tokens) βœ…
  • API key generation with scopes βœ…
  • User dashboard for key management βœ…
  • Implement rate limiting (tier-based with Redis)
  • Setup environment variables management (secrets manager)
  • Enable HTTPS (auto via Railway/Render)
  • Configure CORS for production domain
  • Add monitoring (Sentry/LogRocket)
  • Setup automated backups

Phase 3: Production Scale (10M-51M Profiles) πŸš€ FUTURE

Target: Enterprise-grade with full 51M dataset

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    AWS PRODUCTION STACK                     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  CDN            β”‚ CloudFront (global edge caching)          β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  Load Balancer  β”‚ Application Load Balancer (ALB)           β”‚
β”‚                 β”‚ β†’ Auto-scaling FastAPI containers         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  Compute        β”‚ ECS Fargate (4-16 containers)             β”‚
β”‚                 β”‚ β†’ Horizontal auto-scaling                 β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  Database       β”‚ RDS PostgreSQL 17 (db.r6g.2xlarge)        β”‚
β”‚                 β”‚ β†’ Multi-AZ for high availability          β”‚
β”‚                 β”‚ β†’ 100GB-500GB storage                     β”‚
β”‚                 β”‚ β†’ Read replicas for scaling               β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  Cache          β”‚ ElastiCache Redis (cache.r6g.large)       β”‚
β”‚                 β”‚ β†’ Query result caching (5min TTL)         β”‚
β”‚                 β”‚ β†’ Deduplication bloom filters             β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  Storage        β”‚ S3 (Parquet files)                        β”‚
β”‚                 β”‚ β†’ 51M profiles source data                β”‚
β”‚                 β”‚ β†’ Incremental update pipeline             β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Infrastructure as Code:

# Deploy with Terraform
cd infrastructure/terraform
terraform init
terraform plan -var-file=production.tfvars
terraform apply

# Expected resources:
# - VPC with public/private subnets
# - RDS PostgreSQL 17 (Multi-AZ)
# - ECS Fargate cluster (auto-scaling)
# - Application Load Balancer
# - CloudFront distribution
# - ElastiCache Redis cluster
# - S3 buckets (data + backups)

Phase 3 Features:

  • Performance: <200ms queries with Redis caching
  • Availability: 99.9% uptime (Multi-AZ RDS)
  • Scalability: Auto-scaling 4-16 containers based on load
  • Security: VPC, security groups, IAM roles, SSL/TLS
  • Monitoring: CloudWatch, X-Ray tracing, custom dashboards
  • Backups: Automated daily snapshots + point-in-time recovery

Cost Estimate (AWS):

Service Specs Monthly Cost
RDS PostgreSQL db.r6g.2xlarge (8vCPU, 64GB RAM) ~$480
ECS Fargate 4x 2vCPU, 4GB RAM containers ~$120
ElastiCache Redis cache.r6g.large (2vCPU, 13GB RAM) ~$150
Application Load Balancer Standard ALB ~$25
CloudFront 1TB data transfer ~$85
S3 Storage 100GB + requests ~$15
Data Transfer Outbound ~$50
Total ~$925/month

Cost Optimization:

  • Use Reserved Instances (40% savings): ~$555/month
  • Add Savings Plans: ~$450/month
  • Reduce RDS to db.r6g.xlarge: ~$300/month

πŸ”Œ API Endpoints

Authentication Endpoints πŸ†•

POST /auth/register - Create Account

curl -X POST http://localhost:8000/auth/register \
  -H "Content-Type: application/json" \
  -d '{
    "username": "johndoe",
    "email": "john@example.com",
    "password": "SecurePass123!",
    "full_name": "John Doe"
  }'

POST /auth/login - Login

curl -X POST http://localhost:8000/auth/login \
  -H "Content-Type: application/json" \
  -d '{
    "username": "johndoe",
    "password": "SecurePass123!"
  }'

Response:

{
  "access_token": "eyJhbGciOiJIUzI1NiIs...",
  "refresh_token": "eyJhbGciOiJIUzI1NiIs...",
  "token_type": "bearer",
  "expires_in": 86400
}

POST /auth/api-keys - Create API Key

curl -X POST http://localhost:8000/auth/api-keys \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "key_name": "Production API Key",
    "scopes": ["search:read", "export:read", "pii:read"],
    "tier": "trusted"
  }'

Response:

{
  "api_key": "a1b2c3d4e5f6...full-64-char-key",
  "key_name": "Production API Key",
  "key_prefix": "a1b2c3d4e5f6...",
  "scopes": ["search:read", "export:read", "pii:read"],
  "tier": "trusted",
  "created_at": "2025-10-21T04:30:00Z"
}

⚠️ API key is shown only once! Save it securely.

GET /auth/api-keys - List Your API Keys

curl -X GET http://localhost:8000/auth/api-keys \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN"

Search Endpoints

POST /search - Hybrid Search

Search profiles using semantic vector search + full-text search.

With API Key:

curl -X POST http://localhost:8000/search \
  -H "X-API-Key: your-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "senior software engineer with Python and React experience",
    "location_country": "united states",
    "regions": ["California", "New York"],
    "industries": ["Computer Software", "Internet"],
    "min_years_experience": 5,
    "max_years_experience": 15,
    "skills": ["Python", "React"],
    "limit": 100,
    "offset": 0
  }'

Without API Key (Public Access - 50 results max):

curl -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": "senior software engineer",
    "limit": 50
  }'

Response:

{
  "results": [
    {
      "id": 12345,
      "first_name": "Jane",
      "last_name": "Doe",
      "full_name": "Jane Doe",
      "job_title": "Senior Software Engineer",
      "company_name": "Tech Corp",
      "industry": "Computer Software",
      "location": "San Francisco, CA, United States",
      "location_country": "united states",
      "region": "California",
      "locality": "San Francisco",
      "years_experience": 8,
      "headline": "Senior SWE at Tech Corp | Python, React, AWS",
      "summary": "Experienced software engineer with 8+ years building scalable web applications...",
      "skills": ["Python", "React", "AWS", "Docker", "PostgreSQL"],
      "linkedin_url": "linkedin.com/in/janedoe",
      "email": "jane.doe@example.com",
      "phone": "+1-555-123-4567",
      "website": "janedoe.dev",
      "twitter": "janedoe",
      "github": "janedoe",
      "quality_score": 85.5,
      "data_completeness_pct": 90
    }
  ],
  "total_count": 1247,
  "returned_count": 100,
  "query_time_ms": 847,
  "filters_applied": {
    "keyword": "senior software engineer with Python and React experience",
    "country": "united states",
    "regions": ["California", "New York"],
    "industries": ["Computer Software", "Internet"],
    "min_experience": 5,
    "max_experience": 15,
    "skills": ["Python", "React"]
  }
}

GET /stats - Database Statistics

curl http://localhost:8000/stats

Response:

{
  "total_profiles": 497552,
  "profiles_with_embeddings": 250000,
  "countries": ["united states"],
  "top_industries": [
    "Computer Software",
    "Internet",
    "Information Technology and Services",
    "Financial Services",
    "Marketing and Advertising"
  ],
  "avg_years_experience": 12.4,
  "profiles_with_email": 185432,
  "profiles_with_phone": 98765
}

GET /health - Health Check

curl http://localhost:8000/health

Response:

{
  "status": "healthy",
  "database": "connected",
  "profile_count": 497552,
  "timestamp": "2025-10-14T02:30:15Z"
}

πŸ“Š Data Pipeline

Three-Tier Ingestion Architecture

Tier 1: Simple Loader Tier 2: Optimized Loader Tier 3: Cloud Workers

Use Case: Local development, <1M profiles

Performance:

  • 1,000-2,000 rows/sec
  • ~15 min for 1M profiles

Memory:

  • 2-4 GB RAM

Command:

poetry run python -m \
  backend.data_pipeline.ingestion.load_incremental \
  data/USA_1M_test.parquet

Use Case: Fast local loading, 1M-2M profiles

Performance:

  • 5,000-10,000 rows/sec
  • ~2-3 min for 1M profiles

Memory:

  • <500 MB RAM

Command:

poetry run python -m \
  backend.data_pipeline.ingestion.load_optimized \
  data/USA_1M_test.parquet

5x faster than Tier 1!

Use Case: Production cloud, 10M-51M profiles

Performance:

  • 50,000-100,000 rows/sec
  • ~15 min for 51M profiles

Architecture:

  • ECS Fargate workers
  • Redis bloom filters
  • S3 streaming
  • SQS job queue

Cost:

  • One-time: ~$12
  • Monthly updates: ~$0.25

Embedding Generation

# Generate embeddings for profiles without them
poetry run python -m backend.data_pipeline.embeddings.generate

# Features:
# - Batch processing (100 texts per API call)
# - Exponential backoff retry logic
# - Progress tracking with ETA
# - Bulk updates (500 profiles per transaction)
# - Rate: ~37 profiles/sec

πŸ› οΈ Development

Project Structure

WebApplication/
β”œβ”€β”€ backend/
β”‚   β”œβ”€β”€ api/                      # FastAPI application
β”‚   β”‚   β”œβ”€β”€ app.py               # Main server + CORS + auth routes
β”‚   β”‚   β”œβ”€β”€ database.py          # AsyncPG connection pool
β”‚   β”‚   β”œβ”€β”€ models.py            # Pydantic request/response models
β”‚   β”‚   β”œβ”€β”€ search.py            # Hybrid search logic
β”‚   β”‚   β”œβ”€β”€ auth_routes.py       # πŸ†• Authentication endpoints
β”‚   β”‚   β”œβ”€β”€ jwt_utils.py         # πŸ†• JWT token management
β”‚   β”‚   └── user_manager.py      # πŸ†• User & API key operations
β”‚   β”œβ”€β”€ data_pipeline/
β”‚   β”‚   β”œβ”€β”€ embeddings/          # OpenAI embedding generation
β”‚   β”‚   β”‚   β”œβ”€β”€ generate.py      # Batch embedding generator
β”‚   β”‚   β”‚   └── config.py        # OpenAI settings
β”‚   β”‚   └── ingestion/           # Data loading pipeline
β”‚   β”‚       β”œβ”€β”€ load_incremental.py  # Tier 1 loader
β”‚   β”‚       β”œβ”€β”€ load_optimized.py    # Tier 2 loader (5x faster)
β”‚   β”‚       └── deduplication.py     # Content hash deduplication
β”‚   └── tests/                   # Pytest test suite
β”‚       β”œβ”€β”€ test_api.py
β”‚       β”œβ”€β”€ test_search.py
β”‚       └── test_ingestion.py
β”œβ”€β”€ frontend/
β”‚   β”œβ”€β”€ index.html               # Search page
β”‚   β”œβ”€β”€ results.html             # Results display
β”‚   β”œβ”€β”€ search.js                # Search form logic
β”‚   β”œβ”€β”€ results.js               # Results rendering + CSV export
β”‚   β”œβ”€β”€ styles.css               # Dark theme styles (global CSS vars)
β”‚   β”œβ”€β”€ RotatingText.css         # Hero animation
β”‚   β”œβ”€β”€ login.html               # πŸ†• Login & registration page
β”‚   β”œβ”€β”€ dashboard.html           # πŸ†• User dashboard (API key management)
β”‚   β”œβ”€β”€ api-docs.html            # πŸ†• API documentation
β”‚   β”œβ”€β”€ auth.js                  # πŸ†• Authentication utilities
β”‚   └── dashboard.js             # πŸ†• Dashboard logic
β”œβ”€β”€ migrations/                  # SQL schema migrations
β”‚   β”œβ”€β”€ 001_init_schema.sql
β”‚   β”œβ”€β”€ 002_indexes.sql
β”‚   β”œβ”€β”€ 003_vector_index.sql
β”‚   β”œβ”€β”€ 005_data_completeness.sql
β”‚   └── 008_users_and_api_keys.sql  # πŸ†• Authentication schema
β”œβ”€β”€ scripts/
β”‚   β”œβ”€β”€ prepare_1m_dataset.py    # Extract 1M profiles from 51M
β”‚   β”œβ”€β”€ check_data_quality.py    # Data validation
β”‚   β”œβ”€β”€ run_all_tests.sh         # Test suite runner
β”‚   β”œβ”€β”€ run_api_bg.sh            # πŸ†• Start API in background
β”‚   └── serve_frontend_bg.sh     # πŸ†• Start frontend server
β”œβ”€β”€ docs/                        # All documentation (see docs/README.md)
β”‚   β”œβ”€β”€ architecture/            # System design (ARCHITECTURE.md, roadmap, hybrid track)
β”‚   β”œβ”€β”€ database/                # Schema & index reports
β”‚   β”œβ”€β”€ deployment/              # Deployment & scaling guides
β”‚   β”œβ”€β”€ guides/                  # Quick start, theme, security, coding philosophy
β”‚   β”œβ”€β”€ agents/                  # HANDOFF.md log + agent protocol
β”‚   └── archive/                 # Superseded plans (historical)
β”œβ”€β”€ agent.md                     # πŸ†• AI agent instructions (canonical)
β”œβ”€β”€ infrastructure/
β”‚   └── terraform/               # AWS infrastructure as code
β”‚       β”œβ”€β”€ main.tf
β”‚       β”œβ”€β”€ rds.tf
β”‚       β”œβ”€β”€ ecs.tf
β”‚       └── production.tfvars
β”œβ”€β”€ docker-compose.yml           # PostgreSQL + pgvector
β”œβ”€β”€ pyproject.toml               # Poetry dependencies
β”œβ”€β”€ start_api.sh                 # Start backend services
└── .env.example                 # Environment template

Run Tests

# Run all tests
poetry run pytest backend/tests/ -v

# Run specific test file
poetry run pytest backend/tests/test_search.py -v

# Run with coverage
poetry run pytest backend/tests/ --cov=backend --cov-report=html

# Current status: 35/35 tests passing βœ…

Check Data Quality

# Validate data completeness and quality
poetry run python3 scripts/check_data_quality.py

# Output:
# - Profiles with embeddings: 250,000 / 497,552 (50.2%)
# - Profiles with email: 185,432 (37.3%)
# - Profiles with phone: 98,765 (19.8%)
# - Average quality score: 72.4
# - Average data completeness: 68%

🎯 Use Cases

For GTM Teams

  • 🎯 Build targeted prospect lists by title, industry, location
  • πŸ“€ Export to CSV for CRM import (HubSpot, Salesforce)
  • πŸ€– Enrich leads with professional summaries via Claygent
  • πŸ” Find decision makers at specific companies
  • πŸ“Š Research account penetration and org charts

Example Query:

"VP of Sales at SaaS companies in California with 10+ years experience"

For Recruiters

  • πŸ’Ό Search by skills, experience, location
  • πŸ› οΈ Find candidates with specific tech stacks
  • πŸ“‹ Export candidate lists for outreach campaigns
  • πŸ”— View full LinkedIn profiles and contact info
  • πŸ“ˆ Analyze talent market availability

Example Query:

"Senior React developer with TypeScript and AWS experience in New York"

For Data Scientists

  • πŸ“Š Analyze talent markets by geography
  • πŸ“ˆ Track skill trends across industries
  • πŸ€– Build ML models on professional data
  • πŸ”¬ Research career progression patterns
  • πŸ’‘ Generate market intelligence reports

Example Query:

"Data scientists with PhD and 5+ years in Machine Learning"

For Investors

  • 🏒 Research company employee profiles
  • πŸ“Š Track hiring trends in portfolio companies
  • πŸ” Identify talent concentration in startups
  • πŸ’Ό Discover founders and key executives
  • πŸ“ˆ Analyze competitive hiring landscapes

Example Query:

"Founders of AI startups in San Francisco with previous exits"


πŸ” Security & Compliance

⚠️ Important Legal Considerations

This dataset contains scraped LinkedIn data. Before deploying to production:

  1. Review data source legality in your jurisdiction
  2. βœ… Authentication implemented - JWT tokens with bcrypt password hashing
  3. βœ… API key system - Scoped permissions (search:read, export:read, pii:read)
  4. Add rate limiting to prevent abuse (tier-based: basic 200/min, trusted 1000/min)
  5. Use environment variables for all credentials (never commit .env)
  6. Enable HTTPS for production deployment (mandatory)
  7. Comply with GDPR/privacy laws if serving EU users
  8. Implement data deletion requests mechanism
  9. βœ… Audit logging - API key operations tracked in audit_log table

Security Checklist for Production

  • JWT authentication implemented βœ…
  • API key generation with scopes βœ…
  • Password hashing (bcrypt) βœ…
  • Bearer token authentication βœ…
  • Rate limiting configured (Redis-based tier enforcement)
  • CORS restricted to production domain
  • Database credentials in secrets manager (AWS Secrets Manager)
  • API keys rotation policy
  • HTTPS/TLS enforced (min TLS 1.2)
  • Input validation on all endpoints (Pydantic) βœ…
  • SQL injection prevention (parameterized queries) βœ…
  • XSS protection in frontend
  • Security headers configured (helmet.js equivalent)
  • Automated security scanning (Snyk/Dependabot)
  • Audit logs enabled (audit_log table) βœ…

See docs/guides/SECURITY.md for comprehensive security guide.


πŸ“š Documentation

Full index: docs/README.md

Document Description
docs/architecture/ARCHITECTURE.md Current system architecture
docs/architecture/NEXT_STEPS_ARCHITECTURE.md Active roadmap: tiered warehouse + NL search agent
docs/architecture/HYBRID_SETUP.md Hybrid track setup (Postgres hot tier + Redis + DuckDB)
docs/architecture/INGESTION_ARCHITECTURE.md Three-tier data pipeline
docs/guides/QUICK_START.md Quick start (DuckDB browse API)
docs/deployment/DEPLOYMENT_GUIDE.md Deploying to Railway/Render/Fly.io
docs/deployment/SCALING_TO_51M_GUIDE.md Scaling strategy to 51M profiles
docs/guides/THEME_GUIDELINES.md UI/theme styling standards
docs/guides/SECURITY.md Security best practices
agent.md AI agent instructions (canonical spec)

πŸ”„ Changelog

v1.2.0 (2025-10-21) πŸ†• LATEST

  • βœ… Authentication System: User registration, login with JWT tokens (24h access, 30d refresh)
  • βœ… API Key Management: Generate keys with scopes (search:read, export:read, pii:read)
  • βœ… User Dashboard: Web interface for managing API keys
  • βœ… Tiered Access: Public (50 results), Basic (200 req/min), Trusted (1000 req/min)
  • βœ… Security: bcrypt password hashing, SHA-256 API key hashing, audit logging
  • βœ… Theme Documentation: Comprehensive UI guidelines (THEME_GUIDELINES.md)
  • βœ… API Documentation: Restructured docs with sidebar navigation, code examples
  • βœ… Database Schema: users, api_keys, refresh_tokens, audit_log tables

v1.1.0 (2025-10-14) βœ…

  • Fixed link readability in results table (bright cyan #60d5ff)
  • Added data completeness percentage tracking
  • Fixed parameter indexing bug in hybrid search
  • Added comprehensive deployment roadmap
  • CSV/NDJSON export support

v1.0.0 (2025-10-11) βœ…

  • βœ… 497K profiles loaded and indexed
  • βœ… Hybrid search (80% vector + 20% lexical)
  • βœ… 15+ data fields with social profiles
  • βœ… CSV export functionality (10K row limit)
  • βœ… Professional summaries and skills arrays
  • βœ… Modern dark theme UI with filtering

v0.9.0 (2025-10-08)

  • FastAPI backend with PostgreSQL
  • Full-text search with GIN indexes
  • Web UI with horizontal scrolling tables
  • Filter by country, industry, experience, skills

🀝 Contributing

This is currently a personal project. For questions, suggestions, or bug reports:

  1. Open an issue with detailed description
  2. Include steps to reproduce (for bugs)
  3. Suggest enhancements with use cases

πŸ“„ License

Educational & Personal Use Only

This project is for educational and personal use. The LinkedIn data is subject to LinkedIn's Terms of Service. Use responsibly and in compliance with applicable laws (GDPR, CCPA, etc.).

Disclaimer: This software is provided "as is" without warranty of any kind. Users are responsible for ensuring compliance with all applicable laws and terms of service.


Current Status: βœ… Production Ready (497K Profiles)

Next Step: Deploy to Railway or Scale to AWS


Built with ❀️ using PostgreSQL 17, FastAPI, and OpenAI Embeddings

⬆ Back to Top

About

No description, website, or topics provided.

Resources

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages