Search 497K+ LinkedIn profiles with semantic search and lightning-fast queries
Features β’ Quick Start β’ Deployment β’ API β’ Architecture
PROSPECTIQ is a production-ready talent intelligence platform designed for GTM teams, recruiters, and data-driven professionals. Built on PostgreSQL 17 with pgvector, it delivers sub-second semantic searches across hundreds of thousands of professional profiles.
- π Semantic Search - Natural language queries with vector embeddings (OpenAI text-embedding-3-small)
- β‘ Sub-second Performance - Optimized hybrid search (80% vector + 20% lexical)
- π Rich Data - 15+ fields including summaries, skills, social profiles, and contact info
- π¨ Modern UI - Sleek dark theme with advanced filtering and real-time results
- π€ CSV Export - Bulk export up to 10,000 profiles for CRM integration
- π Production Ready - 497K+ profiles loaded and indexed
|
Search & Discovery
|
Data & Export
|
Authentication & API π
|
| Category | Fields |
|---|---|
| Identity | First Name, Last Name, Full Name |
| Professional | Job Title, Company, Industry, Years Experience |
| Location | Country, Region, City, Full Location |
| Contact | Email, Phone, LinkedIn URL, Website |
| Social | Twitter, GitHub |
| Details | Headline, Professional Summary, Skills (array) |
| Metadata | Quality Score, Data Completeness %, Created/Updated timestamps |
- Docker Desktop (for PostgreSQL + pgvector)
- Python 3.11+ with Poetry
- ~21 GB disk space
- OpenAI API key (for embeddings)
# Clone the repository
git clone <your-repo-url>
cd WebApplication
# Install Python dependencies
poetry install
# Copy environment template
cp .env.example .env
# Edit .env and add your OPENAI_API_KEY# Start PostgreSQL + FastAPI backend
./start_api.sh
# The database is already loaded with 497K profiles!# Serve frontend (Next.js dev server)
cd frontend && bun install && bun run dev
# Then open in browser
open http://localhost:5500# Open login page
open http://localhost:5500/login
# 1. Register a new account
# 2. Login to access dashboard
# 3. Create API key with scopes (search:read, export:read, pii:read)
# 4. Copy API key (shown only once!)Web Interface:
- Enter keywords: "senior software engineer", "product manager", "data scientist"
- Apply filters: US States, Industries, Experience range, Skills
- View results with full summaries and contact information
- Export to CSV/NDJSON for CRM integration (HubSpot, Salesforce, etc.)
API Access:
# Use your API key to search programmatically
curl -X POST http://localhost:8000/search \
-H "X-API-Key: your-api-key-here" \
-H "Content-Type: application/json" \
-d '{"query": "senior software engineer", "limit": 100}'βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β PROSPECTIQ STACK β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Frontend β Vanilla JavaScript + HTML5 + CSS3 β
β β β No framework dependencies β
β β β Modern dark theme UI β
β β β Real-time search with filters β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Backend β FastAPI (Python 3.11+) β
β β β Async/await with asyncpg β
β β β Connection pooling β
β β β Pydantic data validation β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Database β PostgreSQL 17 + pgvector β
β β β HNSW vector index (1536 dimensions) β
β β β GIN full-text search indexes β
β β β Composite indexes for performance β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β ML/AI β OpenAI text-embedding-3-small β
β β β 1536-dimensional embeddings β
β β β Semantic similarity search β
β β β Batch processing for efficiency β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
graph TB
subgraph Client["π₯οΈ Client Layer"]
UI[Web UI<br/>Vanilla JS]
end
subgraph API["β‘ API Layer"]
FastAPI[FastAPI Server<br/>Port 8000]
Pool[AsyncPG Pool<br/>Connection Pool]
end
subgraph Data["πΎ Data Layer"]
PG[(PostgreSQL 17<br/>497K Profiles)]
Vector[pgvector<br/>HNSW Index]
FTS[Full-Text Search<br/>GIN Indexes]
end
subgraph ML["π€ ML Layer"]
OpenAI[OpenAI API<br/>text-embedding-3-small]
end
UI -->|POST /search| FastAPI
FastAPI -->|async queries| Pool
Pool --> PG
PG --> Vector
PG --> FTS
FastAPI -.->|embeddings| OpenAI
style UI fill:#60d5ff,color:#000
style FastAPI fill:#00ff9d,color:#000
style PG fill:#336791,color:#fff
style OpenAI fill:#10a37f,color:#fff
User Query: "senior software engineer in NYC"
β
1. Generate embedding vector (1536 dims) via OpenAI API
β
2. Hybrid Search (PostgreSQL):
- Vector Search (80%): Cosine similarity using pgvector HNSW index
- Lexical Search (20%): Full-text search on title/summary using GIN
β
3. Apply Filters:
- Location: New York, United States
- Experience: min_years_experience, max_years_experience
- Industry: industries[] array
- Skills: skills[] array (AND logic)
β
4. Rank & Paginate:
- Combined score = (0.8 Γ vector_similarity) + (0.2 Γ ts_rank)
- Return top 100 results with offset
β
5. Response (500-1000ms):
- results[] array with 15+ fields
- total_count for pagination
- filters_applied summary
Status: Production-ready with 497K profiles
- Infrastructure: Docker Compose (PostgreSQL)
- Data: 497,552 profiles loaded
- Performance: 500-1000ms queries
- Cost: $0 (local)
- Best for: Development, testing, demos
Target: Deploy to cloud with 1M best profiles
|
Option A: Railway β Recommended # 1-command deployment
railway login
railway init
railway add postgresql
railway upPros:
Cost: ~$25-50/month |
Option B: Render # render.yaml
services:
- type: web
name: prospectiq-api
env: python
buildCommand: poetry install
startCommand: uvicorn backend.api.app:app
databases:
- name: prospectiq-db
plan: standardPros:
Cost: ~$20-40/month |
Option C: Fly.io # Fly.io deployment
fly launch
fly postgres create
fly deployPros:
Cost: ~$15-30/month |
Phase 2 Checklist:
- Extract 1M best profiles (use
scripts/prepare_1m_dataset.py) - Add authentication (JWT tokens) β
- API key generation with scopes β
- User dashboard for key management β
- Implement rate limiting (tier-based with Redis)
- Setup environment variables management (secrets manager)
- Enable HTTPS (auto via Railway/Render)
- Configure CORS for production domain
- Add monitoring (Sentry/LogRocket)
- Setup automated backups
Target: Enterprise-grade with full 51M dataset
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β AWS PRODUCTION STACK β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β CDN β CloudFront (global edge caching) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Load Balancer β Application Load Balancer (ALB) β
β β β Auto-scaling FastAPI containers β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Compute β ECS Fargate (4-16 containers) β
β β β Horizontal auto-scaling β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Database β RDS PostgreSQL 17 (db.r6g.2xlarge) β
β β β Multi-AZ for high availability β
β β β 100GB-500GB storage β
β β β Read replicas for scaling β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Cache β ElastiCache Redis (cache.r6g.large) β
β β β Query result caching (5min TTL) β
β β β Deduplication bloom filters β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Storage β S3 (Parquet files) β
β β β 51M profiles source data β
β β β Incremental update pipeline β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Infrastructure as Code:
# Deploy with Terraform
cd infrastructure/terraform
terraform init
terraform plan -var-file=production.tfvars
terraform apply
# Expected resources:
# - VPC with public/private subnets
# - RDS PostgreSQL 17 (Multi-AZ)
# - ECS Fargate cluster (auto-scaling)
# - Application Load Balancer
# - CloudFront distribution
# - ElastiCache Redis cluster
# - S3 buckets (data + backups)Phase 3 Features:
- Performance: <200ms queries with Redis caching
- Availability: 99.9% uptime (Multi-AZ RDS)
- Scalability: Auto-scaling 4-16 containers based on load
- Security: VPC, security groups, IAM roles, SSL/TLS
- Monitoring: CloudWatch, X-Ray tracing, custom dashboards
- Backups: Automated daily snapshots + point-in-time recovery
Cost Estimate (AWS):
| Service | Specs | Monthly Cost |
|---|---|---|
| RDS PostgreSQL | db.r6g.2xlarge (8vCPU, 64GB RAM) | ~$480 |
| ECS Fargate | 4x 2vCPU, 4GB RAM containers | ~$120 |
| ElastiCache Redis | cache.r6g.large (2vCPU, 13GB RAM) | ~$150 |
| Application Load Balancer | Standard ALB | ~$25 |
| CloudFront | 1TB data transfer | ~$85 |
| S3 Storage | 100GB + requests | ~$15 |
| Data Transfer | Outbound | ~$50 |
| Total | ~$925/month |
Cost Optimization:
- Use Reserved Instances (40% savings): ~$555/month
- Add Savings Plans: ~$450/month
- Reduce RDS to db.r6g.xlarge: ~$300/month
curl -X POST http://localhost:8000/auth/register \
-H "Content-Type: application/json" \
-d '{
"username": "johndoe",
"email": "john@example.com",
"password": "SecurePass123!",
"full_name": "John Doe"
}'curl -X POST http://localhost:8000/auth/login \
-H "Content-Type: application/json" \
-d '{
"username": "johndoe",
"password": "SecurePass123!"
}'Response:
{
"access_token": "eyJhbGciOiJIUzI1NiIs...",
"refresh_token": "eyJhbGciOiJIUzI1NiIs...",
"token_type": "bearer",
"expires_in": 86400
}curl -X POST http://localhost:8000/auth/api-keys \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"key_name": "Production API Key",
"scopes": ["search:read", "export:read", "pii:read"],
"tier": "trusted"
}'Response:
{
"api_key": "a1b2c3d4e5f6...full-64-char-key",
"key_name": "Production API Key",
"key_prefix": "a1b2c3d4e5f6...",
"scopes": ["search:read", "export:read", "pii:read"],
"tier": "trusted",
"created_at": "2025-10-21T04:30:00Z"
}curl -X GET http://localhost:8000/auth/api-keys \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"Search profiles using semantic vector search + full-text search.
With API Key:
curl -X POST http://localhost:8000/search \
-H "X-API-Key: your-api-key-here" \
-H "Content-Type: application/json" \
-d '{
"query": "senior software engineer with Python and React experience",
"location_country": "united states",
"regions": ["California", "New York"],
"industries": ["Computer Software", "Internet"],
"min_years_experience": 5,
"max_years_experience": 15,
"skills": ["Python", "React"],
"limit": 100,
"offset": 0
}'Without API Key (Public Access - 50 results max):
curl -X POST http://localhost:8000/search \
-H "Content-Type: application/json" \
-d '{
"query": "senior software engineer",
"limit": 50
}'Response:
{
"results": [
{
"id": 12345,
"first_name": "Jane",
"last_name": "Doe",
"full_name": "Jane Doe",
"job_title": "Senior Software Engineer",
"company_name": "Tech Corp",
"industry": "Computer Software",
"location": "San Francisco, CA, United States",
"location_country": "united states",
"region": "California",
"locality": "San Francisco",
"years_experience": 8,
"headline": "Senior SWE at Tech Corp | Python, React, AWS",
"summary": "Experienced software engineer with 8+ years building scalable web applications...",
"skills": ["Python", "React", "AWS", "Docker", "PostgreSQL"],
"linkedin_url": "linkedin.com/in/janedoe",
"email": "jane.doe@example.com",
"phone": "+1-555-123-4567",
"website": "janedoe.dev",
"twitter": "janedoe",
"github": "janedoe",
"quality_score": 85.5,
"data_completeness_pct": 90
}
],
"total_count": 1247,
"returned_count": 100,
"query_time_ms": 847,
"filters_applied": {
"keyword": "senior software engineer with Python and React experience",
"country": "united states",
"regions": ["California", "New York"],
"industries": ["Computer Software", "Internet"],
"min_experience": 5,
"max_experience": 15,
"skills": ["Python", "React"]
}
}curl http://localhost:8000/statsResponse:
{
"total_profiles": 497552,
"profiles_with_embeddings": 250000,
"countries": ["united states"],
"top_industries": [
"Computer Software",
"Internet",
"Information Technology and Services",
"Financial Services",
"Marketing and Advertising"
],
"avg_years_experience": 12.4,
"profiles_with_email": 185432,
"profiles_with_phone": 98765
}curl http://localhost:8000/healthResponse:
{
"status": "healthy",
"database": "connected",
"profile_count": 497552,
"timestamp": "2025-10-14T02:30:15Z"
}| Tier 1: Simple Loader | Tier 2: Optimized Loader | Tier 3: Cloud Workers |
|---|---|---|
|
Use Case: Local development, <1M profiles Performance:
Memory:
Command: poetry run python -m \
backend.data_pipeline.ingestion.load_incremental \
data/USA_1M_test.parquet |
Use Case: Fast local loading, 1M-2M profiles Performance:
Memory:
Command: poetry run python -m \
backend.data_pipeline.ingestion.load_optimized \
data/USA_1M_test.parquet5x faster than Tier 1! |
Use Case: Production cloud, 10M-51M profiles Performance:
Architecture:
Cost:
|
# Generate embeddings for profiles without them
poetry run python -m backend.data_pipeline.embeddings.generate
# Features:
# - Batch processing (100 texts per API call)
# - Exponential backoff retry logic
# - Progress tracking with ETA
# - Bulk updates (500 profiles per transaction)
# - Rate: ~37 profiles/secWebApplication/
βββ backend/
β βββ api/ # FastAPI application
β β βββ app.py # Main server + CORS + auth routes
β β βββ database.py # AsyncPG connection pool
β β βββ models.py # Pydantic request/response models
β β βββ search.py # Hybrid search logic
β β βββ auth_routes.py # π Authentication endpoints
β β βββ jwt_utils.py # π JWT token management
β β βββ user_manager.py # π User & API key operations
β βββ data_pipeline/
β β βββ embeddings/ # OpenAI embedding generation
β β β βββ generate.py # Batch embedding generator
β β β βββ config.py # OpenAI settings
β β βββ ingestion/ # Data loading pipeline
β β βββ load_incremental.py # Tier 1 loader
β β βββ load_optimized.py # Tier 2 loader (5x faster)
β β βββ deduplication.py # Content hash deduplication
β βββ tests/ # Pytest test suite
β βββ test_api.py
β βββ test_search.py
β βββ test_ingestion.py
βββ frontend/
β βββ index.html # Search page
β βββ results.html # Results display
β βββ search.js # Search form logic
β βββ results.js # Results rendering + CSV export
β βββ styles.css # Dark theme styles (global CSS vars)
β βββ RotatingText.css # Hero animation
β βββ login.html # π Login & registration page
β βββ dashboard.html # π User dashboard (API key management)
β βββ api-docs.html # π API documentation
β βββ auth.js # π Authentication utilities
β βββ dashboard.js # π Dashboard logic
βββ migrations/ # SQL schema migrations
β βββ 001_init_schema.sql
β βββ 002_indexes.sql
β βββ 003_vector_index.sql
β βββ 005_data_completeness.sql
β βββ 008_users_and_api_keys.sql # π Authentication schema
βββ scripts/
β βββ prepare_1m_dataset.py # Extract 1M profiles from 51M
β βββ check_data_quality.py # Data validation
β βββ run_all_tests.sh # Test suite runner
β βββ run_api_bg.sh # π Start API in background
β βββ serve_frontend_bg.sh # π Start frontend server
βββ docs/ # All documentation (see docs/README.md)
β βββ architecture/ # System design (ARCHITECTURE.md, roadmap, hybrid track)
β βββ database/ # Schema & index reports
β βββ deployment/ # Deployment & scaling guides
β βββ guides/ # Quick start, theme, security, coding philosophy
β βββ agents/ # HANDOFF.md log + agent protocol
β βββ archive/ # Superseded plans (historical)
βββ agent.md # π AI agent instructions (canonical)
βββ infrastructure/
β βββ terraform/ # AWS infrastructure as code
β βββ main.tf
β βββ rds.tf
β βββ ecs.tf
β βββ production.tfvars
βββ docker-compose.yml # PostgreSQL + pgvector
βββ pyproject.toml # Poetry dependencies
βββ start_api.sh # Start backend services
βββ .env.example # Environment template
# Run all tests
poetry run pytest backend/tests/ -v
# Run specific test file
poetry run pytest backend/tests/test_search.py -v
# Run with coverage
poetry run pytest backend/tests/ --cov=backend --cov-report=html
# Current status: 35/35 tests passing β
# Validate data completeness and quality
poetry run python3 scripts/check_data_quality.py
# Output:
# - Profiles with embeddings: 250,000 / 497,552 (50.2%)
# - Profiles with email: 185,432 (37.3%)
# - Profiles with phone: 98,765 (19.8%)
# - Average quality score: 72.4
# - Average data completeness: 68%
Example Query:
|
Example Query:
|
Example Query:
|
Example Query:
|
This dataset contains scraped LinkedIn data. Before deploying to production:
- Review data source legality in your jurisdiction
- β Authentication implemented - JWT tokens with bcrypt password hashing
- β API key system - Scoped permissions (search:read, export:read, pii:read)
- Add rate limiting to prevent abuse (tier-based: basic 200/min, trusted 1000/min)
- Use environment variables for all credentials (never commit .env)
- Enable HTTPS for production deployment (mandatory)
- Comply with GDPR/privacy laws if serving EU users
- Implement data deletion requests mechanism
- β Audit logging - API key operations tracked in audit_log table
- JWT authentication implemented β
- API key generation with scopes β
- Password hashing (bcrypt) β
- Bearer token authentication β
- Rate limiting configured (Redis-based tier enforcement)
- CORS restricted to production domain
- Database credentials in secrets manager (AWS Secrets Manager)
- API keys rotation policy
- HTTPS/TLS enforced (min TLS 1.2)
- Input validation on all endpoints (Pydantic) β
- SQL injection prevention (parameterized queries) β
- XSS protection in frontend
- Security headers configured (helmet.js equivalent)
- Automated security scanning (Snyk/Dependabot)
- Audit logs enabled (audit_log table) β
See docs/guides/SECURITY.md for comprehensive security guide.
Full index: docs/README.md
| Document | Description |
|---|---|
| docs/architecture/ARCHITECTURE.md | Current system architecture |
| docs/architecture/NEXT_STEPS_ARCHITECTURE.md | Active roadmap: tiered warehouse + NL search agent |
| docs/architecture/HYBRID_SETUP.md | Hybrid track setup (Postgres hot tier + Redis + DuckDB) |
| docs/architecture/INGESTION_ARCHITECTURE.md | Three-tier data pipeline |
| docs/guides/QUICK_START.md | Quick start (DuckDB browse API) |
| docs/deployment/DEPLOYMENT_GUIDE.md | Deploying to Railway/Render/Fly.io |
| docs/deployment/SCALING_TO_51M_GUIDE.md | Scaling strategy to 51M profiles |
| docs/guides/THEME_GUIDELINES.md | UI/theme styling standards |
| docs/guides/SECURITY.md | Security best practices |
| agent.md | AI agent instructions (canonical spec) |
- β Authentication System: User registration, login with JWT tokens (24h access, 30d refresh)
- β API Key Management: Generate keys with scopes (search:read, export:read, pii:read)
- β User Dashboard: Web interface for managing API keys
- β Tiered Access: Public (50 results), Basic (200 req/min), Trusted (1000 req/min)
- β Security: bcrypt password hashing, SHA-256 API key hashing, audit logging
- β Theme Documentation: Comprehensive UI guidelines (THEME_GUIDELINES.md)
- β API Documentation: Restructured docs with sidebar navigation, code examples
- β Database Schema: users, api_keys, refresh_tokens, audit_log tables
- Fixed link readability in results table (bright cyan #60d5ff)
- Added data completeness percentage tracking
- Fixed parameter indexing bug in hybrid search
- Added comprehensive deployment roadmap
- CSV/NDJSON export support
- β 497K profiles loaded and indexed
- β Hybrid search (80% vector + 20% lexical)
- β 15+ data fields with social profiles
- β CSV export functionality (10K row limit)
- β Professional summaries and skills arrays
- β Modern dark theme UI with filtering
- FastAPI backend with PostgreSQL
- Full-text search with GIN indexes
- Web UI with horizontal scrolling tables
- Filter by country, industry, experience, skills
This is currently a personal project. For questions, suggestions, or bug reports:
- Open an issue with detailed description
- Include steps to reproduce (for bugs)
- Suggest enhancements with use cases
Educational & Personal Use Only
This project is for educational and personal use. The LinkedIn data is subject to LinkedIn's Terms of Service. Use responsibly and in compliance with applicable laws (GDPR, CCPA, etc.).
Disclaimer: This software is provided "as is" without warranty of any kind. Users are responsible for ensuring compliance with all applicable laws and terms of service.
Next Step: Deploy to Railway or Scale to AWS
Built with β€οΈ using PostgreSQL 17, FastAPI, and OpenAI Embeddings
