Cartoelf

A GitHub-based candidate discovery tool designed for API-first deployment as interconnected modules. Cartoelf transforms the challenging task of finding developers into a data-driven process that identifies candidates through their relationships, contributions, and skills.

Overview

Cartoelf discovers talented open source developers by analyzing GitHub connections, contributions, and skills. It builds a graph of relationships between developers, repositories, and organizations to identify candidates who match specific criteria.

Key Benefits:

API-First Design: Ready for service deployment and integration
Instant Search: Sub-second response times with in-memory graph processing
Smart Filtering: Multi-dimensional filtering with percentile-based ranking
Web Interface: Modern SvelteKit frontend with interactive visualizations
Scalable Architecture: Handles 100K+ users and 1M+ repositories
Production Ready: Completed MVP with proven functionality

Features

🔍 Smart Data Collection: Crawl GitHub starting from users, repos, or organizations
📊 Intelligent Filtering: Focus collection on active, relevant developers
🌐 Graph Analysis: Build connection graphs to find developers through relationships
💻 Skill Extraction: Identify skills from repository languages and contributions
🎯 Unified Search: Flexible search algorithm with parameter tuning
🤖 AI Agent Search: LLM-powered candidate discovery with natural language input
📤 Export Options: CSV for Looker, JSON for programmatic use
🚀 Graph Services: In-memory graph processing for instant search responses
🌐 Web Interface: Modern SvelteKit frontend with interactive visualizations

Quick Start

Prerequisites

Node.js 18+ and npm
GitHub Personal Access Token with appropriate permissions
Anthropic API Key (for AI Agent functionality)
Completed data collection and analysis (see pipeline below)

Installation

# Clone the repository
git clone https://github.com/andyschwab/cartoelf.git
cd cartoelf

# Install dependencies
npm install

# Build the project
npm run build

# Set up your tokens
echo "GITHUB_TOKEN=your_token_here" > .env
echo "ANTHROPIC_API_KEY=your_anthropic_key_here" >> .env

Pipeline Execution

# 1. Collect data from GitHub
cartoelf collect octocat,microsoft --depth 2

# 2. Load data into SQLite database
cartoelf load

# 3. Analyze and build graph with scoring
cartoelf analyze

# 4. Search for candidates with percentile filtering
cartoelf search --nexus microsoft --skills typescript --export csv
cartoelf search --top-technical 10 --top-skill typescript:5 --export json
cartoelf search --top-impact 5 --limit 20 --export csv

# 5. (Optional) Launch web interface for interactive search
npm run dev

## Core Pipeline

### 1. Collect ✅
Gather GitHub data with smart caching and SQLite indexing.

```bash
cartoelf collect <nexus> [options]

# Examples
cartoelf collect torvalds --depth 2
cartoelf collect microsoft/vscode,facebook/react
cartoelf collect github --org-expansion

Features: Depth-based discovery, dual storage strategy, intelligent filtering, rate limit management

2. Load ✅

Transform cached data into normalized SQLite database.

cartoelf load [options]

# Examples
cartoelf load                    # Incremental loading
cartoelf load --force           # Force reload everything
cartoelf load --dry-run         # Preview changes

Features: Resilient processing, inline relationships, phase-based execution, progress tracking

3. Analyze ✅

Calculate comprehensive scores and percentile rankings.

cartoelf analyze [options]

# Examples
cartoelf analyze                    # Full analysis pipeline
cartoelf analyze --force           # Force recalculation
cartoelf analyze --percentiles     # Calculate percentiles only

Features: Modular scoring, incremental processing, percentile calculation, skill extraction

4. Search ✅

Unified search with flexible parameter tuning and web interface.

# CLI Search
cartoelf search [options]

# Examples
cartoelf search --nexus microsoft --skills typescript --export csv
cartoelf search --top-technical 10 --top-skill typescript:5 --export json
cartoelf search --top-impact 5 --limit 20 --export csv

# Unified Server (NEW)
npm run start:unified                 # Full server with graph
npm run start:unified:no-graph        # Fast startup without graph

Features: CLI search, web interface, API endpoints, parameter tuning, percentile filtering

5. Web Interface & APIs 🌐

Modern web interface with comprehensive API access.

Unified Server (http://localhost:3002):

Natural Search: LLM-powered candidate discovery with natural language
Query Builder: Visual boolean query construction (coming soon)
API Explorer: Interactive testing of all V1 endpoints
Graph Visualization: Interactive relationship exploration
Analytics Dashboard: Database statistics and insights

API Endpoints:

Legacy API (/api/*): Backward-compatible search and agent endpoints
V1 API (/api/v1/*): Production-ready REST API with comprehensive search
System API (/health, /graph/*): Health checks and graph management

6. AI Agent Search 🤖

LLM-powered candidate discovery with natural language input.

# Start the unified server with agent support
npm run start:unified

# Access the Natural Search at http://localhost:3002/agent
# Or use the API directly:
curl -X POST http://localhost:3002/api/agent \
  -H "Content-Type: application/json" \
  -d '{"input": "Looking for a senior React developer with TypeScript experience"}'

Features: Natural language input, AI-generated keywords, intelligent ranking, detailed explanations

Architecture

System Overview

GitHub APIs → Collect → Load → Analyze → Search → Results
     ↓           ↓       ↓       ↓        ↓        ↓
  Raw Data   Cache   Database  Scores  Search   Export

Core Modules

src/collect/ - GitHub data collection and caching
src/load/ - Data transformation and database loading
src/analyze/ - Scoring algorithms and percentile calculation
src/search/ - Unified search engine and filtering
src/agent/ - AI-powered candidate search with LLM integration
src/services/ - HTTP API service and graph services
src/client/ - Modern web interface (SvelteKit)
src/cli/ - Command-line interface (thin wrapper)

Key Design Principles

API-First: Core modules ready for service deployment
Dual Storage: File cache + SQLite index for performance + resilience
Incremental Processing: Smart detection of what needs updating
Error Isolation: Individual failures don't stop the entire process
Scalable Architecture: Designed for 100K+ users and 1M+ repositories

Configuration

Create a .env file in the project root:

# Required
GITHUB_TOKEN=your_github_personal_access_token

# Optional
GITHUB_API_URL=https://api.github.com
RATE_LIMIT_REQUESTS_PER_HOUR=4500
CACHE_TTL_DAYS=30
LOG_LEVEL=info

Project Status

✅ MVP Complete - Production Ready

Core Functionality:

Data collection from GitHub with smart caching and SQLite indexing
Resilient data loading with inline relationship processing
Modular scoring algorithms with percentile calculations
Unified search engine with flexible parameter tuning
Graph services for instant search responses
Comprehensive export functionality (CSV/JSON)

Performance Characteristics:

Collection: 1000+ users/hour with rate limit management
Loading: 1000+ entities/minute with inline processing
Analysis: 1000+ users/minute with incremental processing
Search: Sub-second response times for most queries

Architecture Features:

Dual Storage: File cache + SQLite index for optimal performance
In-Memory Graph: Instant search responses with graph services
Error Handling: Graceful degradation and comprehensive recovery
API-First Design: Ready for service deployment and integration
Web Interface: Modern frontend demonstrating API capabilities

🚀 Ready for Production

Core Modules: Business logic separated from CLI concerns
Programmatic Interface: Functions can be imported and used directly
Service Architecture: HTTP API service implemented and ready for production
Scalable Design: Supports distributed processing and deployment
Resilient Processing: Error isolation and resumable operations

Documentation

Quick Start

VISION.md - Project vision and current status
README.md - Documentation navigation and overview

Core Architecture

ARCHITECTURE.md - Complete system architecture overview

Module Documentation

collect.md - GitHub data collection and caching system
load.md - Data transformation and database loading
analyze.md - Scoring algorithms and percentile calculation
search.md - Unified search engine and filtering capabilities
graph.md - Graph services and in-memory relationship analysis
frontend.md - Web interface and API demonstration

Future Development

POST_MVP.md - Future enhancements and planned features

Usage Examples

Basic Search

# Find TypeScript developers
cartoelf search --skills "typescript,react" --hireable

# Find developers connected to Microsoft
cartoelf search --nexus "microsoft" --skills "python" --max-depth 2

# Find top 10% technical performers
cartoelf search --top-technical 10 --skills "javascript"

# Content-based search for specific expertise
cartoelf search --content "machine learning" --skills "python"

Advanced Filtering

# Multiple percentile filters
cartoelf search --top-technical 25 --top-activity 10 --top-impact 15

# Skill-specific percentiles
cartoelf search --top-skill typescript:10 --top-skill python:5

# Content-based search
cartoelf search --content "machine learning" --skills "python"

Export Options

# CSV export for Looker
cartoelf search --skills "react" --export csv --output react_devs.csv

# JSON export for API consumption
cartoelf search --nexus "github" --skills "typescript" --export json

Development

Local Development

# Install dependencies
npm install

# Build project
npm run build

# Run tests
npm test

# Development mode
npm run dev

Web Interface and API Service

# Start development mode (all services)
npm run dev

# Start individual services
npm run start:search-service  # HTTP API service
npm run dev:client           # Frontend development

Contributing

This project is under active development. Contributions are welcome!

Fork the repository
Create your feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

Built with TypeScript, Node.js, and SQLite
Uses Graphology for graph processing
Powered by GitHub's GraphQL API
Designed for API-first deployment and service integration

Name		Name	Last commit message	Last commit date
Latest commit History 95 Commits
config		config
docs		docs
scripts		scripts
src		src
.cursorignore		.cursorignore
.dockerignore		.dockerignore
.env.example		.env.example
.eslintrc.json		.eslintrc.json
.gitignore		.gitignore
.prettierrc		.prettierrc
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
cleanup.sh		cleanup.sh
docker-compose.dev.yml		docker-compose.dev.yml
docker-compose.yml		docker-compose.yml
jsconfig.json		jsconfig.json
package-lock.json		package-lock.json
package.json		package.json
postcss.config.js		postcss.config.js
tailwind.config.js		tailwind.config.js
todo.txt		todo.txt
tsconfig.json		tsconfig.json

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Cartoelf

Overview

Features

Quick Start

Prerequisites

Installation

Pipeline Execution

2. Load ✅

3. Analyze ✅

4. Search ✅

5. Web Interface & APIs 🌐

6. AI Agent Search 🤖

Architecture

System Overview

Core Modules

Key Design Principles

Configuration

Project Status

✅ MVP Complete - Production Ready

🚀 Ready for Production

Documentation

Quick Start

Core Architecture

Module Documentation

Future Development

Usage Examples

Basic Search

Advanced Filtering

Export Options

Development

Local Development

Web Interface and API Service

Contributing

License

Acknowledgments

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Packages