Skip to content

WARP.md

Latest commit

 

History

History
147 lines (113 loc) · 6.42 KB

WARP.md

File metadata and controls

147 lines (113 loc) · 6.42 KB
Raw

WARP.md

This file provides guidance to WARP (warp.dev) when working with code in this repository.

Project Overview

Persona-based LinkedIn connection filtering system that identifies people with specific job titles at enterprise companies. Uses YAML configuration for easy customization of targeting criteria.

Commands

Main Filtering System

# Default usage with innovation_leaders persona
python main.py

# Custom input file
python main.py --input my_connections.csv

# Use different persona (if defined in config.yaml)
python main.py --persona fintech_leaders

# Custom configuration file
python main.py --config my_config.yaml

# Custom output directory
python main.py --output results/

# Debug mode for troubleshooting
python main.py --debug

# Quiet mode (minimal output)
python main.py --quiet

Help and Options

# View all available options
python main.py --help

Architecture Overview

Unified Filter System (main.py)

  • Pipeline: YAML config → CSV input → text normalization → persona-based matching → enterprise filtering → results + analysis
  • Configuration: LinkedInFilter class loads persona-based targeting from config.yaml
  • Innovation Detection: is_innovation_role() - configurable keyword patterns by category, company token filtering, exclusion rules
  • Enterprise Matching: is_enterprise_company() - configurable company lists by category (Brazilian, tech, consulting, financial)
  • Output: Persona-prefixed files with confidence scoring, seniority detection, and comprehensive analysis

Persona Configuration

  • Active Persona: Set in config.yaml via active_persona field
  • Keyword Categories: Organized by innovation type (core_innovation, digital_transformation, ventures_labs, strategy_tech)
  • Language Support: Portuguese/Spanish terms and normalization rules
  • Enterprise Companies: Categorized lists (brazilian, brazilian_tech, global_tech, consulting, financial, innovation_focused)

Key Components

CSV Parsing

  • Library: Built-in csv module with DictReader
  • Encoding: utf-8-sig for BOM handling
  • Columns Used: First Name, Last Name, Company, Position, URL, Email Address, Connected On
  • Special Handling: LinkedIn CSV header detection (read_linkedin_csv() in enhanced/simple filters)

Text Normalization

  • Basic: normalize_text() - lowercase, whitespace normalization
  • Advanced: normalize_text() with Unicode NFD normalization, accent removal (innovation_enterprise_filter.py)
  • Company Names: normalize_company_name() - suffix removal (Inc, LLC, etc.), punctuation cleanup

Innovation Role Detection

  • Keywords: 4 categories - core_innovation, digital_transformation, ventures_labs, strategy_tech
  • Portuguese/Spanish: Includes "transformação digital", "gerente"
  • Company Token Filtering: extract_company_tokens() removes company name from position to avoid false positives
  • Validation: Excludes false positives like "renovation"

Enterprise Company Matching

  • Source: Hardcoded lists + optional enterprise_companies.csv
  • Aliases: ENTERPRISE_ALIASES dict for name variations (e.g., "International Business Machines" → "IBM")
  • Fuzzy Matching: Jaccard similarity on word tokens (jaccard_similarity())
  • Thresholds: Default 0.85 fuzzy threshold, configurable via --fuzzy-threshold

Input/Output Structure

Input

  • File: Connections.csv at repository root (or custom path via --input)
  • Format: LinkedIn connection export with UTF-8 BOM
  • Required Columns: First Name, Last Name, Company, Position
  • Configuration: config.yaml defining personas, keywords, and enterprise companies

Output

  • Directory: output/ (auto-created, configurable via --output)
  • Files: {persona_name}_filtered_connections.csv + {persona_name}_analysis_summary.json
  • CSV Columns: Original fields + innovation_category, matched_keywords, enterprise_match, match_confidence, seniority_level
  • JSON Summary: Statistics, match rates, top companies, configuration used

Configuration Management

config.yaml Structure

  • active_persona: Currently active targeting persona
  • personas: Define multiple targeting strategies with keywords organized by category
  • enterprise_companies: Categorized company lists (brazilian, global_tech, consulting, etc.)
  • language_support: Text normalization and multi-language keyword support
  • matching: Scoring weights, thresholds, and seniority detection rules
  • output: File naming and column specifications
  • validation: Exclusion patterns and quality filters

Creating New Personas

  1. Add new persona section under personas in config.yaml
  2. Define keyword categories relevant to your targeting
  3. Optionally customize enterprise company lists
  4. Run with --persona your_persona_name

Testing and Verification

Nushell Exploration Tools

# Quick data overview and analysis
nu explore.nu

# Search for specific keywords in positions
nu explore.nu search "innovation"

# Deep dive into a specific company
nu explore.nu company "Microsoft"
  • Purpose: Fast data exploration and ad-hoc analysis
  • Benefits: Interactive CSV analysis, quick statistics, company/keyword searches
  • Complement: Use alongside Python main.py for different analysis perspectives

Legacy Scripts (_deprecated/)

  • Location: All original scripts moved to _deprecated/ directory
  • Purpose: Reference implementations and backup approaches
  • Usage: Available for comparison or specific edge cases

Project-Specific Notes

  • File Dependencies: All scripts expect Connections.csv in repository root
  • Output Behavior: Scripts create output/ directory if it doesn't exist
  • Language Support: Enhanced and Enterprise filters include Portuguese/Spanish terms for Brazilian connections
  • Deduplication: Only innovation_enterprise_filter.py includes deduplication based on name/company/position
  • Performance: Enterprise filter supports fuzzy matching threshold tuning for precision/recall tradeoff

Results Summary (Latest Run)

  • Total Connections: 1,536 analyzed
  • Innovation Roles: 29 identified (1.9% of connections)
  • Enterprise Companies: 533 connections (34.7% of all connections)
  • Final Matches: 13 people (0.8% of connections)
  • Top Categories: Strategy & Tech (6), Core Innovation (4), Digital Transformation (3)