Skip to content
This repository was archived by the owner on Apr 17, 2026. It is now read-only.

README.md

Latest commit

 

History

History
214 lines (158 loc) · 5.73 KB

README.md

File metadata and controls

214 lines (158 loc) · 5.73 KB
Raw

Cross-Platform Linker Service

A dedicated microservice for intelligently linking people across GitHub and Bluesky platforms using the Brainy vector database. This service runs independently from scout-search to provide optimal performance for both search and linking operations.

Overview

The Cross-Platform Linker Service identifies and links the same person across different platforms by analyzing:

  • Exact Matches: Email addresses, website URLs, explicit cross-references
  • Heuristic Matches: Name similarity, handle similarity, bio similarity, skill overlap, location consistency
  • Graph Matches: Mutual connections, shared interactions, repository mention patterns
  • Temporal Matches: Account creation timing, activity correlation patterns

Architecture

  • Write-Only Service: Reads from shared S3 storage, enriches data, writes back
  • Background Processing: Runs on adaptive intervals (15s-5m) based on activity
  • Zero Configuration: Auto-detects new people, no manual setup required
  • Performance Optimized: Dedicated resources without impacting search performance

Quick Start

Local Development

# Install dependencies
npm install

# Build the service
npm run build

# Start the service
npm start

Docker

# Build Docker image
docker build -t cross-platform-linker .

# Run container
docker run -p 8083:8083 --env-file .env cross-platform-linker

Google Cloud Run

# Deploy to Cloud Run
./deploy-to-cloud-run.sh

Configuration

The service uses environment variables for configuration:

# Google Cloud Storage (shared with other services)
GCS_ACCESS_KEY_ID=your_access_key
GCS_SECRET_ACCESS_KEY=your_secret_key
GCS_BUCKET_NAME=brainy
GCS_ENDPOINT=https://storage.googleapis.com

# Service configuration
PORT=8083
LOG_LEVEL=info
AUTO_START_PROCESSING=true

API Endpoints

Health Check

GET /health

Returns service status and statistics.

Metrics

GET /metrics

Returns performance metrics and processing statistics.

Manual Trigger

POST /trigger-linking

Manually triggers the cross-platform linking process (for testing).

Linking Algorithm

1. Candidate Discovery

  • Name-based queries (full name, first+last name)
  • Handle-based queries (cleaned of platform suffixes)
  • Email extraction from bios
  • Website/domain matching
  • Unique bio phrase matching

2. Confidence Scoring

  • Exact Matches (40% weight): Email, website, explicit mentions
  • Heuristic Matches (25% weight): Name, handle, bio, skills, location similarity
  • Graph Matches (25% weight): Mutual connections, interaction patterns
  • Temporal Matches (10% weight): Account timing, activity correlation

3. Link Creation

  • High Confidence (>0.9): Automatic linking
  • Medium Confidence (0.7-0.9): Queued for review
  • Low Confidence (<0.7): Ignored

4. Data Enrichment

Links are stored in the metadata.enhanced.identity.alternateIds structure:

{
  "metadata": {
    "enhanced": {
      "identity": {
        "alternateIds": [
          {
            "platform": "github",
            "id": "github_user_123",
            "handle": "johndoe",
            "verified": true,
            "confidence": 0.95,
            "linkingMethod": "cross-platform-linker-service"
          }
        ]
      }
    }
  }
}

Performance Features

  • Adaptive Processing: Speeds up when activity detected, slows down when idle
  • Memory Management: LRU cache cleanup to prevent memory leaks
  • Batch Processing: Processes multiple people in optimized batches
  • Timeout Protection: Prevents hanging operations from blocking the service
  • Error Resilience: Continues processing even if individual links fail

Monitoring

The service provides comprehensive metrics:

  • Processing statistics (people processed, links created)
  • Performance metrics (processing time, memory usage)
  • Error tracking and logging
  • Health check endpoints for load balancer integration

Integration with Other Services

Scout-Search

  • Read-Only Mode: Scout-search reads enriched data with cross-platform links
  • Performance Isolation: No background processing in scout-search
  • Data Consistency: Both services use the same S3 storage backend

GitHub/Bluesky Packages

  • Data Source: These packages write raw person data
  • Link Enhancement: Cross-platform-linker enriches the data with connections
  • Confidence Scores: Leverages the confidence-scored verbs from both packages

Deployment

The service is designed for cloud deployment with:

  • Google Cloud Run: Serverless container deployment
  • Shared Storage: Same GCS bucket as other services
  • Auto-scaling: Scales based on processing load
  • Health Monitoring: Built-in health checks and metrics

Development

Project Structure

src/
├── index.ts                    # Main service entry point
├── services/
│   └── CrossPlatformLinker.ts  # Core linking logic
└── augmentations/
    └── CrossPlatformLinkingAugmentation.ts  # Brainy augmentation

Adding New Platforms

To add support for linking additional platforms:

  1. Update candidate discovery queries in generateSearchQueries()
  2. Add platform-specific matching logic in scoring methods
  3. Update the isPerson() method for new platform detection
  4. Add platform-specific metadata extraction

License

MIT License - see LICENSE file for details.

Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Add tests if applicable
  5. Submit a pull request

Support

For issues and questions:

  • Create an issue in this repository
  • Check the health endpoint for service status
  • Review logs using gcloud run logs read --service=cross-platform-linker