Skip to content
This repository was archived by the owner on Apr 17, 2026. It is now read-only.

CLAUDE.md

Latest commit

 

History

History
225 lines (179 loc) · 10 KB

CLAUDE.md

File metadata and controls

225 lines (179 loc) · 10 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Development Commands

Building and Running

  • npm run build - Compile TypeScript and copy polyfill files to dist/
  • npm start - Build and start the service (requires 8GB memory)
  • npm run dev - Start TypeScript watch mode for development
  • npm run dev:start - Run with nodemon watching dist/ changes

Testing

  • npm test - Run all tests with Vitest
  • npm run test:watch - Run tests in watch mode
  • npm run test:coverage - Generate test coverage reports
  • npm run test:skills - Run skills detection tests specifically
  • npm run test:scout-search - Run scout-search integration tests
  • npm run test:github-service - Run GitHub service tests
  • npm run test:server - Run server integration tests
  • npm run test:cloud-run - Run Cloud Run compatibility tests

Code Quality

  • npm run lint - Lint TypeScript files in src/
  • npm run lint:fix - Auto-fix linting issues

Deployment

  • npm run deploy:cloud - Deploy to Google Cloud Run (uses .env file directly, no secrets manager)
  • npm run release - Build and create GitHub release

Git Commit Process

IMPORTANT: When committing changes, use Conventional Commit format without any Claude Code attribution or co-author information.

For commits, use one of these approaches:

  1. Simple wrapper script: ./commit (stages changes and opens editor)
  2. Manual conventional commit: 
    git add -A
git commit -m "<type>(<scope>): <description>

Detailed explanation of changes and their impact."

Conventional Commit Format:

  • Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
  • Keep first line under 72 characters
  • Use imperative mood (e.g., 'add' not 'added')
  • No Claude Code attribution or co-author information

Examples:

  • feat(auth): add GitHub App authentication support
  • fix(models): resolve Universal Sentence Encoder loading failures
  • docs(readme): update deployment instructions

When Claude Code says "commit this", always use the conventional format without any AI attribution.

Architecture Overview

This is a Node.js/TypeScript service that processes GitHub data into knowledge graphs using the Brainy framework. The service runs on Google Cloud Run and provides scout-search APIs for talent discovery.

Core Components

Main Service (src/index.ts)

  • Express.js web server with health checks and API endpoints
  • Brainy initialization with Universal Sentence Encoder model (requires 8GB memory)
  • GitHub service integration with rate limiting
  • Scout-search API endpoints for talent discovery

Data Processing Pipeline

  • GitHubService - Fetches data from GitHub API with rate limit handling
  • GitHubMappingService - Maps GitHub data to Brainy nouns/verbs
  • PlaceholderResolutionService - Resolves placeholder references
  • Augmentations transform raw data into standardized formats

Key Augmentations

  • GitHubDataTransformationAugmentation - Transforms GitHub API data using standardized cross-platform schema
  • SkillsAndJobDetectionAugmentation - Detects developer skills and job-seeking signals
  • PlaceholderManagementAugmentation - Manages data references and relationships

Data Architecture

Standardized Cross-Platform Schema (v1.1.0)

  • Compatible with bluesky-package and scout-search platforms
  • Enhanced with Brainy 0.36.0 verb type standardization
  • Consistent identifiers (github_user_123, canonical IDs) with type inference
  • Standardized metadata structure with confidence scoring and relationship tracking
  • Cross-platform talent discovery capabilities with improved reliability

Storage Integration

  • Uses Brainy framework with Google Cloud Storage backend
  • Structured as nouns (entities) and verbs (relationships)
  • Write-only mode reduces memory requirements for large datasets

Environment Configuration

Required Variables

# GitHub Authentication (one of these approaches)
GITHUB_TOKEN=your_github_token                    # Personal access token
# OR GitHub App (recommended for higher rate limits)
GITHUB_APP_ID=your_app_id
GITHUB_PRIVATE_KEY=your_private_key
GITHUB_INSTALLATION_ID=your_installation_id

# Storage Configuration
GCS_BUCKET_NAME=your_storage_bucket
GCS_ACCESS_KEY_ID=your_access_key
GCS_SECRET_ACCESS_KEY=your_secret_key
GCS_ENDPOINT=https://storage.googleapis.com       # Optional
GCS_REGION=auto                                   # Optional

Optional Variables

PORT=8080                                         # Server port
NODE_ENV=production                              # Environment
AUTO_START_PROCESSING=true                       # Auto-start data processing
LOG_LEVEL=error                                  # Logging level (error=lowest cost, info=development)
LOCAL_MONITORING=true                            # Display monitoring info

# Model Configuration (DO NOT MODIFY - Float32 enforced for data consistency)
# PREFER_COMPRESSED_MODELS=false                 # Automatically set by config
# FORCE_FLOAT32_MODELS=true                      # Automatically set by config

Production Logging Cost Reduction

IMPORTANT: Cloud Run logging charges per log entry. To minimize costs:

  • LOG_LEVEL=error: Only logs errors, warnings, and critical milestones
  • Aggressive Console Filtering: Suppresses high-volume Brainy processing logs
  • Blocked Patterns: Verb creation, skill mapping, transformations, cache operations
  • Cost Impact: Can reduce logging costs by 90-95% in high-volume scenarios

For Debugging: Temporarily set LOG_LEVEL=info and redeploy, then revert to error.

API Endpoints

Scout-Search Talent Discovery

  • GET /api/scout-search/talent - Search developers with filters (skills, experience, job-seeking status)
  • GET /api/scout-search/skills - Get all available skills for filtering
  • GET /api/scout-search/stats - Get talent discovery statistics

Service Management

  • GET /health - Health check with service status
  • GET /stats - Processing statistics
  • POST /start - Start GitHub data processing
  • POST /full-sync - Perform full GitHub synchronization

Placeholder Resolution

  • GET /api/placeholder-resolution/stats - Resolution statistics
  • POST /api/placeholder-resolution/resolve-all - Resolve all placeholders
  • POST /api/placeholder-resolution/resolve-immediate - Resolve specific placeholders

Development Notes

Memory Requirements

  • Universal Sentence Encoder model requires minimum 8GB memory
  • Cloud Run deployment requires --memory="8Gi" configuration

Testing Strategy

  • Uses Vitest with single-threaded execution for stability
  • Test files in test/ directory with .test.ts extension
  • Coverage reports generated in ./test-results/
  • Separate test scripts for different components

Code Style

  • ESLint configuration with TypeScript rules
  • Prettier formatting with specific preferences (no semicolons, single quotes)
  • Uses ES modules ("type": "module") with NodeNext module resolution

Key Dependencies

  • @soulcraft/brainy (v0.52.0) - Knowledge graph framework with automatic high-volume optimization and memory management
  • @octokit/rest - GitHub API client
  • Express.js - Web server framework
  • Vitest - Testing framework
  • TypeScript with strict compilation

Docker Deployment

The service includes shell scripts for deployment automation:

  • deploy-to-cloud-run.sh - Simplified Cloud Run deployment using .env file directly (no Google Secret Manager)
  • Various test scripts for verifying fixes and deployments

Deployment Approach: Uses environment variables directly from .env file instead of Google Cloud Secret Manager for simpler debugging and configuration management.

Brainy Integration (v0.52.0)

  • Uses augmentation pipeline for data transformation
  • Implements standardized schema v1.1.0 for cross-platform compatibility
  • Write-only mode enabled to reduce memory usage during processing
  • NEW in 0.52.0: Automatic socket exhaustion prevention and memory pressure management

Brainy 0.52.0 Features:

  • Automatic High-Volume Optimization: Socket exhaustion prevention and intelligent connection pooling
  • Memory Pressure Management: Automatic memory-aware batch sizing and backpressure handling
  • Error-Based Throttling: Self-adjusting rate limiting based on system feedback
  • Zero-Configuration: All optimizations work automatically without manual tuning
  • Enhanced Reliability: Built-in safeguards for production high-volume scenarios

Previous Brainy 0.39.0 Features:

  • Ultra-Fast Search Performance: Revolutionary caching and optimization system
    • Smart result caching with <1ms repeated query performance
    • Cursor-based pagination with O(k) performance for large datasets
    • Real-time data sync with automatic cache invalidation
    • Performance monitoring with built-in hit rate tracking
    • Intelligent auto-configuration that detects environment and optimizes automatically
  • Enhanced API Endpoints: New performance monitoring capabilities
    • /api/brainy/performance - Cache statistics and performance metrics
    • /api/brainy/health - Enhanced distributed health status
    • /api/scout-search/talent - Now supports cursor-based pagination for better performance
  • Improved Cache System: Advanced caching with intelligence
    • Increased cache sizes (15k hot cache, optimized eviction thresholds)
    • Smart result caching with automatic TTL management
    • Performance monitoring with adaptive optimization
    • Selective cache invalidation strategies
  • Enhanced Real-Time Updates: Better coordination and synchronization
    • Automatic cache invalidation when data changes
    • Faster sync intervals (30 seconds) for better responsiveness
    • Selective invalidation strategy for optimal performance
  • Previous v0.38.0 Features:
    • Distributed mode with writer role for optimized write operations
    • Enhanced cache system with auto-tuning capabilities
    • Coordinated heartbeat system for distributed instance management
    • Enhanced retry policy with exponential backoff