Skip to content

docxology/opentir

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
comprehensive_analysis_results/visualizations
comprehensive_analysis_results/visualizations
 
 
examples
examples
 
 
multi_repo_analysis
multi_repo_analysis
 
 
scripts
scripts
 
 
src
src
 
 
tests
tests
 
 
.cursorignore
.cursorignore
 
 
.cursorrules
.cursorrules
 
 
.env.example
.env.example
 
 
.gitattributes
.gitattributes
 
 
.gitignore
.gitignore
 
 
DOCUMENTATION_UPDATES.md
DOCUMENTATION_UPDATES.md
 
 
IMPROVEMENTS_SUMMARY.md
IMPROVEMENTS_SUMMARY.md
 
 
README.md
README.md
 
 
STATISTICAL_ANALYSIS_SUMMARY.md
STATISTICAL_ANALYSIS_SUMMARY.md
 
 
comprehensive_analysis.py
comprehensive_analysis.py
 
 
demo.py
demo.py
 
 
main.py
main.py
 
 
pytest.ini
pytest.ini
 
 
requirements.txt
requirements.txt
 
 
run.sh
run.sh
 
 
setup.py
setup.py
 
 

Repository files navigation

Opentir - Comprehensive Palantir OSS Ecosystem Tool

Opentir is a comprehensive open source project that builds on, with, and among Palantir's open source technologies. It provides a complete toolkit for analyzing, organizing, and documenting Palantir's extensive open source ecosystem of 250+ repositories, 6.2+ million lines of code, and 500,000+ functions and classes.

🌟 Palantir Ecosystem Overview

📊 Complete Palantir Ecosystem Guide

Palantir has built one of the most comprehensive enterprise open source ecosystems, spanning:

🎯 Flagship Projects

  • Blueprint - React UI toolkit (20,000+ stars)
  • TSLint - TypeScript linter (6,000+ stars)
  • Plottable - D3 charting library (2,900+ stars)
  • AtlasDB - Distributed database (800+ stars)

📈 By Technology Stack

🌟 Opentir Features

🔍 Comprehensive Repository Management

  • Automated Discovery: Fetch all 250+ Palantir repositories from GitHub
  • Organized Structure: Automatically organize repos by language, category, and popularity
  • Smart Cloning: Efficient cloning with rate limiting and error handling
  • Package Categories: Deep categorization by function and technology

📊 Advanced Code Analysis

  • Multi-Language Support: Analyze Python, JavaScript, TypeScript, Java, Go, Rust, and more
  • Method Extraction: Extract and catalog all 500,000+ functions, classes, and methods
  • Complexity Analysis: Calculate code complexity and quality metrics across all repos
  • Functionality Mapping: Generate comprehensive functionality matrices

📚 Documentation Generation

🚀 Easy-to-Use Interface

  • CLI Tool: Powerful command-line interface for all operations
  • Python API: Programmatic access to all functionality
  • Async Support: High-performance async operations for large-scale analysis

🎯 Why Opentir? Beyond Simple Repository Cloning

🔬 Enterprise-Grade Analysis Engine

While you could manually clone 250+ repositories, Opentir transforms raw code into actionable intelligence:

📊 Statistical Rigor

  • ANOVA Testing: Statistical significance across repository metrics
  • Principal Component Analysis: Identify the most influential code patterns
  • Clustering Analysis: Automatically group repositories by functionality
  • Confidence Intervals: Quantify uncertainty in code quality metrics
  • Effect Size Calculations: Measure practical significance of differences

🧠 Pattern Recognition & Insights

  • Cross-Repository Dependencies: Map how 250+ repos interconnect
  • Functionality Overlap Detection: Find redundant implementations across projects
  • Architecture Pattern Analysis: Identify common design patterns at scale
  • Code Evolution Tracking: Understand how Palantir's practices have evolved
  • Technical Debt Assessment: Quantify maintenance burden across the ecosystem

🎨 Professional Visualizations

  • Interactive Network Graphs: Explore repository relationships dynamically
  • Complexity Heatmaps: Visualize code complexity across the entire ecosystem
  • Dependency Trees: Navigate intricate inter-project dependencies
  • Timeline Analysis: Track project evolution and activity patterns
  • Technology Stack Distribution: Understand language and framework adoption

🏢 Enterprise Decision Support

Strategic Technology Assessment

  • Technology Maturity Scoring: Evaluate stability and adoption readiness
  • Maintenance Risk Analysis: Identify projects with sustainability concerns
  • Integration Complexity Mapping: Plan implementation strategies
  • Resource Allocation Insights: Understand development effort requirements
  • Compliance & Security Analysis: Assess enterprise readiness

Developer Productivity Enhancement

  • Smart Package Discovery: Find the right tool for specific use cases
  • Integration Examples: See how packages work together in practice
  • Best Practice Extraction: Learn from Palantir's engineering excellence
  • Performance Benchmarking: Compare alternatives with data-driven insights
  • Documentation Quality Assessment: Evaluate learning curve and support

🚀 Time & Effort Multiplication

Manual Approach vs. Opentir

Task Manual Effort With Opentir Time Saved
Repository Discovery 2-3 hours browsing GitHub 5 minutes automated 95% faster
Code Analysis 40+ hours per repo × 250 2 hours total 99.8% faster
Documentation Generation 200+ hours writing docs Automated 100% saved
Dependency Mapping 100+ hours manual tracing Instant visualization 100% saved
Statistical Analysis Weeks of data science work Built-in analytics 95% faster
Cross-Reference Building Months of manual linking Automated cross-refs 100% saved

What You Get Instantly:

  • ✅ 500,000+ Functions Cataloged - Searchable database of all capabilities
  • ✅ Comprehensive Metrics - Complexity, quality, and performance data
  • ✅ Integration Roadmaps - How to combine packages effectively
  • ✅ Executive Summaries - Business-level insights for decision makers
  • ✅ Technical Deep Dives - Engineer-level implementation details
  • ✅ Risk Assessments - Maintenance, security, and compliance analysis

🎯 Multi-Layered Value Proposition

Layer 1: Code Intelligence

  • Extract and analyze 500,000+ code elements
  • Calculate complexity metrics across 6.2M+ lines of code
  • Identify architectural patterns and design principles
  • Generate quality scorecards for every repository

Layer 2: Ecosystem Understanding

  • Map inter-project dependencies and relationships
  • Discover functionality overlaps and integration opportunities
  • Analyze technology stack evolution and adoption patterns
  • Identify key maintainers and community health metrics

Layer 3: Strategic Insights

  • Technology Roadmap Planning: Which packages align with your architecture
  • Risk Mitigation: Identify deprecated or poorly maintained projects
  • Investment Prioritization: Focus on high-impact, well-supported tools
  • Team Skills Development: Understand learning paths and complexity curves

Layer 4: Operational Excellence

  • Implementation Playbooks: Step-by-step integration guides
  • Performance Optimization: Benchmarks and tuning recommendations
  • Security Compliance: Vulnerability assessments and best practices
  • Monitoring & Maintenance: Long-term sustainability planning

💡 Unique Analytical Capabilities

🔍 Deep Code Analysis

  • AST Parsing: Beyond regex - true semantic understanding
  • Cyclomatic Complexity: Scientific measurement of code difficulty
  • Documentation Coverage: Quantified maintainability metrics
  • API Surface Analysis: Complete public interface cataloging

📈 Statistical Modeling

  • Regression Analysis: Predict project success and maintenance needs
  • Correlation Studies: Understand relationships between metrics
  • Outlier Detection: Identify exceptional projects for deeper study
  • Trend Analysis: Track ecosystem evolution over time

🌐 Network Analysis

  • Dependency Graphs: Visualize the entire ecosystem's structure
  • Centrality Metrics: Identify the most critical packages
  • Community Detection: Find natural groupings of related projects
  • Path Analysis: Understand technology migration routes

🎖️ Professional Standards

Enterprise-Ready Deliverables

  • Executive Briefings: C-suite appropriate technology assessments
  • Technical Specifications: Detailed integration requirements
  • Risk Registers: Comprehensive risk/benefit analysis
  • ROI Calculations: Quantified business impact projections

Scientific Rigor

  • Peer-Reviewable Methods: Reproducible analysis techniques
  • Statistical Validation: Hypothesis testing and significance analysis
  • Confidence Metrics: Uncertainty quantification throughout
  • Methodology Documentation: Complete analytical transparency

🚀 Getting Started is Just the Beginning

Opentir doesn't just give you code repositories - it gives you:

  1. 🔬 Research-Grade Analysis - Publication-quality insights
  2. 📊 Data-Driven Decisions - Move beyond gut feelings
  3. ⚡ Instant Expertise - Years of ecosystem knowledge in hours
  4. 🎯 Strategic Clarity - Clear technology adoption roadmaps
  5. 💼 Professional Deliverables - Enterprise-ready documentation
  6. 🔄 Continuous Intelligence - Keep analysis current as ecosystem evolves

Transform your approach from ad-hoc exploration to systematic intelligence.

📁 Project Structure

opentir/
├── src/                        # Core Opentir package
│   ├── __init__.py            # Package initialization
│   ├── github_client.py       # GitHub API client with rate limiting
│   ├── repo_manager.py        # Repository cloning and organization
│   ├── code_analyzer.py       # Multi-language code analysis
│   ├── docs_generator.py      # Documentation generation
│   ├── cli.py                 # Command-line interface
│   ├── main.py                # Main orchestrator
│   ├── config.py              # Configuration management
│   ├── utils.py               # Utility functions and logging
│   └── templates/             # Documentation templates
├── repos/                     # Cloned repositories (created during execution)
│   ├── all_repos/            # All Palantir repositories
│   ├── by_language/          # Organized by programming language
│   ├── by_category/          # Organized by functionality
│   └── popular/              # Popular repositories (1000+ stars)
├── docs/                      # Generated documentation
│   ├── index.md              # Main documentation
│   ├── repositories/         # Repository-specific docs
│   ├── api_reference/        # API documentation
│   ├── analysis/             # Analysis reports
│   └── mkdocs.yml           # MkDocs configuration
├── examples/                  # Usage examples
├── tests/                     # Test suite
├── requirements.txt           # Python dependencies
├── setup.py                  # Package configuration
└── README.md                 # This file

🔍 Deep Dive into Palantir's Ecosystem

📋 Complete Package Catalog

View All 250+ Packages

Major Enterprise Platforms

By Package Size & Complexity

🏗️ Architecture Patterns

Palantir Architecture Deep Dive

🎯 Use Case Scenarios

Palantir Solutions by Use Case

  • Enterprise Data Platform

    • Components: Hadoop + Spark + AtlasDB + Foundry
    • Scale: Petabyte-scale data processing

  • Modern Web Applications

    • Components: Blueprint + TypeScript + Conjure APIs
    • Scale: Complex, data-dense interfaces

  • Microservices Architecture

    • Components: Witchcraft + Dialogue + Service mesh
    • Scale: High-throughput distributed systems

  • DevOps & CI/CD

    • Components: gödel + Gradle plugins + Quality tools
    • Scale: Large-scale software development

📊 Technical Analysis

Comprehensive Technical Reports

🚀 Quick Start

For Palantir Ecosystem Exploration

# Quick exploration of Palantir's ecosystem
opentir build-complete

# View ecosystem overview
open docs/palantir/index.md

# Explore specific categories
open docs/palantir/categories/data-analytics.md
open docs/palantir/categories/web-development.md

For Development with Palantir Packages

# Example: Setting up Blueprint development
npm install @blueprintjs/core @blueprintjs/icons
# See: docs/palantir/flagship/blueprint.md

# Example: Using Conjure for APIs
# See: docs/palantir/enterprise/conjure.md

# Example: Setting up data pipeline with Spark
# See: docs/palantir/categories/data-analytics.md

Installation

# Clone the repository
git clone https://github.com/username/opentir.git
cd opentir

# Install dependencies
pip install -r requirements.txt

# Install the package
pip install -e .

GitHub Token Setup (Recommended)

Opentir works with or without a GitHub API token, but a token is highly recommended for downloading all 250+ repositories:

  • Without token: 60 requests/hour (will hit rate limits quickly)
  • With token: 5,000 requests/hour (smooth operation)

Option 1: Interactive Setup (Easiest)

Just run the command - Opentir will guide you through token setup:

opentir build-complete

The tool will:

  1. Detect if no token is available
  2. Show you exactly how to get a GitHub token
  3. Let you paste the token securely
  4. Allow you to skip and continue without a token

Option 2: Environment Variable

export GITHUB_TOKEN=your_github_token_here
opentir build-complete

Option 3: Command Line

opentir build-complete --token your_github_token_here

How to Get a GitHub Token:

  1. Go to: https://github.com/settings/tokens
  2. Click "Generate new token" → "Generate new token (classic)"
  3. Give it a name like opentir-access
  4. Select scope: "public_repo" (read access to public repositories)
  5. Click "Generate token" and copy the token
  6. Use it with any of the options above

Single Command Download

Download all 250+ Palantir repositories with one command:

opentir build-complete

This single command will:

  • ✅ Prompt for GitHub token (if needed)
  • ✅ Download all Palantir repositories
  • ✅ Organize them by language and category
  • ✅ Analyze all code and extract functionality
  • ✅ Generate comprehensive documentation

Basic Usage

# Complete build (recommended - does everything)
opentir build-complete

# Check what's been downloaded
opentir status

# View generated documentation
cd docs && mkdocs serve

Python API

import asyncio
from src.main import build_complete_ecosystem

async def main():
    results = await build_complete_ecosystem(
        github_token="your_token",
        force_reclone=False
    )
    print(f"Analyzed {results['summary']['repositories_analyzed']} repositories!")

asyncio.run(main())

📋 Commands

🚀 Main Command (Recommended)

# Complete workflow - does everything in one command
opentir build-complete

Interactive token setup included - just run and follow prompts!

🔧 Step-by-Step Commands

If you prefer to run individual steps:

# Fetch repository information
opentir fetch-repos

# Clone all repositories
opentir clone-all

# Analyze code and extract functionality
opentir analyze

# Generate comprehensive documentation
opentir generate-docs

All commands will prompt for GitHub token if needed

📊 Utilities

# Show workspace status
opentir status

# Clean up repositories
opentir cleanup --keep-popular

🔑 Token Options

All commands support multiple ways to provide GitHub tokens:

# Interactive prompt (easiest)
opentir build-complete

# Command line argument
opentir build-complete --token YOUR_TOKEN

# Environment variable
export GITHUB_TOKEN=YOUR_TOKEN
opentir build-complete

📊 What You Get

After running the complete workflow, you'll have:

  1. 250+ Cloned Repositories organized by language and category
  2. Comprehensive Code Analysis with extracted methods and functionality
  3. Interactive Documentation with searchable API reference
  4. Functionality Matrix showing capabilities across all repositories
  5. Analysis Reports with metrics, dependencies, and patterns

📦 Specific Palantir Packages Deep Dive

🏆 Top 20 Palantir Repositories by Impact

Complete Repository Rankings

Repository Lines of Code Files Primary Language Key Features
hadoop 2.18M 10,650 Java/Python Distributed storage & processing
cassandra 417K 1,892 Java/Python NoSQL distributed database
atlasdb 378K 3,226 Java Distributed transactional DB
spark 300K 1,538 Java/Python Unified analytics engine
react-native 222K 1,755 JavaScript Cross-platform mobile
blueprint 110K 816 TypeScript React UI toolkit
parquet-mr 122K 768 Java/Python Columnar storage format
conjure-java 88K 636 Java API code generation
plottable 49K 286 TypeScript D3-based charting
gradle-baseline 41K 307 Java Gradle build standards

🔧 Essential Developer Tools

Complete Developer Tools Guide

🌐 Web & Frontend Ecosystem

Web Development Stack Guide

⚙️ Backend & Infrastructure

Backend Services Guide

🔐 Security & Configuration

Security Tools Documentation

🐹 Go Ecosystem

Go Development Tools

  • godel - Build, test, and distribute Go projects
  • distgo - Go application distribution
  • okgo - Go code quality checks
  • pkg - Common Go utilities
  • go-baseapp - Base application framework

🐍 Python Ecosystem

Python Development Tools

🎯 Package Selection Guide

Choose the Right Palantir Packages

For Data Engineering Teams

  • Core: Hadoop + Spark + Parquet + AtlasDB
  • Streaming: Kafka integrations + Spark Streaming
  • Storage: HDFS + Cassandra + Iceberg

For Frontend Development Teams

  • Core: Blueprint + TypeScript + React
  • Visualization: Plottable + D3 integrations
  • Mobile: React Native components

For Backend Development Teams

  • Core: Conjure + Dialogue + Witchcraft
  • Java: Spring integrations + Gradle plugins
  • Go: gödel + Service frameworks

For DevOps & Platform Teams

  • Build: gödel + Gradle baseline + Formatters
  • Security: Encrypted config + Auth tokens
  • Monitoring: Tracing + Metrics + Logging

🎯 Key Capabilities

Repository Management

  • Fetches all Palantir repositories via GitHub API
  • Organizes repositories by language, category, and popularity
  • Handles rate limiting and error recovery
  • Provides cleanup and update functionality

Code Analysis

  • Multi-Language Parsing: Python (AST), JavaScript/TypeScript (regex), Java, Go
  • Element Extraction: Functions, classes, methods, variables
  • Complexity Analysis: Cyclomatic complexity calculation
  • Pattern Recognition: Common naming patterns and functionality categories

Documentation Generation

  • MkDocs Integration: Beautiful, searchable documentation
  • Functionality Tables: Comprehensive tables of all methods and capabilities
  • Cross-References: Links between repositories and functionality
  • Export Formats: JSON, CSV, and Markdown outputs

Performance Features

  • Async Operations: Concurrent API calls and processing
  • Rate Limiting: Respectful GitHub API usage
  • Caching: Intelligent caching of analysis results
  • Progress Tracking: Real-time progress indicators

🛠️ Configuration

Create a .env file:

GITHUB_TOKEN=your_github_token_here
LOG_LEVEL=INFO
ANALYSIS_DEPTH=comprehensive

🩺 Troubleshooting

Rate Limit Issues

If you encounter rate limit errors:

  1. Get a GitHub token (most common solution):

    opentir build-complete  # Will prompt for token

  2. Check your current rate limit:

    curl -H "Authorization: token YOUR_TOKEN" https://api.github.com/rate_limit

  3. Wait and retry (rate limits reset hourly):

    # Rate limits reset every hour
opentir build-complete --token YOUR_TOKEN

Common Issues

"No repositories found"

  • Run opentir status to check workspace state
  • Ensure you have internet connection
  • Try with a GitHub token

"Permission denied"

  • Check your GitHub token has public_repo scope
  • Regenerate token if it's expired

"Analysis failed"

  • Check disk space (repos can be several GB)
  • Run opentir cleanup to free space
  • Run individual steps to isolate issues

📈 Analysis Capabilities

Opentir provides deep insights into Palantir's ecosystem:

  • Language Distribution: Which languages are most used
  • Functionality Patterns: Common patterns across repositories
  • Code Quality Metrics: Complexity, documentation coverage
  • Dependency Analysis: Inter-repository dependencies
  • Activity Metrics: Most active and popular projects

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

📄 License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

📚 Complete Documentation Index

🌐 Palantir Ecosystem Documentation

🔍 By Technology Stack

📊 Technical Analysis

🎯 Usage Guides

🔗 API & Reference

🔗 Related Links

📊 Analysis Results Summary

After complete analysis of Palantir's ecosystem, you'll have access to:

  • 🏛️ 250+ repositories fully analyzed and categorized
  • 📊 6.2M+ lines of code across all programming languages
  • 🔧 500K+ code elements (functions, classes, methods) cataloged
  • 📈 Complexity metrics and quality assessments for each repository
  • 🔗 Dependency mapping showing inter-repository relationships
  • 📚 Interactive documentation with full-text search capabilities
  • 🎯 Usage patterns and integration examples
  • 📋 Functionality matrix showing capabilities across all projects

📋 Quick Reference

Command Purpose Documentation
opentir build-complete 🚀 Complete ecosystem analysis Quick Start
opentir status 📊 Check analysis progress Status Guide
opentir clone-all 📥 Download all repositories Repository Management
opentir analyze 🔍 Code analysis & extraction Analysis Guide
opentir generate-docs 📚 Generate documentation Documentation Guide

Token Setup Options:

  • 🎯 Interactive: opentir build-complete (guided setup)
  • 🔧 Environment: export GITHUB_TOKEN=your_token
  • ⚡ Command line: opentir build-complete --token your_token

🎓 Learning Path

1. Explore the Ecosystem

Start with Palantir Ecosystem Overview

2. Choose Your Focus

3. Deep Dive

Explore specific packages like Blueprint, Conjure, or gödel

4. Implementation

Follow integration examples and best practices

🌟 Built with ❤️ for the open source community and Palantir's incredible ecosystem of 250+ packages

About

OpenTIR — Open-source tool for multi-repo analysis, code visualization, and software research intelligence

Topics

visualization python open-source code-analysis software-analysis multi-repo research-tools

Resources

Readme
Activity

Stars

3 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages