Skip to content

ROADMAP.md

Latest commit

 

History

History
199 lines (142 loc) · 6.27 KB

ROADMAP.md

File metadata and controls

199 lines (142 loc) · 6.27 KB

SOLLOL Roadmap

Vision: Make distributed AI compute accessible to independent engineers — bridging the gap between one-machine demos and real cluster orchestration without cloud dependency.

Current Status: v0.9.x (Alpha)

Production-Stable Features:

  • ✅ Intelligent task distribution across Ollama nodes
  • ✅ Auto-discovery and failover
  • ✅ Real-time observability dashboard
  • ✅ GPU/CPU resource-aware routing
  • ✅ Priority-based scheduling
  • ✅ Response caching and HTTP/2 support
  • ✅ Batch processing API

Experimental Features:

  • 🔬 Distributed inference via llama.cpp RPC (proof-of-concept)

v1.0 — Production Hardening (Q1 2025)

Goal: Production-ready for teams running multi-node AI infrastructure

Core Stability

  • Performance validation - Independent multi-node benchmarks
  • Error handling - Comprehensive retry logic and circuit breakers
  • Production deployment - Docker Compose + Kubernetes manifests
  • Security hardening - TLS/SSL, API authentication, rate limiting
  • Monitoring - Grafana dashboards, alerting rules

Developer Experience

  • Python SDK improvements - Better error messages, type stubs
  • CLI enhancements - Interactive setup wizard, node management
  • Documentation - Video tutorials, deployment playbooks
  • Examples - LangChain, CrewAI, AutoGPT integrations

API Stability

  • API versioning - Backwards compatibility guarantees
  • Schema validation - OpenAPI 3.1 specs
  • Migration guide - Upgrade path from v0.9.x

Release Target: March 2025

v1.x — Feature Expansion (Q2-Q3 2025)

Enhanced Intelligence

  • ML-based routing - Learn optimal node selection from historical patterns
  • Cost-aware routing - Consider energy/cloud costs in decisions
  • Predictive scaling - Auto-scale based on queue depth and patterns
  • A/B testing framework - Compare routing strategies in production

Multi-Cloud & Hybrid

  • Cloud provider integrations - AWS Bedrock, Azure OpenAI fallback
  • Geographic routing - Latency-aware multi-region support
  • Hybrid deployments - Mix local + cloud seamlessly

Developer Tools

  • VSCode extension - Monitor cluster from IDE
  • Jupyter integration - Notebook-native cluster management
  • Webhooks - Event notifications for failures, scaling events

v2.0 — Advanced Distributed Compute (Q4 2025+)

True Model Weight Sharding

Current limitation: llama.cpp coordinator requires full model in RAM

v2.0 Goal: Run 70B+ models with NO single node needing full model

Research Track (Dedicated Engineering Effort Required):

  • GGUF tensor-level distribution - Split model weights across nodes
  • Ray-based pipeline parallelism - Activation passing via object store
  • Quantization-aware sharding - Smart layer distribution
  • Production validation - 70B-405B models on consumer hardware

Impact: Enables sovereign AI deployment at scale without cloud dependency

Advanced Features

  • Model fine-tuning pipeline - Distributed training workflows
  • Multi-tenancy - Isolated workspaces with quotas
  • GraphQL API - Flexible query interface
  • WebSockets - Real-time streaming and bidirectional communication

Research Tracks

1. Distributed Inference Optimization (High Priority)

Status: Experimental (5x slower than local, manual setup)

Path to Production:

  • Reduce startup time: 2-5min → <30s
  • Improve throughput: 5x slower → 2x slower
  • Automated version management and binary compatibility
  • Comprehensive testing across model sizes (13B-70B+)

Blocker: Requires dedicated cluster access and optimization time

2. Zero-Config Deployment

Goal: pip install sollol && sollol start → instant cluster

Requirements:

  • Auto-install Ollama on discovered nodes
  • Intelligent model distribution (which models go where)
  • Self-healing configuration
  • One-click cloud deployment (AWS/GCP/Azure)

3. Agent Marketplace

Vision: Share and discover SOLLOL-optimized agent configurations

Features:

  • Pre-configured routing strategies for common use cases
  • Tested multi-agent orchestrations (research, coding, analysis)
  • Community ratings and benchmarks
  • One-click deployment

Community Priorities

Vote on features via GitHub Discussions

Most Requested:

  1. LangChain native integration
  2. Model fine-tuning support
  3. Cost tracking per request
  4. Slack/Discord notifications
  5. Windows native support

Contributing to the Roadmap

Want to influence direction?

Versioning Strategy

Semantic Versioning (SemVer):

  • Major (v1.0, v2.0): Breaking API changes
  • Minor (v1.1, v1.2): New features, backwards compatible
  • Patch (v1.0.1, v1.0.2): Bug fixes only

Release Cycle:

  • Patch releases: Weekly (as needed for critical bugs)
  • Minor releases: Monthly (new features)
  • Major releases: Quarterly to annually (breaking changes)

Success Metrics

v1.0 Success Criteria:

  • 500+ GitHub stars
  • 10+ production deployments (documented case studies)
  • <1% critical bug rate
  • 90%+ test coverage
  • Sub-50ms routing overhead

v2.0 Success Criteria:

  • Enable 70B models on 3×16GB consumer GPU clusters
  • 1000+ GitHub stars
  • 50+ production deployments
  • Official integrations with 3+ popular frameworks

Long-Term Vision (2026+)

SOLLOL becomes the standard for:

  • Independent AI researchers running frontier models
  • Small teams building sovereign AI applications
  • Universities conducting distributed AI research
  • Hobbyists experimenting with large models

Impact:

"Any engineer with 3 consumer GPUs can run models that previously required enterprise infrastructure."

Last Updated: November 2025 Current Version: v0.9.65 Next Milestone: v1.0 (March 2025)

For detailed technical architecture and current implementation status, see ARCHITECTURE.md.