@@ -11,88 +11,104 @@ Kubin uses a microservices architecture optimized for high-throughput uploads an
1111## Core Services
1212
1313### API Gateway
14+
1415- Single entry point with TLS, auth, rate limiting
1516- Routes traffic to appropriate services
1617- 100 concurrent uploads, 1000 queries/min limits
1718
18- ### Upload Orchestrator
19+ ### Upload Orchestrator
20+
1921- Coordinates multi-step upload workflow
2022- Generates snapshot IDs and immediate shareable URLs
2123- Manages S3 pre-signed URLs and async processing
2224- States: ` INITIATED ` → ` UPLOADING ` → ` PROCESSING ` → ` COMPLETED `
2325
2426### Metadata Service
27+
2528- Fast PostgreSQL storage for K8s resource metadata
2629- Optimized for listing, filtering, and search operations
2730- Handles user management and sharing permissions
2831
2932### Storage Service
33+
3034- Manages S3 file operations and organization
3135- Generates pre-signed URLs for direct uploads/downloads
3236- Implements lifecycle policies and multipart uploads
3337
3438### Log Processor
39+
3540- Background parsing of raw logs into structured data
3641- Batch inserts to ClickHouse for analytics
3742- Handles various log formats with parallel processing
3843
3944### Query Service
45+
4046- Orchestrates data from multiple services
4147- Parallel requests with intelligent caching (Redis)
4248- Provides unified responses for web UI
4349
4450### Analytics Service
51+
4552- Advanced log search and filtering on ClickHouse
4653- Time-series analysis and aggregations
4754- Materialized views for common queries
4855
4956## Data Storage
5057
5158** PostgreSQL** : User data, snapshot metadata, K8s resources
59+
5260- ACID transactions for immediate consistency
5361- Optimized indexes for fast queries
5462- Handles user auth and sharing permissions
5563
56- ** ClickHouse** : Log data and analytics
64+ ** ClickHouse** : Log data and analytics
65+
5766- Columnar storage for fast aggregations
5867- Partitioned by snapshot_id and time
5968- Materialized views for common patterns
6069
6170** S3** : Raw files and YAML resources
71+
6272- Direct upload/download via pre-signed URLs
6373- Organized hierarchy: ` snapshots/{id}/{type}/... `
6474- Lifecycle policies for cost optimization
6575
6676** Redis** : Multi-layer caching
77+
6778- Request cache (15min), component cache (5min)
6879- Search cache (1hr), CDN cache (global)
6980
7081## Communication Patterns
7182
7283** Synchronous (HTTP/gRPC)** :
84+
7385- CLI ↔ Upload Orchestrator (immediate response)
7486- Web UI ↔ Query Service (real-time interaction)
7587- Inter-service calls for request fulfillment
7688
7789** Asynchronous (Kafka)** :
90+
7891- S3 upload completion events
7992- Background processing coordination
8093- Cross-service event propagation
8194
8295## Upload Flow
8396
8497### Phase 1: Immediate Response (< 2s)
98+
85991 . CLI sends cluster metadata to Upload Orchestrator
86- 2 . Generate snapshot ID, create PostgreSQL record
100+ 2 . Generate snapshot ID, create PostgreSQL record
871013 . Request S3 pre-signed URLs from Storage Service
881024 . Return shareable URL and upload URLs to CLI
89103
90104### Phase 2: Parallel Upload (5-120s)
105+
911061 . CLI uploads files directly to S3 using pre-signed URLs
921072 . Multipart uploads for large files, progress reporting
931083 . S3 events trigger Kafka notifications
94109
95110### Phase 3: Background Processing (30-300s)
111+
961121 . Log Processor parses and indexes logs
971132 . Metadata status updates: ` uploading ` → ` processing ` → ` completed `
981143 . Query Service cache warming
0 commit comments