Skip to content

Commit f646642

Browse files
committed
docs: Move architecture and development to docs/
1 parent ca8bade commit f646642

2 files changed

Lines changed: 31 additions & 4 deletions

File tree

ARCHITECTURE.md renamed to docs/architecture.md

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,88 +11,104 @@ Kubin uses a microservices architecture optimized for high-throughput uploads an
1111
## Core Services
1212

1313
### API Gateway
14+
1415
- Single entry point with TLS, auth, rate limiting
1516
- Routes traffic to appropriate services
1617
- 100 concurrent uploads, 1000 queries/min limits
1718

18-
### Upload Orchestrator
19+
### Upload Orchestrator
20+
1921
- Coordinates multi-step upload workflow
2022
- Generates snapshot IDs and immediate shareable URLs
2123
- Manages S3 pre-signed URLs and async processing
2224
- States: `INITIATED``UPLOADING``PROCESSING``COMPLETED`
2325

2426
### Metadata Service
27+
2528
- Fast PostgreSQL storage for K8s resource metadata
2629
- Optimized for listing, filtering, and search operations
2730
- Handles user management and sharing permissions
2831

2932
### Storage Service
33+
3034
- Manages S3 file operations and organization
3135
- Generates pre-signed URLs for direct uploads/downloads
3236
- Implements lifecycle policies and multipart uploads
3337

3438
### Log Processor
39+
3540
- Background parsing of raw logs into structured data
3641
- Batch inserts to ClickHouse for analytics
3742
- Handles various log formats with parallel processing
3843

3944
### Query Service
45+
4046
- Orchestrates data from multiple services
4147
- Parallel requests with intelligent caching (Redis)
4248
- Provides unified responses for web UI
4349

4450
### Analytics Service
51+
4552
- Advanced log search and filtering on ClickHouse
4653
- Time-series analysis and aggregations
4754
- Materialized views for common queries
4855

4956
## Data Storage
5057

5158
**PostgreSQL**: User data, snapshot metadata, K8s resources
59+
5260
- ACID transactions for immediate consistency
5361
- Optimized indexes for fast queries
5462
- Handles user auth and sharing permissions
5563

56-
**ClickHouse**: Log data and analytics
64+
**ClickHouse**: Log data and analytics
65+
5766
- Columnar storage for fast aggregations
5867
- Partitioned by snapshot_id and time
5968
- Materialized views for common patterns
6069

6170
**S3**: Raw files and YAML resources
71+
6272
- Direct upload/download via pre-signed URLs
6373
- Organized hierarchy: `snapshots/{id}/{type}/...`
6474
- Lifecycle policies for cost optimization
6575

6676
**Redis**: Multi-layer caching
77+
6778
- Request cache (15min), component cache (5min)
6879
- Search cache (1hr), CDN cache (global)
6980

7081
## Communication Patterns
7182

7283
**Synchronous (HTTP/gRPC)**:
84+
7385
- CLI ↔ Upload Orchestrator (immediate response)
7486
- Web UI ↔ Query Service (real-time interaction)
7587
- Inter-service calls for request fulfillment
7688

7789
**Asynchronous (Kafka)**:
90+
7891
- S3 upload completion events
7992
- Background processing coordination
8093
- Cross-service event propagation
8194

8295
## Upload Flow
8396

8497
### Phase 1: Immediate Response (< 2s)
98+
8599
1. CLI sends cluster metadata to Upload Orchestrator
86-
2. Generate snapshot ID, create PostgreSQL record
100+
2. Generate snapshot ID, create PostgreSQL record
87101
3. Request S3 pre-signed URLs from Storage Service
88102
4. Return shareable URL and upload URLs to CLI
89103

90104
### Phase 2: Parallel Upload (5-120s)
105+
91106
1. CLI uploads files directly to S3 using pre-signed URLs
92107
2. Multipart uploads for large files, progress reporting
93108
3. S3 events trigger Kafka notifications
94109

95110
### Phase 3: Background Processing (30-300s)
111+
96112
1. Log Processor parses and indexes logs
97113
2. Metadata status updates: `uploading``processing``completed`
98114
3. Query Service cache warming

DEVELOPMENT.md renamed to docs/development.md

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ The development environment includes:
2828
## Makefile Commands
2929

3030
### Environment Management
31+
3132
```bash
3233
make dev # Start all services
3334
make dev-down # Stop all services
@@ -37,6 +38,7 @@ make dev-status # Check service status
3738
```
3839

3940
### Database Management
41+
4042
```bash
4143
make db # Ensure database is running
4244
make db-reset # Reset database
@@ -45,19 +47,22 @@ make db-restore # Restore from backup
4547
```
4648

4749
### Storage Management
50+
4851
```bash
4952
make storage # Ensure storage is running
5053
make storage-reset # Reset MinIO storage
5154
make storage-ls # List storage contents
5255
```
5356

5457
### Development Tools
58+
5559
```bash
5660
make tools # Start pgAdmin
5761
make tools-down # Stop tools
5862
```
5963

6064
### Component Development
65+
6166
```bash
6267
# Server
6368
make server-run # Run server
@@ -75,6 +80,7 @@ make ui-test # Test UI
7580
```
7681

7782
### Full Development Setup
83+
7884
```bash
7985
make full-dev # Complete setup
8086
```
@@ -140,22 +146,27 @@ After starting the environment:
140146
## Troubleshooting
141147

142148
### Reset Everything
149+
143150
```bash
144151
make clean # Stop and remove everything
145152
make full-dev # Start fresh
146153
```
147154

148155
### Database Issues
156+
149157
```bash
150158
make db-reset # Reset database
151159
```
152160

153161
### Storage Issues
162+
154163
```bash
155164
make storage-reset # Reset MinIO
156165
```
157166

158167
### Service Status
168+
159169
```bash
160170
make dev-status # Check all services
161-
```
171+
```
172+

0 commit comments

Comments
 (0)