Skip to content

Commit 97ab391

Browse files
author
Marvin Zhang
committed
feat(specs): add detailed documentation for gRPC file sync migration and release 0.7.0
- Introduced README.md for the file sync issue after gRPC migration, outlining the problem, root cause, and proposed solutions. - Added release notes for Crawlab 0.7.0 highlighting community features and improvements. - Created a README.md for the specs directory to provide an overview and usage instructions for LeanSpec.
1 parent 18c5eb3 commit 97ab391

14 files changed

Lines changed: 5564 additions & 0 deletions

File tree

‎specs/002-context-patterns/README.md‎

Lines changed: 445 additions & 0 deletions
Large diffs are not rendered by default.

‎specs/003-task-system/20250930-task-worker-configuration.md‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# Crawlab Task Worker Configuration Examples
2+
3+
## Basic Configuration
4+
5+
### config.yml
6+
```yaml
7+
# Task execution configuration
8+
task:
9+
workers: 20 # Number of concurrent task workers (default: 10)
10+
11+
# Node configuration (optional)
12+
node:
13+
maxRunners: 50 # Maximum total tasks per node (0 = unlimited)
14+
```
15+
16+
### Environment Variables
17+
```bash
18+
# Set via environment variables
19+
export CRAWLAB_TASK_WORKERS=20
20+
export CRAWLAB_NODE_MAXRUNNERS=50
21+
```
22+
23+
## Configuration Guidelines
24+
25+
### Worker Count Recommendations
26+
27+
| Scenario | Task Workers | Queue Size | Memory Usage |
28+
|----------|-------------|------------|--------------|
29+
| Development | 5-10 | 25-50 | ~100MB |
30+
| Small Production | 15-20 | 75-100 | ~200MB |
31+
| Medium Production | 25-35 | 125-175 | ~400MB |
32+
| Large Production | 40-60 | 200-300 | ~800MB |
33+
34+
### Factors to Consider
35+
36+
1. **Task Complexity**: CPU/Memory intensive tasks need fewer workers
37+
2. **Task Duration**: Long-running tasks need more workers for throughput
38+
3. **System Resources**: Balance workers with available CPU/Memory
39+
4. **Database Load**: More workers = more database connections
40+
5. **External Dependencies**: Network-bound tasks can handle more workers
41+
42+
### Performance Tuning
43+
44+
#### Too Few Workers (Queue Full Errors)
45+
```log
46+
WARN task queue is full (50/50), consider increasing task.workers configuration
47+
```
48+
**Solution**: Increase `task.workers` value
49+
50+
#### Too Many Workers (Resource Exhaustion)
51+
```log
52+
ERROR failed to create task runner: out of memory
53+
ERROR database connection pool exhausted
54+
```
55+
**Solution**: Decrease `task.workers` value
56+
57+
#### Optimal Configuration
58+
```log
59+
INFO Task handler service started with 20 workers and queue size 100
60+
DEBUG task[abc123] queued, queue usage: 5/100
61+
```
62+
63+
## Docker Configuration
64+
65+
### docker-compose.yml
66+
```yaml
67+
version: '3'
68+
services:
69+
crawlab-master:
70+
image: crawlab/crawlab:latest
71+
environment:
72+
- CRAWLAB_TASK_WORKERS=25
73+
- CRAWLAB_NODE_MAXRUNNERS=100
74+
# ... other config
75+
76+
crawlab-worker:
77+
image: crawlab/crawlab:latest
78+
environment:
79+
- CRAWLAB_TASK_WORKERS=30 # Workers can be different per node
80+
- CRAWLAB_NODE_MAXRUNNERS=150
81+
# ... other config
82+
```
83+
84+
### Kubernetes ConfigMap
85+
```yaml
86+
apiVersion: v1
87+
kind: ConfigMap
88+
metadata:
89+
name: crawlab-config
90+
data:
91+
config.yml: |
92+
task:
93+
workers: 25
94+
node:
95+
maxRunners: 100
96+
```
97+
98+
## Monitoring Worker Performance
99+
100+
### Log Monitoring
101+
```bash
102+
# Monitor worker pool status
103+
grep -E "(workers|queue usage|queue is full)" /var/log/crawlab/crawlab.log
104+
105+
# Monitor task throughput
106+
grep -E "(task.*queued|task.*finished)" /var/log/crawlab/crawlab.log | wc -l
107+
```
108+
109+
### Metrics to Track
110+
- Queue utilization percentage
111+
- Average task execution time
112+
- Worker pool saturation
113+
- Memory usage per worker
114+
- Task success/failure rates
115+
116+
## Troubleshooting
117+
118+
### Queue Always Full
119+
1. Increase worker count: `task.workers`
120+
2. Check task complexity and optimization
121+
3. Verify database performance
122+
4. Consider scaling horizontally (more nodes)
123+
124+
### High Memory Usage
125+
1. Decrease worker count
126+
2. Optimize task memory usage
127+
3. Implement task batching
128+
4. Add memory monitoring alerts
129+
130+
### Slow Task Processing
131+
1. Profile individual tasks
132+
2. Check database query performance
133+
3. Optimize external API calls
134+
4. Consider async task patterns
135+
136+
## Testing Configuration Changes
137+
138+
```bash
139+
# Test new configuration
140+
export CRAWLAB_TASK_WORKERS=30
141+
./scripts/test_goroutine_fixes.sh 900 10
142+
143+
# Monitor during peak load
144+
./scripts/test_goroutine_fixes.sh 3600 5
145+
```
146+
147+
## Best Practices
148+
149+
1. **Start Conservative**: Begin with default values and monitor
150+
2. **Load Test**: Always test configuration changes under load
151+
3. **Monitor Metrics**: Track queue utilization and task throughput
152+
4. **Scale Gradually**: Increase worker count in small increments
153+
5. **Resource Limits**: Set appropriate memory/CPU limits in containers
154+
6. **High Availability**: Configure different worker counts per node type

‎specs/003-task-system/README.md‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
---
2+
status: complete
3+
created: 2025-09-30
4+
tags: [task-system]
5+
priority: medium
6+
---
7+
8+
# Task Reconciliation Improvements
9+
10+
## Overview
11+
12+
This document describes the improvements made to task reconciliation in Crawlab to handle node disconnection scenarios more reliably by leveraging worker-side status caching.
13+
14+
## Problem Statement
15+
16+
Previously, the task reconciliation system was heavily dependent on the master node to infer task status during disconnections using heuristics. This approach had several limitations:
17+
18+
1. **Fragile heuristics**: Status inference based on stream presence and timing could be incorrect
19+
2. **Master node dependency**: Worker nodes couldn't maintain authoritative task status during disconnections
20+
3. **Status inconsistency**: Risk of status mismatches between actual process state and database records
21+
4. **Poor handling of long-running tasks**: Network issues could cause incorrect status assumptions
22+
23+
## Solution: Worker-Side Status Caching
24+
25+
### Key Components
26+
27+
#### 1. TaskStatusSnapshot Structure
28+
```go
29+
type TaskStatusSnapshot struct {
30+
TaskId primitive.ObjectID `json:"task_id"`
31+
Status string `json:"status"`
32+
Error string `json:"error,omitempty"`
33+
Pid int `json:"pid,omitempty"`
34+
Timestamp time.Time `json:"timestamp"`
35+
StartedAt *time.Time `json:"started_at,omitempty"`
36+
EndedAt *time.Time `json:"ended_at,omitempty"`
37+
}
38+
```
39+
40+
#### 2. TaskStatusCache
41+
- **Local persistence**: Status cache survives worker node disconnections
42+
- **File-based storage**: Cached status persists across process restarts
43+
- **Automatic cleanup**: Cache files are cleaned up when tasks complete
44+
45+
#### 3. Enhanced Runner (`runner_status_cache.go`)
46+
- **Status caching**: Every status change is cached locally first
47+
- **Pending updates**: Status changes queue for sync when reconnected
48+
- **Persistence layer**: Status cache is saved to disk asynchronously
49+
50+
### Workflow Improvements
51+
52+
#### During Normal Operation
53+
1. Task status changes are cached locally on worker nodes
54+
2. Status is immediately sent to master node/database
55+
3. If database update fails, status remains cached for later sync
56+
57+
#### During Disconnection
58+
1. Worker node continues tracking actual task/process status locally
59+
2. Status changes accumulate in pending updates queue
60+
3. Task continues running with authoritative local status
61+
62+
#### During Reconnection
63+
1. Worker triggers sync of all pending status updates
64+
2. TaskReconciliationService prioritizes worker cache over heuristics
65+
3. Database is updated with authoritative worker-side status
66+
67+
### Enhanced TaskReconciliationService
68+
69+
#### Priority Order for Status Resolution
70+
1. **Worker-side status cache** (highest priority)
71+
2. **Direct process status query**
72+
3. **Heuristic detection** (fallback only)
73+
74+
#### New Methods
75+
- `getStatusFromWorkerCache()`: Retrieves cached status from worker
76+
- `triggerWorkerStatusSync()`: Triggers sync of pending updates
77+
- Enhanced `HandleNodeReconnection()`: Leverages worker cache
78+
79+
## Benefits
80+
81+
### 1. Improved Reliability
82+
- **Authoritative status**: Worker nodes maintain definitive task status
83+
- **Reduced guesswork**: Less reliance on potentially incorrect heuristics
84+
- **Better consistency**: Database reflects actual process state
85+
86+
### 2. Enhanced Resilience
87+
- **Disconnection tolerance**: Tasks continue with accurate status tracking
88+
- **Automatic recovery**: Status sync happens automatically on reconnection
89+
- **Data persistence**: Status cache survives process restarts
90+
91+
### 3. Better Performance
92+
- **Reduced master load**: Less dependency on master node for status inference
93+
- **Faster reconciliation**: Direct access to cached status vs. complex heuristics
94+
- **Fewer database inconsistencies**: More accurate status updates
95+
96+
## Implementation Details
97+
98+
### File Structure
99+
```
100+
core/task/handler/
101+
├── runner.go # Main task runner
102+
├── runner_status_cache.go # Status caching functionality
103+
└── service_operations.go # Service methods for runner access
104+
105+
core/node/service/
106+
└── task_reconciliation_service.go # Enhanced reconciliation logic
107+
```
108+
109+
### Configuration
110+
- Cache directory: `{workspace}/.crawlab/task_cache/`
111+
- Cache file pattern: `task_{taskId}.json`
112+
- Sync trigger: Automatic on reconnection
113+
114+
### Error Handling
115+
- **Cache failures**: Logged but don't block task execution
116+
- **Sync failures**: Failed updates re-queued for retry
117+
- **Type mismatches**: Graceful fallback to heuristics
118+
119+
## Usage
120+
121+
### For Workers
122+
Status caching is automatic and transparent. No configuration required.
123+
124+
### For Master Nodes
125+
The reconciliation service automatically detects worker-side cache availability and uses it when possible.
126+
127+
### Monitoring
128+
- Log messages indicate when cached status is used
129+
- Failed sync attempts are logged with retry information
130+
- Cache cleanup is logged for debugging
131+
132+
## Future Enhancements
133+
134+
1. **Batch sync optimization**: Group multiple status updates for efficiency
135+
2. **Compression**: Compress cache files for large deployments
136+
3. **TTL support**: Automatic cache expiration for very old tasks
137+
4. **Metrics**: Expose cache hit/miss rates for monitoring
138+
139+
## Migration
140+
141+
This is a backward-compatible enhancement. Existing deployments will:
142+
1. Gradually benefit from improved reconciliation
143+
2. Fall back to existing heuristics when cache unavailable
144+
3. Require no configuration changes

0 commit comments

Comments
 (0)