[FEATURE] Track rental lifecycle as part of validation process

[epappas]

[FEATURE] Track rental lifecycle as part of validation process

Problem Statement

The validator needs to monitor and validate rental performance throughout the rental lifecycle, not just during initial executor verification. Currently, validation only happens periodically without considering active rentals. This creates blind spots where:

• Executors might perform well during validation but poorly during rentals
• No feedback loop between rental performance and executor scoring
• Cannot detect issues like container crashes or network problems during rentals
• No way to ensure Quality of Service (QoS) for active rentals

Proposed Solution

Integrate rental lifecycle tracking into the validation system:

1. Active Monitoring - Continuously validate executors with active rentals
2. Performance Metrics - Collect rental-specific performance data
3. Score Adjustment - Update executor scores based on rental performance
4. Issue Detection - Identify and respond to problems in real-time
5. QoS Enforcement - Ensure rentals meet performance requirements

Component

Validator

Priority Level

High

Checklist

Phase 1: Define Rental Validation Metrics

• Performance metrics
  • Container startup time
  • Resource allocation accuracy
  • Network latency and bandwidth
  • GPU utilization and availability
  • Uptime and reliability
• User experience metrics
  • SSH connection success rate
  • Command execution latency
  • File transfer speeds
  • Overall responsiveness
• Compliance metrics
  • Resource limit adherence
  • Security policy compliance
  • Billing accuracy
  • SLA violations

Phase 2: Implement Monitoring Infrastructure

• Create rental monitoring service
  • Track all active rentals
  • Schedule periodic checks
  • Handle monitoring failures
• Extend validation engine
  • Add rental-aware validation
  • Prioritize executors with rentals
  • Adjust validation frequency
• Implement health checks
  • Container health probes
  • Network connectivity tests
  • Resource availability checks

Phase 3: Data Collection Pipeline

• Executor-side collection
  • Container metrics agent
  • System resource monitoring
  • Network traffic analysis
• Validator-side aggregation
  • Receive metrics streams
  • Store time-series data
  • Calculate aggregates
• Real-time processing
  • Stream processing for alerts
  • Anomaly detection
  • Threshold monitoring

Phase 4: Score Integration

• Update scoring algorithm
  • Weight rental performance
  • Consider rental history
  • Penalize failures
• Dynamic score updates
  • Real-time adjustments
  • Sliding window calculations
  • Confidence intervals
• Score recovery
  • Allow improvement over time
  • Gradual penalty decay
  • Second chance policy

Phase 5: Issue Detection and Response

• Define issue types
  • Container crashes
  • Network outages
  • Resource exhaustion
  • Security breaches
• Implement detection
  • Pattern matching
  • Threshold alerts
  • Predictive warnings
• Automated responses
  • Restart containers
  • Migrate rentals
  • Notify users
  • Update scores

Phase 6: QoS Enforcement

• Define SLA levels
  • Uptime guarantees
  • Performance thresholds
  • Response time limits
• Monitor compliance
  • Track SLA metrics
  • Calculate violations
  • Generate reports
• Enforcement actions
  • Automatic remediation
  • Executor penalties
  • User compensation

Phase 7: Reporting and Analytics

• Rental performance reports
  • Per-executor statistics
  • Aggregate network health
  • Trend analysis
• User-facing metrics
  • Rental quality scores
  • Historical performance
  • Availability forecasts
• Operator dashboards
  • Real-time monitoring
  • Alert management
  • Capacity planning

Phase 8: Integration with Weight Setting

• Include rental metrics
  • Factor into weights
  • Prioritize reliable executors
  • Penalize poor performers
• Emission distribution
  • Reward quality service
  • Incentivize availability
  • Balance network load
• Feedback loops
  • Adjust weights frequently
  • Respond to changes
  • Maintain stability

Phase 9: Testing and Validation

• Simulation testing
  • Mock rental scenarios
  • Failure injection
  • Load testing
• Integration testing
  • End-to-end monitoring
  • Score calculation
  • Response automation
• Performance testing
  • Monitoring overhead
  • Data volume handling
  • Scalability limits

Implementation Ideas

Monitoring architecture:

1. Executor runs monitoring agent in each rental container
2. Agent streams metrics to validator every 30 seconds
3. Validator aggregates metrics and updates scores
4. Issues trigger immediate validation and potential migration
5. Historical data used for trend analysis and predictions

Example metrics collection:

struct RentalMetrics {
    container_id: String,
    timestamp: DateTime<Utc>,
    cpu_usage: f64,
    memory_usage: f64,
    gpu_utilization: f64,
    network_rx_bytes: u64,
    network_tx_bytes: u64,
    ssh_sessions: u32,
    errors: Vec<String>,
}

Additional Context

Benefits:

• Better quality of service for users
• More accurate executor scoring
• Proactive issue resolution
• Data-driven network optimization
• Improved user trust

Challenges:

• Monitoring overhead on executors
• Large data volumes to process
• Real-time processing requirements
• Privacy considerations

Related Files

• crates/validator/src/validation/ - Validation engine
• crates/validator/src/metrics/ - Metrics collection
• crates/executor/src/container_manager/ - Container monitoring
• crates/common/src/metrics/ - Metrics traits

Priority

High - Essential for production-quality rental service