[FEATURE] Split validator config into network and service configurations

[epappas]

[FEATURE] Split validator config into network and service configurations

Problem Statement

The validator configuration at crates/validator/src/config/ (example: config/validator.correct.toml) combines both network-wide parameters and instance-specific settings in a single file. This creates problems:

• Network parameters that must be identical across all validators are mixed with personal settings
• Risk of misconfiguration when validators use different network parameters
• Difficult to distribute standard network configs
• Hard to manage instance-specific settings separately
• No clear separation between what's shared and what's unique

Proposed Solution

Split the configuration into two distinct types:

1. Network Config (--network flag) - Static, shared across all network participants

   • Built into the binary or loaded from a standard location
   • Contains parameters that MUST be the same for all validators
   • Version controlled and distributed with releases

2. Service Config (--service flag) - Dynamic, unique to each instance

   • Contains instance-specific settings like IPs, ports, paths
   • Managed by individual operators
   • Not shared between instances

Component

Validator

Priority Level

High

Checklist

Phase 1: Define Configuration Split

• Identify network-wide parameters
  • Bittensor network settings (netuid, chain endpoint)
  • Verification intervals and timeouts
  • Scoring weights and thresholds
  • Emission percentages and allocations
  • Protocol version requirements
• Identify service-specific parameters
  • Server host and port bindings
  • Database connection strings
  • File paths and directories
  • API keys and authentication
  • Local resource limits
  • Logging configuration

Phase 2: Create New Config Structures

• Design network config schema

  # network.toml - Shared across all validators
  [network]
  name = "finney"
  netuid = 39
  chain_endpoint = "wss://entrypoint-finney.opentensor.ai:443"
  
  [protocol]
  version = "1.0.0"
  min_compatible = "0.9.0"
  
  [verification]
  interval_secs = 600
  challenge_timeout_secs = 120
  min_score_threshold = 0.1
  
  [emission]
  burn_percentage = 95.0
  gpu_allocations = { H100 = 60.0, H200 = 40.0 }

• Design service config schema

  # service.toml - Unique per validator
  [server]
  host = "0.0.0.0"
  port = 8080
  advertised_host = "192.168.1.100"
  
  [database]
  url = "sqlite:/opt/basilica/data/validator.db"
  
  [paths]
  data_dir = "/opt/basilica/data"
  ssh_keys = "/opt/basilica/keys"

Phase 3: Implement Config Loading

• Create NetworkConfig struct
  • Define all network parameters
  • Add validation for required fields
  • Implement version checking
• Create ServiceConfig struct
  • Define all service parameters
  • Add defaults for optional fields
  • Implement path resolution
• Update ValidatorConfig
  • Compose from NetworkConfig + ServiceConfig
  • Maintain backwards compatibility
  • Add migration logic

Phase 4: Update Config Loading Logic

• Modify CLI arguments
  • Add --network flag for network config
  • Add --service flag for service config
  • Keep --config for backwards compatibility
  • Add deprecation warnings
• Implement config resolution
  • Load network config from binary/file
  • Load service config from file
  • Merge configurations properly
  • Validate combined config
• Add config commands
  • validator config generate-service
  • validator config show-network
  • validator config validate
  • validator config migrate

Phase 5: Build Process Integration

• Embed network configs in binary
  • Create configs directory in crate
  • Use include_str! for embedding
  • Support multiple networks
• Update build scripts
  • Copy network configs during build
  • Generate config documentation
  • Validate configs at build time
• Create distribution mechanism
  • Package network configs with releases
  • Version network configs properly
  • Document upgrade process

Phase 6: Migration and Documentation

• Create migration tool
  • Split existing configs automatically
  • Preserve all settings
  • Generate both config files
• Update documentation
  • Document new config structure
  • Provide migration guide
  • Update deployment docs
• Update examples
  • Provide sample configs
  • Show common patterns
  • Include troubleshooting

Implementation Ideas

Config loading precedence:

1. Built-in network config (for known networks)
2. Custom network config file (for testing)
3. Service config file (required)
4. Environment variables (overrides)
5. Command line arguments (highest priority)

Additional Context

This change improves:

• Consistency: All validators use identical network parameters
• Safety: Reduces misconfiguration risks
• Deployment: Easier to distribute standard configs
• Maintenance: Clear separation of concerns
• Upgrades: Network params can be updated with binary

Current pain points:

• Validators with different emission allocations
• Mismatched verification intervals
• Inconsistent scoring weights
• Manual config distribution

Related Files

• crates/validator/src/config/main_config.rs
• crates/validator/src/config/emission.rs
• config/validator.correct.toml
• Build scripts in scripts/validator/

Priority

High - Critical for network consistency and operational safety