Skip to content

DUAL_CONFIG_SYSTEM.md

Latest commit

 

History

History
158 lines (111 loc) · 4.64 KB

DUAL_CONFIG_SYSTEM.md

File metadata and controls

158 lines (111 loc) · 4.64 KB
Raw

Dual-Layer Configuration System

Design Philosophy

The SALF project adopts a dual-layer configuration system that balances flexibility and ease of use:

  1. Python Configuration Layer (config/settings.py) - Provides defaults, suitable for quick development
  2. Environment Variable Layer (config.sh) - Overrides defaults, suitable for batch processing

Why Two Configuration Layers?

Problem

  • During development, want to run quickly without setting environment variables every time
  • In production, want centralized configuration management to avoid hardcoding

Solution

Two configuration layers, each serving its purpose:

Development Scenario: Python script → Directly uses config/settings.py defaults
Production Scenario: Bash script → Loads config.sh → Overrides defaults

Use Case Comparison

Scenario Recommended Method Reason
Quick test of single script Python config No need to set environment variables
Batch processing large data Bash config Centralized management, bash scripts auto-load
Development debugging Python config Easy to modify, takes effect immediately
Production environment Bash config Unified configuration, easy to maintain
Temporarily use different model Command-line args Flexible override

Configuration Priority Examples

Example 1: Using Only Python Defaults

# Don't set any environment variables
python run_generation.py --input_file data/sample.json --output_file results/gen.json

# Uses defaults from config/settings.py:
# - DEFAULT_LLM_MODEL = "gpt-4o-mini-2024-07-18"
# - MAX_ITERATIONS = 3

Example 2: Override with Environment Variables

# Load config.sh
source config.sh

# Run script
python run_generation.py --input_file data/sample.json --output_file results/gen.json

# Uses environment variables from config.sh (overrides Python defaults):
# - DEFAULT_LLM_MODEL = environment variable value
# - MAX_ITERATIONS = environment variable value

Example 3: Override with Command-line Arguments

# Temporarily use different model
python run_generation.py \
    --input_file data/sample.json \
    --output_file results/gen.json \
    --llm_model deepseek-chat \
    --max_iterations 5

# Command-line arguments have highest priority, override all configurations

Best Practices

Development Phase

  1. Modify config/settings.py to set your API key
  2. Run Python scripts directly for testing
  3. Rapid iteration development
# Edit configuration
vim config/settings.py

# Run directly
python run_generation.py --input_file data/sample.json --output_file results/gen.json --max_samples 1

Production Phase

  1. Create config.sh (or copy template)
  2. Set production environment configuration
  3. Use bash scripts for batch processing
# Edit production configuration
vim config.sh

# Run batch processing
bash scripts/run_full_pipeline.sh

Multi-Environment Management

Create different configuration files:

# Test environment
cp config.sh config_test.sh
vim config_test.sh  # Set test API key and parameters

# Production environment
cp config.sh config_prod.sh
vim config_prod.sh  # Set production API key and parameters

# Use different environments
source config_test.sh && bash scripts/run_test.sh
source config_prod.sh && bash scripts/run_full_pipeline.sh

FAQ

Q: Why two configuration files?

A: To accommodate different use cases:

  • config/settings.py: Python defaults, suitable for quick development
  • config.sh: Environment variable configuration, suitable for batch processing

Q: Which one should I use?

A:

  • If only occasionally running Python scripts → Modify config/settings.py
  • If batch processing or using bash scripts → Use config.sh

Q: Will the two configuration files conflict?

A: No. Environment variables (config.sh) have higher priority and will override Python defaults.

Q: Can I use only one?

A: Yes, but you'll lose flexibility:

  • Only Python config → Bash scripts need to manually set environment variables
  • Only Bash config → Python scripts need to source config.sh every time

Summary

Advantages of the dual-layer configuration system:

✅ Flexibility: Three-tier priority, adapts to different scenarios ✅ Ease of Use: No need to set environment variables during development ✅ Maintainability: Centralized configuration management in production ✅ Backward Compatibility: Doesn't break existing usage patterns

This design allows you to:

  • Run directly during quick development
  • Unified configuration for batch processing
  • Flexible override for temporary adjustments