BATCH_SCRIPTS_SUMMARY.md

Batch Scripts Summary

New Files

Based on user requirements, we have added a decoupled batch processing script system:

Python Scripts

1. run_generation.py (8.6KB)

   • Independent batch generation script
   • Saves after processing each item
   • Supports checkpoint recovery
   • Automatic error handling

2. run_detection.py (8.3KB)

   • Independent batch detection script
   • Saves after processing each item
   • Automatic retry mechanism (default 3 times)
   • Supports using original or optimized content

Bash Scripts

Located in scripts/ directory:

1. run_test.sh - Quick test
2. run_batch_generation.sh - Batch generation
3. run_batch_detection.sh - Batch detection
4. run_full_pipeline.sh - Full pipeline

Documentation

• BATCH_PROCESSING.md - Detailed usage guide

---

Core Advantages

1. Decoupled Design

Old Method (main.py):

Generate → Detect → Optimize → Generate → Detect → ...
(Saves only after completing all steps for one item)

New Method (batch processing scripts):

Batch Generation:
  Sample 1 → Optimize → Save
  Sample 2 → Optimize → Save
  ...

Batch Detection:
  Sample 1 → Debate → Save
  Sample 2 → Debate → Save
  ...

2. Improved Fault Tolerance

Feature	main.py	Batch Scripts
Save Frequency	After processing all samples	After each item
Checkpoint Recovery	✅ Supported	✅ Supported
Error Handling	Single error may interrupt	Logs error and continues
Retry Mechanism	❌	✅ Auto-retry 3 times
Use Case	Small-scale (<100)	Large-scale (1000+)

3. Flexibility

• ✅ Can run only generation, not detection
• ✅ Can run only detection, not generation
• ✅ Can run detection multiple times on same generation results
• ✅ Can process different batches in parallel

---

Usage Examples

Scenario 1: Quick Test

bash scripts/run_test.sh

Scenario 2: Large-Scale Batch Processing

# Step 1: Batch generation (may take hours)
python run_generation.py \
    --input_file data/train.json \
    --output_file results/generated_news.json \
    --max_iterations 3

# Step 2: Batch detection
python run_detection.py \
    --input_file results/generated_news.json \
    --output_file results/detection_results.json \
    --use_optimized

Scenario 3: Checkpoint Recovery

# If generation interrupted at item 150
python run_generation.py \
    --input_file data/train.json \
    --output_file results/generated_news.json \
    --max_iterations 3 \
    --resume_index 150  # Continue from item 151

Scenario 4: Parallel Processing

# Terminal 1: Process first 500 items
python run_generation.py \
    --input_file data/train.json \
    --output_file results/gen_part1.json \
    --max_samples 500

# Terminal 2: Process next 500 items
python run_generation.py \
    --input_file data/train.json \
    --output_file results/gen_part2.json \
    --resume_index 500 \
    --max_samples 500

---

Comparison with Original Code

main.py (Original)

Advantages:

• ✅ Clear logic, follows paper Algorithm 1
• ✅ Suitable for workflow verification
• ✅ Suitable for small-scale data

Disadvantages:

• ❌ Generation and detection coupled, cannot run independently
• ❌ Low save frequency, poor fault tolerance
• ❌ Not suitable for large-scale batch processing

Batch Scripts (New)

Advantages:

• ✅ Decoupled design, high flexibility
• ✅ Saves each item, strong fault tolerance
• ✅ Supports checkpoint recovery
• ✅ Automatic retry mechanism
• ✅ Suitable for large-scale batch processing

Disadvantages:

• ❌ Need to manually run two scripts (or use bash scripts)

---

Recommended Use Cases

Scenario	Recommended Solution
Workflow verification, small-scale testing	main.py
Large-scale batch processing (1000+)	Batch scripts
Unstable network environment	Batch scripts
Need parallel processing	Batch scripts
Only need generation or detection	Batch scripts

---

Summary

Based on user requirements, we successfully implemented:

1. ✅ Decoupled Design: Generation and detection completely independent
2. ✅ Improved Fault Tolerance: Saves after processing each item
3. ✅ Checkpoint Recovery: Supports resume from any position
4. ✅ Automatic Retry: Detection script auto-retries 3 times
5. ✅ Flexible Usage: Provides 4 bash script examples
6. ✅ Detailed Documentation: Complete usage guide

Now users can:

• Use main.py for small-scale verification
• Use batch scripts for large-scale production environment processing
• Flexibly choose to run only generation or detection as needed
• Safely process large amounts of data even with unstable network