Skip to content

CLUSTER_SETUP.md

Latest commit

 

History

History
181 lines (136 loc) · 4.12 KB

CLUSTER_SETUP.md

File metadata and controls

181 lines (136 loc) · 4.12 KB
Raw

NEU Explorer Cluster Setup Guide

Overview

This guide covers the setup and improvements made to run your parallel video recognition experiments on NEU Explorer cluster.

1. SLURM Script

GPU Resource Configuration

Scripts now use flexible GPU requests that work with both A100 and V100:

#SBATCH --gres=gpu:2  # Request any 2 GPUs

# Commented alternatives for specific GPU types:
# #SBATCH --gres=gpu:a100:2  # For A100 specifically
# #SBATCH --gres=gpu:v100:2  # For V100 specifically

Quick Start Guide

Step 1: Test GPU Access

Before running full experiments, verify your setup:

# Make script executable
chmod +x scripts/slurm/test_gpu.sh

# Submit test job
sbatch scripts/slurm/test_gpu.sh

# Check status
squeue -u $USER

# View output when complete
cat logs/gpu/test_gpu_<job_id>.out

Step 2: Run Individual Experiments

Single GPU with Mixed Precision

sbatch scripts/slurm/gpu/single_gpu.sh

DDP 2 GPUs (Frozen Backbone)

sbatch scripts/slurm/gpu/ddp_2gpu_frozen.sh

DDP 4 GPUs (Frozen Backbone)

sbatch scripts/slurm/gpu/ddp_4gpu_frozen.sh

Full Fine-tuning with FSDP

sbatch scripts/slurm/gpu/fsdp1_2gpu.sh
sbatch scripts/slurm/gpu/fsdp1_4gpu.sh

Step 3: Run All Experiments

# Make script executable
chmod +x scripts/slurm/run_all.sh

# Submit all with dependencies
bash scripts/slurm/run_all.sh

Monitoring Jobs

Check Job Status

squeue -u $USER

View Job Output (Real-time)

tail -f logs/gpu/single_gpu_<job_id>.out
tail -f logs/gpu/ddp_2gpu_frozen_<job_id>.out

Cancel Jobs

# Cancel specific job
scancel <job_id>

# Cancel all your jobs
scancel -u $USER

Check Job History

sacct -u $USER --format=JobID,JobName,Partition,State,Elapsed,MaxRSS

GPU Selection

If you want to use a specific GPU type, edit the SLURM script:

For A100 GPUs

Uncomment the A100 line:

#SBATCH --gres=gpu:a100:2

For V100 GPUs

Uncomment the V100 line:

#SBATCH --gres=gpu:v100:2

Expected Results Location

After experiments complete, results will be in:

GPU Training Results

  • results/metrics/gpu_training_results.csv - Master CSV tracker
  • results/metrics/single_gpu_metrics.json - Single GPU details
  • results/metrics/ddp_2gpu_metrics.json - DDP 2 GPU details
  • results/metrics/fsdp1_2gpu_metrics.json - FSDP1 2 GPU details

CPU Preprocessing Results

  • results/metrics/all_results.csv - CPU preprocessing tracker
  • results/metrics/baseline.json - Baseline metrics
  • results/metrics/joblib_comparison.csv - Joblib results
  • results/metrics/dask_comparison.csv - Dask results

Performance Expectations

Single GPU (with Mixed Precision)

  • Training time: ~2-3 hours for 10,000 videos, 10 epochs
  • Throughput: ~8-10 videos/sec
  • Memory usage: ~20-25 GB

DDP 2 GPUs (Frozen Backbone)

  • Training time: ~1-1.5 hours for 10,000 videos, 10 epochs
  • Throughput: ~16-20 videos/sec
  • Speedup: ~2x vs single GPU

DDP 4 GPUs (Frozen Backbone)

  • Training time: ~0.5-1 hour for 10,000 videos, 10 epochs
  • Throughput: ~30-40 videos/sec
  • Speedup: ~3.5-4x vs single GPU

Troubleshooting

Job Fails Immediately

  • Check: cat logs/gpu/<job_name>_<job_id>.err
  • Common issues:
    • Conda environment not activated
    • Missing modules
    • Incorrect partition

Out of Memory Error

  • Reduce batch size in SLURM script
  • Try with mixed precision training
  • Use gradient accumulation (would need to add this)

Job Stuck in Queue

  • Check partition availability: sinfo -p multigpu
  • Consider using different partition
  • Check job dependencies

NCCL Errors in Multi-GPU

  • Check NCCL environment variables are set
  • Verify all GPUs are on same node
  • Check network connectivity between GPUs

Best Practices

  1. Start Small: Test with test_gpu.sh first
  2. Use Interactive Jobs for debugging: 
    srun --partition=gpu-interactive --gres=gpu:1 --pty bash
  3. Monitor Resources: Use nvidia-smi in your jobs
  4. Save Checkpoints: For long-running jobs (not yet implemented)
  5. Log Everything: All scripts save detailed logs