Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions .github/workflows/benchmarks.yaml
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
name: Benchmarks

on:
# Run on schedule - weekly benchmarks
schedule:
- cron: '0 2 * * 0' # Every Sunday at 2 AM UTC

# Manual trigger with options
workflow_dispatch:
inputs:
Expand All @@ -22,8 +18,16 @@ on:
default: 'cx23'
type: choice
options:
- 'cx23' # 2 vCPU, 4GB RAM
- 'cx43' # 8 vCPU, 16GB RAM
- 'cx23' # 2 vCPU, 4GB RAM (shared)
- 'cx33' # 4 vCPU, 8GB RAM (shared)
- 'cx43' # 8 vCPU, 16GB RAM (shared)
- 'cx53' # 16 vCPU, 32GB RAM (shared)
- 'ccx13' # 2 vCPU, 8GB RAM (dedicated AMD EPYC)
- 'ccx23' # 4 vCPU, 16GB RAM (dedicated AMD EPYC)
- 'ccx33' # 8 vCPU, 32GB RAM (dedicated AMD EPYC)
- 'ccx43' # 16 vCPU, 64GB RAM (dedicated AMD EPYC)
- 'ccx53' # 32 vCPU, 128GB RAM (dedicated AMD EPYC)
- 'ccx63' # 48 vCPU, 192GB RAM (dedicated AMD EPYC)
commit_range:
description: 'Commit range to benchmark (e.g., HEAD~5..HEAD)'
required: false
Expand Down Expand Up @@ -96,7 +100,7 @@ jobs:

# Full benchmarks on Hetzner Cloud
benchmark-hetzner:
if: github.event_name == 'schedule' || (github.event_name == 'workflow_dispatch' && github.event.inputs.use_hetzner == 'true') || (github.event_name == 'push' && github.ref == 'refs/heads/main')
if: (github.event_name == 'workflow_dispatch' && github.event.inputs.use_hetzner == 'true') || (github.event_name == 'push' && github.ref == 'refs/heads/main')
runs-on: ubuntu-latest

steps:
Expand Down
69 changes: 68 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,6 @@ python -m csp_benchmarks.hetzner.cli cleanup

Benchmarks run automatically:

- **Weekly**: Full benchmarks on Hetzner Cloud (Sunday 2 AM UTC)
- **On push to main**: Benchmarks for the new commit
- **Manual trigger**: Via workflow_dispatch with custom options

Expand All @@ -158,7 +157,75 @@ View the latest results at: <https://csp-community.github.io/csp-benchmarks/>

## Contributing

### Adding New Benchmarks

1. Add new benchmarks to the `benchmarks/` directory
1. Follow ASV naming conventions (`bench_*.py`, class names ending in `Suite`)
1. Use parameterized benchmarks for testing across different configurations
1. Run `make benchmark-local` to test your benchmarks before submitting

### Contributing Machine Results

You can contribute benchmark results from your own machine to help the community understand csp performance across different hardware configurations.

#### Step 1: Register Your Machine

Add your machine to `csp_benchmarks/asv-machine.json`. Use a unique, descriptive name:

```json
{
"timkpaine-framework-13": {
"arch": "x86_64",
"cpu": "AMD Ryzen AI 9 HX 370 (24 cores)",
"machine": "timkpaine-framework-13",
"num_cpu": "24",
"os": "Ubuntu 24.04",
"ram": "64GB"
}
}
```

Machine naming convention: `username-device-model` (e.g., `timkpaine-framework-13`, `johndoe-mbp-m3`)

#### Step 2: Run Benchmarks

```bash
# Install dependencies
pip install -e ".[develop]"

# Copy machine config to ASV location
cp csp_benchmarks/asv-machine.json ~/.asv-machine.json

# Run benchmarks with your machine name
python -m asv run --config csp_benchmarks/asv.conf.json --machine your-machine-name

# Or use make (runs with local Python)
make benchmark-local
```

#### Step 3: Verify Results

```bash
# Check that results were created
ls csp_benchmarks/results/your-machine-name/

# Preview the results locally
make benchmark-view
```

#### Step 4: Submit a Pull Request

1. Fork the repository
1. Create a branch: `git checkout -b add-machine-results`
1. Commit your changes:
- `csp_benchmarks/asv-machine.json` (your machine entry)
- `csp_benchmarks/results/your-machine-name/` (your result files)
1. Open a PR with a description of your hardware

#### Tips for Consistent Results

- Close other applications during benchmarking
- Run on AC power (not battery) for laptops
- Ensure stable CPU frequency (disable turbo boost for more consistent results)
- Run multiple times and verify results are stable
- Include your OS version and Python version in the PR description
64 changes: 64 additions & 0 deletions csp_benchmarks/asv-machine.json
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,14 @@
"os": "Ubuntu 24.04",
"ram": "4GB"
},
"hetzner-cx33": {
"arch": "x86_64",
"cpu": "Intel/AMD (4 vCPU)",
"machine": "hetzner-cx33",
"num_cpu": "4",
"os": "Ubuntu 24.04",
"ram": "8GB"
},
"hetzner-cx43": {
"arch": "x86_64",
"cpu": "Intel/AMD (8 vCPU)",
Expand All @@ -23,6 +31,62 @@
"os": "Ubuntu 24.04",
"ram": "16GB"
},
"hetzner-cx53": {
"arch": "x86_64",
"cpu": "Intel/AMD (16 vCPU)",
"machine": "hetzner-cx53",
"num_cpu": "16",
"os": "Ubuntu 24.04",
"ram": "32GB"
},
"hetzner-ccx13": {
"arch": "x86_64",
"cpu": "AMD EPYC (2 dedicated vCPU)",
"machine": "hetzner-ccx13",
"num_cpu": "2",
"os": "Ubuntu 24.04",
"ram": "8GB"
},
"hetzner-ccx23": {
"arch": "x86_64",
"cpu": "AMD EPYC (4 dedicated vCPU)",
"machine": "hetzner-ccx23",
"num_cpu": "4",
"os": "Ubuntu 24.04",
"ram": "16GB"
},
"hetzner-ccx33": {
"arch": "x86_64",
"cpu": "AMD EPYC (8 dedicated vCPU)",
"machine": "hetzner-ccx33",
"num_cpu": "8",
"os": "Ubuntu 24.04",
"ram": "32GB"
},
"hetzner-ccx43": {
"arch": "x86_64",
"cpu": "AMD EPYC (16 dedicated vCPU)",
"machine": "hetzner-ccx43",
"num_cpu": "16",
"os": "Ubuntu 24.04",
"ram": "64GB"
},
"hetzner-ccx53": {
"arch": "x86_64",
"cpu": "AMD EPYC (32 dedicated vCPU)",
"machine": "hetzner-ccx53",
"num_cpu": "32",
"os": "Ubuntu 24.04",
"ram": "128GB"
},
"hetzner-ccx63": {
"arch": "x86_64",
"cpu": "AMD EPYC (48 dedicated vCPU)",
"machine": "hetzner-ccx63",
"num_cpu": "48",
"os": "Ubuntu 24.04",
"ram": "192GB"
},
"timkpaine-mbp-m2": {
"arch": "arm64",
"cpu": "Apple M2 (8 cores)",
Expand Down
9 changes: 0 additions & 9 deletions csp_benchmarks/results/TP11251-M2/machine.json

This file was deleted.