Skip to content

Commit c7edfd6

Browse files
ryan-williamsryan-williams
committed
Add demos
1 parent ad85cc4 commit c7edfd6

15 files changed

Lines changed: 623 additions & 40 deletions

‎.github/test-scripts/gpu-benchmark.py‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
#!/usr/bin/env python3
2+
"""GPU benchmark script for testing PyTorch CUDA capabilities.
3+
4+
Used by ``demo-gpu.yml``.
5+
"""
6+
7+
import sys
8+
import time
9+
import torch
10+
11+
12+
def main():
13+
print(f'PyTorch version: {torch.__version__}')
14+
print(f'CUDA available: {torch.cuda.is_available()}')
15+
16+
if not torch.cuda.is_available():
17+
print('ERROR: CUDA not available!')
18+
sys.exit(1)
19+
20+
# Get GPU information
21+
device = torch.cuda.get_device_properties(0)
22+
print(f'GPU: {device.name}')
23+
print(f'Memory: {device.total_memory / 1e9:.1f} GB')
24+
print(f'CUDA version: {torch.version.cuda}')
25+
26+
# Matrix multiplication benchmark
27+
print('\nRunning matrix multiplication benchmark...')
28+
size = 8192
29+
x = torch.randn(size, size, dtype=torch.float32).cuda()
30+
torch.cuda.synchronize()
31+
32+
# Warmup
33+
print(' Warming up...')
34+
for _ in range(3):
35+
_ = torch.matmul(x, x.T)
36+
torch.cuda.synchronize()
37+
38+
# Benchmark
39+
print(' Running benchmark...')
40+
start = time.time()
41+
y = torch.matmul(x, x.T)
42+
torch.cuda.synchronize()
43+
elapsed = time.time() - start
44+
45+
gflops = (2 * size**3) / (elapsed * 1e9)
46+
print(f' Matrix size: {size}x{size}')
47+
print(f' Time: {elapsed:.3f} seconds')
48+
print(f' Performance: {gflops:.1f} GFLOPS')
49+
print(f' Memory used: {torch.cuda.max_memory_allocated()/1e9:.2f} GB')
50+
51+
# Quick training simulation
52+
print('\nSimulating model training...')
53+
model = torch.nn.Sequential(
54+
torch.nn.Linear(1024, 512),
55+
torch.nn.ReLU(),
56+
torch.nn.Linear(512, 256),
57+
torch.nn.ReLU(),
58+
torch.nn.Linear(256, 10)
59+
).cuda()
60+
61+
optimizer = torch.optim.Adam(model.parameters())
62+
data = torch.randn(32, 1024).cuda()
63+
64+
for i in range(10):
65+
output = model(data)
66+
loss = output.sum()
67+
loss.backward()
68+
optimizer.step()
69+
optimizer.zero_grad()
70+
if i == 0 or i == 9:
71+
print(f' Iteration {i+1}: loss = {loss.item():.4f}')
72+
73+
print('\nGPU workload completed successfully!')
74+
return 0
75+
76+
77+
if __name__ == '__main__':
78+
sys.exit(main())

‎.github/workflows/README.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# `ec2-gha` Demos
2+
This directory contains the reusable workflow and demo workflows for ec2-gha, demonstrating various capabilities.
3+
4+
For documentation about the main workflow, [`runner.yml`](runner.yml), see [the main README](../../README.md).
5+
6+
<!-- toc -->
7+
- [`demos` – run all demo workflows](#demos)
8+
- [GPU demos](#gpu)
9+
- [`gpu-minimal` – `nvidia-smi` "hello world"](#gpu-minimal)
10+
- [`gpu-job-seq` – GPU train/test/eval (sequential jobs)](#gpu-job-seq)
11+
- [Real-world example: Mamba installation testing](#mamba)
12+
- [Architecture & Parallelization](#arch)
13+
- [`archs` – launch x86 and ARM nodes](#archs)
14+
- [`multi-instance` – launch multiple instances, use in matrix](#multi-instance)
15+
- [`multi-job` – launch multiple instances, use individually](#multi-job)
16+
<!-- /toc -->
17+
18+
## [`demos`](demos.yml) – run all demo workflows <a id="demos"></a>
19+
Useful regression test, demonstrates and verifies features.
20+
21+
[![](../../img/demos%2325%201.png)][demos#25]
22+
23+
## GPU demos <a id="gpu"></a>
24+
25+
### [`gpu-minimal`](demo-gpu-minimal.yml) – `nvidia-smi` "hello world" <a id="gpu-minimal"></a>
26+
- **Instance type:** `g4dn.xlarge`
27+
28+
### [`gpu-job-seq`](demo-gpu-job-seq.yml) – GPU train/test/eval (sequential jobs) <a id="gpu-job-seq"></a>
29+
- Runs 3 jobs sequentially on the same GPU instance (prepare, train, evaluate)
30+
- Uses pre-installed PyTorch from Deep Learning AMI's conda environment
31+
- Runs GPU benchmark with matrix operations and training simulation
32+
- Verifies same GPU is used across all jobs
33+
- Demonstrates instance reuse for multi-stage ML workflows
34+
- **Instance type:** `g4dn.xlarge`
35+
- **Use case:** ML/AI workflow testing with GPU acceleration
36+
37+
### Real-world example: [Mamba installation testing](https://github.com/Open-Athena/mamba/blob/gha/.github/workflows/install.yaml) <a id="mamba"></a>
38+
- Tests different versions of `mamba_ssm` package on GPU instances
39+
- **Customizes `instance_name`**: `"$repo/$name==${{ inputs.mamba_version }} (#$run_number)"`
40+
- Results in descriptive names like `"Open-Athena/mamba/install==2.2.5 (#123)"`
41+
- Makes it easy to identify which version is being tested on each instance
42+
- Uses pre-installed PyTorch from DLAMI conda environment
43+
- **Use case:** Package compatibility testing across versions
44+
45+
[![](../../img/mamba%2312.png)][mamba#12]
46+
47+
## Architecture & Parallelization <a id="arch"></a>
48+
49+
### [`archs`](demo-archs.yml) – launch x86 and ARM nodes <a id="archs"></a>
50+
- Verify architecture-specific behavior
51+
- **Instance types:** `t3.medium` (x86), `t4g.medium` (ARM)
52+
53+
### [`multi-instance`](demo-multi-instance.yml) – launch multiple instances, use in matrix <a id="multi-instance"></a>
54+
- Creates configurable number of instances (default: 3)
55+
- Uses matrix strategy to run jobs in parallel
56+
- Each job runs on its own EC2 instance
57+
- **Customizes `instance_name`**: `"$repo/$name-$idx (#$run_number)"`
58+
- Uses `$idx` template variable (0-based index) to distinguish instances
59+
- Results in names like `"ec2-gha/multi-instance-0 (#123)"`, `"ec2-gha/multi-instance-1 (#123)"`, etc.
60+
- **Instance type:** `t3.medium`
61+
- **Use case:** Parallel test execution
62+
63+
### [`multi-job`](demo-multi-job.yml) – launch multiple instances, use individually <a id="multi-job"></a>
64+
- Launch 2 instances
65+
- Run build job on first instance
66+
- Run test job on second instance
67+
- Aggregate results from both instances
68+
- **Customizes `instance_name`**: `"$repo/$name-$idx (#$run_number)"`
69+
- Results in names like `"ec2-gha/multi-job-0 (#123)"` and `"ec2-gha/multi-job-1 (#123)"`
70+
- **Instance type:** `t3.medium`
71+
- **Use case:** Pipeline with dedicated instances per stage
72+
73+
[mamba#12]: https://github.com/Open-Athena/mamba/actions/runs/16972369660/
74+
[demos#25]: https://github.com/Open-Athena/ec2-gha/actions/runs/17004697889
75+

‎.github/workflows/demo-00-minimal.yml‎

Lines changed: 0 additions & 8 deletions
This file was deleted.

‎.github/workflows/demo-archs.yml‎

Lines changed: 74 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,78 @@
1-
name: Demo – 2 GPU instances with different architectures
1+
name: Demo – x86 and ARM instances
22
on:
33
workflow_dispatch:
4+
workflow_call: # Tested by `demos.yml`
5+
permissions:
6+
id-token: write # Required for AWS OIDC authentication
7+
contents: read # Required for actions/checkout
48
jobs:
5-
placeholder:
6-
runs-on: ubuntu-latest
9+
# Launch EC2 runners for each architecture
10+
x86:
11+
name: Launch x86
12+
uses: ./.github/workflows/runner.yml
13+
with:
14+
ec2_instance_type: t3.medium
15+
ec2_image_id: ami-0e86e20dae9224db8 # Ubuntu 24.04 LTS x86_64 (us-east-1)
16+
secrets: inherit
17+
arm:
18+
name: Launch ARM
19+
uses: ./.github/workflows/runner.yml
20+
with:
21+
ec2_instance_type: t4g.medium
22+
ec2_image_id: ami-0aa307ed50ca3e58f # Ubuntu 24.04 LTS arm64 (us-east-1)
23+
secrets: inherit
24+
25+
# Run jobs directly on launched instances
26+
test-x86:
27+
needs: x86
28+
if: needs.x86.outputs.id != ''
29+
name: Test x86
30+
runs-on: ${{ needs.x86.outputs.id }}
731
steps:
8-
- run: echo "Placeholder workflow – being developed in \#2 / rw/hooks branch"
32+
- name: Architecture test on t3.medium
33+
run: |
34+
echo "Architecture: $(uname -m)"
35+
echo "Processor: $(lscpu | grep 'Architecture:' | awk '{print $2}')"
36+
echo "Instance type: $(curl -s http://169.254.169.254/latest/meta-data/instance-type)"
37+
test-arm:
38+
needs: arm
39+
if: needs.arm.outputs.id != ''
40+
name: Test ARM
41+
runs-on: ${{ needs.arm.outputs.id }}
42+
steps:
43+
- name: Architecture test on t4g.medium
44+
run: |
45+
echo "Architecture: $(uname -m)"
46+
echo "Processor: $(lscpu | grep 'Architecture:' | awk '{print $2}')"
47+
echo "Instance type: $(curl -s http://169.254.169.254/latest/meta-data/instance-type)"
48+
49+
# Use a `matrix` to run jobs on multiple instances
50+
test-matrix:
51+
needs: [x86, arm]
52+
if: always() && (needs.x86.outputs.id != '' || needs.arm.outputs.id != '')
53+
name: Test ${{ matrix.arch }} (matrix)
54+
continue-on-error: true
55+
strategy:
56+
matrix:
57+
include:
58+
- arch: x86
59+
runner: ${{ needs.x86.outputs.id }}
60+
- arch: arm
61+
runner: ${{ needs.arm.outputs.id }}
62+
fail-fast: false
63+
runs-on: ${{ matrix.runner }}
64+
steps:
65+
- name: Matrix architecture test
66+
run: |
67+
echo "Running on ${{ matrix.arch }}"
68+
echo "Architecture: $(uname -m)"
69+
echo "Instance type: $(curl -s http://169.254.169.254/latest/meta-data/instance-type)"
70+
71+
# Verify expected architecture
72+
if [[ "${{ matrix.arch }}" == "x86" ]]; then
73+
echo "Verifying x86_64 architecture..."
74+
[[ "$(uname -m)" == "x86_64" ]] && echo "✓ Confirmed x86_64" || exit 1
75+
elif [[ "${{ matrix.arch }}" == "arm" ]]; then
76+
echo "Verifying ARM architecture..."
77+
[[ "$(uname -m)" == "aarch64" ]] && echo "✓ Confirmed aarch64" || exit 1
78+
fi

‎.github/workflows/demo-gpu-dbg.yml‎

Lines changed: 0 additions & 8 deletions
This file was deleted.

‎.github/workflows/demo-gpu-job-seq.yml‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
name: Demo – comprehensive GPU workload with sequential jobs
2+
on:
3+
workflow_dispatch:
4+
inputs:
5+
sleep:
6+
description: "Sleep for this many seconds at the end of each job (optional, for SSH debugging)"
7+
required: false
8+
type: number
9+
default: 0
10+
ssh_pubkey:
11+
description: "Add this SSH public key to instance's `~/.ssh/authorized_keys` (optional, for debugging)"
12+
required: false
13+
type: string
14+
workflow_call: # Tested by `demos.yml`
15+
inputs:
16+
sleep:
17+
required: false
18+
type: number
19+
default: 0
20+
ssh_pubkey:
21+
required: false
22+
type: string
23+
24+
permissions:
25+
id-token: write # Required for AWS OIDC authentication
26+
contents: read # Required for actions/checkout
27+
28+
jobs:
29+
ec2:
30+
uses: ./.github/workflows/runner.yml
31+
secrets: inherit
32+
with:
33+
ec2_instance_type: g4dn.xlarge
34+
ec2_image_id: ami-00096836009b16a22 # Deep Learning OSS Nvidia Driver AMI GPU PyTorch
35+
ssh_pubkey: ${{ inputs.ssh_pubkey }}
36+
37+
# Job 1: Prepare environment and verify GPU
38+
prepare:
39+
needs: ec2
40+
runs-on: ${{ needs.ec2.outputs.id }}
41+
outputs:
42+
gpu-uuid: ${{ steps.gpu-info.outputs.uuid }}
43+
gpu-name: ${{ steps.gpu-info.outputs.name }}
44+
steps:
45+
- uses: actions/checkout@v4
46+
47+
- name: Get GPU info
48+
id: gpu-info
49+
run: |
50+
echo "=== Preparing GPU environment ==="
51+
nvidia-smi
52+
uuid=$(nvidia-smi --query-gpu=gpu_uuid --format=csv,noheader)
53+
name=$(nvidia-smi --query-gpu=name --format=csv,noheader)
54+
echo "uuid=$uuid" >> $GITHUB_OUTPUT
55+
echo "name=$name" >> $GITHUB_OUTPUT
56+
echo "GPU UUID: $uuid"
57+
echo "GPU Name: $name"
58+
59+
- name: Setup PyTorch environment
60+
run: |
61+
echo "=== Setting up PyTorch environment ==="
62+
# The DLAMI already has PyTorch installed in a conda environment
63+
# Set up environment for GitHub Actions to use the conda env
64+
echo "/opt/conda/envs/pytorch/bin" >> $GITHUB_PATH
65+
echo "CONDA_DEFAULT_ENV=pytorch" >> $GITHUB_ENV
66+
67+
# Verify PyTorch is available
68+
/opt/conda/envs/pytorch/bin/python -c "import torch; print(f'PyTorch {torch.__version__} with CUDA {torch.version.cuda}')"
69+
echo "PyTorch environment ready"
70+
71+
- name: Sleep (${{ inputs.sleep }}s)
72+
if: ${{ inputs.sleep > 0 }}
73+
run: |
74+
echo "Sleeping for ${{ inputs.sleep }} seconds..."
75+
echo "Instance IP available in GitHub Actions log for SSH"
76+
sleep ${{ inputs.sleep }}
77+
78+
# Job 2: Training simulation
79+
train:
80+
needs: [ec2, prepare]
81+
runs-on: ${{ needs.ec2.outputs.id }}
82+
steps:
83+
- uses: actions/checkout@v4
84+
85+
- name: Verify same GPU
86+
run: |
87+
echo "=== Training on GPU ==="
88+
current_uuid=$(nvidia-smi --query-gpu=gpu_uuid --format=csv,noheader)
89+
if [[ "$current_uuid" == "${{ needs.prepare.outputs.gpu-uuid }}" ]]; then
90+
echo "✅ Confirmed: Using same GPU as preparation job"
91+
echo "GPU: ${{ needs.prepare.outputs.gpu-name }}"
92+
else
93+
echo "❌ ERROR: Different GPU!"
94+
exit 1
95+
fi
96+
97+
- name: Run training benchmark
98+
run: |
99+
echo "=== Running GPU Training Benchmark ==="
100+
# Use the conda environment's Python which has PyTorch pre-installed
101+
/opt/conda/envs/pytorch/bin/python .github/test-scripts/gpu-benchmark.py
102+
103+
- name: Sleep (${{ inputs.sleep }}s)
104+
if: ${{ inputs.sleep > 0 }}
105+
run: sleep ${{ inputs.sleep }}
106+
107+
# Job 3: Evaluation/testing
108+
evaluate:
109+
needs: [ec2, train]
110+
runs-on: ${{ needs.ec2.outputs.id }}
111+
steps:
112+
- name: System diagnostics
113+
run: |
114+
echo "=== Final Evaluation ==="
115+
echo "System Info:"
116+
uname -a
117+
lscpu | grep "Model name" || true
118+
free -h
119+
120+
echo -e "\nGPU Status:"
121+
nvidia-smi --query-gpu=name,memory.total,driver_version,utilization.gpu,temperature.gpu --format=csv
122+
123+
echo -e "\nDisk Usage:"
124+
df -h /
125+
126+
echo -e "\nInstance Metadata:"
127+
curl -s -H "X-aws-ec2-metadata-token: $(curl -X PUT -H 'X-aws-ec2-metadata-token-ttl-seconds: 300' http://169.254.169.254/latest/api/token 2>/dev/null)" http://169.254.169.254/latest/meta-data/instance-type || echo "Metadata unavailable"
128+
129+
- name: Verify multi-job reuse
130+
run: |
131+
echo "=== Verifying EC2 Instance Reuse ==="
132+
echo "This job successfully ran on the same EC2 instance as the previous jobs"
133+
echo "The instance will terminate automatically after idle timeout"
134+
135+
- name: Sleep (${{ inputs.sleep }}s)
136+
if: ${{ inputs.sleep > 0 }}
137+
run: sleep ${{ inputs.sleep }}

0 commit comments

Comments
 (0)