GPU Computing with CUDA — from NumPy to Custom Kernels

Hands-on GPU programming and performance optimization: moving numerical workloads from a NumPy CPU baseline to CuPy and custom CUDA kernels written with Numba, then profiling and tuning them with NVIDIA Nsight Systems.

What's in this repo

Two parts: (1) getting kernels working on GPU and benchmarking them against a CPU baseline, and (2) getting them fast — profiling with Nsight Systems, testing different block/thread configurations, and fixing synchronization anti-patterns.

Part 1 — From NumPy to custom CUDA kernels

• vec_add.py, elem1d_todo_profile.py — baseline element-wise operations: NumPy (CPU) → CuPy (GPU array library) → custom Numba CUDA kernel
• matmul_profile.py — matrix multiplication, CPU vs GPU
• stencil_profile.py, stencil5_todo_profile.py, stencil9_profile.py, stencil49_profile.py — 2D stencil kernels (3×3 convolution and larger 5-, 9-, and 49-point neighborhoods)
• lab1Scrimali.ipynb — walkthrough notebook: CPU baseline → CuPy → custom kernel, with benchmarking

Part 2 — Profiling and optimization with Nsight Systems

• stencil49_sweep.py — block-size sweep for the 49-point stencil kernel, to find the configuration with the best throughput
• reduction_profile.py — parallel reduction, comparing a CuPy built-in reduction against a hand-written Numba kernel
• sync_antipattern.py — a deliberately introduced synchronization anti-pattern, used to show its performance cost under the profiler
• parse_nsys.py — utility script to parse Nsight Systems (.nsys-rep/exported) profiling output into a usable summary
• lab2Scrimali.ipynb — walkthrough notebook: profiling methodology, reading the Nsight Systems timeline, and interpreting the block-size sweep and reduction results

Key takeaways

• Moving from NumPy to CuPy already recovers most of the easy speedup for large arrays with no kernel-writing required; custom Numba kernels pay off once the operation isn't already covered by a CuPy primitive (e.g. the stencils).
• Block/thread configuration matters more than it looks: the stencil49_sweep.py results show throughput is far from flat across block sizes, so a "default" launch configuration leaves real performance on the table.
• Nsight Systems makes synchronization anti-patterns visible as idle GPU time in the timeline — sync_antipattern.py is a minimal repro used to demonstrate that cost directly, rather than just asserting it.

Tech stack

CUDA · Numba (@cuda.jit) · CuPy · NumPy · NVIDIA Nsight Systems

Requirements

• CUDA-capable GPU + NVIDIA drivers
• numba, cupy (matching your CUDA toolkit version), numpy
• NVIDIA Nsight Systems (nsys) for profiling the *_profile.py / *_sweep.py scripts

---

Part of my AI/ML portfolio — see profile README for other projects.