🚀 CUDA Insight: Profile, Analyze, Optimize

Write CUDA. Understand it. Fix it. Ship faster.

(Placeholder for asciinema terminal recording)

CUDA Insight is an all-in-one educational toolkit and profiler designed to demystify GPU performance. Whether you're learning CUDA or optimizing a production kernel, CUDA Insight wraps NVIDIA's powerful toolchain into a simple, actionable workflow.

---

✨ Features

• 📊 Hardware Discovery (gpu-info): Instantly retrieve specs like theoretical TFLOPS, memory bandwidth, and compute capability.
• 🔍 PTX Annotation (ptx): Automatically map generated PTX assembly instructions directly to your original C++ source lines.
• ⚡ One-Click Profiling (profile): Automatically runs ncu (Nsight Compute), captures metrics, and generates a structured analysis report.
• 🧠 Expert Suggestions Engine: Identifies bottlenecks (e.g., Bank Conflicts, Register Pressure) and provides actionable code snippets to fix them.
• 📈 Roofline Modeling: Automatically generates visual Roofline charts (roofline.png) to show how close your kernel is to the theoretical hardware limit.
• 🔥 PyTorch Integration: Easily profile your custom PyTorch C++ extensions via our torch_ext API.

---

🛠️ Installation

pip install -e ".[dev]"

Requirements: NVIDIA GPU, nvcc, and ncu (Nsight Compute) must be in your PATH.

---

🚀 Quick Start

1. View GPU Specs

cuda-insight gpu-info

2. Inspect Assembly (PTX/SASS)

# See how your CUDA maps to PTX
cuda-insight ptx kernels/matmul/naive.cu --annotate

3. Profile & Analyze a Kernel

# Compile and profile a binary. Generates HTML report and Roofline chart.
nvcc kernels/matmul/naive.cu -o naive_matmul
cuda-insight profile naive_matmul --html --roofline

---

📚 Reference Kernels Library

Inside the kernels/ directory, you'll find a progressive library of algorithms showing the evolution from naive to state-of-the-art:

• Matrix Multiplication (matmul/): Naive $\rightarrow$ Shared Memory Tiling $\rightarrow$ cuBLAS Reference
• Parallel Reduction (reduction/): Atomic $\rightarrow$ Block Shared Mem $\rightarrow$ Warp Primitives
• Softmax (softmax/): Two-Pass $\rightarrow$ Online Normalizer (FlashAttention style)
• Attention (flash_attention_minimal/): Block-Sparse Fused Attention
• Quantization (quantization/): INT8 dp4a Gemm

Each kernel includes an educational header block explaining the optimization, expected speedup, and further reading links!

---

🏗️ Architecture & Workflow

graph TD
    A[User writes .cu file] -->|cuda-insight ptx| B(Compiler Wrapper)
    B --> C[PTX / SASS Extractor]
    C --> D[PTX Annotator]
    D -.-> E((Rich CLI Output))
    
    F[Compiled Binary] -->|cuda-insight profile| G(NCU Profiler Wrapper)
    G --> H[CSV Parser]
    H --> I[Analyzer & Roofline Model]
    I --> J[Suggestions Engine]
    J --> K[Jinja2 HTML Reporter]
    K -.-> L((Interactive Dashboard))

Loading

---

🧠 Example Output

Bottleneck Detection

If you run the naive matmul, CUDA Insight will tell you:

Bottleneck: Memory Bandwidth Bound
Kernel is bottlenecked by global memory bandwidth. Utilizing 98% of peak.
Fix: Use shared memory tiling to reduce global memory reads.

Automated Roofline Charts

CUDA Insight automatically generates beautiful roofline charts to visualize your kernel's performance limits.

The roofline model shows whether your kernel is bottlenecked by memory bandwidth (the sloped line) or compute throughput (the horizontal flat line). Notice how the optimized tiled matmul shifts the arithmetic intensity rightwards and reaches higher performance!

---

🤝 Contributing

Contributions are welcome! If you want to add a new reference kernel (e.g., Prefix Scan, GEMV, Convolution), feel free to open a PR.

---

🆚 Comparison

Feature	CUDA Insight	Nsight Systems	Manual Profiling (nvcc/ncu)
Setup Time	< 1 minute	Heavy GUI install	Needs scripting
Output	Actionable text + HTML	Complex timeline	Raw CSV/JSON
Fix Suggestions	Built-in Expert Engine	None	StackOverflow
PyTorch Ops	1-line wrapper	Requires NVTX markers	Complex
Roofline	Auto-generated chart	Manual	Manual

---

⚡ Benchmark Results

Applying the automated suggestions from CUDA Insight yields massive speedups on our reference kernels on an NVIDIA A100 (sm_80):

Kernel	Naive Throughput	Optimized Throughput	Speedup	Bottleneck Fixed
Matrix Multi (4096)	~1.1 TFLOPS	~15.2 TFLOPS	13.8x	Global Memory Bandwidth
Reduction (10^8)	~12 GB/s	~1600 GB/s	133x	Atomic Contention -> Warp Shfl
Softmax	400 GB/s	1250 GB/s	3.1x	Two-pass Global Mem Reads

---

🚨 Troubleshooting

No profiling data captured (Google Colab)

If you run cuda-insight profile on Google Colab and get an error saying No profiling data captured, this is because Colab's hypervisor forcibly blocks hardware performance counters (ERR_NVGPUCTRPERM). Fix: You can use our software PyTorch profiler as an alternative:

cuda-insight profile-torch examples/profile_matmul.py

Invalid NCU Metrics

If your GPU architecture is older or newer, ncu may reject certain metrics requested by the tool. To see the exact error:

ncu --csv --page raw --metrics sm__warps_active.avg.pct_of_peak_sustained_active ./naive_matmul

---

🏢 Used By

• Deep Learning Researchers
• AI Performance Engineers

---

📝 Citation

If you use cuda-insight in your research, please cite:

@software{cuda_insight,
  author = {Paramveer Singh},
  title = {CUDA Insight: A Kernel Profiling, Debugging & Optimization Toolkit},
  year = {2026},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/Paramveersingh-S/cuda-insight}}
}

📝 License

MIT License.