GEMM (General Matrix Multiplication) is the operation behind every layer of a neural network: every linear layer, attention projection, and fully-connected layer eventually reduces to it. This project builds a CUDA implementation of GEMM from first steps: starting from a naive, one-thread-per-output kernel, profiling it with Nsight Compute to find the exact bottleneck, and then rebuilding it with shared-memory tiling.
The tile size (TILE_WIDTH=16 vs. 32) was not guessed.It was decided using Nsight-measured Duration, occupancy, and DRAM bandwidth data. Shared-memory access patterns were checked for bank conflicts at both sizes (zero found at either). The final kernel is wrapped with pybind11 so it is directly callable from Python on NumPy arrays, and cross-validated for identical output correctness against a CPU reference and PyTorch/cuBLAS across five matrix sizes.
At 4096×4096, the kernel reaches 932.9 GFLOPS which is about 11.5% of the T4's FP32 hardware peak, and roughly 26% of PyTorch/cuBLAS's measured throughput on the same hardware. That remaining gap comes from register blocking, double buffering, and hand-tuned instruction scheduling that cuBLAS has and this kernel doesn't which the next layer i'm trying to close.
The trade: use the slow global-memory cost once per tile,
then read from fast shared memory TILE_WIDTH times.
Every element loaded into shared memory gets reused by many threads
before being evicted, instead of every thread re-reading global
memory independently which would be a waste of occupancy.
Benchmarked TILE_WIDTH=16 vs. 32 with Nsight Compute at 2048×2048. Results: TILE=32 won, about ~9.4% faster Duration, higher occupancy, and lower DRAM bandwidth which reflects greater shared-memory data reuse per global load, not reduced throughput.
| Metric | TILE=16 | TILE=32 |
|---|---|---|
| Duration (Nsight) | 46.26ms | 41.89ms |
| Achieved occupancy | 99.63% | 99.97% |
| DRAM read bandwidth | 92.49 GB/s | 39.43 GB/s |
Shared-memory access patterns were profiled for bank conflicts
at both tile sizes using Nsight Compute's load/store conflict
metrics.
Zero conflicts were found at either size, meaning the kernel's
existing access pattern was already conflict-free (no
[TILE][TILE+1] padding fix was needed.)
| Tile Size | Load Conflicts | Store Conflicts |
|---|---|---|
| 16 | 0 | 0 |
| 32 | 0 | 0 |
All five sizes cross-validated for identical output correctness: CPU, naive CUDA, and tiled CUDA all produce the same result at every size tested.
| Matrix Size | CPU ms | Naive CUDA ms | Tiled CUDA ms | PyTorch CUDA ms | Tiled GFLOPS |
|---|---|---|---|---|---|
| 256×256 | 20.647 | 0.218 | 0.207 | 0.057 | 162.1 |
| 512×512 | 203.569 | 1.005 | 0.599 | 0.116 | 448.1 |
| 1024×1024 | 3509 | 5.574 | 5.355 | 0.608 | 401.0 |
| 2048×2048 | 65316.9 | 69.062 | 28.7* | 4.203 | 598.6 |
| 4096×4096 | 813709 | 330.641 | 147.31* | 38.039 | 932.9 |
*Averaged warm-run timing after discarding cold-start outliers on Colab's shared T4.
Simon Boehm's "How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance" walks through the naive-to-tiled-to-register-blocked progression this project focuses on.
NVIDIA's own Developer Blog covers high-performance matrix multiply techniques in CUDA in more depth than this project currently implements register blocking and double buffering being the next layer to fix the gap to cuBLAS.
CUTLASS / CCCL is NVIDIA's own template library for exactly this class of kernel, and a natural next reference point once this project moves past hand-written tiling.
