CUDA | ECOSYSTEM Research

CUDA works so well because it’s not just a library or API—it’s a co-designed hardware–software architecture where the GPU silicon, memory system, thread model, and programming model are all tuned for one thing: massive, fine-grained data parallelism. ieeexplore.ieee Below is what’s special about CUDA, at the level that matters if you’re thinking about “parallel states of computation.” 1. Hardware built for thousands of threads, not a few big ones 1.1. Many small cores instead of a few big ones - A modern NVIDIA GPU has dozens of Streaming Multiprocessors (SMs), each containing many CUDA cores (ALUs). nfina - Each CUDA core is relatively simple compared to a CPU core: - Optimized for arithmetic (FP32/FP64/INT, tensor ops). - Less logic for branch prediction, out-of-order execution, etc. This lets NVIDIA pack thousands of cores on one die, all capable of running in parallel. supermicro 1.2. Massive thread capacity with zero-cost context switching Key points from the CUDA architecture: - A single GPU can support hundreds of thousands of active threads concurrently (e.g., 2048 active threads per SM × 80 SMs → 160,000 threads). docs.nvidia - Each thread has its own register file allocation; when the scheduler switches between threads, there’s no register saving/restoring like on a CPU. docs.nvidia - The hardware can sleep warps waiting on memory and instantly run other ready warps, with effectively zero context-switch overhead. forums.developer.nvidia That’s very different from a CPU, where: - You might have 8–32 heavy cores. - Each core runs a small number of threads (often 1–2 hardware threads per core). - Context switching involves saving/restoring state and has non-trivial cost. On a GPU, the hardware is designed so that thousands of threads can be “live” at once, and the scheduler just picks which warp to issue each cycle. 2. Execution model: warps, SIMT, and latency hiding 2.1. Warps and SIMT execution - The smallest schedulable unit on a CUDA GPU is a warp: 32 threads that execute in lockstep. docs.nvidia - The warp scheduler issues one instruction for all 32 threads at once (SIMT: Single Instruction, Multiple Threads). forums.developer.nvidia - Modern GPUs dispatch these 32-wide SIMD instructions to many pipelines (e.g., 16–32 ALUs) per SM. forums.developer.nvidia This gives you: - Perfect fit for data-parallel workloads: same operation on many data elements. - Very high throughput when threads in a warp follow the same control flow. 2.2. Latency hiding via massive parallelism GPUs have high memory latency, but CUDA hides it by: - Having many more warps than can run at once on an SM. - When a warp stalls on a memory load, the scheduler instantly switches to another ready warp. forums.developer.nvidia - Because there are so many warps, the SM is almost always doing useful work, even if individual threads are waiting on memory. This is why CUDA wants you to launch thousands to tens of thousands of threads: it gives the hardware enough “slack” to hide latency and keep the pipelines full. isfpga 3. Programming model: grids, blocks, and kernels CUDA’s software model mirrors the hardware in a way that makes parallelism natural and scalable. 3.1. Kernels = “do this in parallel” You write a kernel: Then launch it like: Key ideas: milvus - Kernel: a function that runs on the GPU, executed by many threads in parallel. - Threads are grouped into blocks, blocks into a grid. - You specify how many threads, not which core runs which thread; the hardware schedules blocks onto SMs automatically. This matches your “parallel states of computation” intuition: you describe one computation (the kernel) and then say “run it over this many independent data elements,” and CUDA maps that to thousands of hardware threads. 3.2. Hierarchical decomposition = easy scaling - A problem is decomposed into: - Grid: global problem. - Blocks: chunks that fit on an SM and can share fast memory. - Threads: individual elements (e.g., one pixel, one particle, one matrix element). - This structure lets the same code scale from small to huge GPUs without changes; the runtime just maps blocks to available SMs. isfpga 4. Memory hierarchy tuned for parallel math CUDA GPUs have a deep, explicit memory hierarchy that’s crucial for performance: milvus - Global memory: large, high-latency DRAM shared by all SMs. - Shared memory: small, very fast on-chip memory per SM, shared by threads in a block. - Registers: per-thread, fastest storage. - Caches: L1/L2 for global memory. Why this matters: - Threads in a block can cooperate via shared memory, reducing expensive global memory accesses. milvus - Algorithms like matrix multiply, convolutions, and particle simulations are structured so that: - Each block loads a tile of data into shared memory. - Threads operate on that tile with many reuses. - This yields high arithmetic intensity (lots of FLOPs per byte fetched), which is where GPUs crush CPUs. nfina 5. Software ecosystem: libraries, compilers, and too…