CUDA

The GPU and TPU articles covered AI hardware. This article covers something different but equally essential — the software layer that actually lets programmers use a GPU’s parallel hardware for anything beyond graphics at all: CUDA.

The simple definition

CUDA is Nvidia’s software platform that lets programmers write code directly for a GPU’s parallel cores, using ordinary programming languages, rather than having to speak the specialized graphics-only instructions GPUs originally understood. Recall from the GPU article’s core distinction — a GPU’s thousands of cores happen to suit AI computation well. CUDA is the actual, practical bridge that makes using those cores for AI (or any general computation) genuinely possible, rather than a GPU remaining locked into graphics-only rendering commands.

Why a GPU needed this software layer at all

Recall from the GPU article that graphics chips were originally built and programmed exclusively for rendering — taking in geometry and textures, outputting pixels. Before CUDA, using a GPU’s parallel cores for anything else meant disguising your computation as a graphics problem, a genuinely awkward, indirect workaround few researchers were willing to attempt. CUDA solved this directly, in 2006, by giving developers a real, general-purpose programming interface to the GPU’s parallel cores — write code in a language much closer to ordinary C, and CUDA handles distributing the work across thousands of cores underneath.

flowchart LR
    A[GPU hardware: thousands of parallel cores] --> B[CUDA: general-purpose programming interface]
    B --> C[Developers write ordinary code]
    C --> D[CUDA distributes execution across the GPU's cores automatically]

The real story: a six-year bet with no clear payoff in sight

This deserves to be told directly, because it’s a genuine, well-documented example of a company investing heavily in something years before the market justified it. Nvidia released CUDA in 2006, explicitly framed as opening up GPU parallel processing to science and research generally, not gaming.

For years afterward, CUDA had no obvious, massive commercial application — Nvidia’s revenue remained overwhelmingly tied to gaming graphics cards, and building and maintaining an entirely separate software platform was a genuine, ongoing cost with no proven return. It wasn’t until 2012, when AlexNet — covered in the Computer Vision article — trained on Nvidia GPUs using CUDA and triggered the modern deep learning revolution, that the bet’s real value became visible.

Nvidia’s own corporate history explicitly marks 2012 as the moment the company “sparked the era of modern AI,” a full six years after CUDA had originally shipped with no guarantee that moment would ever come.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of building and maintaining an entire universal translator years before you know which foreign country you’ll actually need to negotiate with — a genuine, sustained investment made on the belief that being able to communicate broadly will eventually matter, well before any specific, profitable use case has emerged.

Where this breaks down: A universal translator is a fixed, general tool. CUDA is a continuously developed, deeply technical software platform, with Nvidia investing ongoing engineering effort into optimizing it specifically for the kinds of computation that turned out to matter most — increasingly, the exact matrix multiplication covered throughout the Neural Networks phase — rather than a one-time tool built once and simply left unchanged.

Why this software advantage became a genuine, durable competitive moat

This is worth being direct about, since it explains a real, current market reality. Because CUDA had a decade-long head start, the vast majority of AI research software, frameworks, and tooling — PyTorch, TensorFlow, and the libraries most of this glossary’s real, named models were built with — was written and optimized specifically for CUDA.

This creates a genuine, self-reinforcing advantage: even a competitor building excellent GPU hardware would still need an equivalent software ecosystem, built up over years, before researchers could easily switch — a major, real reason AMD’s competing GPUs, despite genuine hardware capability, have struggled to gain equivalent AI market share.

A concrete example, layered

For a simple beginner example: a researcher writing a small neural network in PyTorch, covered indirectly throughout this glossary’s real-system examples, is relying on CUDA underneath every single training step, translating PyTorch’s high-level code into instructions the GPU’s parallel cores can actually execute. For a production example: virtually every major AI lab referenced throughout this glossary — OpenAI, Anthropic, Google DeepMind, Meta — trains and runs its models on Nvidia GPUs using CUDA as the underlying software layer, a real, dominant market position directly traceable to Nvidia’s 2006 bet.

Common misconception

The CUDA layers, from highest to lowest

PyTorch model
    -> CUDA-enabled library such as cuBLAS or cuDNN
    -> CUDA runtime and driver
    -> GPU kernel
    -> NVIDIA GPU hardware

• A kernel is a GPU function executed by many parallel threads.
• A library provides optimized kernels for common operations such as matrix multiplication.
• The runtime manages work such as launching kernels and allocating device memory.
• The driver communicates with the physical GPU.

A developer may write PyTorch code without writing CUDA kernels, while still relying on all these CUDA layers underneath.

A named-model example

NVIDIA documented running Falcon-180B with TensorRT-LLM on one H200 GPU after applying INT4 AWQ optimization. CUDA supplies the NVIDIA execution foundation, while TensorRT-LLM adds model-specific kernels, quantization, batching, and KV-cache management.

This example shows why “the model uses CUDA” is incomplete. Its performance comes from the model, numerical format, GPU, CUDA libraries, serving engine, and request settings working together.

What happens when PyTorch uses CUDA

Most developers do not write every GPU instruction themselves. A simplified path looks like this:

Python / PyTorch operation
        -> CUDA-enabled library chooses a GPU kernel
        -> kernel runs many threads on the NVIDIA GPU
        -> result remains in VRAM for the next operation

A kernel here means a small program executed many times in parallel on the GPU. Moving data between ordinary RAM and VRAM can be expensive, so frameworks try to keep related calculations and tensors on the GPU.

CUDA is not the GPU

• The GPU is physical hardware.
• CUDA is NVIDIA’s platform, programming model, compiler tools, runtime, and libraries for using compatible NVIDIA GPUs.
• PyTorch and TensorFlow are higher-level frameworks that can call CUDA underneath.

CUDA code does not run on every accelerator unchanged. Google TPUs use a different hardware and compiler stack, while AMD GPUs commonly use ROCm. This software compatibility is one reason hardware choice affects much more than chip specifications.

A real production connection

NVIDIA’s current TensorRT-LLM stack uses CUDA-based GPU kernels together with features such as in-flight batching, paged KV caching, multi-GPU execution, and FP8, INT8, or INT4 quantization. This shows CUDA’s role clearly: the model sits above it, the NVIDIA GPU sits below it, and optimized serving software connects the two.

Verified sources

• NVIDIA CUDA documentation
• NVIDIA TensorRT-LLM documentation
• NVIDIA Falcon-180B TensorRT-LLM example

A frequent beginner assumption: that “GPU” and “CUDA” are simply two names for the same thing. As this article has explained, they’re genuinely distinct — a GPU is the physical hardware, capable of massive parallel computation regardless of any specific software; CUDA is Nvidia’s own particular software platform for programming that hardware, and it’s specifically tied to Nvidia’s chips, which is exactly why competing GPU makers need their own separate software platforms to compete.

Where this fits in what comes next

You now understand the hardware and the software layer that together make AI computation possible. The next article, VRAM, covers a genuinely practical, everyday constraint anyone working with GPUs runs into directly — how much memory is actually available on the chip, and why that number ends up limiting nearly everything else covered throughout this phase.

In one sentence

CUDA is Nvidia’s software platform for programming GPUs for general-purpose computation, released in 2006 as a genuine, years-long bet with no clear payoff in sight, and its decade-long head start — validated only when AlexNet trained on it in 2012 — became the durable software advantage underlying Nvidia’s current dominance of the AI hardware market.

Related Terms

• GPU
• TPU
• Computer Vision
• Neural Network
• VRAM