Jensen Huang heavily invested in CUDA (Compute Unified Device Architecture) at a time when most graphics card buyers had no use for it. This software platform enabled graphics processing units (GPUs) to be used not just for gaming and rendering, but also for accelerating parallel computing workloads.
Supporting these new capabilities drove up chip production costs, even as shareholders demanded higher profitability. However, a few years later, this strategic bet positioned NVIDIA at the forefront of the deep learning revolution, making its hardware the cornerstone of modern neural network training.
By the early 2000s, graphics processing units (GPUs) were already capable of rapidly executing massive numbers of identical operations in parallel. This architecture was highly suitable for scientific computing: for instance, during matrix multiplication, different elements of the resulting matrix could be calculated simultaneously.
However, accessing this computational power was strictly tied to graphics pipelines. Researchers were forced to encode numerical data into textures and trigger computations using rendering commands. To solve a mathematical problem, one first had to frame it as an image processing task — essentially, manipulating triangles. [1]
One of the pioneers attempting to break down this barrier was Ian Buck at Stanford University. In 2004, his team introduced Brook for GPUs, a system that allowed developers to describe parallel computations using extensions to the C programming language. That same year, Buck joined NVIDIA, where he continued to expand upon this foundational work. [2]
The next major milestone was the CUDA platform, introduced by NVIDIA in 2006. In February 2007, the company released a public beta of its developer tools. Programmers were finally given a direct way to describe parallel computations, a compiler to translate that code into executable programs, and robust tools for memory management and task execution on the GPU. [3]. [4]
Developers could now write a function in C using CUDA extensions and execute it simultaneously across thousands of array elements. The CPU managed the overall application, while the GPU handled the specific workloads best suited for hardware acceleration. While developers still had to manually distribute computations and account for memory architecture, they no longer needed to disguise their mathematical tasks as 3D rendering operations.
To drive platform adoption, Jensen Huang made CUDA support a standard feature in mass-market GeForce graphics cards, which were primarily designed for gaming. This allowed researchers to access high-performance compute power using readily available consumer hardware. Simultaneously, NVIDIA partnered with universities and developed comprehensive educational resources.
The company was effectively building a future market from both sides: supplying compatible silicon and training a new generation of developers to write software for it. However, mass-market demand for such software did not yet exist; developers still had to discover practical, real-world applications for these new GPU capabilities. [5]
The mass adoption of CUDA came at a steep price. Most consumers bought graphics cards strictly for gaming and had no intention of paying a premium for scientific computing capabilities. According to Huang, these additional hardware features drove up manufacturing costs and compressed gross margins — the percentage of revenue remaining after deducting the cost of goods sold. [6]
He later recalled:
“Our shareholders were skeptical of CUDA and preferred that we focus on improving profitability.” [7]
The strategy required immense patience: NVIDIA bore the R&D costs upfront, while future demand depended entirely on what practical applications third-party developers could invent. And invent they did. One of the earliest and most significant breakthroughs for CUDA emerged in the field of computer vision and image recognition.
AlexNet Proved CUDA’s Potential
In Geoffrey Hinton’s research group at the University of Toronto, GPUs were already being leveraged to train neural networks, including for speech recognition tasks. Alex Krizhevsky and Ilya Sutskever turned their focus to image recognition using Convolutional Neural Networks (CNNs). These networks apply mathematical filters across different regions of an image to detect salient features and patterns. During training, the network's predictions are compared against ground-truth labels, and the algorithm iteratively adjusts the underlying numerical parameters (weights) to improve accuracy. This cycle is repeated across massive datasets, requiring vast amounts of highly parallelizable computations — a workload perfectly suited for GPU architecture. [8], [9]
Krizhevsky had already developed a CUDA-based framework called cuda-convnet specifically for training these networks. Sutskever proposed applying it to ImageNet, a massive, meticulously labeled dataset of images. Notably, this dataset was the foundation of a prestigious annual object recognition competition. [10], [11]
To tackle this larger-scale challenge, Krizhevsky modified the code to support multi-GPU training. Conducted under Hinton’s supervision, this work culminated in the creation of the neural network famously known as AlexNet. Remarkably, the rig—equipped with two NVIDIA GPUs — was set up in Krizhevsky’s bedroom at his parents’ house. [12].
Training the network, as detailed in their seminal paper, took five to six days running on two GeForce GTX 580 GPUs. In 2012, the AlexNet-based system decisively won the ImageNet competition: its top-5 error rate was just 15.3%, drastically outperforming the runner-up’s 26.2%. [13].
CUDA’s contribution was undeniable: it enabled the massive optimization of neural network computations on the GPU. While the model's ultimate accuracy was driven by its architecture, training methodologies, and the dataset, it was CUDA that made it possible to execute the entire training pipeline at a speed that was unprecedented at the time.
AlexNet’s victory provided a compelling demonstration of deep learning’s technical potential. From a business perspective, however, a critical question remained: how large of a commercial market could actually emerge from these academic research results? The industry still had to identify practical applications, translate them into viable products, and cultivate sustainable market demand. NVIDIA continued to pour resources into CUDA, knowing full well that academic success did not guarantee a return on investment. According to Huang, early applications in scientific and engineering computing helped sustain the platform’s development, but for a long time, they failed to deliver the massive business scale NVIDIA had envisioned. [14].
In 2013, NVIDIA gained a new shareholder: Starboard Value, an activist investment fund known for aggressively intervening in corporate management to drive up share prices. As Tae Kim details in his book The Nvidia Way, NVIDIA executives feared the new investor might push for a board shakeup and demand cuts to CUDA funding, given that its commercial prospects remained highly uncertain at the time.
However, that conflict never materialized. Starboard’s primary objective was a share buyback program—returning capital directly to shareholders. NVIDIA announced the buyback, its stock price surged, and by March 2014, the fund had divested its stake. Meanwhile, the company remained fully committed to advancing CUDA. Years later, following NVIDIA’s exponential growth in market value, Starboard’s founder, Jeff Smith, publicly admitted: “We shouldn’t have exited that investment.” [15]
NVIDIA consistently worked to lower the barrier to entry for neural network development. In 2014, the company introduced the CUDA Deep Neural Network (cuDNN) library, a suite of highly optimized, pre-built primitives for deep learning. By integrating cuDNN with frameworks like Caffe, researchers could define complex network architectures without having to write custom CUDA code for basic computational operations. [16]
The underlying logic of this strategy was clear: NVIDIA took on the heavy lifting of hardware acceleration, freeing developers to focus on experimenting with models and discovering practical, real-world applications. The company was essentially building the foundational tools for a market that was still in its infancy.
This simplification extended to hardware. In 2016, NVIDIA unveiled the DGX-1: a turnkey server featuring eight Tesla P100 accelerators, NVLink interconnects for high-speed GPU-to-GPU communication, and pre-installed deep learning software. Teams received a fully integrated compute system powered by the CUDA ecosystem, including cuDNN, allowing them to bypass infrastructure setup and move straight to experimentation. [17]
That August, Jensen Huang personally delivered the very first production DGX-1 to OpenAI, which was then operating as a non-profit research organization. Among its researchers was Ilya Sutskever: a key contributor to the original AlexNet project was now receiving a purpose-built system for his next wave of neural network experiments. [18]
In May 2017, NVIDIA introduced Volta, the seventh generation of its GPU microarchitecture, succeeding Pascal. The microarchitecture dictates the processor's internal design: its compute units, memory handling, and data communication pathways. The first accelerator based on this architecture was the data center-focused Tesla V100. According to NVIDIA, thousands of engineers worked on Volta for over three years, specifically addressing the escalating demands of deep learning and scientific computing. [19].
The V100 combined 5,120 standard CUDA cores with 640 new Tensor Cores, which were highly specialized for matrix operations. NVIDIA also redesigned the compute blocks and parallel thread scheduling. Fast HBM2 memory provided a bandwidth of up to 900 GB/s, a significant jump from the previous generation P100's 732 GB/s. This ensured the compute units were continuously fed with the massive amounts of data required for efficient operation.
To interconnect the accelerators, NVIDIA utilized the second generation of NVLink: the aggregate bandwidth of a single GPU's connections surged from 160 GB/s to 300 GB/s. These improvements addressed all three critical pillars of training large neural networks: compute execution, memory data access, and GPU-to-GPU communication. The V100 subsequently became the foundation for the updated, eight-accelerator DGX-1 system. [20]
Support for Tensor Cores was integrated directly into CUDA 9, as well as the cuBLAS and cuDNN libraries. In this way, NVIDIA tightly coupled hardware evolution with established software tools: developers gained immediate access to cutting-edge hardware capabilities through familiar, out-of-the-box library functions. [21]
The emergence of the Transformer architecture marked a pivotal turning point in the evolution of neural networks. Later that same year, in 2017, Google researchers introduced the Transformer, an architecture featuring an attention mechanism capable of modeling relationships between all parts of an input sequence.
During training, it could process sequence positions in parallel. This eliminated a major bottleneck of sequential computation inherent in the then-dominant Recurrent Neural Networks (RNNs), allowing for vastly more efficient GPU utilization. A significant portion of a Transformer's workload boils down to matrix operations—the exact type of computation NVIDIA’s accelerators and software libraries were already perfectly optimized to handle. The initial models described in the seminal Attention Is All You Need paper were trained on eight P100 GPUs. [21]. [22]
Transformers paved the way for a new scale of language models. They could be pre-trained on massive corpora of text and subsequently fine-tuned for a wide variety of downstream tasks. This created a uniquely favorable convergence for NVIDIA: a highly promising architecture that demanded colossal amounts of compute, met by a company that had spent over a decade meticulously preparing the hardware and software ecosystem to accelerate exactly that.
In 2018, OpenAI applied the Transformer architecture to create the Generative Pre-trained Transformer (GPT). The model was first trained to predict the next word in a massive corpus of books, after which it was fine-tuned for specific language tasks. This initial pre-training phase took a full month running on eight GPUs. [23]
Further scaling required distributing both the computations and the model itself across multiple accelerators. In 2020, GPT-3, with its 175 billion parameters (the numerical weights adjusted during training), was trained on V100 GPUs within a Microsoft cluster. The authors distributed both the matrix operations and the network layers across the GPUs to fit within the available memory constraints. [24]
In November 2022, OpenAI launched ChatGPT, built on the GPT-3.5 family of models. Transformer development had now evolved into consumer-facing services that users could interact with using natural language. [25].
By 2024, NVIDIA was explicitly attributing the explosive growth of its data center business to the surging demand for training and deploying generative models from cloud providers and enterprises across diverse industries. [26]
By this point, NVIDIA could offer developers a complete, end-to-end computing ecosystem: accelerators, pre-built libraries, native framework support, and systems designed for seamless multi-GPU collaboration. For teams recognizing the potential of these new models, it was vastly easier to start building with tools that were already battle-tested and ready to use. NVIDIA’s years of foundational work on CUDA had finally found its ultimate application in a rapidly expanding market.
For years, the company nurtured a platform whose mass-market demand remained highly uncertain, actively helping developers discover practical applications for it. Deep learning, and subsequently the Transformer architecture, transformed this long-term preparation into an insurmountable strategic advantage. By the time demand for neural network compute exploded, NVIDIA already possessed a fully aligned ecosystem: compatible hardware, mature software tools, and a thriving community of developers who knew exactly how to leverage them.
Admittedly, competitors have since introduced their own accelerators and software platforms, most notably AMD’s ROCm ecosystem. However, persuading developers to migrate away from CUDA means overcoming massive switching costs: porting legacy code, replacing dependencies, rigorously validating correctness, and re-tuning the system to achieve optimal performance. Even automated migration tools cannot entirely eliminate this friction. [27]
Consequently, NVIDIA’s lead continues to compound. Developers protect their existing investments in a familiar environment, while NVIDIA continuously evolves its silicon and software, layering new capabilities into that same entrenched ecosystem. Competitors are forced to battle not just a single product, but an entire, deeply integrated system, and they must constantly convince engineering teams that the pain of migration will ultimately justify the cost.
A software platform built long before mass-market demand existed ultimately became the primary reason NVIDIA captured that demand when it finally arrived. This is precisely how CUDA changed NVIDIA’s fate. [28]