GPUs evolved from fixed-function graphics pipelines to programmable parallel processors for AI. Shaders, then CUDA/OpenCL, exposed ALUs for general computing. AlexNet catalyzed deep learning on GPUs; vendors added tensor cores, mixed precision, HBM, and interconnects. Training now pursues throughput and memory, while inference targets latency and efficiency using quantization and sparsity. The stack matured—TensorFlow, PyTorch, TensorRT, XLA—driving data center scale and edge deployments. Consumer GPUs recycle AI features like DLSS. Today’s landscape spans GPUs, NPUs, CIM, neuromorphic, and CPUs across training, cloud, edge, vehicles, and IoT, and all these aspects are intertwined between many human endeavors including games, AI, content creation, and education.
The evolution of graphics processing units (GPUs) has been remarkable, transforming from specialized graphics rendering engines to powerful parallel processor accelerators and AI training and inference engines.
Graphics processors began as fixed-function pipelines for rasterizing triangles, shading pixels, and compositing frames. Early GPUs introduced hard-wired stages like transform, lighting, and texture mapping to maximize frame rates under tight memory and power limits. The first inflection point arrived with programmable shaders: vertex and pixel shaders (later geometry, hull, domain, and compute) exposed vector Arithmetic Logic Units (ALUs) and register files through high-level languages such as High Level Shading Language (HLSL) and OpenGL Shading Language (GLSL). Suddenly developers had the keys to the palace. They could write their own shaders to control the GPU. Further, developers wrote short programs that ran in parallel on thousands of lightweight threads, establishing the single instruction, multiple thread (SIMT/SIMD) execution style that still defines GPUs. Around 2004, labs started to push scientific kernels through shader pipelines. The fit looked natural: wide vectors, high throughput, and predictable access patterns.
Nvidia answered with CUDA in 2006 and invited C/C++ programmers to launch kernels straight at their GPU cores. OpenCL followed and opened the door across graphics hardware vendors. Scientists ported climate models, fluid dynamics, cryptography, and finance to GPUs. Universities leaned in. By 2009, teams at Stanford and NYU trained neural networks on GPUs and cut experiments from weeks to days. General-purpose GPU computing stepped out of the lab and into mainstream workflows.
Deep learning then hit the accelerator. AlexNet, developed by Alex Krizhevsky, Ilya Sutskeyer, and Geoffrey Hinton, demonstrated how powerful deep learning and image recognition accelerated by GPUs could be. AlexNet opened the door for facial recognition, object detection, and automated image tagging. AlexNet’s 2012 ImageNet win showed that convolutional networks (CNNs) plus GPUs could crush prior records and training times. Vendors doubled down. Nvidia’s Tesla line targeted compute first, not gaming. Pascal added mixed precision to speed training with FP16 while preserving accuracy through clever accumulation. Volta introduced Tensor Cores, hard units for matrix multiply-accumulate that turbocharged training loops. Ampere and Ada broadened the menu with FP32, FP16, BF16, FP8, INT8, and even INT4 so you could trade precision for speed and energy headroom. In answer, AMD advanced ROCm open-source software platform and MI-series GPUs. Toolchains such as cuDNN, MIOpen, and oneDNN wrapped tuned kernels for convolutions, attention, and normalization.
Figure 1. AI-based ray tracing brought a new era to game playing. (Source: CD Projekt)
Memory and interconnects had to keep up. GDDR pushed bandwidth for years, then High Bandwidth Memory (HBM) arrived with stacked DRAM and terabytes-per-second per package. Architects widened crossbars, fattened caches, and tuned schedulers to hide latency. NVLink and NVSwitch tied multiple GPUs into one logical pool. Across nodes, InfiniBand and Ethernet with RDMA carried gradients during data, tensor, and pipeline parallelism. You could slice a single GPU into isolated instances for multi-tenant jobs or fuse eight of them into a training appliance. Checkpointing, ZeRO-style optimizers, and sharding let teams scale to models with hundreds of billions of parameters.
Training and inference then diverged in priorities. Training chases throughput, robust numerics, and memory capacity. Inference chases latency, cost per token, and power envelopes that fit crowded racks or quiet edge boxes. You tune the same silicon differently. For training, you drive high utilization with long fused kernels and large batches. For inference, you quantize to INT8 or INT4, prune with structured sparsity, and fuse kernels to cut launch overheads. You also stream keys and values through paged caches to serve long contexts without blowing memory budgets. The outcome feels simple to users: faster answers and cheaper tokens.
The GPU story also looped back to gaming in a virtuous circle. Tensor units that trained data center models began to upscale frames in your living room. DLSS and FSR predict missing pixels, generate intermediate frames, and denoise ray-traced scenes. Ray-tracing cores traverse bounding-volume hierarchies, while tensor units clean the image, so games gain fidelity without throwing performance out the window. One stack feeds the other; gamers benefit from AI research, and AI models benefit from years of graphics optimization.
A strong software ecosystem had held everything together. TensorFlow, PyTorch, and JAX abstracted device specifics and scheduled graphs across thousands of threads. Compilers and runtimes—TensorRT, XLA, TVM, and friends—fused ops, placed buffers, and mapped math to tensor units. Vulkan and DirectX 12 exposed explicit compute paths next to graphics queues, while Khronos’ SYCL and AMD’s HIP C++ programming models broadened portability. Profilers visualized stalls, captured graphs, and helped chase stray milliseconds. The result feels friendly at the keyboard: Write high-level code, get near-metal performance.
Vendors continued to stretch the definition of a GPU. Data center parts emphasize memory capacity, high-speed interconnects, and multi-instance features over gaming luxuries. Consumer graphics boards kept the display output, stream encoders, and ray-tracing blocks, yet share the tensor engines that serve LLMs. Chiplet designs promise larger logical GPUs with better yields. CXL aims to pool memory beyond the socket. Optical links inch closer to the package to push bandwidth without cooking the board. Every step favors dense math, fast memory, and efficient movement of tensors.
Edge deployments add its own flavor. Cars, drones, cameras, and phones run tight loops under strict power caps. Developers squeeze latency with on-chip SRAM reuse, tile-friendly layouts, and domain codecs for vision and speech. They also hand off work to small NPUs when that path saves energy or wins determinism. The boundary between GPU, NPU, and accelerator blurs as designs specialize for the same math with slightly different trade-offs.
AI processing capabilities spread from training and cloud inferencing to edge inferencing, vehicles including amatory and autonomous robots, to AI IoT devices, smart and adaptable sensors, and smart home devices including toy companions.
Figure 2. Population of AIP suppliers.
We found 138 companies making or designing AI processors (AIPs). They’re scattered across 18 countries and are focused on the five primary segments: AI training, cloud inferencing, AI edge devices, vehicles and robots, and AI IoT devices. There are actually two more categories: AI in vision, which falls somewhere between edge and IoT, and Network-on-Chip (NoC), an on-chip interconnect where IP blocks (“tiles” like CPU cores, NPUs, SRAM, HBM controllers, DMA, etc.) sit on a 2D grid.
The type of AIPs has also expanded, and there are four major processor types with three subcategories: GPUs, NPUs, compute-in-memory (CIM), and neuromorphic; while CPUs subdivide into x86, RISC-V, and Arm.
Figure 3. Heat map of AIP function/capabilities.
The future of GPUs looks promising, with advancements in denser tensor fabrics, smarter compilers, and tighter coupling across nodes. Sustainability will become a key focus, with operators prioritizing efficiency and regulators pushing for greater transparency and disclosure. Consumer hardware will increasingly rely on AI for upscaling, creator tools, and gameplay logic.
Looking back, GPUs evolved from specialized graphics engines to general-purpose parallel computers. Scientists and machine learning applications leveraged this architecture, leading to the addition of tensor units, high-bandwidth memory, and fast links. Today, GPUs balance three identities: real-time graphics engines, parallel math accelerators, and AI training and inference platforms.
This journey has reshaped industries, enabling richer games, faster medical imaging, and real-time risk scoring. The core idea behind GPUs—doing the same math across many data elements at once—has been amplified and specialized. As the industry continues to evolve, GPUs will remain a crucial component, driving innovation from gaming to data centers and beyond. The future of GPUs is bright, with potential applications in AI, sustainability, and more.
LIKE WHAT YOU’RE READING? DON’T KEEP IT A SECRET, TELL YOUR FRIENDS AND FAMILY, AND PEOPLE YOU MEET IN COFFEE SHOPS.