Skip to main content

GPU (Graphics Processing Unit) Architecture

What is a GPU​

GPU (Graphics Processing Unit) was originally designed for graphics rendering. Its parallel architecture (thousands of small ALUs) naturally suits AI matrix operations. In 2007, NVIDIA released CUDA, transforming GPUs into GPGPU (General-Purpose GPU), kicking off the AI acceleration era.

Currently 90%+ of AI training and 70%+ of inference runs on GPUs. The CUDA ecosystem is the biggest moat.

GPU vs Other AI Chips​

DimensionGPUTPUASICNPU
General-purposeStrongestMedium (Google Cloud only)WeakMedium
Compute densityHighExtremely highHighMedium
EcosystemCUDA dominantJAX/TFVendor-proprietaryFragmented
Programming modelCUDA/OpenCL/SYCLXLAVendor SDKVendor SDK
PriceHighHigh (cloud only)MediumMedium
Best forGeneral AI / training / inferenceData center trainingInference optimizationEdge/on-device

Major Vendors​

NVIDIA (~90% AI GPU market share)​

  • Data Center: H100 / H200 / B100 / B200 / B300 / A100
  • Inference: L2 / L4 / L40S / T4
  • Workstation/Consumer: RTX 4090 / RTX 5090 / RTX 5080 / RTX 6000 Ada
  • Edge: Jetson Orin / Jetson Thor

AMD (~5% AI GPU market share)​

  • Data Center Training: MI250 / MI300X / MI300A / MI325X / MI350
  • Ecosystem: ROCm (CUDA alternative, performance lags)

Intel (~3% AI GPU market share)​

  • Data Center: Intel Max Series (Ponte Vecchio) / Flex Series
  • Consumer: Arc series
  • Integrated GPU: Meteor Lake / Lunar Lake NPU

Mainstream GPU Spec Comparison​

GPUArchitectureMemoryFP16 TensorTDPUse Case
NVIDIA B300 UltraBlackwell Ultra288GB HBM3e15 PFLOPS1,400WTop-tier training
NVIDIA B200Blackwell192GB HBM3e2.25 PFLOPS1,000WData center
NVIDIA H200Hopper141GB HBM3e1.98 PFLOPS700WTraining/inference
NVIDIA H100Hopper80GB HBM31.98 PFLOPS700WTraining/inference
NVIDIA RTX 5090Blackwell32GB GDDR7419 TFLOPS575WConsumer flagship
AMD MI355XCDNA 4288GB HBM3E10 PFLOPS1,400WTraining
AMD MI300XCDNA 3192GB HBM31.5 PFLOPS750WTraining

Programming Models​

CUDA (NVIDIA)​

  • CUDA C/C++ — low-level API
  • cuDNN — neural network primitives
  • cuBLAS — matrix operations
  • Triton — Python high-level compiler
  • Ecosystem: PyTorch / TensorFlow / JAX / vLLM

ROCm (AMD)​

  • HIP — CUDA-compatible API
  • MIOpen — deep learning library
  • Performance: ~70-90% of NVIDIA (workload-dependent)

SYCL / oneAPI (Intel)​

  • DPC++ — C++ + SYCL
  • oneMKL — math library
  • Smaller ecosystem, but open source

GPU Use Cases​

  • ✅ General AI training / inference (mature CUDA ecosystem)
  • ✅ LLM training (GPT-3 / LLaMA / Mixtral)
  • ✅ Stable Diffusion training
  • ✅ Scientific computing (HPC)
  • ✅ Multi-workload data center
  • ❌ Extreme energy efficiency (use ASIC)
  • ❌ Edge/on-device (use NPU)

Selection Guide​

WorkloadRecommended GPU
Training GPT-4 class modelsB200 / B300 / H200
Training 70B LLMH100 8-way / MI300X 8-way
Training 13B LLMH100 / A100
Inference 70B+ LLMH100 NVL / H200
Inference 13B LLML40S / L4
Stable Diffusion XLRTX 4090 / RTX 5090
Edge AI inferenceJetson Orin / Thor
HPC + AI jointMI300A / H100

Detailed Product Pages​

NVIDIA Data Center (H Series / A Series)​

NVIDIA Blackwell (B Series)​

NVIDIA Vera Rubin Platform (2026 H2)​

  • NVIDIA Vera Rubin R200 - 6-chip CoWoS-L package, 288GB HBM4 22 TB/s, 50 PFLOPS FP4 sparse, ConnectX-9 28.8 TB/s
  • Rubin NVL72 (1 rack): 72×R200 + 36×Vera, 1.4 EFLOPS FP4 sparse
  • Rubin NVL576 (8 racks): 576×R200 + 288×Vera, 28.8 EFLOPS FP4 sparse, 1.1 MW per room

NVIDIA Inference / Edge​

AMD​

  • AMD MI210 - CDNA 2 64GB HBM2e 22.6 TF FP64 PCIe, preferred for Europe's LUMI supercomputer
  • AMD MI250 - CDNA 2 128GB HBM2e
  • AMD MI300X - CDNA 3 192GB HBM3 5.3 TB/s
  • AMD MI300A - CDNA 3 + Zen 4 APU 128GB HBM3
  • AMD MI325X - 256GB HBM3e 6 TB/s 1.3 PF FP8
  • AMD MI355X - 288GB HBM3E 8 TB/s 4.6 PF FP8 UALoF 600 GB/s
  • AMD MI350 - CDNA 4 288GB HBM3E
  • AMD MI400 - CDNA Next 432GB HBM4 40 PF FP4 dense, Helios 72-GPU rack

Intel​