Skip to main content

GPU Memory System & Interconnect: How Data Travels Inside a Modern GPU

Image
By

GPU Memory System & Interconnect: How Data Travels Inside a Modern GPU

Modern GPUs deliver incredible computing power by processing thousands of operations simultaneously. While compute cores often receive the spotlight, the real performance of a GPU depends heavily on how efficiently data moves between memory and compute units.

A GPU memory system is designed to provide extremely high bandwidth while keeping latency as low as possible. This is achieved through a collection of specialized hardware components working together, including the Memory Controller, HBM/GDDR memory, Crossbar, Network-on-Chip (NoC), DMA Engine, PCIe Interface, and NVLink.

In this article, we'll explore each component in detail and understand how they collectively enable high-performance parallel computing.


Why Does GPU Memory Matter?

Every GPU application—from gaming and graphics rendering to AI model training and scientific simulations—requires continuous movement of enormous amounts of data.

For example:

  • A deep learning model may read hundreds of gigabytes of weights.
  • A graphics application constantly streams textures.
  • Scientific simulations repeatedly load matrices into compute cores.

If the compute cores wait for data, performance drops dramatically.

The GPU memory system is therefore designed to keep thousands of processing cores busy all the time.

1. Memory Controller

What is it?

The Memory Controller acts as the traffic manager between the GPU and external memory (HBM or GDDR).

It determines:

  • Which memory request should be served first
  • Which memory channel to use
  • How addresses are translated
  • When refresh operations occur
  • Error detection and correction (ECC)

Without the memory controller, the GPU would have no organized way of accessing external memory.

Main Responsibilities

Address Mapping

Programs use logical addresses.

The memory controller converts them into physical DRAM addresses.



Request Scheduling

Thousands of GPU threads generate memory requests simultaneously.

The controller schedules them efficiently by:

  • minimizing row activations
  • reducing conflicts
  • maximizing bandwidth utilization

ECC Handling

Many AI and HPC GPUs use ECC memory.

The memory controller:

  • generates ECC bits during writes
  • checks ECC during reads
  • detects memory corruption

Why It Matters

An efficient memory controller ensures:

  • High throughput
  • Low latency
  • Better utilization of HBM bandwidth
  • Reduced memory conflicts

2. HBM / GDDR Memory

External memory stores almost all application data.

Examples include:

  • Neural network weights
  • Input tensors
  • Images
  • Textures
  • Buffers
  • Simulation data

HBM (High Bandwidth Memory)



Advantages

  • Extremely wide buses
  • Lower power
  • Very high bandwidth
  • Short electrical connections

Modern HBM interfaces may exceed 1000 GB/s of bandwidth per stack.

GDDR Memory

Gaming GPUs usually use GDDR memory.

Characteristics:

  • Lower cost
  • High clock frequency
  • Slightly higher latency
  • Narrower interface compared to HBM

HBM vs GDDR

FeatureHBMGDDR
BandwidthExtremely HighHigh
PowerLowerHigher
Packaging3D StackedDiscrete Chips
CostExpensiveAffordable
Primary UseAI, HPCGaming

3. Crossbar

The Crossbar is an internal switching network.

It connects:

  • Memory Controllers
  • L2 Cache slices
  • Compute Clusters
  • DMA Engines
  • Other GPU subsystems

Instead of fixed connections, the crossbar dynamically routes data where it is needed.

Why Is It Important?

Suppose:

  • SM0 needs Memory Controller 1
  • SM1 needs Memory Controller 2
  • SM2 needs L2 Cache
  • DMA Engine needs HBM

All of these transfers can happen simultaneously through the crossbar.

SM0 ──►
SM1 ──►
SM2 ──► Crossbar ──► Memory Controllers
DMA ──►

This greatly improves parallelism.

4. Network-on-Chip (NoC)

As GPUs grow larger, connecting every component with dedicated wires becomes impractical.

Instead, modern GPUs use a Network-on-Chip (NoC).

A NoC works similarly to a computer network.

Instead of packets traveling between computers, packets travel between hardware blocks inside the chip.

Common NoC Topologies

Mesh

Most modern GPUs use a 2D mesh.

Advantages:

  • Scalable
  • Easy routing
  • Good bandwidth

Ring

Used in smaller chips.

Advantages:

  • Simple
  • Low hardware cost

Disadvantage:

  • Poor scalability

Hierarchical

Combines multiple smaller networks.

Useful for very large AI accelerators.

Benefits

  • Scalable architecture
  • Lower wiring complexity
  • High throughput
  • Better power efficiency

5. DMA (Direct Memory Access)

Normally, moving data requires compute cores to participate.

DMA removes this overhead.

A DMA engine copies data directly between memory locations without involving Streaming Multiprocessors (SMs).







Common Uses

  • Loading AI tensors
  • Copying textures
  • Host-to-GPU transfers
  • GPU-to-host transfers
  • Memory initialization

Advantages

  • Faster transfers
  • Reduced CPU workload
  • Better overlap between computation and communication

6. PCIe Interface

The GPU communicates with the CPU through PCI Express (PCIe).

PCIe is the standard interface found in nearly every desktop, server, and workstation.

Data That Travels Over PCIe

  • Input datasets
  • Program instructions
  • Model parameters
  • Images
  • Results
  • Driver commands

Typical Communication


PCIe Generations

GenerationApproximate Speed (x16)
PCIe Gen3~16 GB/s
PCIe Gen4~32 GB/s
PCIe Gen5~64 GB/s
PCIe Gen6Higher with PAM4 signaling

Although PCIe is fast, it is still much slower than on-board HBM bandwidth.

7. NVLink

NVLink is NVIDIA's high-speed GPU-to-GPU interconnect.

Instead of sending data through the CPU and PCIe, GPUs communicate directly.




Benefits
  • Higher bandwidth than PCIe
  • Lower latency
  • Faster multi-GPU communication
  • Better scaling for AI training

Large language model (LLM) training often requires multiple GPUs to share parameters and gradients. NVLink enables this communication efficiently.


High-Level GPU Data Path



When a kernel executes:

  1. The CPU launches the kernel.
  2. Input data is transferred over PCIe.
  3. DMA places the data into GPU memory.
  4. The Memory Controller schedules memory accesses.
  5. The Crossbar and NoC route data to the appropriate Streaming Multiprocessors (SMs).
  6. Compute cores process the data.
  7. Results are written back to HBM or returned to the CPU.



How These Components Work Together

Imagine training a neural network:

  • The CPU launches the training kernel.
  • Training data is transferred via PCIe.
  • DMA copies the data into HBM.
  • The Memory Controller distributes memory requests across HBM channels.
  • The Crossbar routes requests to the correct L2 cache slices.
  • The NoC delivers data to the appropriate SMs.
  • Tensor Cores perform matrix multiplications.
  • Updated weights are stored back in HBM.
  • In multi-GPU systems, NVLink synchronizes model parameters between GPUs.

This pipeline allows billions of operations to occur every second without starving the compute units for data.

Quick Comparison



Key Takeaways

  • The GPU memory system is just as important as the compute cores for overall performance.
  • The Memory Controller efficiently schedules and manages all accesses to external memory.
  • HBM provides massive bandwidth through 3D-stacked memory, while GDDR offers a cost-effective solution for graphics workloads.
  • The Crossbar enables simultaneous communication between multiple GPU subsystems.
  • The Network-on-Chip (NoC) provides a scalable, high-bandwidth communication fabric across the chip.
  • The DMA Engine offloads data movement, allowing compute resources to focus on execution.
  • PCIe connects the GPU to the host CPU, while NVLink enables high-speed communication between GPUs.
  • Together, these components ensure a continuous flow of data, keeping thousands of GPU cores fed with work and maximizing throughput for graphics, AI, and scientific computing.


 

Labels:

Comments

Post a Comment

Popular posts from this blog

Module 1 — SystemVerilog Basics

By
VLSI Verification  •  SystemVerilog Series Module 1 — SystemVerilog Basics A complete, example-driven walkthrough of SystemVerilog fundamentals: data types, variables vs nets, arrays of every flavor, strings, enums, structs, unions, and typedef — the building blocks every verification engineer needs before UVM. 🧩 Beginner → Intermediate ⏱ ~25 min read 💻 Code Examples Included 📑 Table of Contents Introduction to SystemVerilog Verilog vs SystemVerilog Data Types: 2-State vs 4-State Logic Variables and Nets Arrays: Packed vs Unpacked Dynamic Arrays Queues Associative Arrays Strings Enumerations Structures Unions Typedef 1 Introduction to SystemVerilog SystemVerilog (IEEE 1800) is a unified hardware description and verification language (HDVL). It extends Verilog (IEEE 1364) with powerful constructs for design (interfaces, always_comb/always_ff, enhanced d...
Post a Comment
Read more

Clock Domain Crossing (CDC): A Complete Beginner's Guide to Metastability, Synchronizers & Safe Data Transfer

By
Clock Domain Crossing (CDC): A Complete Beginner's Guide to Metastability, Synchronizers & Safe Data Transfer Modern System-on-Chip (SoC) designs rarely operate with a single clock. Instead, different modules such as CPUs, GPUs, memory controllers, communication interfaces, and peripherals often run at different clock frequencies or even use completely independent clock sources . When data moves from one clock domain to another, timing uncertainty can cause serious problems such as metastability , data corruption , and unpredictable system behaviour . This challenge is known as Clock Domain Crossing (CDC) . CDC is one of the most important topics in ASIC Design , RTL Design , ASIC Verification , FPGA Design , and SoC Architecture . Every modern chip—from smartphones to automotive processors—uses CDC techniques to ensure reliable communication between multiple clock domains. In this article, we'll explore CDC fundamentals, metastability, synchronization techniques, asynchro...
Post a Comment
Read more

AXI Verification Interview Questions and Real-World Debugging

By
AMBA Protocol Series · Part 10 of 10 AXI Verification Interview Questions and Real-World Debugging Common protocol violations, how to read them off a waveform, and a curated set of interview questions with concise, correct answers — the capstone of the core 10-day series. AMBA Protocol Series: From APB & AHB to AXI — A Complete Guide for VLSI Engineers This wraps up the core 10-day series. Everything here draws directly on Days 1–9 — if any answer below feels unfamiliar, that's a good signal for which earlier post to revisit. (If you want to go further, see the 20-day extended list at the end of this post.) In This Article Common Protocol Violations Debugging From a Waveform Interview Questions and Answers Series Recap Extending the Series 1. Common Protocol Violations VALID deasserted before handshake — a source drops VALID before READY ever went high (violates the Day 8 stabi...
Post a Comment
Read more

Cache Basics and Cache Mapping Techniques

By
  Cache Basics and Cache Mapping Techniques  Modern processors execute billions of instructions every second. However, processor speed has increased much faster than main memory (RAM) speed. This difference, known as the memory wall , creates a performance bottleneck because the CPU often has to wait for data from slower memory. To solve this problem, computer systems use cache memory —a small, high-speed memory located between the CPU and the main memory. In this article, we will understand the fundamentals of cache memory, the principle of locality, cache organization, and the three major cache mapping techniques. What is Cache Memory? Cache memory is a small, high-speed memory that stores copies of frequently accessed data from the main memory. Since accessing cache is much faster than accessing RAM, the CPU first checks whether the required data is available in the cache. If the data is found in the cache, it is called a cache hit . Otherwise, the CPU fetches the dat...
Post a Comment
Read more

Timing Basics in Static Timing Analysis (STA): Delay Models, Sequential Timing, and Wire Delays

By
Timing Basics in Static Timing Analysis (STA) Static Timing Analysis (STA) is one of the most important verification techniques used in modern VLSI design. Before a chip is manufactured, engineers must verify that every signal reaches its destination within the required clock period. Even if a design passes functional simulation, it may still fail after fabrication because of timing violations. Unlike simulation, Static Timing Analysis mathematically analyzes every timing path without applying input test vectors. This makes STA significantly faster and capable of analyzing millions of timing paths efficiently. Table of Contents What is Static Timing Analysis? Why STA is Required Propagation Delay Contamination Delay Cell Delay Models Combinational Gate Delay Sequential Timing Setup Time Hold Time Wire Delay RC Delay Model PVT Corners Worked Examples Interview Questions Key Takeaways What is Static Timing Analysis? Static Ti...
Post a Comment
Read more

Computer Architecture Interview Question Part 1

By
Skip to content Computer Architecture Interview Series Part 1 · CPU Basics · RISC vs CISC · ISA 50 Most Asked Computer Architecture Interview Questions (Part 1) The foundational round every VLSI, ASIC, and CPU design interview opens with: instruction set architecture, RISC vs CISC trade-offs, Von Neumann vs Harvard design, registers, endianness, and the fetch-decode-execute cycle — explained the way interviewers actually expect you to answer.  40 Questions (Q1–Q40) ⌛ 22 min read  VLSI & Computer Architecture Interviews On this page Foundations of Computer Architecture RISC vs CISC — Core Differences Von Neumann, Harvard & the Semantic Gap Architecture Styles, Endianness & Registers ILP, VLIW, Superscalar & Amdahl's Law Register Files, Microcontrollers & the Fetch Cycle Tip...
Post a Comment
Read more