Module 04: Hardware & Math

GPU Architecture & VRAM Memory Math

Interactive technical curriculum synthesized from FAANG and Tier-1 AI startup interview rounds.

GPU Architecture, Hardware Sizing & Memory Mathematics

Comprehensive coverage of QA-04, QA-06, and QA-12. Master Grouped-Query Attention (GQA) memory equations, the Roofline Model, and emergency capacity levers.

Whiteboard Memory Math Step-Through

Click each component to walk through the exact equations required in senior engineering technical screens:

Static Model Weights
Parameters × Precision Bytes

GQA KV Cache Per Token
2 × L × H_kv × d_h × Bytes

Concurrent Stream Sizing
Batch × Context Length × KV_Token

Tensor Parallelism (TP) Split
VRAM Per GPU & Headroom Buffer

Step 1: Static Weights Footprint

Formula: Memory = Parameters × Bytes_Per_Weight

For Llama-3-70B in 16-bit (FP16/BF16, 2 bytes):
70 × 10^9 × 2 bytes = 140,000,000,000 bytes ≈ 140 GB.

This memory is completely static and independent of how many users are actively querying the server.

QA-06 • Whiteboard Math

Llama-3-70B VRAM & GQA KV-Cache Math (64 streams)

Source: Candidate Technical Screen • Track: AI DevOps / LLMOps • Difficulty: Hard

▼

Interview Scenario: "We need to serve Llama-3-70B in 16-bit precision across 80GB GPUs with 64 concurrent streams and a 4,096-token context window. Exactly how many GPUs do we need, and what is the exact memory breakdown?"

Live Hardware Sizing Simulator

Active Streams (Batch Size): Max Context Length (Tokens): KV Precision:

Model Weights:

140.0 GB

KV Cache Size:

83.9 GB

Total VRAM (+12GB Activations):

235.9 GB

Recommended Hardware Cluster:

4x 80GB GPUs (TP=4)

QA-04 • Hardware Profiling

GPU Utilization vs Occupancy vs Memory Bandwidth (The Roofline Model)

Source: Pallavi Ahuja • Track: AI Platform / DevOps • Difficulty: Hard

▼

The Scenario: "Your nvidia-smi shows 100% GPU utilization while serving an LLM, but token throughput is disappointingly low. Your manager asks if buying faster GPUs is the only fix. How do you explain why 100% GPU utilization is deceptive?"

The Deception:

nvidia-smi's utilization metric only measures the percentage of time that any kernel was executing on the GPU. If a single warp is fetching memory from DRAM while the other 128 SMs sit completely idle, nvidia-smi still reports 100% Utilization!

Decode Phase is Memory-Bound: Each generated token must load 140GB of weights from HBM to register files. The GPU is memory-bandwidth bound ($O(1)$ arithmetic intensity).
Nsight Profiling (nsys): Profile DRAM throughput vs Tensor Core active cycles. If DRAM throughput is at 95% while SM compute is at 15%, you cannot fix it with faster clocks—you need Continuous Batching, Quantization (FP8/AWQ), or PagedAttention.