Module 03: Serving Engines

LLM Serving: vLLM, PagedAttention & Kernels

Interactive technical curriculum synthesized from FAANG and Tier-1 AI startup interview rounds.

LLM Serving: vLLM, PagedAttention, Kernels & Triton

Comprehensive coverage of QA-05, QA-27, QA-31, QA-32, QA-36, and QA-44. Master iteration-level scheduling, FlashAttention online softmax, speculative decoding, and prefill-decode disaggregation.

Interactive Speculative Decoding Simulator

See how a small 1B draft model collaborates with a 70B target model to double token generation speeds without loss of accuracy:

Speculative Lookahead Tokens (K):

K = 4 tokens

Draft Acceptance Probability (α):

Acceptance = 75%

Expected Accepted Tokens Per Forward Pass:

3.25 tokens

Net Generation Speedup:

2.41x Faster

Target model executes 1 parallel prefill verification step instead of 4 sequential memory-bound decode steps!

QA-05 • Core Serving

Continuous Batching & PagedAttention in vLLM

Source: H2LooP / Enterprise Infra • Track: AI DevOps • Difficulty: Hard

▼

The Question: "Why does static request-level batching waste >70% of GPU compute, and how does vLLM schedule at the iteration level?"

Static Batching Inefficiency: The entire batch is bound by the slowest sequence. When a 50-token query finishes, it must output padding tokens while waiting for a 2,000-token sequence to complete.
Continuous Batching: Scheduling occurs at the token iteration level. The moment an sequence emits an <EOS>, it is evicted immediately. A newly queued request enters on the very next forward pass, keeping GPU tensor cores saturated.
PagedAttention Virtual Tables: Standard serving pre-allocates maximum context VRAM (causing up to 80% fragmentation). PagedAttention partitions KV cache into 16-token virtual blocks, allocating physical pages on demand.

QA-27 • Google Round

FlashAttention Tiling & The Online Softmax Algorithm

Source: Ashutosh Gupta (Google) • Track: AI Systems • Difficulty: Hard

▼

The Core Asymmetry: High Bandwidth Memory (HBM) bandwidth is 2.0 TB/s, whereas on-chip SRAM is 20 TB/s (10x faster).

Mechanism:

Standard attention calculates S = Q * K^T, writes the intermediate N × N matrix to HBM, reads it back for Softmax, writes P back to HBM, and reads it for O = P * V. Memory complexity is O(N²).
FlashAttention tiles Q, K, and V into blocks that fit entirely within SRAM.
Online Softmax: Keeps running scale factors (max value m and sum l). When a new tile arrives, existing output accumulator is scaled by e^{m_prev - m_new}, completing attention in a single pass without writing N × N matrices to HBM!

QA-32 • Scale Architecture

Prefill-Decode Disaggregation (PD Disaggregation) at Enterprise Scale

Source: Mohit Kumar Dubey • Track: Architect / Principal • Difficulty: Hard

▼

Why Colocation Fails: Prefill is compute-bound (massive parallel GEMM operations). Decode is memory-bound (loading all weights per single token). Running both on the same GPU means a sudden burst of long prompts halts active decoding streams, spiking Inter-Token Latency (ITL) and creating stuttering user experiences.

Disaggregated Solution:

Prefill Nodes: Clustered high-compute GPUs run prompt phase at 100% FLOPS occupancy.
KV-Cache Streaming: Computed KV cache is transmitted directly into decode GPU memory over high-speed RDMA / InfiniBand.
Decode Nodes: Devoted 100% to continuous generation streams with zero latency spikes.