Module 08: Orchestration

Kubernetes GPU Infrastructure & Autoscaling

Interactive technical curriculum synthesized from FAANG and Tier-1 AI startup interview rounds.

Kubernetes GPU Infrastructure, Volcano & Fast Cold Starts

Comprehensive coverage of QA-07, QA-13, QA-21, and QA-40. Master queue-depth autoscaling with KEDA, Volcano gang scheduling, JuiceFS distributed NVMe caching, and eBPF kube-proxy replacements.

Interactive 70B Cold Start Optimization Pipeline

Click each stage to see how a naive 8-minute pod launch is collapsed to under 45 seconds:

Naive Baseline (8m 10s)
140GB image pull from ECR/DockerHub

Image Decoupling (2m 15s)
Base runtime image (<1.5GB)

Distributed POSIX Cache (55s)
JuiceFS / Fluid on local NVMe

GPUDirect Storage (14s)
Direct NVMe → GPU HBM bypass

Stage 1: Naive Monolithic Image (8 min 10 sec)

The Bottleneck: Teams bake 140GB of safetensors weights directly into the Docker image. Pulling a 140GB image across network interfaces and unpacking container layers takes 7 to 10 minutes on new nodes, completely missing burst traffic.

QA-07 • KEDA Autoscaling

Why HPA CPU/GPU Utilization Fails for LLMs

Source: Rishabh Tripathi • Track: AI DevOps / SRE • Difficulty: Medium-Hard

▼

The Trap: Setting HPA to scale on 80% GPU memory causes instant scale-out to max replicas because vLLM pre-allocates 90% of VRAM for PagedAttention at boot time.

The Solution: Use KEDA targeting vllm:num_requests_waiting:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-queue-autoscaler
spec:
  scaleTargetRef:
    name: vllm-llama3-deployment
  minReplicaCount: 2
  maxReplicaCount: 16
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring:9090
      metricName: vllm_num_requests_waiting
      query: sum(vllm:num_requests_waiting{model="llama3-70b"})
      threshold: '4'