Module 09: Production War Stories
Cloud SRE, Networking & Production Failures
Interactive technical curriculum synthesized from FAANG and Tier-1 AI startup interview rounds.
Cloud SRE, Networking & Production Incident War Stories
Comprehensive coverage of QA-01, QA-08, QA-09, QA-10, QA-14, and QA-22. Master the STAR interview framework for 2 AM production failures, TCP keepalive 502 race conditions, and thundering herd mitigations.
QA-01 • The STAR Framework
The "Tell Me About a 2 AM Production Failure" Incident
STAR War Story Delivery:
- Situation: At 2:15 AM, p99 latency on our customer service AI agent spiked from 800ms to 48 seconds, triggering PagerDuty alerts across the platform team.
- Task: Identify the root cause, restore service without dropping customer chats, and eliminate the vulnerability permanently.
- Action: Low-level tracing revealed a downstream CRM API was returning HTTP 429 HTML rate-limit error pages. The agent interpreted the HTML body as new instructions and entered an unbounded reflection loop. We applied emergency circuit breakers, capped
max_iterations = 4, and enforced exponential backoff with full jitter. - Result: P99 latency recovered to 650ms. Incident post-mortem introduced automated CI/CD chaos tests, saving estimated $45k/month in runaway token loops.
QA-09 • Network Race Condition
Intermittent 502 Bad Gateway Under AI Inference Load
The Mystery: 1.5% of requests fail with 502 Bad Gateway, but backend CPU is low and pods are not restarting.
The TCP Keepalive Race Condition:
When the upstream reverse proxy (Nginx / ALB) has a keepalive timeout longer than or equal to the backend server (FastAPI/Uvicorn), the backend can send a TCP FIN at the exact microsecond the proxy forwards a new request, resulting in an immediate TCP RST.
The Golden Rule: Backend keepalive timeout must ALWAYS be strictly greater than proxy timeout:
# Nginx Ingress: keepalive_timeout 65s;
# Uvicorn: uvicorn.run(..., timeout_keep_alive=75) # 75s > 65s!