# The Visual Guide — Every Concept, Drawn ### Companion to BEGINNER_GUIDE.md · pure ASCII graphics · read top to bottom > Each picture = one idea. Caption tells you the one sentence to remember. > Open in any Markdown viewer / editor (monospace font keeps the shapes aligned). --- ## 1. The Whole Story — One Machine, Many Knobs ``` YOUR DATA (fixed, cannot touch) THE NETWORK (a box of KNOBS) ┌──────────────────────────┐ ┌───────────────────────────────┐ │ x = [2.0, 3.0] │ │ ( w1 )──┐ │ └────────────┬─────────────┘ │ ├──▶ GUESS: 0.3 │ │ │ ( w2 )──┘ │ │ └──────────────┬────────────────┘ │ │ ▼ ▼ ╔═══════════════════════════════════════════════════════════╗ ║ TRUE ANSWER = 1.0 ║ ║ LOSS = (guess − answer)² = (0.3 − 1.0)² = 0.49 ║ ║ "How wrong? 49% wrong" ║ ╚═══════════════════════════════╦═══════════════════════════╝ ║ blame flows BACKWARD ↩ ▼ ┌───────────────────────────────────────────────────────────┐ │ w1.grad = −3.2 → "turn me UP, by 0.32" │ │ w2.grad = +1.1 → "turn me DOWN, by 0.11" │ └───────────────────────────────────────────────────────────┘ │ nudge EVERY knob slightly, then repeat │ ╔══════════════▶ LOOP until LOSS ≈ 0 ◀══════════════╗ ``` > **One sentence:** predict → measure wrongness → blame every knob → nudge → repeat. --- ## 2. Analogy 1 — The Shower Thermostat (learning rate) ``` TEMPERATURE hot ┤ ╳ ╳ │ ╱ ╲ ╱ ╲ CRANK IT FULLY │ ╱ ╲ ╱ ╲ = bounce hot↔cold ✓ ┤ ─ ─ ─ ─ ─ ─ ─╱ ─ ─ ╲ ─ ─ ─ ─ ─╱ ─ ─ ─╱ ─ FOREVER (lr too big) │ ╳ ╲ ╱ │ ╱ ╲ ╲ ╱ cold ┤ ╱ ╲ ╳ └────────────────────────────────────→ knob turns over time tiny turns (lr too small): · · · · · · · · · · → still shivering 😥 smart turns (lr just right): ●───●───● → perfect shower ✓ ``` > **Remember:** the learning rate = "how far do I turn the knob each time?" --- ## 3. Analogy 2 — Tasting the Soup (loss + the training loop) ``` ┌─────────────────────────────────────────────────┐ │ │ ▼ │ ┌────────┐ ┌─────────┐ ┌──────────┐ ┌───────────┐ │ COOK │────▶│ TASTE │────▶│ "TOO │────▶│ ADD A │ │ (fwd) │ │ (loss) │ │ BLAND" │ │ PINCH │ └────────┘ └─────────┘ └──────────┘ │ (update) │ └─────┬─────┘ │ loop until delicious ←────┘ The gradient is a MAGIC TONGUE: instead of just "too bland", it says "add exactly 0.32 g of salt" — for EVERY ingredient at once. ``` --- ## 4. Analogy 3 — Golf in Thick Fog (gradient descent) ``` LOSS │ A● │ ●● "I can't SEE the valley (fog = no direct formula) │ ● but I CAN FEEL the slope under my feet (gradient)" │ ● │ ● C● │ ● ● │ ● ● │ ● ● │ ●__ __● │ ●▄▄▄▄▄▄▄● ← THE VALLEY (minimum loss) ⛳ └───────────────────────────────────→ weight value At A: ground tilts → right → step RIGHT At B: ground tilts → right → step RIGHT (smaller step, gentler slope) At C: ground tilts ← left → step LEFT At the bottom: ground FLAT (grad = 0) → STOP. You're home. ``` > **Remember:** always step in the direction that feels downhill. That's the whole algorithm. --- ## 5. Analogy 4 — The Blame Meeting (backpropagation) ``` ╔══════════════════════════╗ ║ LOSS: $100k this month ║ ← "how wrong are we?" ╚═════════════╦════════════╝ ║ blame flows DOWN ↓ ┌───────────────┴───────────────┐ ▼ ▼ ┌─────────────┐ ┌─────────────┐ │ SALES 40% │ │ FACTORY 60% │ blame ∝ each └──────┬──────┘ └──────┬──────┘ department's ┌────┴────┐ ┌─────┼─────┐ INFLUENCE ▼ ▼ ▼ ▼ ▼ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ │Alice │ │ Bob │ │line 1│ │line 2│ │line 3│ │ 25% │ │ 15% │ │ 30% │ │ 20% │ │ 10% │ └──────┘ └──────┘ └──────┘ └──────┘ └──────┘ ▲ ▲ ▲ ▲ ▲ every employee gets a blame score ← every WEIGHT gets a gradient High blame = change behavior the MOST ← big |grad| = bigger update ``` --- ## 6. Analogy 5 — Currency Exchange (the chain rule) ``` $1 ────[ rate 0.9 ]───▶ €0.90 ────[ rate 90 ]───▶ ₹81 How much do the final RUPEES move if a rate wiggles? wiggle rate 1 by 0.01 → ₹ move by 0.01 × 90 = 0.9 wiggle rate 2 by 1 → ₹ move by 0.9 × 1 = 0.9 total sensitivity $→₹ = 0.9 × 90 = 81 └───┬───┘ the per-step rates MULTIPLY along the path ════════════════ THAT IS THE CHAIN RULE ════════════════ ``` --- ## 7. The Knob Problem (why gradients exist) ``` ┌─────────────────────────────────┐ input ───▶ │ ⎡‾‾‾‾‾‾‾⎤ │ x = 3.0 │ ⎢ w=2.0 ⎥ ───▶ output = 6.0 │ ───▶ "should be 9.0!" │ ⎣_______⎦ │ └─────────────────────────────────┘ ONE knob → just try 3.0 → output 9.0 ✓ easy! ┌──────────────────────────────────────────────────────────────┐ │ w1 ─┐ │ │ w2 ─┤ ╔══════════════════╗ │ │ w3 ─┼──▶║ THE NETWORK ║──▶ output │ │ ... │ ╚══════════════════╝ │ │ w1M ┘ │ └──────────────────────────────────────────────────────────────┘ 1,000,000 tangled knobs → you need a formula per knob saying "up a little / down a little". THAT FORMULA = THE GRADIENT. ``` --- ## 8. Loss = Archery (mean squared error) ``` MSE = (distance to bullseye)² ╭───╮ ╭─┤ ◉ ├─╮ ◉ bullseye → loss 0 perfect │ ╰───╯ │ ╰───┬───╯ ◦ 1 cm off → loss 1 fine │ ● 7 cm off → loss 49 OUCH! big misses hurt 49× MORE than small ones (that's WHY we square — bad shots must hurt bad) ``` --- ## 9. Derivative = The Slope You Can Feel ``` y │ tangent line = "the slope right here" 9 ┤ _●_ its steepness = the DERIVATIVE │ ⁄ ╲ │ ⁄ ╲ at x = 3 : slope = 6 (steep!) 6 ┤ ⁄ ╲ at x = 1 : slope = 2 (gentler) │ ⁄ ╲ at x = 0 : slope = 0 (flat bottom) 3 ┤ ⁄ ╲ │ ⁄ ╲ 0 ┤⁄──────────────────────────╲──────→ x y = x² slope = 2x (the famous rule) ``` > **Remember:** a derivative is not scary symbols. It's "nudge the input a hair — how much did the output move?" --- ## 10. A Value Object = A Box With 5 Slots (`engine.py`) ``` you type: c = a * b (a=2.0, b=3.0) ┌─────────────────────────────┐ │ c │ ├─────────────────────────────┤ │ .data 6.0 ◀── the number itself (2 × 3) │ .grad 0.0 ◀── blame score (starts 0, filled later) │ ._children {a, b}◀── "I was made FROM these two" │ ._op '*' ◀── "via multiplication" │ ._backward (fn) ◀── "when blame arrives, I'll send └─────────────────────────────┘ a → blame×3 and b → blame×2" leaf node (like a or b): ._children = {} ._op = '' = "I was born directly; I'm an input or a weight" ``` > **Remember:** every operation stamps its result with "who made me + how to forward blame." --- ## 11. The Graph Grows As You Compute ``` You type The graph grows LEFT → RIGHT (the FORWARD pass) c = a * b a=2.0 ──┐ ├──[ × ]──▶ c = 6.0 b=3.0 ──┘ d = a + b a=2.0 ──┐ ├──[ + ]──▶ d = 5.0 b=3.0 ──┘ e = c + d c=6.0 ──┐ ├──[ + ]──▶ e = 11.0 ◀── OUTPUT d=5.0 ──┘ All together: a ──┬──[×]──▶ c ──┐ └──[+]──▶ d ──┼──[+]──▶ e b ──┴─────────────┘ Every box secretly recorded its parents + blame rule. That's the graph. ``` --- ## 12. Blame Rules Per Operation (arrows!) ``` ADD z = a + b blame(z) ══▶ a (full copy) blame(z) ══▶ b (full copy) "addition is a COPY MACHINE — both parents get the whole blame" MUL z = a × b a=2, b=3, blame(z)=1: a ◀══ blame(z) × b a.grad = 1×3 = 3 b ◀══ blame(z) × a b.grad = 1×2 = 2 "SWAP the values — your blame = incoming × the OTHER guy's size" TANH z = tanh(a) a ◀══ blame(z) × (1 − z²) z=0.46 → blame 0.79 (mid-range: wide open) z=0.99 → blame 0.02 (near the edge: tanh is FLAT there) RELU z = max(0, a) a > 0 : blame ═══▶ passes right through ╔═════════╗ a ≤ 0 : blame ✗ BLOCKED (bouncer!) ══▶ ✗ ║ REJECTED║ a ≤ 0 ╚═════════╝ ``` --- ## 13. Backpropagation = 3 Moves (`backward()` in engine.py) ``` MOVE 1 ▶ LINE THEM UP (topological sort — "socks before shoes") forward order: [a] [b] [c] [d] [e] leaves first... ...output LAST MOVE 2 ▶ SEED THE OUTPUT e.grad = 1.0 "the loss blames itself exactly 1" MOVE 3 ▶ WALK IN REVERSE, each box forwards blame to its parents e ──▶ "c and d, my blame of 1.0 is partly yours" → c=1.0, d=1.0 d ──▶ "a and b, take my 1.0" → a += 1, b += 1 c ──▶ "a: 1.0×b(=3) b: 1.0×a(=2)" → a += 3, b += 2 ▲ a was blamed from TWO paths → that's why we ADD (+=) RESULT: a.grad = 1+3 = 4.0 b.grad = 1+2 = 3.0 ``` --- ## 14. The Neuron — Drawn (`step2_one_neuron.py`) ``` x1 = 2.0 ──▶ [ × w1=−3.0 ]──┐ │ x2 = 0.0 ──▶ [ × w2=+1.0 ]──┼───▶( + )───▶[ tanh ]───▶ OUTPUT │ ▲ 0.707 │ │ └── b = 6.88 (the bias) └─ inputs ─┘ └─ WEIGH each ─┘ └ add all ┊ └ squash ┊ inputs = the evidence (fixed — you cannot change data) weights = the neuron's OPINIONS about each input (LEARNED) bias = its default mood (LEARNED) tanh = squash the total into (−1, +1) Learning = changing the OPINIONS (weights), never the evidence (inputs). ``` --- ## 15. The Umbrella Neuron (same picture, real life) ``` cloudy? 0.8 ──▶ [ × +2.0 ]──┐ ← "clouds STRONGLY predict rain" │ summer? 0.7 ──▶ [ × −0.5 ]──┼──▶( + )──▶[ tanh ]──▶ +0.95 │ = 1.85 │ rained 0.9 ──▶ [ × +1.0 ]──┘ ▲ ▼ yesterday? b=−0.3 "BRING THE UMBRELLA" ☂ The weights ARE the lesson learned from past rainy days. ``` --- ## 16. The MLP — Your 41-Knob Machine (`step3_full_network.py`) ``` INPUT(3) HIDDEN 1 (tanh) HIDDEN 2 (tanh) OUTPUT (linear) x1 ───┬──▶ (N1) ──┐ x2 ───┼──▶ (N2) ──┼──▶ (N5) ──┐ x3 ───┼──▶ (N3) ──┼──▶ (N6) ──┼──▶ (N9) ──▶ prediction └──▶ (N4) ──┤ (N7) ──┤ └──▶ (N8) ──┘ knob budget: N1–N4: 4×(3w+1b) = 16 N5–N8: 4×(4w+1b) = 20 TOTAL = 41 knobs N9 : 1×(4w+1b) = 5 every one gets blame every one gets nudged ``` --- ## 17. MSE vs Hinge — Two Different Loss Shapes ``` MSE (predict a NUMBER) HINGE (predict a CLASS ±1) loss loss │ ●●● │━━━━━━╮ │ ●● ●● │ ╲ │ ●● ●● │ ╲ │ ●● ●● │ ╲ 0 └─────────────────────→ error 0 └──────────→ margin = y×pred a BOWL around the target a CLIFF: zero only if you're "get as close as possible" right AND confident (margin>1) regression → MSE classification → hinge ``` --- ## 18. The Training Loop — The Only Flowchart You Need ``` ┌────────────────────────┐ │ start: random knobs │ └───────────┬────────────┘ ▼ ┌──────▶ ┌─────────────────────┐ │ │ 1) FORWARD: guess │ (also secretly builds the graph) │ └──────────┬──────────┘ │ ▼ │ ┌─────────────────────┐ │ │ 2) ZERO all grads │ ← ERASE THE WHITEBOARD! │ └──────────┬──────────┘ │ ▼ │ ┌─────────────────────┐ │ │ 3) BACKWARD: blame │ ← chain rule fills every .grad │ └──────────┬──────────┘ │ ▼ │ ┌─────────────────────┐ │ │ 4) UPDATE │ w.data −= lr × w.grad │ └──────────┬──────────┘ (every knob steps downhill) │ ▼ │ loss ≈ 0 ? ──no──┐ │ │ │ │ yes │ │ ▼ │ │ 🎉 trained! │ └──────────────────────────────┘ ``` --- ## 19. Learning Rate — Three Races Down the Same Valley ``` loss TOO BIG (lr=1.0) JUST RIGHT (lr=0.1) │ ● ● ● │ ● ● ● ● │ ● ● ● ● │ ● ● ● ● │ ● ● ● ● │ X ← bounces across the ● ● │ valley FOREVER ● ● ← arrives! └──────────────────── w └──────────────────────────── w TOO SMALL (lr=0.001): │ ·→·→·→·→·→·→·→·→·→·→·→·→·→·→·→·→·→ ... still nowhere near └──────────────────────────── w the bottom (needs ~10,000 steps) ``` --- ## 20. Real Training Curve — Actual Numbers From YOUR step3 Run ``` loss 1.0 ┤ ●1.011 ← accuracy 40% (2 of 5 right) │ ╲ 0.8 ┤ ● 0.787 ← accuracy 60% │ ╲ 0.6 ┤ ● 0.591 ← accuracy 80% │ ╲ 0.4 ┤ ● 0.403 ← accuracy 100% ← all 5 correct here! │ ╲ 0.2 ┤ ╲___ │ ╲____ 0.0 ┤ ●━━━━●━━━● 0.002 at step 99 └────┬────┬────┬────┬────┬────┬───→ training step 0 10 20 30 50 70 99 loss = the SMOOTH compass │ accuracy = the CHOPPY scoreboard (train on loss; report accuracy) ``` --- ## 21. Where Blame Comes From — One Full Path Drawn ``` target=1.0, guess=0.3 → error = −0.7 [square] blame for "guess": 2 × (−0.7) = −1.4 ← local derivative │ ┌───────────────┴────────────────┐ ▼ ▼ [× node] w1's blame = −1.4 × x1 [× node] w2's blame = −1.4 × x2 = −1.4 × 2 = −2.8 = −1.4 × 3 = −4.2 ▲ ▲ "w1: increasing you "w2: increasing you makes it WORSE — makes it BETTER — turn me DOWN" turn me UP" update: w2 −= lr × (−4.2) every blame = incoming × local derivative (chain rule, drawn) ``` --- ## 22. What "Learning" Actually Changes ``` BEFORE training AFTER 100 steps ┌──────────────┐ ┌──────────────┐ │ w1 = +0.8566 │ │ w1 = +0.8584 │ ← tiny changes... │ w2 = +0.2348 │ ── training ──▶ │ w2 = +0.2375 │ ...repeated │ b = +0.0000 │ (nudges) │ b = +0.0022 │ 100 times └──────────────┘ └──────────────┘ output = 0.0997 (loss 0.81) output = 0.9892 (loss 0.0001) No single nudge is impressive. A MILLION tiny correct nudges = "the network learned". That's the magic. ``` --- ## 23. The One Picture To Keep Forever ``` ┌──────────────────────────┐ │ THE WHOLE THING │ └────────────┬─────────────┘ │ ┌──────────────┬───────────────┼───────────────┬──────────────┐ ▼ ▼ ▼ ▼ ▼ DATA (fixed) KNOBS (learn) GUESS LOSS BLAME │ │ │ │ (gradient) └──────────────┴───────┬───────┴───────────────┘ │ ▼ │ nudge every knob ◀───────────────────────────────┘ │ ▼ repeat until loss ≈ 0 ``` > Training = tiny correct nudges, repeated. The gradient is *what makes each nudge correct.* > Everything else — PyTorch, GPUs, GPT-4 — is this picture at larger scale.