A from-scratch, zero-black-box curriculum for the people who keep clusters alive: Linux internals & security, DISA STIGs, HPC benchmarking, eBPF, Ansible, Warewulf, RDMA fabric, parallel storage, containers, kernel tuning, anomaly detection, fraud detection, key management, SOC & detection engineering, NOC & incident command, Splunk/SIEM, chaos engineering, and red/blue/purple teaming — with runnable pure-Python labs that model the real thing.
Every module follows the same shape: three runnable stepN_*.py
labs that model the real mechanism with no dependencies, a
BEGINNER_GUIDE with everyday analogies, a
DEEP_DIVE with production war-stories and real configs,
and an INTERVIEW file of micro-detail Q&A.
Part 1 — Fundamentals: the twenty-one topics below as
21 chapters × 6 levels, from plain-English analogy to
an Ultimate level of real production scenarios.
Part 2 — Linux Kernel Internals & Performance Engineering:
16 chapters × 5 levels on scheduler internals, NUMA,
cgroups v2, perf/eBPF profiling, kernel bypass, live-patching.
Part 3 — HPC Cluster Orchestration at Scale: 16
chapters × 5 levels on Slurm internals, Warewulf, MPI
placement, RDMA fabric, parallel filesystems, GPU scheduling.
Part 4 — Security Hardening & Compliance Engineering:
16 chapters × 5 levels on STIG/CIS automation,
SELinux, kernel hardening, supply-chain security, PAM, OpenSCAP.
Part 5 — Production Systems Operations: 16
chapters × 5 levels on fleet observability, centralized
logging, anomaly detection, chaos engineering, on-call & postmortems —
each ending in an interview drill.
What actually happens between power-on and a login prompt. The
virtual filesystem, the page cache, how fork()
and the CFS/EEVDF scheduler share a CPU, and why "free memory" is
a lie you should stop worrying about.
Turning a default install into a defensible one. Baseline scoring,
SSH and PAM, least-privilege sudo, SELinux vs AppArmor, kernel
sysctl hardening, and how a small
misconfiguration becomes root.
How the DoD ships a hardened OS. The XCCDF/OVAL data model,
CAT I/II/III severity, oscap scanning
and remediation, SSG/ComplianceAsCode, tailoring, POA&Ms, and
keeping an accredited system compliant over time.
Measuring a machine honestly. Sizing HPL for the Top500 run, STREAM and the memory wall, OSU latency/bandwidth, IOR/mdtest for storage, the roofline model, and Amdahl vs Gustafson when your scaling curve bends.
Safe programs in the kernel. Why the verifier rejects your loop,
how maps and the ring buffer move data to userspace, tracepoints
vs kprobes vs fentry, CO-RE and BTF, and building a syscall
latency histogram the way bpftrace does.
Configuration as a repeatable fact. Inventory and group_vars
precedence, what "idempotent" really demands of a task, handlers,
serial rolling updates with health
gates, and why free strategy and
fact-caching decide your window.
Booting a thousand diskless nodes from one image. The DHCP → TFTP → iPXE → kernel/initramfs handshake, container images as node images, system vs runtime overlays, and why stateless provisioning makes a node reboot instead of a ticket.
How a batch scheduler turns a room of servers into a shared supercomputer. Partitions and QOS, the backfill algorithm, fairshare, MPI ranks and collectives, process binding, and the control-plane daemons that must never go down.
The wires that make MPI fast. RDMA vs the kernel TCP path,
InfiniBand vs RoCEv2, fat-tree bisection bandwidth and
oversubscription, the subnet manager, adaptive routing, and
reading ibstat / perfquery.
Feeding thousands of ranks from one filesystem. MDS/OSS/OST
anatomy, stripe count and size, the metadata bottleneck and the
ls -l that took the cluster down,
progressive file layouts, and why small random I/O is the enemy.
Making a run reproduce in five years. Apptainer/Singularity's rootless single-file model vs Docker, binding the host MPI and fabric into a container, Spack concretization and the build DAG, and environment modules / Lmod.
The last 20% of performance. NUMA locality and
numactl, cgroups v2 resource control,
hugepages and the TLB, C-states and the governor, CPU isolation
and IRQ affinity, and tuned profiles
for latency vs throughput.
The kernel guardrails a process cannot argue with. "root" split into 40 capabilities, a syscall allowlist that deletes 80% of the attack surface, SELinux type enforcement and the domain transition, and why a confined RCE goes nowhere.
When the payload is encrypted, the shape still talks. Direction/size/timing features that separate web from exfil from C2, TLS client fingerprints against an egress allowlist, and jitter-resistant beacon detection — no decryption.
Judging behaviour you can't fully see: sliding-window velocity and impossible travel, linkage graphs that expose a ring one account can't, device/proxy/bot tells, and why you tune the threshold by dollars, not accuracy.
The vault under everything. KEK/DEK hierarchies where rotation re-wraps keys not petabytes, crypto-shredding for "delete my data", k-of-n secret sharing and the key ceremony, and signing-key rotation without an auth outage.
The room that watches. Alert triage and analyst tiers, detection-as -code and the precision/recall trade, an ATT&CK coverage matrix weighted by prevalence, and why a noisy rule is worse than no rule.
Running it well when it is down. The four golden signals and why you page on symptoms not causes, multi-window burn-rate alerting, incident command roles, and the blameless postmortem.
Logs into answers. SPL as a Unix pipeline for events, indexes and sourcetypes, the stats/eventstats/streamstats confusion settled, data models & CIM, and correlation searches with throttling and risk-based alerting.
Breaking things on purpose to find the weaknesses first. The steady-state hypothesis test, bounding blast radius by error budget, the staged ramp with an abort button, and the four things that stop a dependency failure from cascading.
Every module is explained three ways — plain-English analogy, a production deep-dive with real configs and outage stories, and an interview drill of micro-details.
| Module | Beginner | Deep Dive | Interview |
|---|---|---|---|
| Linux internals | Guide | Deep Dive | Q&A |
| Security hardening | Guide | Deep Dive | Q&A |
| STIGs & compliance | Guide | Deep Dive | Q&A |
| HPC benchmarking | Guide | Deep Dive | Q&A |
| eBPF & observability | Guide | Deep Dive | Q&A |
| Ansible automation | Guide | Deep Dive | Q&A |
| Warewulf provisioning | Guide | Deep Dive | Q&A |
| HPC / Slurm architecture | Guide | Deep Dive | Q&A |
| Networking & RDMA fabric | Guide | Deep Dive | Q&A |
| Parallel storage | Guide | Deep Dive | Q&A |
| Containers & reproducibility | Guide | Deep Dive | Q&A |
| Kernel & performance tuning | Guide | Deep Dive | Q&A |
| Linux security (LSM/seccomp/caps) | Guide | Deep Dive | Q&A |
| Anomaly detection on encrypted activity | Guide | Deep Dive | Q&A |
| Fraud & circumvention detection | Guide | Deep Dive | Q&A |
| Key-management infrastructure | Guide | Deep Dive | Q&A |
| SOC & detection engineering | Guide | Deep Dive | Q&A |
| NOC & incident command | Guide | Deep Dive | Q&A |
| Splunk & SIEM | Guide | Deep Dive | Q&A |
| Chaos engineering & resilience | Guide | Deep Dive | Q&A |
| Red / blue / purple teaming | Guide | Deep Dive | Q&A |
Five course parts in this repo — Part 1 from first principles, Parts 2–5 operating the standard tools at production depth.
Linux internals → hardening & STIGs → HPC benchmarking → eBPF → Ansible → Warewulf → Slurm → fabric & storage → containers & tuning → SOC/NOC & red/blue teaming.
Open Part 1Scheduler internals, memory management & NUMA, cgroups v2, perf/eBPF profiling in production, I/O schedulers, kernel bypass, live-patching, fleet-wide sysctl tuning.
Open Part 2Slurm scheduler internals, Warewulf provisioning at scale, MPI placement & topology awareness, RDMA/InfiniBand fabric design, parallel filesystems, GPU scheduling, burst-to-cloud HPC.
Open Part 3DISA STIG automation at fleet scale, CIS benchmarks as code, SELinux/AppArmor policy authoring, kernel hardening, supply-chain security, secrets management, intrusion detection, OpenSCAP.
Open Part 4Fleet-wide observability, centralized logging at scale, anomaly detection, chaos engineering for bare-metal/HPC, on-call & incident response, capacity planning, postmortems.
Open Part 5AI/ML engineering from scratch through production system design and enterprise-scale AI, and infrastructure from IaC through Kubernetes platform engineering and observability — the same "from scratch, zero black box" approach applied to two other stacks.
Each INTERVIEW.md is a drill of the exact
micro-details that come up in Linux / HPC / DevSecOps screens — asked
as a question, answered in a few tight sentences, with the "why it
matters" that separates a memorised answer from an understood one.
Definitions you must be able to give cold, with the one detail interviewers probe for.
Symptom → hypotheses → the command that confirms it, for the classic failure modes.
The formulas — HPL N, bisection bandwidth, Amdahl limit, stripe count — worked in your head.
No cluster required. Every lab is pure Python 3 with zero dependencies — it models the real mechanism so you can read and modify it.
git clone https://github.com/satyabhan007/linux-hpc-security.git
cd linux-hpc-securitypython3 run_all.pypython3 benchmarking/step1_hpl_problem_size.py
python3 hpc/step1_slurm_backfill.py
python3 ebpf/step3_latency_hist.py