A Dell OptiPlex from a recycling pile, three Raspberry Pi 4s, and a $35 eBay switch: that's the $220 stack running a 4-node bare-metal Kubernetes cluster. The author, a professional AWS user, didn't want another managed EKS cluster. They wanted to break things.
The Architecture
The control plane runs on the OptiPlex (Intel i5-6500, 16GB RAM, 256GB SSD) with Talos Linux. Three Raspberry Pi 4s (4GB RAM each) serve as workers, booting from USB SSDs (not SD cards). The network stack: Cilium with eBPF, MetalLB in BGP mode, Longhorn for storage, ArgoCD for GitOps, and Prometheus/Grafana for monitoring.
Failure 1: SD Cards Die Under etcd
Three 64GB SanDisk Ultra SD cards died in six weeks. Root cause: etcd's frequent small writes overwhelm SD card wear-leveling designed for sequential photo writes. The fix: every node boots from a $25 USB3 SSD. Talos supports installing to SSD natively. The cluster has run 8 months without storage crashes.
Failure 2: Invisible eBPF Lockout
A CiliumNetworkPolicy that allowed ingress only from pods with label app=frontend locked the author out of every node. The policy didn't include kube-system or Cilium health checks. eBPF enforces policies before packets reach the host network stack, so no logs appear in dmesg or iptables. Recovery required booting with talos.config=none kernel parameter. The procedural fix: always use cilium policy trace before applying, keep a default-allow-all policy in Git, and never apply policies on Friday evening.
Failure 3: Longhorn Disk Math
A 10Gi PostgreSQL volume with 3 replicas and 2 snapshots consumed 60Gi of actual disk. Each Pi had 120Gi SSDs. Combined with Talos overhead (20Gi) and container images/logs (40Gi), the cluster hit 85% disk pressure and kubelet evicted all pods. The fix: enforce volume quotas (max 2 replicas for non-critical), automatic snapshot cleanup (retain 2, delete older than 7 days), Prometheus alerts at 70% usage, and container image garbage collection at imageGCThreshold: 70.
Failure 4: MetalLB IP Conflict
MetalLB's Layer 2 mode assigned IPs inside the DHCP range, causing conflicts. The fix: move MetalLB to 192.168.1.200-192.168.1.250 outside the DHCP pool. Better fix: configure BGP mode with OpenWrt router peering for dynamic route learning.
Failure 5: Upgrade Without etcd Backup
An OS upgrade from Talos v1.6.0 to v1.7.0 while the cluster was under disk pressure corrupted etcd WAL files. No etcd backup existed. Recovery took 6 hours with a 3-week-old snapshot. The fix: automated etcd snapshots every 6 hours to S3, pre-upgrade health checks (disk <70%, all nodes Ready, Longhorn volumes healthy), and a custom Go CLI that wraps talosctl upgrade with these checks.
Design Decisions
The author chose Talos Linux over Ubuntu + kubeadm for its immutable, API-driven OS. Cilium over Flannel/Calico for eBPF-based kube-proxy replacement and Hubble observability. Longhorn over NFS for distributed block storage with snapshots. ArgoCD over Flux for its UI and team familiarity. MetalLB with BGP over NodePort for true LoadBalancer IPs. USB SSDs over SD cards for reliability.
Why Not Managed Kubernetes?
The author runs AWS professionally but built this cluster to break things deliberately: "If I can't self-host it, do I really know it?" The homelab exposes gaps that CKA certification doesn't cover — like what happens when etcd runs out of disk or Cilium silently drops traffic.
Takeaway
Bare-metal Kubernetes on scrap hardware is feasible but demands understanding storage media, eBPF debugging, disk math, IP management, and backup hygiene. The author's cluster now runs production side projects with zero downtime since implementing these fixes.
