Lesson 8 — Cluster networking: CNI, Cilium, eBPF fundamentals

2026-09-16

What interviewers are really probing

Lessons 05 and 06 built the cloud fabric; 07 was the public front door. This lesson is east-west inside the cell — what happens after the packet is past the edge and lands on a worker.

They want an operator story, not a CNCF buzzword bingo: CNI ADD/DEL/CHECK and IPAM modes, why kube-proxy iptables melts at Service scale, what Cilium’s eBPF datapath replaces (maps, identities, endpoints), and how you prove policy with Hubble instead of guessing from tcpdump. Bonus if you name DatapathMode (veth vs native routing), MTU/jumbo traps with overlay, ENI IP exhaustion, and refuse “NetworkPolicy exists so we’re secure” without identity-aware defaults.

Core diagram: east-west after the front door

Read it top-down: Lesson 7 delivers north-south to the node. On the node, eBPF (XDP/tc) and either kube-proxy or Cilium kube-proxy-replacement steer Service traffic using BPF maps. The CNI plugin (conf under /etc/cni/net.d/) plus cilium-agent allocate IPs, create endpoints, and stamp security identities from labels. Policy (CNP/CCNP) and Hubble sit on that identity plane — east-west between app and inference pods is where you isolate model endpoints after the packet is already inside the cell.

Cross-link: VPC/TGW (05) and Interconnect/NAT (06) are between cells and clouds. Edge LB (07) is ingress scrubbing. This lesson is pod-to-pod / pod-to-Service inside one cluster.

CNI interface & IPAM — what actually runs on ADD

CNI is a contract, not a brand. On pod create, kubelet invokes the configured plugin with ADD (create veth/move link, assign IP, write routes/DNS); on delete, DEL; readiness/liveness of the setup can use CHECK. Misconfigured conf files under /etc/cni/net.d/ (wrong binary, stale 10-*.conflist, dual plugins fighting) show up as FailedCreatePodSandBox long before “Cilium is slow.”

IPAM modes (say which one you run in interviews):

# Which CNI won on the node?
ls -la /etc/cni/net.d/
cat /etc/cni/net.d/*.conflist 2>/dev/null | head -80
 
# Cilium agent alive?
kubectl get pods -n kube-system -l k8s-app=cilium -o wide
kubectl -n kube-system exec ds/cilium -- cilium status

kube-proxy vs Cilium kube-proxy-replacement

kube-proxy implements Services with iptables (default) or IPVS. At hyperscale Service/Endpoint counts, iptables rule programming becomes a control-plane + node CPU tax: every Endpoint update rewrites chains; conntrack tables swell; NodePort hairpinning surprises you.

Cilium kube-proxy-replacement moves ClusterIP / NodePort / ExternalIP (and more) into eBPF: socket-level or tc/XDP LB, BPF maps for backends, fewer netfilter crossings. You still need a clear story on hostPort, externalTrafficPolicy: Local, and health-check NodePorts — replacement is not “delete kube-proxy and forget Services.”

# Are we still running kube-proxy?
kubectl get ds -n kube-system kube-proxy -o wide 2>/dev/null
kubectl -n kube-system exec ds/cilium -- cilium status --verbose | rg -i 'kube-proxy|masquerade|routing|ipam'
 
# Node iptables pressure (if not fully replaced)
sudo iptables-save | wc -l
sudo conntrack -C 2>/dev/null || cat /proc/sys/net/netfilter/nf_conntrack_count

Interview nuance: IPVS helps kube-proxy scale, but Cilium’s pitch is one datapath for routing + LB + policy + observability — not “IPVS bad.” Say what you measured (rule count, programming latency, P99) rather than slogan.

Cilium operator depth — identity, endpoints, maps

Cilium does not primarily policy on IP:port. Labels resolve to a numeric security identity; endpoints (pods) carry that identity; BPF policy maps allow/deny by identity. IPs churn; identities are stable for a label set — that is why identity-aware policy survives rolling deploys better than brittle IP ACL fantasy.

kubectl -n kube-system exec ds/cilium -- cilium endpoint list
kubectl -n kube-system exec ds/cilium -- cilium identity list | head
kubectl get cep -A | head
kubectl get cnp,ccnp -A

eBPF fundamentals for operators (not kernel hackers)

You do not write C in the interview. You do know: cilium status red is a stop-ship; bpftool prog show / bpftool map show prove programs are loaded; map pressure and verifier rejections show up as agent logs, not as “kubectl network flake.”

# On a node (or via debug pod with privileges)
bpftool prog show | head
bpftool map show | head
ip -d link show | head
# Cilium’s view of the datapath
kubectl -n kube-system exec ds/cilium -- cilium status

Encryption, DatapathMode, hyperscale quirks

• WireGuard vs IPsec: WireGuard is often the operationally simpler transparent encryption; IPsec shows up for compliance matrices and older requirements. Encryption costs CPU and can interact with MTU — always re-validate MSS/MTU after enabling.
• DatapathMode: veth (classic) vs native routing (pods routable without overlay). Overlay (VXLAN/Geneve) eases IP continuity across L2 domains but burns MTU and buries headers; native routing wants cloud route/ENI alignment (05).
• MTU / jumbo: Overlay + jumbo + GPU RDMA multi-NIC fleets is a classic foot-gun — east-west training traffic is not the same as ClusterIP HTTP.
• Conntrack pressure: Even with eBPF CT, abuse and short-lived probes fill tables; NodePort to the world amplifies.
• hostPort / NodePort: hostPort pins host ports and fights scheduling density; externalTrafficPolicy: Local needs real local backends or you blackhole.
• ENI IP exhaustion: AWS VPC CNI / Cilium ENI mode — instance type limits secondary IPs; prefix delegation changes the math; pending pods with no obvious CPU pressure → check ENI/IP inventory.
• ClusterMesh (tease only): multi-cluster identities + service export; next-level after a single cell is healthy — do not hand-wave it as “just another peering.”

Concrete commands & config

# Agent health + feature flags
kubectl get pods -n kube-system -l k8s-app=cilium -o wide
kubectl -n kube-system exec ds/cilium -- cilium status
kubectl -n kube-system exec ds/cilium -- cilium-health status
 
# Endpoints & identities
kubectl -n kube-system exec ds/cilium -- cilium endpoint list
kubectl -n kube-system exec ds/cilium -- cilium bpf tunnel list 2>/dev/null | head
kubectl -n kube-system exec ds/cilium -- cilium bpf lb list 2>/dev/null | head
 
# Hubble — see allows/denies with reason
kubectl -n kube-system exec ds/cilium -- hubble observe --since 5m --verdict DROPPED
kubectl -n kube-system exec ds/cilium -- hubble observe --since 5m \
  --from-pod tenant-a/frontend --to-pod tenant-a/backend
 
# CNI conf + link detail on the node
ls /etc/cni/net.d/
ip -d link show
bpftool prog show | rg -i cilium

# CiliumNetworkPolicy — identity-aware L3/L4 (illustrative)
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: tenant-a
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
---
# Clusterwide default: allow DNS, deny world from sensitive NS (sketch)
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: sensitive-default-deny-egress
spec:
  endpointSelector:
    matchLabels:
      security: sensitive
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP

# Apply + confirm endpoint policy status
kubectl apply -f cnp-frontend-backend.yaml
kubectl -n kube-system exec ds/cilium -- cilium endpoint list | rg tenant-a

Failures & fixes

Security & hardening

East-west is where inference pods and model endpoints get scraped by anything that already cleared the front door. CNI/eBPF policy is your microsegmentation; privileged agents and CRDs are part of the attack surface — treat them like control-plane.

# Who can edit fleet-wide policy?
kubectl auth can-i create ciliumclusterwidenetworkpolicies --as=system:serviceaccount:ci:deployer
kubectl get ccnp -A
 
# Prove model NS is not world-open east-west
kubectl -n kube-system exec ds/cilium -- hubble observe --since 15m \
  --to-namespace model-serving --verdict FORWARDED

AI/ML fleets: after Lesson 7’s edge authn, assume a compromised tenant pod will probe /v1/completions on neighbor ClusterIPs. Identity-aware default-deny + Hubble on the model namespace is how you keep weights and GPU capacity from becoming an in-cluster free API.

Interview soundbites

• “CNI is ADD/DEL/CHECK plus IPAM — brands are implementations.” Start from the contract and the failure modes.
• “kube-proxy iptables is a rule compiler; eBPF LB is a map lookup.” Scale story beats product loyalty.
• “Cilium policies identities, not forever-IPs.” Labels → security identity → BPF policy maps.
• “Hubble drop reason before opening the firewall.” Observability is part of the datapath sale.
• “Native routing vs overlay is an MTU and cloud-route decision.” Not a religion.
• “Privileged CNI agents are trusted computing base.” RBAC the CRDs; pin versions; watch exec.
• “Encryption without policy is confidentiality without authorization.” WireGuard ≠ allowlist.

How this maps to the JD

Hyperscale compute roles own cell networking under churn: IPAM that does not strand capacity, Service LB that survives Endpoint storms, and east-west isolation that still lets platform traffic (DNS, metrics) work. You should sound fluent moving from VPC/TGW (05) and edge (07) down into CNI/eBPF, then forward into NetworkPolicy depth.

Next in the arc: NetworkPolicy, multi-NIC, sFlow.

Watch (talks & demos)

Real-world talks and demos that map to this lesson — watch after the Failures & fixes table.

Liberating Kubernetes From Kube-proxy and Iptables — Martynas Pumputis, Cilium (CNCF)

Why iptables/netfilter struggle as the Service datapath and how eBPF + Cilium replace kube-proxy — foundational for the “maps not chains” interview answer.

Open on YouTube (opens in a new tab)

Cilium: Connecting, Observing, and Securing Kubernetes and Beyond with eBPF — B. Mulligan & N. Bisht (CNCF)

Hyperscale-shaped story (large node/pod counts, east-west QPS) on adopting Cilium for connectivity, Hubble-style observability, and security — how the mental model holds past a demo cluster.

Open on YouTube (opens in a new tab)

Kubernetes Network Observability with Cilium and Hubble — Tomasz Tarczyński

Day-2 operator view: flows L2–L7, metrics that matter, using observability to implement microsegmentation, and watching the CNI dataplane itself — pairs with the Hubble commands above.

Open on YouTube (opens in a new tab)

References

• CNI specification (opens in a new tab)
• Cilium documentation (opens in a new tab)
• Cilium kube-proxy replacement (opens in a new tab)
• CiliumNetworkPolicy (opens in a new tab)
• Hubble observability (opens in a new tab)
• eBPF docs (opens in a new tab)
• bpftool (opens in a new tab)
• Kubernetes Network Policies (opens in a new tab)
• Prior lessons: 05 — VPC / TGW, 06 — Interconnect / NAT / BGP, 07 — Edge LB / DDoS

Quick self-check

Out loud, in under two minutes:

1. Walk CNI ADD on a new pod — where does the IP come from in cluster-pool vs ENI mode, and what breaks when that pool is empty?
2. Why does kube-proxy iptables hurt at high Service/Endpoint churn, and what does Cilium put in BPF maps instead?
3. How does a Cilium security identity differ from “allow this pod IP,” and how would Hubble show a policy drop?
4. Name two hyperscale foot-guns: MTU with overlay vs native routing, and NodePort/externalTrafficPolicy: Local — what do you check first?