Lesson 1 — Hyperscale mental model: 100+ clusters / 10K+ nodes

2026-09-08 · refreshed 2026-09-10

What the requirement actually means (operator view)

Interviewers saying “100+ clusters / 10K+ nodes” are not asking if you can kubectl get nodes on a big GKE cluster. They are asking whether you have run a fleet under partial failure: many independent control planes, continuous node death, and automation that treats clusters as cattle — not pets you SSH into.

I have woken up to dashboards where three cells were red, twelve were yellow, and the fleet was still fine because capacity headroom and cell isolation were designed that way. That is the bar.

At hyperscale, the unit of thinking shifts from pod → cluster (cell) → fleet.

Core diagram: fleet topology

Read the diagram top-down:

1. Fleet plane — inventory, lifecycle, policy, observability across all clusters. This is where you page, not into a random bastion.
2. Regions / cells — many small(ish) clusters instead of one mega-cluster. A cell is a blast-radius boundary.
3. Inside a cell — node pools by SKU (general, GPU, spot); noisy neighbors stop at the pool or the cell, not the company.

Why not one giant cluster? (what I tell architects)

Rule of thumb: prefer more clusters that are boring over fewer clusters that are heroic. Heroic clusters create heroic outages.

Idiosyncrasies you only learn in production

• Cell size is a product decision, not a kube default. I have seen teams aim for ~200–800 nodes/cell so etcd stays under a few GB and upgrade windows fit a night. GPU cells often smaller because driver/NPD noise is higher.
• Naming is an API. use1-prod-cell-07 beats cluster-blue. Encode region, env, purpose, generation — your paging, Terraform state, and CI matrices all hang off it.
• CIDR planning is fleet work. Overlap pod CIDRs across cells and you cannot do east-west without NAT hell. Reserve supernets per region; allocate per cell from a registry (IPAM as a service).
• Spot/preemptible pools lie about capacity. At 10K nodes, “max: 2000 spot” is marketing until you model AZ supply and interruption rate. Keep on-demand floor capacity per critical workload class.
• Fleet inventory drifts. Cloud consoles, Cluster API, and homegrown CMDB disagree. The source of truth must be reconciler-driven (desired Cluster CR → actual cloud resources), not a spreadsheet.
• “Cluster Ready” ≠ “serving traffic.” Control plane healthy while CNI, DNS, or node-problem-detector is wedged is a classic false green. Gate readiness on a synthetic probe Deployment + SLO, not just kubectl get --raw='/readyz'.

Concrete commands & config (operator muscle memory)

# --- Fleet inventory sanity (Cluster API example) ---
kubectl get clusters.cluster.x-k8s.io -A -o wide
kubectl get machinedeployments,machines -A \
  -l cluster.x-k8s.io/cluster-name=use1-prod-cell-a
 
# How many nodes per cell, labeled the way paging expects
kubectl get nodes -L topology.kubernetes.io/zone,node.kubernetes.io/instance-type,node-pool \
  --context use1-prod-cell-a | awk 'NR>1 {z[$3]++; n++} END {print "nodes",n; for (i in z) print i,z[i]}'
 
# Spot interruption / NotReady burn (last hour) — adjust LogQL/PromQL to your stack
# Prometheus: rate of NotReady
# sum(rate(kube_node_status_condition{condition="Ready",status="false"}[15m])) by (cluster)
 
# Cap a bad rollout: cordon a canary cell's worker pools without touching control plane
kubectl cordon -l node-pool=general --context use1-prod-cell-canary
kubectl get nodes -l node-pool=general --context use1-prod-cell-canary
 
# Synthetic readiness: does DNS + a probe Deployment still work?
kubectl -n fleet-probes get deploy/probe --context use1-prod-cell-a
kubectl -n fleet-probes rollout status deploy/probe --timeout=60s --context use1-prod-cell-a

# Conceptual fleet inventory — what a lifecycle system tracks
apiVersion: fleet.example.com/v1
kind: Cluster
metadata:
  name: use1-prod-cell-a
  labels:
    region: us-east-1
    env: prod
    purpose: general-compute
    cell: "a"
spec:
  region: us-east-1
  environment: prod
  purpose: general-compute   # vs gpu | batch | sandbox
  kubernetesVersion: "1.30"
  networking:
    cni: cilium
    # Allocated from regional supernet registry — never ad hoc
    podCidr: 10.40.0.0/16
    serviceCidr: 10.96.0.0/16
  nodePools:
    - name: general
      min: 50
      max: 800
      instanceType: m7i.4xlarge
      capacityType: on-demand
    - name: spot-batch
      min: 0
      max: 2000
      instanceType: m7i.2xlarge
      capacityType: spot
  readiness:
    # Don't trust control-plane Ready alone
    requireSyntheticProbe: true
    maxNotReadyFraction: 0.05
status:
  phase: Ready
  nodeCount: 612
  controlPlaneHealthy: true
  syntheticProbeHealthy: true
  lastUpgrade: "2026-09-01T04:00:00Z"
  lastUpgradeCellWave: "2026-09-01-wave-2"

Map this in interviews to Cluster API, GKE Fleet / Anthos, EKS + custom controllers, or an internal Borg/Omega-style allocator. The CR shape matters less than: desired state, reconciliation, readiness beyond apiserver.

Failure is the steady state

At 10K+ nodes you should assume:

• Nodes die every day (hardware, preemption, kernel panics).
• A region or AZ degrades while others are fine.
• A bad rollout hits some cells first (if your pipeline is correct).
• Humans cannot SSH-and-fix; controllers + runbooks + SLOs are the product.

Design for partial failure: retries with jitter, zone-aware scheduling, multi-cell capacity headroom, and clear ownership of “who pages when cell-17’s API server is wedged.”

Failures & fixes

Security & hardening

At fleet scale, blast-radius isolation is a security control, not only an availability story. A compromised cell, leaked deploy SA, or poisoned admission webhook should not become a company-wide incident — that is why cells exist.

# Inventory: which identities can mutate every cell? (smell = one SA everywhere)
kubectl --context fleet-mgmt get clusterrolebindings -o wide
# Prefer per-cell RoleBindings + short-lived OIDC tokens over long-lived shared kubeconfigs

AI/ML fleets: treat model-weight buckets, checkpoint stores, and GPU node pools as high-value tenancy. Sandbox training cells must not share pull credentials or registry write paths with prod inference cells — isolate weights/checkpoints/images the same way you isolate blast radius.

Interview soundbites (steal these)

1. “Cluster = blast-radius boundary.” I size cells so a control-plane or CNI incident doesn’t take the company down.
2. “Fleet plane ≠ workload plane.” Provisioning, policy, and observability are cross-cutting products; apps stay in cells.
3. “Change is a pipeline, not a ticket.” Progressive delivery across cells (canary → 10% → 50% → rest) with automatic halt on SLO burn.
4. “Scale problems move up a layer.” At 10K nodes the hard problems are identity, networking topology, and lifecycle automation — not “how do I schedule a pod.”
5. “Ready means serving, not apiserver 200.” I gate cell membership on synthetic probes and error budgets, not vanity health endpoints.

How this maps to the rest of your JD

Watch (talks & demos)

Real-world talks and demos that map to this lesson — watch after the Failures & fixes table.

Cluster Management at Google with Borg — John Wilkes (GOTO 2016)

Cells, Borglets, partial failure, and why “more boring clusters” is a product decision — the vocabulary behind Lesson 1.

Open on YouTube (opens in a new tab)

Complexities of Capacity Management for Distributed Services — Google SRE Tech Talk

How SREs provision and plan capacity across large distributed services; pairs with fleet headroom and cell sizing.

Open on YouTube (opens in a new tab)

References

• Large-scale cluster management at Google with Borg (opens in a new tab) — the classic “why cells / why Borg”
• Omega: flexible, scalable schedulers (opens in a new tab) — shared-state vs monolithic schedulers
• Kubernetes scalability thresholds (opens in a new tab) — official large-cluster guidance
• Cluster API book (opens in a new tab) — declarative cluster lifecycle (industry pattern for fleets)
• Google SRE Book — Embracing Risk (opens in a new tab) — error budgets as the language of hyperscale change
• GKE Fleet management (opens in a new tab) — multi-cluster fleet plane in the wild

Quick self-check

If someone asks: “Would you run 10K nodes in one cluster or fifty clusters of ~200?” — answer with blast radius, upgrade safety, API server/etcd load, tenancy, CIDR/IPAM, and how you’d measure before deciding (watch count, etcd DB size, NotReady burn rate, error budget). Then say you’d still keep a fleet plane either way.

Next: Lesson 2 — Kubernetes control plane internals at scale