Lesson 24 — AI/ML platform: multi-cluster inference routing & capacity planning across cells

2026-10-02

What interviewers are really probing

Lesson 18 gave you SKU pools. Lesson 19 made NVLink / RDMA / NCCL schedulable. Lesson 20 admitted gangs. Lesson 21 made KV cache the capacity wallet and taught TTFT / TPOT / goodput. Lesson 22 split prefill vs decode. Lesson 23 gave you eyes — DCGM, NCCL canaries, roofline. This lesson is the fleet layer: how do you route inference traffic and plan capacity across many independent clusters/cells without turning “multi-cluster mesh” into a single blast-radius fantasy — and without shipping unsigned weights or a train SA onto the routers?

Interviewers at hyperscale AI platforms (fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab)) want evidence you have operated multi-cell inference under partial failure, not only “we put Envoy in front of one cluster”:

• Cell / region / AZ topology — independent control planes; blast-radius cells; why “one mesh for 100 clusters” is usually wrong (L1)
• Edge → cell routing — Anycast / GLB (L7); Gateway API / Envoy / optional Istio multi-cluster (L10); sticky vs free routing for KV affinity vs PD disagg (L21/L22)
• Capacity across cells — KV seats, prefill vs decode pools, roofline inputs (L23); headroom & admission ceilings
• Health-aware routing — TTFT/TPOT SLO burn; DCGM/XID cell quarantine; NCCL fabric health gates
• Model rollout / canary — digest-pinned weights across cells; weighted traffic shift; never “latest” tags
• Failover — cell drain, brownout, retry budgets; why you usually don’t ship decode KV cross-cell
• Discovery / east-west — MCS / Gateway composite backends / DNS / mesh optional — pick deliberately
• Observability — per-cell goodput, route-decision metrics, capacity dashboards
• Security floor — signed weights, private registries, RO mounts, GetObject-only WI, digests, NetPol across cell boundaries, admission/RBAC, no train SA on routers, mTLS between cells

Interview sentence: “I treat each inference cell as an independent blast-radius control plane; edge GLB + regional Gateway API/Envoy routes by capacity and SLO burn (KV seats, prefill/decode headroom, DCGM/NCCL health), keep decode affinity inside a cell and avoid cross-cell KV by default, canary digest-pinned weights with weighted traffic shift, and never put train SAs or writable weight mounts on the routing control plane.”

Core diagram: Edge → Gateway → Cells (PD + KV) → capacity/health CP (+ security floor)

Read it top-down, then the feedback loop: Clients hit an edge GLB (Anycast / geo / DDoS — L7). A regional Gateway (Gateway API / Envoy; mesh optional — L10) picks a cell using weights plus capacity and health signals. Each cell owns prefill, decode, and a local KV wallet (L21/L22). A fleet capacity & health control plane aggregates KV seats, roofline headroom (L23), SLO burn, DCGM/XID quarantine, and NCCL fabric health — then feeds route decisions and canary traffic shifts. Everything sits on a security floor: signed digest-pinned weights, NetPol across cell boundaries, mTLS east-west, no train SA on routers.

Idiosyncrasies / operator depth

1. Cell vs region vs AZ — blast radius first

Gotcha: “Multi-cluster service mesh with one trust root and automatic failover everywhere” sounds elegant in a diagram and becomes a correlated outage when a control-plane bug or CA mishap hits. Hyperscale mental model (L1): many small control planes, deliberate cross-cell contracts.

2. Edge → cell routing patterns (pick deliberately)

Interview trap: Equating “we have Istio” with “we have multi-cell inference routing.” Mesh gives identity and policy; capacity-aware routing still needs a decision plane fed by KV seats, SLO burn, and silicon health.

3. Sticky vs free routing — KV, sessions, PD disagg

Default posture: Free at the edge for cell selection (capacity + health), sticky inside the cell for decode. If you must move a live decode session, treat it as brownout / drain with reconnect, not silent KV teleport.

4. Capacity planning across cells (the wallet that interviews care about)

Cell capacity (operator formula):

cell_admit = min(kv_seats_free, prefill_flops_headroom, decode_hbm_headroom, fabric_ok, slo_budget_ok)

Fleet planner routes proportional to cell_admit (or ranked tiers: green / yellow / red), never pure RPS round-robin.

5. Health-aware routing — what actually moves weight to zero

6. Model rollout / canary across cells

7. Failover, drain, brownout, retries

8. Multi-cluster discovery without one mandatory mesh

Operator stance: mTLS and NetPol are mandatory between cells that speak; full mesh is optional. Many fleets run Gateway + SPIFFE/mTLS ferrymen only on the cross-cell paths that exist.

9. Observability — route decisions you can defend in an incident

Gotcha: Logging full prompts or tenant IDs into route-decision metrics recreates the L23 cardinality / privacy footgun. Keep cell + pool + digest_short + reason_code.

10. Interview trap — “just fail over the KV”

If decode affinity dies with a cell, re-prefill or reconnect. Shipping KV across cells “for availability” couples failure domains, adds a high-bandwidth trust boundary, and often loses the TTFT you were trying to save under load. Say that out loud.

Concrete commands & config

Fleet inventory — cells, pools, weights

# Cell registry (platform CRD / ConfigMap — shape only)
kubectl --context fleet-admin get inferencecells -A \
  -o custom-columns=NAME:.metadata.name,REGION:.spec.region,WEIGHT:.status.routeWeight,ADMIT:.status.admitCeiling,HEALTH:.status.health
 
# Per-cell pool snapshot
for CTX in cell-a cell-b cell-c; do
  echo "=== $CTX ==="
  kubectl --context "$CTX" -n inference get deploy,svc -l app.kubernetes.io/component in prefill,decode
  kubectl --context "$CTX" -n inference get cm cell-capacity -o yaml
done

Capacity snapshot → admit ceiling (runbook sketch)

# Pull binding constraints (replace with your Prom / API)
# kv_free, prefill_flops_headroom, decode_hbm_headroom, fabric_ok (0/1), slo_ok (0/1)
python3 - <<'PY'
kv, flops, hbm, fabric, slo = 1200, 0.35, 0.22, 1, 1
# headroom floors: keep 20% KV, 15% compute/mem as policy
admit = min(kv * 0.8, flops, hbm) * fabric * slo
print(f"cell_admit_score={admit:.3f}")
PY

Gateway API — weighted cell backends + canary digest header

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-inference
  namespace: edge-gw
spec:
  parentRefs:
    - name: regional-gw
  hostnames: ["infer.example.com"]
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1/
      backendRefs:
        - name: cell-a-infer
          port: 443
          weight: 40
        - name: cell-b-infer
          port: 443
          weight: 45
        - name: cell-c-infer
          port: 443
          weight: 15   # canary cell / digest
      filters:
        - type: RequestHeaderModifier
          requestHeaderModifier:
            add:
              - name: x-model-digest
                value: "sha256:REPLACE_CANARY_OR_PROD"
              - name: x-routing-policy
                value: "capacity-v1"
---
# Optional: sticky for decode resume paths (cookie / header hash) — cell-local preference
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llm-decode-sticky
  namespace: edge-gw
spec:
  parentRefs:
    - name: regional-gw
  hostnames: ["infer.example.com"]
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1/sessions/
      backendRefs:
        - name: cell-a-infer
          port: 443
          weight: 1
      # Real sticky often lives in Envoy consistent-hash or Inference Extension picker;
      # document which layer owns affinity — don’t double-stick edge + mesh.

Envoy cluster sketch — outlier detection + retry budget

# Fragment — wire via Gateway / Helm values; pin Envoy image digest
clusters:
  - name: cell_a
    connect_timeout: 1s
    type: STRICT_DNS
    lb_policy: LEAST_REQUEST
    load_assignment:
      cluster_name: cell_a
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address: { address: cell-a.infer.svc, port_value: 443 }
    outlier_detection:
      consecutive_5xx: 5
      interval: 10s
      base_ejection_time: 30s
      max_ejection_percent: 100
    # Circuit breakers / retry budgets belong at edge — prevent retry storms into a red cell

Helm values — regional router (shape)

# Shape only — digests, not tags, in prod
regionalRouter:
  image: registry.example.com/infer-router@sha256:REPLACE
  serviceAccount: infer-router   # NOT train-sa
  env:
    CAPACITY_API: https://capacity.fleet.example.com
    DEFAULT_POLICY: min_kv_flops_hbm_fabric_slo
    CROSS_CELL_KV: "false"
  resources:
    requests: { cpu: "2", memory: 4Gi }
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels: { app: infer-router }
          topologyKey: topology.kubernetes.io/zone

Cell drain / brownout / weight shift

CELL=cell-b
 
# 1) Mark yellow in capacity API / CR
kubectl --context fleet-admin patch inferencecell "$CELL" --type merge \
  -p '{"status":{"health":"yellow","routeWeight":5}}'
 
# 2) Stop new sticky admits; allow in-flight drain window
kubectl --context "$CELL" -n inference annotate deploy/decode \
  drain.platform/mode=brownout-prefill-first --overwrite
 
# 3) Hard drain when in-flight < threshold
kubectl --context fleet-admin patch inferencecell "$CELL" --type merge \
  -p '{"status":{"health":"red","routeWeight":0}}'
 
# 4) After fix: NCCL canary + DCGM green + digest verify before weight restore ([L23])

Per-cell goodput / route metrics (PromQL sketches)

# Goodput by cell
sum by (cell) (rate(inference_goodput_requests_total[5m]))

# Binding admit term (custom exporter)
min by (cell) (
  cell_admit_kv_seats_ratio,
  cell_admit_prefill_flops_ratio,
  cell_admit_decode_hbm_ratio,
  cell_admit_fabric_ok,
  cell_admit_slo_budget_ratio
)

# Edge selected cell vs planned weight
sum by (cell) (rate(router_selected_cell_total[5m]))
  /
sum by (cell) (router_desired_weight)

Kyverno / VAP — no train SA on routers (sketch)

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: router-privilege-floor
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["deployments"]
  validations:
    - expression: >
        !has(object.metadata.labels) ||
        object.metadata.labels["trust-domain"] != "infer-router" ||
        (
          object.spec.template.spec.serviceAccountName in
            ["infer-router", "edge-gw", "capacity-reader"] &&
          !object.spec.template.spec.serviceAccountName.startsWith("train")
        )
      message: "infer-router trust-domain must use router SAs — never train SA"

Failures & fixes

Security & hardening

The routing plane sits in front of every model you serve. A router with train credentials, a flat NetPol across cells, or unsigned “latest” weights turns multi-cluster routing into a fleet-wide exfil and supply-chain fan-out. Same spirit as L11–L14, applied to edge + regional + cell ingress.

AI/ML weight, model & deployment hardening on the multi-cell path (required)

Multi-cell routing does not relax L13/L14. Each cell hydrates signed, digest-pinned weights onto RO volumes with GetObject-only identity. Regional routers never mount /models, never reuse train SAs, and never hold PutObject on checkpoint buckets. Cross-cell NetPol + mTLS protect the ferry. Deployment admission still requires RuntimeClass, digest pins, and pool nodeSelectors on every cell — including the “temporary” canary cell.

# Sketch: multi-cell inference routing security floor (platform CR shape)
apiVersion: policy.platform/v1
kind: MultiCellInferGate
metadata:
  name: infer-routing-floor
spec:
  match:
    labels:
      trust-domain: infer-router
  require:
    imageDigestPin: true
    weightDigestPin: true
    cosignVerify: true
    privateRegistryOnly: true
    weightMountOnRouter: false
    weightMountOnServe: "readOnly"
    serveWorkloadIdentity:
      objectGetOnly: true
      putObject: false
    allowedServiceAccounts: ["infer-router", "edge-gw", "capacity-reader"]
    trainSaOnRouters: false
    crossCellKv: false
    networkPolicy:
      defaultDeny: true
      allowEastWest:
        - from: "namespace:edge-gw"
          to: "cell-ingress:443"
          mtls: required
    admission:
      runtimeClass: nvidia
      auditRouteWeightChanges: true
    canary:
      autoPauseOnSloBurn: true
      maxMixedDigestInStickyPool: 0

Punchline: Multi-cell routing is a security-critical control plane. Digest-pinned signed weights, GetObject-only hydrate, no train SA on routers, NetPol + mTLS across cell boundaries, and no casual cross-cell KV are what keep “smart capacity routing” from becoming a fleet-wide compromise path.

Interview soundbites

• “Cells are blast-radius units — independent control planes, deliberate cross-cell contracts.”
• “Edge GLB for geo/DDoS; Gateway/Envoy for weighted capacity routes — mesh is optional identity, not magic routing.”
• “Free at the edge for cell pick; sticky inside the cell for decode KV.”
• “cell_admit = min(KV seats, prefill FLOPs, decode HBM, fabric, SLO budget) — the minimum wins.”
• “XID / NCCL / TTFT burn all move weight — GPUs green ≠ cell green.”
• “Canary digest-pinned weights with weighted shift; never :latest across cells.”
• “Don’t teleport decode KV cross-cell by default — reconnect/re-prefill beats coupled failure domains.”
• “Retry budgets + outlier ejection, or one red cell becomes a retry amplifier.”
• “No train SA on routers; RO weights; GetObject-only; NetPol + mTLS between cells.”
• “Next: multi-tenant GPU isolation and quota economics across cells.”

How this maps to the JD

Hyperscale + AI/ML platform JDs that mention 100+ clusters, inference serving, and GPU fleets are asking whether you can route and plan capacity across cells under partial failure — not run a single-cluster Envoy demo. This lesson lifts L18–L23 into the fleet layer with L1/L5–L7/L9/L10/L13–L14 controls. Fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab). Next up — Lesson 25: multi-tenant GPU isolation & quota economics across cells.

Watch (talks & demos)

Real talks — watch after Failures & fixes.

Building the Next Generation of Multi-Cluster with Gateway API

KubeCon talk (Mike Morris & Alan Grosskurth) on ClusterIP Gateways, endpoint selectors, and composite backends for aggregating backends across clusters. Operator takeaway: multi-cluster inference needs an explicit backend aggregation story in Gateway API terms — not “one Service DNS name forever.” Pair with capacity signals from this lesson; APIs alone don’t know your KV seats.

Open on YouTube (opens in a new tab)

NVIDIA Dynamo — LLM Inference in Multi-Node Distributed Environments

Hands-on / overview of Dynamo’s distributed serving stack (multi-node inference, routing adjacent to PD pools). Operator takeaway: treat KV-aware / disagg routers as cell-local brains feeding a fleet weight plane — don’t confuse engine-level KV routing with cross-cell GLB decisions (L21/L22).

Open on YouTube (opens in a new tab)

Tutorial: Configuring Your Service Mesh with Gateway API

Gateway API + mesh (Linkerd/Istio) workshop for routing, resilience, and progressive delivery. Operator takeaway: use Gateway for portable canary weights; use mesh selectively for mTLS/identity between cells — don’t merge them into one mandatory monolith (L10).

Open on YouTube (opens in a new tab)

References

• Gateway API (opens in a new tab) · Gateway API Inference Extension (opens in a new tab) · kubernetes-sigs/gateway-api-inference-extension (opens in a new tab)
• Multi-Cluster Services (MCS) (opens in a new tab) · Istio multi-cluster (opens in a new tab) · Envoy outlier detection (opens in a new tab)
• NVIDIA Dynamo docs — KV routing (opens in a new tab) · Routing concepts (opens in a new tab)
• wafer-ai/gpu-perf-engineering-resources (opens in a new tab) — fleet / inference arithmetic fuel
• Prior: 23 — GPU observability · 22 — Prefill/decode disagg / MoE · 21 — Inference serving SLOs · 18 — GPU node pools · 14 — ML deploy hardening · 13 — ML weights · 10 — Service mesh · 09 — NetworkPolicy · 07 — Edge LB / DDoS · 05 — Cloud networking · 01 — Hyperscale mental model
• Next: Lesson 25 — AI/ML platform: multi-tenant GPU isolation & quota economics across cells

Quick self-check

Out loud, in under two minutes:

1. Define a cell vs a region vs a pool — and why independent control planes matter for blast radius.
2. Explain free-at-edge vs sticky-in-cell for decode KV, and why cross-cell KV is usually a bad default.
3. Write cell_admit = min(...) with five inputs and say which signal moves weight to zero first in an XID storm.
4. Walk a digest-pinned canary: hydrate → weight shift → auto-pause on SLO burn → rollback.
5. Drain a yellow cell without retry-amplifying it (outlier eject, retry budget, brownout).
6. List five security controls on the routing path — signed digests, GetObject-only, no train SA on routers, NetPol + mTLS across cells, no casual cross-cell KV.