Lesson 35 — AI/ML platform: fleet-wide incident response & on-call runbooks for GPU, inference and agent workloads

2026-10-13

What interviewers are really probing

Lessons 18–34 built the machine: GPU pools and topology (L18/L19), serving SLOs (L21), PD/MoE (L22), eyes (L23), cells (L24), weights (L27), autoscaling (L28), training reliability (L29/L30), release gates (L31–L33) and sandboxes (L34). This lesson is about 3 a.m. on a Saturday, when one of those pieces breaks on 10K nodes and you are the one holding the pager.

The interview version is usually a scenario: "TTFT p99 just tripled in one region", "a 4K-GPU training run has hung for 40 minutes", "security says an agent emailed a customer list somewhere". They are not checking whether you know what a GPU is. They want to see whether you can run an incident: size it, contain it without making it bigger, mitigate before you root-cause, talk to people while it's happening, and leave the fleet better afterwards.

Cross-links: L1 (cells = blast radius), L2 (control-plane overload, APF), L8–L11 (Cilium/Hubble, NetworkPolicy, mTLS, RBAC/admission), L13/L14 (weight + deploy hardening), L15/L16 (GitOps + workflow rollbacks), and every lesson from 19 to 34. Fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab): platform depth, not CUDA.

What they check for:

• Severity on ML-specific axes. Users affected and SLO burn, yes. But also GPU-hours burning (a hung 16K-GPU job at about $2/GPU-hr wastes about $32K an hour with zero user impact), model-quality incidents (bad weights that are fast and wrong), and security incidents (weight exfil, unapproved side effects), which are SEV1 regardless of latency.
• Mitigate first, diagnose second. Do you reach for the fastest reversible lever (a route weight, a CatalogBundle label, a Kueue hold, a kill switch) before you open dmesg?
• Command structure. IC / Ops / Comms (Google IMAG), plus the ML roles: model/release owner, capacity lead, security lead. Who decides to roll back weights?
• Domain runbooks. GPU hardware (XID/ECC/NVLink/thermal), inference SLO collapse (KV pressure, preemption storms, PD imbalance), training storms (hangs, checkpoint storms, stragglers), agent sandbox blowups (egress spikes, injection), release incidents (bad canary, registry outage, hydrate storm), and cell failures.
• Blast-radius discipline. Do you know that automation can be the outage (a remediation controller cordoning half a cell because of a bad exporter), and that a careless failover can push a second cell over?
• Incident-time security. Break-glass that is time-boxed and audited, containment that narrows access instead of widening it, evidence preserved, no "just make the bucket public so the hydrate works."

Interview sentence: "I run ML platform incidents like any SRE incident, with IMAG roles, SLO-burn paging and mitigate-before-root-cause, and I add three ML-specific axes: GPU-hours burning, model quality, and security side effects. Every runbook starts with 'what changed' across weights, serving image, CatalogBundle, driver, node image and policy. Then I pull the cheapest reversible lever with the smallest blast radius: drain one cell after checking headroom, flip a bundle label, hold a Kueue queue, or kill-switch a tool class. Automation gets rate limits so it can't become the outage. Break-glass is time-boxed, two-person and audited, and containment always narrows access. I never fix a hydrate by turning off signature checks or widening bucket IAM."

Core diagram: Signals → Triage & SEV → Incident command → Domain levers → Verify → Learn (+ security floor)

How to read it: signals come in from the left already grouped per cell, so 400 nodes throwing XIDs open one incident, not 400 pages. Triage decides severity on four axes and asks what changed. The IC hands out levers by role: Ops owns traffic, queues and nodes, the model owner owns weight and bundle rollback, and security owns kill switches. Every lever is reversible and scoped to a cell or tenant. The loop closes with postmortems that produce new alerts, runbook steps and automation limits. The PNG adds the per-domain runbooks, SLIs and the incident-time security floor.

Idiosyncrasies / operator depth

1. A SEV matrix for ML platforms (impact is four-dimensional)

Two rules interviewers like hearing. First, security and quality incidents don't wait for latency: a model that is fast and wrong is worse than one that is slow. Second, SEV can only go up during the incident. Downgrade in the postmortem, never mid-incident to dodge process.

2. Page vs ticket at 10K nodes: alert design is incident prevention

• Page on symptoms with burn rate, ticket on causes. Page when TTFT/goodput SLO burn is 14.4× over 1h and 5m, or 6× over 6h and 30m (the SRE workbook multiwindow pattern). A single XID 79 on one node is a ticket for automation, not a human page. NVIDIA reported about 19 remediations per 1,000 nodes per day on GeForce NOW, so at 10K nodes that is about 190 events a day. If those page, on-call burns out in a week.
• Group by cell, inhibit downstream. When CellTTFTBurnFast fires, inhibit per-pod and per-node alerts in that cell. When NodeNotReady fires, inhibit that node's DCGM alerts.
• Page on the remediation system, not the remediation. Alert when the remediation rate is anomalous: more than N quarantines per cell per hour, the same node remediated twice in 90 minutes (NVIDIA's "flapping node" alert), or the quarantined fraction above budget.
• Security tripwires always page: any sandbox connection to metadata or the kube API, any money tool without approval, any read of models/* by a non-serving principal.

3. Roles: IMAG plus the ML-specific seats

4. "What changed?" The ML platform has six change streams, not one

Most incidents are changes. Check all of these, per cell, for the last 2–6 hours:

1. Weights digest (L27/L31): canary %, Argo Rollout step.
2. CatalogBundle (L33): prompt / toolset / serving-config digests. A max_num_seqs or gpu_memory_utilization change here causes KV OOMs.
3. Serving image (vLLM/SGLang/TRT-LLM version, CUDA, NCCL).
4. Node layer: driver, firmware, GPU Operator, kernel, node image, DCGM exporter version.
5. Network / policy: Cilium, NetworkPolicy, egress proxy allowlists, sandbox template version (L34).
6. Demand: a tenant launch, a new agent product, a traffic shift from another cell that drained earlier.

The trap: stream 6 looks exactly like a regression. If TTFT rose in cell A right after cell B drained, the fix is capacity, not a rollback.

5. Domain runbooks: symptom, first lever, and the gotcha

GPU hardware / driver / fabric (L19, L23, L29)

Fleet rule: automate cordon + drain + reset/reboot (NVSentinel, Medik8s NHC, or your own controller), but put a budget on it: max N quarantines per cell per hour, and a minHealthy floor. A buggy DCGM exporter release that reports phantom XIDs must not be able to cordon a cell.

Inference SLO fires (L21, L22, L28)

Gotcha: autoscalers fight you during incidents. If the registry is down, scale-in destroys warm capacity you cannot hydrate back. Pause KEDA/HPA scale-in first.

Training job storms (L20, L29, L30)

Gotcha: restart is not free. A 4K-GPU restart costs R minutes of the whole gang (L29). Don't restart into the same bad node. Exclude it first.

Agent / tool sandbox blowups (L33, L34)

Weight / release incidents (L27, L31, L32)

Multi-cluster / multi-cell (L1, L24)

• Drain is a capacity decision. Before setting a cell's weight to 0, the capacity lead confirms the remaining cells have KV-seat and prefill headroom for its traffic plus the cold-cache penalty (prefix-cache misses mean the receiving cells run more prefill per request).
• Shift in steps (100, 50, 0), watching the receiving cells' TTFT burn. Failover cascades are the classic multi-cell outage: drain A, overload B, drain B, and now it's a region.
• Quarantine vs drain. Quarantine stops new routing and new scheduling but keeps pods for evidence and fast return. Drain moves everything off. Security incidents usually want quarantine.
• Never fix one cell by changing global config. If the mitigation edits fleet-wide state (global route table, shared policy, base bundle), it needs a second approver.

6. Rollback order: fastest and smallest first

Rollback targets must be previously signed and attested digests. A rollback is a deploy and goes through the same admission path.

7. Communication

• First update within 15 minutes of SEV1/2 declaration, even if all it says is "investigating elevated latency for model X in region Y."
• Status page granularity = product × region, never cell IDs or node names.
• The incident channel is a data store. No customer prompts, tokens or PII pasted in. Link to the audit-store query instead.
• Handoffs are explicit: "IC handing to @name at 04:00 UTC; open threads: A, B; next update 04:30."

8. Game days and chaos for GPU and agent fleets

9. Postmortems that change the fleet

Blameless, with a timeline from the scribe. Metrics: TTD, TTM, TTR, GPU-hours lost, requests failed, tokens refunded. Every postmortem produces at least one of these: an alert, a runbook step, a guardrail on automation, or a game-day scenario. Cluster incidents by root-cause class quarterly. Meta's "reliability flywheel" work found recurring cross-team patterns that no single on-call would spot.

Concrete commands & config

Triage: what changed in this cell?

CELL=us-east-3
# Recent cluster events, newest last
kubectl --context "$CELL" get events -A --sort-by=.lastTimestamp | tail -50
 
# Weight / image rollouts and their current step
kubectl --context "$CELL" argo rollouts list rollouts -A
kubectl --context "$CELL" argo rollouts get rollout llm-70b -n serve-chat
 
# CatalogBundle digests (L33): what is live vs last-known-good
kubectl --context "$CELL" get catalogbundles -A \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,VER:.spec.catalogVer,PROMPT:.spec.prompt.digest,WEIGHTS:.spec.weightDigest
 
# GitOps history (Argo CD) for serving + node layers
argocd app history serve-chat-$CELL | tail -5
argocd app history gpu-operator-$CELL | tail -5
 
# Driver / image skew across GPU nodes
kubectl --context "$CELL" get nodes -l nvidia.com/gpu.present=true \
  -L nvidia.com/cuda.driver-version.full,node.corp/image-version --no-headers \
  | awk '{print $(NF-1), $NF}' | sort | uniq -c

PromQL an on-call actually runs

# TTFT p99 by cell (vLLM)
histogram_quantile(0.99,
  sum by (cell, le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))

# Fast-burn check: fraction of requests over the 2s TTFT SLO, 1h vs 5m
(
  1 - sum by (cell) (rate(vllm:time_to_first_token_seconds_bucket{le="2.0"}[1h]))
    / sum by (cell) (rate(vllm:time_to_first_token_seconds_count[1h]))
) / (1 - 0.99)

# KV pressure + preemption churn (batching collapse)
max by (cell, pod) (vllm:kv_cache_usage_perc) > 0.95
sum by (cell) (rate(vllm:num_preemptions_total[5m])) > 0

# Queueing vs running
sum by (cell) (vllm:num_requests_waiting) / sum by (cell) (vllm:num_requests_running)

# GPU fleet health
count by (cell) (DCGM_FI_DEV_XID_ERRORS > 0)
sum by (cell, Hostname) (increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[1h])) > 0
max by (cell, Hostname) (DCGM_FI_DEV_ROW_REMAP_PENDING) > 0

# Quarantine budget: fraction of GPU nodes tainted in each cell
count by (cell) (kube_node_spec_taint{key="gpu.corp/quarantine"})
  / count by (cell) (kube_node_labels{label_nvidia_com_gpu_present="true"})

# Training: pending vs admitted in Kueue
sum by (cluster_queue) (kueue_pending_workloads{status="active"})

Alertmanager: page on symptoms, group by cell, inhibit the noise

route:
  receiver: ticket-queue
  group_by: ["cell", "alertname"]
  group_wait: 30s
  group_interval: 5m
  routes:
  - matchers: ['severity="page"']
    receiver: pagerduty-ml-platform
  - matchers: ['category="security"']
    receiver: pagerduty-security
    continue: true
inhibit_rules:
- source_matchers: ['alertname="CellTTFTBurnFast"']
  target_matchers: ['alertname=~"PodTTFTHigh|VllmQueueDeep|KVCacheHigh"']
  equal: ["cell"]
- source_matchers: ['alertname="NodeNotReady"']
  target_matchers: ['alertname=~"DCGM.*|GPUThermal.*"']
  equal: ["node"]

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: {name: ml-platform-ir, namespace: monitoring}
spec:
  groups:
  - name: inference-slo
    rules:
    - alert: CellTTFTBurnFast
      expr: |
        cell:ttft_slo_burn:rate1h > 14.4 and cell:ttft_slo_burn:rate5m > 14.4
      labels: {severity: page, team: inference}
      annotations:
        runbook_url: https://runbooks.corp/ml/inference-ttft-burn
        summary: "TTFT SLO fast burn in {{ $labels.cell }}"
  - name: remediation-guardrails
    rules:
    - alert: QuarantineRateAnomalous
      expr: sum by (cell) (increase(gpu_quarantine_actions_total[1h])) > 20
      labels: {severity: page, team: gpu-fleet}
      annotations:
        summary: "Auto-remediation quarantined >20 nodes/h in {{ $labels.cell }}: suspect exporter or rollout, consider pausing automation"
    - alert: NodeRemediationFlapping
      expr: increase(gpu_quarantine_actions_total[90m]) >= 2
      labels: {severity: ticket, team: gpu-fleet}

GPU node: confirm, quarantine, drain, remediate

NODE=gpu-a3-0142
# Kernel-side truth
kubectl debug node/$NODE -it --image=registry.corp/ops/toolbox@sha256:... -- \
  chroot /host bash -c 'journalctl -k --since "2 hours ago" | grep -iE "NVRM: Xid|nvlink|fallen off"'
 
# On the node (or via the GPU Operator driver pod)
nvidia-smi -q -d ECC,ROW_REMAPPER,PERFORMANCE,TEMPERATURE,POWER | less
nvidia-smi --query-remapped-rows=gpu_bus_id,remapped_rows.correctable,remapped_rows.uncorrectable,remapped_rows.pending,remapped_rows.failure --format=csv
dcgmi health -s a && dcgmi health -c          # enable all watches, then check
dcgmi diag -r 2                                # medium diag; -r 3/4 only on a drained node
systemctl status nvidia-fabricmanager          # NVSwitch systems: FM down = NVLink domain down
ibstat | grep -E "State|Rate"; ibdev2netdev
 
# Quarantine with a reason (taint + label so dashboards and automation see it)
kubectl taint nodes $NODE gpu.corp/quarantine=xid79:NoSchedule --overwrite
kubectl label nodes $NODE gpu.corp/quarantine-reason=xid79 gpu.corp/quarantine-ticket=INC-4812 --overwrite
 
# Drain GPU workloads only; leave DaemonSets (DCGM, Cilium) for evidence
kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data --timeout=10m \
  --pod-selector='workload.corp/class in (inference,training)'
 
# After reset/repair: burn-in before return (L29)
dcgmi diag -r 3 && /opt/nccl-tests/build/all_reduce_perf -b 1G -e 8G -f 2 -g 8
kubectl taint nodes $NODE gpu.corp/quarantine- && kubectl uncordon $NODE

Automation with a budget (Medik8s NodeHealthCheck)

apiVersion: remediation.medik8s.io/v1alpha1
kind: NodeHealthCheck
metadata:
  name: gpu-nodes-us-east-3
spec:
  selector:
    matchLabels: {nvidia.com/gpu.present: "true"}
  minHealthy: "90%"            # stop remediating if more than 10% of GPU nodes are already unhealthy
  unhealthyConditions:
  - type: Ready
    status: "False"
    duration: 300s
  - type: GpuXidCritical       # set by NVSentinel / NPD custom plugin
    status: "True"
    duration: 60s
  remediationTemplate:
    apiVersion: self-node-remediation.medik8s.io/v1alpha1
    kind: SelfNodeRemediationTemplate
    namespace: medik8s
    name: gpu-reboot-template

minHealthy is the most important line in that file. If a bad exporter rollout flips the condition on 40% of nodes, remediation stops and a human gets paged by QuarantineRateAnomalous. NVSentinel gives you the same kind of control with dryRun and per-module enablement.

NCCL hang: name the rank, exclude it, restart

# Env baked into training images (L29): flight recorder on, dump on timeout
#   TORCH_NCCL_TRACE_BUFFER_SIZE=2000  TORCH_NCCL_DUMP_ON_TIMEOUT=1
#   TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/ckpt/fr/rank_
kubectl get pods -n train-llm -l jobset.sigs.k8s.io/jobset-name=pretrain-405b \
  -o custom-columns=POD:.metadata.name,PHASE:.status.phase,NODE:.spec.nodeName,RESTARTS:.status.containerStatuses[0].restartCount \
  | sort -k4 -rn | head
# Analyze flight-recorder dumps: reports the collective each rank last entered and who is missing
torchfrtrace /ckpt/fr/ -p rank_
 
# Suspend the JobSet (stops the bleed, keeps the object + history)
kubectl patch jobset pretrain-405b -n train-llm --type=merge -p '{"spec":{"suspend":true}}'
# Exclude the bad node, then resume from the last COMMITTED checkpoint
kubectl taint nodes gpu-a3-0217 gpu.corp/quarantine=nccl-hang:NoSchedule --overwrite
kubectl patch jobset pretrain-405b -n train-llm --type=merge -p '{"spec":{"suspend":false}}'

Training storm: stop admissions without killing running work

# Hold: no new admissions, running workloads continue
kubectl patch clusterqueue research-h100 --type=merge -p '{"spec":{"stopPolicy":"Hold"}}'
# HoldAndDrain: also evicts admitted workloads (use for runaway quota/preemption cascades)
kubectl patch clusterqueue research-h100 --type=merge -p '{"spec":{"stopPolicy":"HoldAndDrain"}}'
# Release
kubectl patch clusterqueue research-h100 --type=json -p '[{"op":"remove","path":"/spec/stopPolicy"}]'
 
kubectl get workloads -A --sort-by=.metadata.creationTimestamp | tail -20

Inference: freeze autoscalers, brown out, shift traffic

# Registry or hydrate incident: freeze KEDA at current replicas so scale-in can't destroy warm capacity
kubectl annotate scaledobject llm-70b -n serve-chat \
  autoscaling.keda.sh/paused-replicas="$(kubectl get deploy llm-70b -n serve-chat -o jsonpath='{.spec.replicas}')" --overwrite
# Undo
kubectl annotate scaledobject llm-70b -n serve-chat autoscaling.keda.sh/paused-replicas-
 
# Weight canary went bad: abort, then confirm stable gets 100%
kubectl argo rollouts abort llm-70b -n serve-chat
kubectl argo rollouts get rollout llm-70b -n serve-chat --watch
 
# CatalogBundle revert (L33): move the label back to the last-known-good digest via GitOps
git -C catalog revert --no-edit HEAD && git -C catalog push   # Argo CD syncs; signed bundle re-admitted

# Cell drain in steps: edge HTTPRoute weights (Gateway API). Capacity lead signs off on headroom first.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: chat-global, namespace: edge}
spec:
  parentRefs: [{name: global-gw}]
  rules:
  - backendRefs:
    - {name: cell-us-east-1, kind: Service, port: 443, weight: 50}
    - {name: cell-us-east-2, kind: Service, port: 443, weight: 50}
    - {name: cell-us-east-3, kind: Service, port: 443, weight: 0}   # was 33 → 15 → 0, watching receivers' TTFT burn

Agents: audit, evidence, kill switches

# Which sessions hit the proxy deny path in the last hour, by tenant and destination
jq -r 'select(.ts > (now-3600) and .decision=="deny") | [.tenant,.session,.dest_host] | @tsv' \
  /audit/egress-proxy/$(date -u +%F).jsonl | sort | uniq -c | sort -rn | head -20
 
# Tool gateway: write/money side effects for a tenant, with approval binding
jq -c 'select(.tenant=="acme" and (.side_effect=="write" or .side_effect=="money"))
       | {ts, session, tool, args_hash, decision, approver, toolset_digest}' \
  /audit/toolgw/$(date -u +%F).jsonl | tail -50
 
# Hubble: dropped flows and DNS from the tenant's sandboxes
hubble observe --namespace sbx-tenant-acme --verdict DROPPED --since 1h -o jsonpb > /evidence/INC-4812/hubble-drops.json
hubble observe --namespace sbx-tenant-acme --protocol dns --since 1h -o compact | awk '{print $NF}' | sort | uniq -c | sort -rn | head
 
# Kill switches (L34), narrowest first
kubectl patch configmap toolgw-flags -n toolgw --type=merge -p '{"data":{"acme.max_side_effect":"read"}}'
kubectl patch sandboxwarmpool code-exec-v7-pool -n sbx-tenant-acme --type=merge -p '{"spec":{"replicas":0}}'
# Fleet-wide proxy deny-all is a SEV1 lever with IC + security approval

Quarantine a sandbox or pod for forensics without deleting it

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata: {name: ir-quarantine}
spec:
  endpointSelector:
    matchLabels: {ir.corp/quarantine: "true"}
  ingress:
  - fromEndpoints:
    - matchLabels: {k8s:io.kubernetes.pod.namespace: ir-forensics}
  egressDeny:
  - toEntities: ["all"]

kubectl label pod -n sbx-tenant-acme sess-7f3a9c-0 ir.corp/quarantine=true
kubectl patch sandbox sess-7f3a9c -n sbx-tenant-acme --type=merge -p '{"spec":{"shutdownPolicy":"Retain"}}'

One label cuts the pod off. Deny wins in Cilium, so nothing else can reopen it. The pod, its filesystem and its flows all stay around for the security lead.

Game day: inject a phantom XID on a canary node

# Inject into DCGM's view only (no real hardware fault); 230 = DCGM_FI_DEV_XID_ERRORS
dcgmi test --inject --gpuid 0 -f 230 -v 79
# Expect: alert in under 2 min, quarantine taint, drain, ticket, NO page, budget counter +1
# Registry blackout drill (Chaos Mesh)
kubectl apply -f - <<'EOF'
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata: {name: registry-blackout, namespace: chaos-testing}
spec:
  action: partition
  mode: all
  selector:
    namespaces: [serve-canary]
  direction: to
  externalTargets: ["registry.corp"]
  duration: "20m"
EOF

Failures & fixes

Security & hardening

The worst security decisions happen during incidents. People are tired, the pressure is real, and "just open it up for now" sounds reasonable. On an ML platform, that kind of shortcut can leak weights worth more than the cluster, or let an injected agent finish what it started. Incident-time security means the safe path has to be the fast path, and it has to be set up before anything breaks.

Model weights, checkpoints & ML assets during incidents

Registries & signing / provenance

Bucket IAM & secrets

Network isolation & containment without widening blast radius

Admission, RBAC & break-glass

# On-call gets exactly the runbook verbs: cordon/taint, drain (evict), scale, patch rollouts/queues/flags
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: {name: ml-oncall-runbook}
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch", "patch"]          # cordon + taint
- apiGroups: [""]
  resources: ["pods/eviction"]
  verbs: ["create"]                                 # drain
- apiGroups: ["argoproj.io"]
  resources: ["rollouts"]
  verbs: ["get", "list", "watch", "patch"]          # abort / pause
- apiGroups: ["kueue.x-k8s.io"]
  resources: ["clusterqueues"]
  verbs: ["get", "list", "patch"]                   # stopPolicy
- apiGroups: ["keda.sh"]
  resources: ["scaledobjects"]
  verbs: ["get", "list", "patch"]                   # paused-replicas
- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["toolgw-flags"]
  verbs: ["get", "patch"]                           # kill switches
# Deliberately absent: secrets, pods/exec in sandbox namespaces, clusterrolebindings, webhook configs
---
# Audit everything a break-glass identity does at full fidelity
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
  userGroups: ["corp:breakglass"]
- level: RequestResponse
  resources:
  - group: "admissionregistration.k8s.io"
    resources: ["validatingwebhookconfigurations", "mutatingwebhookconfigurations"]
- level: Metadata
  resources: [{group: "", resources: ["secrets"]}]

# Check what on-call can do before you need it
kubectl auth can-i --list --as=oncall@corp --as-group=corp:ml-oncall | head -40
kubectl auth can-i create pods/exec -n sbx-tenant-acme --as-group=corp:ml-oncall   # must be "no"

Break-glass rules: a two-person request (IC + security lead for SEV1), a binding with an expiry that a controller reaps (or an access broker such as Teleport / cloud PAM issuing short-lived certs), every use pages security, and the identity is excluded from weight buckets unless explicitly added. The per-cell offline break-glass kubeconfig from L1 exists for IdP outages and lives in a sealed vault with check-out audit.

Agent-specific incident controls

Freeze CatalogBundle / toolset promotions for the affected product. Downgrade the tenant to read-class tools. Preserve sessions and gateway + proxy audit. Identify the injecting content by hash (doc, URL, ticket ID) and block it at retrieval. Run registered compensations for any completed writes. Loop in privacy/legal if customer data moved. Sandboxes and their identities have no route to weights, checkpoints, golden sets or catalog write paths (L34), so the blast radius is bounded before the incident starts.

Interview soundbites

• "Severity on an ML platform is four-dimensional: users, GPU-hours, model quality and security. A hung 16K-GPU job is a SEV2 with zero users affected."
• "Mitigate first, root-cause second, and the first question is always what changed: weights, bundle, image, driver, policy or demand."
• "Page on symptoms with multiwindow burn rate. Hardware events are tickets for automation. At 10K nodes, about 190 GPU remediations a day can't be human pages."
• "Automation can be the outage. Every remediation controller gets a minHealthy floor and a rate budget, and I page on the remediation rate, not the remediation."
• "Draining a cell is a capacity decision. I check headroom plus the cold-prefix-cache penalty and shift in steps, or I turn one bad cell into a regional outage."
• "Rollback order is fastest and smallest first: kill switch, bundle label, route weight, Rollout abort, queue hold, node quarantine, and the driver last."
• "During a registry outage, step one is freezing scale-in. Warm capacity you can't hydrate back is the real outage."
• "Containment narrows access, never widens it: quarantine labels instead of deletes, a pre-signed mirror instead of public pulls, time-boxed audited break-glass instead of cluster-admin."
• "A model that's fast and wrong is worse than one that's slow. Quality incidents get the SEV path too."

How this maps to the JD

100+ clusters / 10K+ nodes is exactly where incident response stops being heroics and becomes a system: cell-scoped alerting and blast radius (L1/L24), automation with budgets, GitOps rollbacks through Atlantis/Argo (L15/L16), and K8s internals you lean on under pressure (APF, eviction, taints, admission, L2/L11). The networking (Cilium/Hubble quarantine, cell routing) and security depth (break-glass, audit, signed rollback) are what make a DevSecOps/platform candidate credible as the person holding the pager.

Watch (talks & demos)

All-Your-GPUs-Are-Belong-to-Us: An Inside Look at NVIDIA's Self-Healing GeForce NOW Infrastructure

Ryan Hallisey & Piotr Prokop (NVIDIA), KubeCon NA 2024. Tens of thousands of GPUs across regional K8s/KubeVirt clusters, a maintenance API, and self-healing operators. It's the source of the "19 remediations per 1,000 nodes per day" number and the "alert when a node is remediated twice in 90 minutes" pattern. Operator takeaway: section 2 (page on the remediation system) and the automation budget.

Open on YouTube (opens in a new tab)

Navigating Failures in Pods With Devices: Challenges and Solutions

Sergey Kanzhelev (Google) & Mrunal Patel (Red Hat), KubeCon NA 2024. Why Kubernetes keeps a pod bound to a GPU it knows is unhealthy, what device plugins and the Pod Resources API do and don't tell you, node health controllers, and Pod failure policies for device errors. Operator takeaway: the gaps your GPU runbooks and remediation controllers have to fill.

Open on YouTube (opens in a new tab)

GenAI Training in Production: Software, Hardware & Network Considerations

Adi Gangidi, Jenya Lee & KR Kishore (Meta), @Scale Systems 2024. How Meta built and operated the 16K-GPU clusters behind Llama 3, including the reliability and performance problems they hit at every layer and how they fixed them. Operator takeaway: the training-storm runbooks in section 5 and why GPU-hours are a severity axis.

Open on YouTube (opens in a new tab)

Also read: Google SRE Workbook, Incident Response (opens in a new tab), and the Incident Management Guide (opens in a new tab) (IMAG roles), plus Navigating Failures in Pods With Devices (opens in a new tab) on the Kubernetes blog.

References

• Incident management: Google SRE Workbook: Incident Response (opens in a new tab) · Google Incident Management Guide (opens in a new tab) · SRE Workbook: Alerting on SLOs (multiwindow burn rate) (opens in a new tab) · SRE Book: Postmortem Culture (opens in a new tab)
• GPU faults: NVIDIA XID Errors catalog (opens in a new tab) · DCGM documentation (opens in a new tab) · NVIDIA/NVSentinel (opens in a new tab) · Medik8s Node Health Check (opens in a new tab) · nccl-tests (opens in a new tab) · PyTorch flight recorder (opens in a new tab)
• Training reliability: The Llama 3 Herd of Models (interruption taxonomy) (opens in a new tab) · Revisiting Reliability in Large-Scale ML Research Clusters (Meta) (opens in a new tab)
• Kubernetes levers: Kueue ClusterQueue stopPolicy (opens in a new tab) · JobSet (opens in a new tab) · KEDA pausing autoscaling (opens in a new tab) · Argo Rollouts (opens in a new tab) · Gateway API HTTPRoute (opens in a new tab) · Safely drain a node (opens in a new tab)
• Observability & alerting: vLLM metrics (opens in a new tab) · Alertmanager configuration (inhibit rules) (opens in a new tab) · Hubble (opens in a new tab) · Kubernetes auditing (opens in a new tab)
• Security: Cilium deny policies (opens in a new tab) · Kyverno PolicyException (opens in a new tab) · Kyverno cleanup TTL (opens in a new tab) · Sigstore cosign (opens in a new tab) · Chaos Mesh NetworkChaos (opens in a new tab)
• Teaching AI to Fight Fires: Building the Reliability Flywheel at Meta (opens in a new tab) — incident clustering and AI-assisted investigation
• wafer-ai/gpu-perf-engineering-resources (opens in a new tab) — AI/ML platform fuel
• Prior: 34 — Agent/tool sandboxing / egress · 33 — Prompt/tool catalog / serving config · 32 — Offline/online eval · 31 — Release gating / canary weights · 30 — Training data / I/O · 29 — Training reliability · 28 — Inference autoscaling · 27 — GPU supply chain · 24 — Multi-cluster routing · 23 — GPU observability · 22 — PD / MoE · 21 — Inference SLOs · 20 — Kueue / Volcano · 19 — Topology / NCCL · 14 — ML deploy hardening · 13 — ML weights · 11 — PSS / admission / RBAC · 9 — NetworkPolicy · 8 — Cilium / eBPF · 2 — Control plane · 1 — Mental model
• Next: Lesson 36 — AI/ML platform: multi-region DR & failover for inference — traffic shift, cross-region weight hydrate, RTO/RPO for GPU cells

Quick self-check

Out loud, in under two minutes each:

1. TTFT p99 tripled in us-east-3 ten minutes after cell us-east-1 was drained for a driver rollout. Walk through triage, the "what changed" streams, who decides what, and the first lever you pull.
2. A 4K-GPU pretraining run has made zero progress for 40 minutes, with no crash and no XID page. Name the rank, contain it, and restart without picking the same node again. What severity, and why, with no users affected?
3. DCGM alerts fire on 30% of the nodes in one cell within 10 minutes, right after a DCGM exporter upgrade. What does your automation do, what should it do, and what config makes the difference?
4. Security reports a spike in egress-proxy denials for tenant acme, with destinations that look like base64 subdomains. Walk through containment, evidence, kill switches and comms without breaking other tenants.
5. The weight registry is down for a region and a traffic peak is 90 minutes away. Give the ordered runbook, and explain which tempting shortcuts (public mirror, admission bypass, bucket IAM) you refuse and what you do instead.