Lesson 39 — AI/ML platform: GPU fleet lifecycle — node acceptance, burn-in, firmware/driver rollouts & RMA at 10K+ GPUs

2026-10-17

What interviewers are really probing

Lesson 38 ended with a rule: new capacity is untrusted until it passes acceptance. This lesson is what "acceptance" actually means, and what happens for the next three to five years of that node's life: burn-in, a steady stream of driver and firmware rollouts, a slow trickle of hardware faults, and the RMA loop that sends parts back to the vendor and brings nodes back into the fleet. At 10K+ GPUs this is not an occasional event. On any given day some percentage of your fleet is being burned in, upgraded, quarantined or waiting on a replacement part, and the platform team's job is to make that a boring, automated state machine instead of a Slack thread.

The interview version: "2,000 new H100 nodes land next month, and the same week NVIDIA publishes a driver security bulletin. Walk me through how those nodes get into production and how you roll the fix across 10,000 GPUs without killing a 4,000-GPU training run." They are listening for an explicit node state machine, burn-in that tests what production actually uses (NVLink, the fabric, sustained thermals, not just "GPU detected"), one versioned compatibility tuple for driver, CUDA, fabric manager and firmware, waves scoped by blast radius with automatic halts, failure classification so you don't RMA a GPU for a user's bug, and a repair budget that stops automation from draining the fleet when the real problem is systemic.

Cross-links: L03 (cluster lifecycle), L12 (node hardening), L15 (change as code), L16 (workflow engines), L18 (SKU pools), L19 (NVLink/NCCL), L20 (Kueue), L23 (DCGM, Xid, NCCL canaries), L27 (hydrate), L29 (checkpoints, hot spares), L35 (runbooks), L37 (drills), L38 (failure and maintenance tax). Fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab).

What they check for:

• Lifecycle as a state machine. Racked, burn-in, accepted, in service, suspect, repair, RMA, retired. Each state is a label or taint with exactly one controller allowed to move a node between states.
• Burn-in that matches production. Long DCGM diagnostics, sustained stress with thermal and power checks, single-node NVLink and multi-node fabric bandwidth against a baseline, and silent-data-corruption checks. Most hardware failures are infant mortality, so this is where you catch them cheaply.
• One compatibility tuple. Driver, CUDA user-mode, fabric manager, GPU VBIOS, NVSwitch firmware, NIC firmware and driver, BMC and kernel are versioned together and rolled together. Fabric manager must match the driver version.
• Waves with halts. Canary racks, then a small slice of one cell, then cells, each with a bake time and automatic stop on Xid rate, NCCL bandwidth or TTFT regressions. Training rolls at job or checkpoint boundaries.
• Classify before you remediate. Xid 13 or 31 is usually the application; Xid 79 is usually the hardware. A remediation ladder (reset, reboot, reimage, reseat, RMA) with repeat-offender tracking.
• Budgets. A repair budget and a maintenance budget per pool, and a circuit breaker that stops auto-remediation when too many nodes look broken at once.
• Security across the lifecycle. Firmware is supply chain, BMCs are a high-value target, diagnostics run privileged, and nodes going out for RMA carry your weights and credentials unless you wipe them.

Interview sentence: "Every node is a state machine encoded as labels and taints. New and repaired nodes come in tainted, run 24 to 72 hours of burn-in (DCGM level 3 or 4, sustained stress, NVLink and fabric NCCL against the SKU baseline, an SDC check) on a signed golden image with the fleet's firmware tuple, and only then does Kueue count them or do they see a weight. Driver and firmware changes are one versioned tuple rolled in waves: canary racks, five percent of a cell, then cell by cell with maxUnavailable scoped per pool, a bake window, and an automatic halt on Xid rate or NCCL bandwidth. Training pools roll between jobs or at checkpoints. Faults get classified by Xid before anything happens, go through a reset-reboot-reimage-RMA ladder, and a MachineHealthCheck-style maxUnhealthy breaker stops automation from draining the fleet when the real cause is a bad rollout."

Core diagram: Rack → Burn-in → Accepted → Suspect → Repair/RMA → back to Burn-in (+ rollout waves and security floor)

How to read it: there are two ways back into burn-in, and both are the point. A node that comes back from RMA is a new node (new parts, possibly new firmware) and goes through the full gauntlet. A node that takes a major driver or firmware wave gets a short re-validation before it rejoins. The only arrow that skips burn-in is a light-touch fix on the ladder (a GPU reset that clears a pending row remap, a reboot), and even that must pass a short diagnostic before the taint comes off.

Idiosyncrasies / operator depth

1. The node is a state machine; encode it, and give it one owner

Gotchas:

• Two controllers fighting. The GPU Operator's upgrade controller uncordons a node at uncordon-required, while your health controller just cordoned it for a fault. Pick one owner per transition, and make the others check the lifecycle label before acting. The operator's nvidia.com/gpu-driver-upgrade.skip=true label exists for exactly this.
• Cordon is not a state. kubectl cordon is a flag anyone can flip. Use a named taint plus a label so you can query "how many nodes are in repair" and so a human uncordoning a node by habit does not silently put a bad GPU back in service.
• Kueue quota follows the accepted count, not the node count (L38). If quota includes nodes in burn-in or repair, workloads get admitted and sit Pending.

2. Burn-in: test what production uses, for long enough

Hardware failures follow a bathtub curve: a spike of infant mortality in the first days or weeks, a long flat middle, then wear-out. Burn-in exists to move the infant-mortality spike out of your training runs and into a tainted node pool where it costs nothing but time.

How long? AMD's reference design for Instinct fleets gates new nodes on a 24-hour burn-in with continuous telemetry; many operators run 24 to 72 hours for new deliveries. The trade-off is explicit: each extra day of burn-in is a day of capacity you paid for but cannot use, against a node that fails mid-run and costs a multi-thousand-GPU job its progress since the last checkpoint (L29). A common split: full burn-in for new and RMA-returned nodes, a short (minutes to an hour) re-validation after reimage or major upgrades, and a periodic short active check between jobs.

Gotcha: passing dcgmi diag -r 1 means the GPU is present and responding. It says nothing about whether it can hold clocks for six hours at 700 W in a hot aisle.

Gotcha: slow is worse than dead. A dead node gets evicted. A node with one degraded NVLink lane or a flapping IB port runs at 70% bus bandwidth and drags every collective in a synchronous job down to its speed. Acceptance must compare against a baseline number, not just pass/fail.

3. One compatibility tuple, versioned like code

A GPU node's software and firmware stack is a dependency chain where most pairs have hard constraints:

Treat the whole thing as one artifact: a baseline id (for example h100-hgx-2026.10.1) that maps to exact versions and digests, stored in Git, referenced by the golden image build, the GPU Operator config and the firmware workflow. Nodes carry a baseline label, and a drift report lists every node not on the current tuple.

Meta's description of how they maintain hundreds of thousands of GPUs makes the same point from the other direction: keep the job layer (the CUDA libraries inside the training container) consistent across hosts, roll the host layer (driver, firmware) underneath it gradually, and serialise all disruptive operations through one orchestrator so overlapping maintenance never takes too much of one domain down at once.

4. Rollout waves: blast radius by cell, bake by signal

Operator details that come up:

• Training pools roll between jobs. A driver reload kills every process on the GPU. The GPU Operator's waitForCompletion.podSelector holds a node until matching jobs finish. For long runs, coordinate with the job: checkpoint, release a domain, upgrade, re-validate, rejoin. Roll whole NVLink domains or racks together so a job never straddles two driver versions mid-run.
• Inference pools roll by draining through the router (L24): keep N+1 replicas per model in the cell before taking a node, and hydrate weights on the upgraded node before it takes traffic (L27).
• Scope maxUnavailable per pool. The NVIDIADriver custom resource lets each node pool carry its own upgrade policy; a single ClusterPolicy applies one policy to every GPU node in the cluster.
• Maintenance and repair share a budget. The upgrade controller counts already-cordoned nodes toward maxUnavailable, so a pool with many nodes in repair stops upgrading. That is a feature: do not raise the limit to "get the rollout done."
• Firmware is out-of-band. VBIOS, NVSwitch and BMC updates go through the BMC (Redfish or the vendor's tool) and usually need a power-cycle. Sequence it as: drain, firmware via BMC, power-cycle, driver if needed, short re-validation, uncordon.

5. Detection: passive signals, active checks, and the Xid table

Passive (always on, L23): DCGM health watches, Xid events from the kernel log, ECC counters, row-remap state, throttle reasons, IB port counters. Active (scheduled): short DCGM diag and a single-node NCCL test between jobs, rack-level NCCL canaries on idle capacity. Databricks' DBRX talk and the NVIDIA fault-tolerance talk (see Watch) both land on the same design: passive monitoring catches the loud failures, active checks catch the slow ones.

Look up the current table in NVIDIA's Xid catalogue when you build rules; codes are added with new architectures and driver branches.

6. The remediation ladder and repeat offenders

1. Retry / reschedule the job on other nodes (app-class Xids, transient NCCL timeouts).
2. GPU reset (no processes on the GPU; clears pending row remaps and some GSP states).
3. Reboot, then power-cycle via BMC (fallen off bus, some firmware states).
4. Reimage to the golden baseline (driver drift, corrupted node state).
5. Physical: reseat GPU or cable, swap transceiver (NVLink and IB errors).
6. RMA: collect evidence, wipe, open vendor ticket, swap part or whole node.

After every rung, run a short re-validation before the node is accepted again. Keep a repeat-offender rule: a second hardware-class failure on the same GPU serial inside, say, 30 days skips the ladder and goes straight to RMA. Track by GPU serial and UUID, not hostname, because parts move between chassis.

Run the ladder in a durable workflow engine (L16) with one workflow per node, not a cron script: steps take hours, involve humans in the datacenter, and must survive controller restarts.

7. RMA mechanics nobody tells you about

• Evidence is the ticket. Vendors want nvidia-bug-report.sh output, DCGM diag JSON, and often their own field diagnostic results before accepting an RMA. Automate the evidence bundle at quarantine time, while the fault is still reproducible.
• Spares change everything. On-site spare GPUs, boards or whole nodes turn a multi-week vendor turnaround into hours of downtime. Size spares from your observed failure rate, and budget the repair-in-flight count in the capacity plan (L38 failure tax).
• A returned node is a new node. New serials, possibly newer firmware than your baseline. It goes back through full burn-in and firmware sync to the fleet tuple.
• Cloud is different but not exempt. On managed GPU instances you cannot RMA a GPU, but you still quarantine, collect evidence, and replace the instance (terminate and relaunch, or the provider's repair API). The ladder becomes: reset, reboot, replace. Track which instance ids you returned so the same bad host does not come straight back.

8. Budgets and the circuit breaker

When the breaker trips, page a human and freeze rollouts in that pool. This is the single most important guardrail interviewers want to hear, because the classic outage is automation correctly following its rules into a fleet-wide drain.

9. Classic gotchas

• Accepting nodes on "GPU detected" or dcgmi diag -r 1 without sustained stress and a bandwidth baseline.
• Driver upgraded but fabric manager left on the old version: CUDA fails on every HGX node in the wave.
• Kernel auto-updated by the OS image pipeline, driver modules fail to load on reboot.
• Rolling a pool with a 4,000-GPU job straddling it, so half the ranks run a different driver.
• Quarantining on Xid 13 and draining nodes for a user's out-of-bounds kernel.
• Health controller and upgrade controller both cordoning and uncordoning the same node.
• RMA-returned node rejoining on its shipped firmware, two versions behind the fleet.
• Tracking faults by hostname, so a swapped GPU inherits the old GPU's history (or loses its own).
• No breaker: a broken exporter marks 800 nodes unhealthy and automation drains them all.

Concrete commands & config

Lifecycle encoding

# New or returned node joins tainted
kubectl taint node "$NODE" lifecycle.acme.io/burnin=true:NoSchedule
kubectl label node "$NODE" lifecycle.acme.io/state=burnin --overwrite
 
# After a PASS verdict (burn-in controller only)
kubectl taint node "$NODE" lifecycle.acme.io/burnin:NoSchedule-
kubectl label node "$NODE" lifecycle.acme.io/state=accepted \
  lifecycle.acme.io/baseline=h100-hgx-2026.10.1 \
  lifecycle.acme.io/accepted-on=$(date +%F) --overwrite
 
# Fleet view: how many nodes in each state, per pool
kubectl get nodes -L lifecycle.acme.io/state,lifecycle.acme.io/baseline,nvidia.com/gpu-driver-upgrade-state \
  --no-headers | awk '{print $6}' | sort | uniq -c

Burn-in and acceptance checks (per node)

# Inventory and firmware: compare to the build manifest
nvidia-smi --query-gpu=index,uuid,serial,vbios_version,driver_version,pci.bus_id --format=csv
nvidia-smi -q | grep -iE "Inforom|VBIOS|GSP Firmware"
ibstat | grep -E "CA '|Firmware version|State|Rate"
systemctl is-active nvidia-fabricmanager && nvidia-smi -q | grep -iA3 "Fabric"
 
# Long diagnostics (machine-readable)
dcgmi diag -r 3 -j > /results/$NODE-dcgm-r3.json
jq -r '.. | objects | select(.status? and .status != "Pass" and .status != "Skip") ' /results/$NODE-dcgm-r3.json
 
# ECC and row remapping must be clean at the end of burn-in
nvidia-smi -q -d ECC | grep -iE "Uncorrectable|Volatile|Aggregate" | head -20
nvidia-smi -q -d ROW_REMAPPER   # "Pending: Yes" needs a GPU reset; "Remapping Failure Occurred: Yes" is an RMA
 
# Single-node NVLink bandwidth vs baseline (nccl-tests)
./build/all_reduce_perf -b 8M -e 8G -f 2 -g 8 | tee /results/$NODE-nccl-8gpu.log
 
# Fabric: port errors and link state for the rack
ibdiagnet --pc -o /results/ibdiag-$RACK

Burn-in Job (privileged, locked namespace)

apiVersion: batch/v1
kind: Job
metadata:
  name: burnin-gpu-n0412
  namespace: gpu-lifecycle            # no tenant workloads, no access to weights
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 172800       # 48 h hard stop
  template:
    spec:
      serviceAccountName: burnin-runner   # can only patch node labels via the controller, not secrets
      nodeName: gpu-n0412
      restartPolicy: Never
      tolerations:
      - {key: lifecycle.acme.io/burnin, operator: Exists, effect: NoSchedule}
      - {key: nvidia.com/gpu, operator: Exists, effect: NoSchedule}
      containers:
      - name: burnin
        image: registry.internal/gpu-lifecycle/burnin@sha256:9f2c...   # pinned digest, cosign-verified at admission
        args: ["--dcgm-level=3", "--stress-hours=24", "--nccl-baseline=h100-8gpu", "--results=s3://gpu-lifecycle-results/"]
        securityContext:
          privileged: true            # DCGM diag needs it; this is why the namespace is locked down
        resources:
          limits: {nvidia.com/gpu: 8}

GPU Operator: per-pool driver rollout policy

apiVersion: nvidia.com/v1alpha1
kind: NVIDIADriver
metadata:
  name: h100-train
spec:
  version: 580.95.05               # from the baseline tuple, bumped by a reviewed PR
  nodeSelector:
    pool: h100-train
  upgradePolicy:
    autoUpgrade: true
    maxParallelUpgrades: 8         # one rack of 8-GPU nodes at a time
    maxUnavailable: 5%             # includes nodes already cordoned for repair
    waitForCompletion:
      podSelector: "workload.acme.io/kind=training"
      timeoutSeconds: 0            # wait for training pods to finish; never kill a run
    podDeletion:
      force: false
      timeoutSeconds: 300
    drain:
      enable: false                # pod deletion first; drain evicts unrelated pods too

# Watch the wave
kubectl get node -l nvidia.com/gpu.present \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.nvidia\.com/gpu-driver-upgrade-state}{"\n"}{end}' \
  | awk '{print $2}' | sort | uniq -c
 
# Halt: pause auto-upgrade for this pool only
kubectl patch nvidiadrivers.nvidia.com/h100-train --type merge -p '{"spec":{"upgradePolicy":{"autoUpgrade":false}}}'
 
# Exclude a node under investigation from the rollout
kubectl label node gpu-n0412 nvidia.com/gpu-driver-upgrade.skip=true

Firmware via the BMC (Redfish)

# Inventory: what firmware is actually on this box (short-lived creds from the vault, never inline)
curl -sk -u "$BMC_USER:$BMC_TOKEN" https://$BMC/redfish/v1/UpdateService/FirmwareInventory \
  | jq -r '.Members[]."@odata.id"' \
  | xargs -I{} curl -sk -u "$BMC_USER:$BMC_TOKEN" https://$BMC{} | jq -r '[.Id, .Version] | @tsv'
 
# Stage a vendor-signed bundle from the internal mirror (hash checked before this call)
curl -sk -u "$BMC_USER:$BMC_TOKEN" -X POST \
  https://$BMC/redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdate \
  -H 'Content-Type: application/json' \
  -d '{"ImageURI":"https://fw-mirror.internal/hgx/h100/2026.10.1/bundle.fwpkg","TransferProtocol":"HTTPS"}'

Vendors also ship their own tools for HGX bundles (NVIDIA's nvfwupd, OEM utilities). Whatever the tool, the workflow owns drain, update, power-cycle, re-validation and uncordon as one durable unit.

Xid → NodeCondition (Node Problem Detector custom kernel monitor)

{
  "plugin": "kmsg",
  "logPath": "/dev/kmsg",
  "lookback": "5m",
  "bufferSize": 10,
  "source": "gpu-xid-monitor",
  "conditions": [
    {"type": "GPUHardwareFault", "reason": "NoFatalXid", "message": "no hardware-class Xid seen"}
  ],
  "rules": [
    {"type": "permanent", "condition": "GPUHardwareFault", "reason": "XidFallenOffBus", "pattern": "NVRM: Xid \\(PCI:[0-9a-fA-F:.]+\\): 79,.*"},
    {"type": "permanent", "condition": "GPUHardwareFault", "reason": "XidRowRemapFailure", "pattern": "NVRM: Xid \\(PCI:[0-9a-fA-F:.]+\\): 64,.*"},
    {"type": "permanent", "condition": "GPUHardwareFault", "reason": "XidUncontainedECC", "pattern": "NVRM: Xid \\(PCI:[0-9a-fA-F:.]+\\): 95,.*"},
    {"type": "temporary", "reason": "XidNVLink", "pattern": "NVRM: Xid \\(PCI:[0-9a-fA-F:.]+\\): 74,.*"},
    {"type": "temporary", "reason": "XidAppClass", "pattern": "NVRM: Xid \\(PCI:[0-9a-fA-F:.]+\\): (13|31|43),.*"}
  ]
}

App-class Xids become events, not conditions, so they are counted and correlated but do not quarantine a node on their own. NVIDIA's open-source NVSentinel packages the same idea (GPU health monitors, analysis, cordon and drain, NodeConditions, remediation hooks) if you would rather not build it.

The breaker: MachineHealthCheck with maxUnhealthy

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineHealthCheck
metadata: {name: h100-train-gpu-health, namespace: fleet}
spec:
  clusterName: train-use1-a
  selector:
    matchLabels: {pool: h100-train}
  maxUnhealthy: 5%                 # above this, stop remediating and page a human
  nodeStartupTimeout: 45m
  unhealthyConditions:
  - {type: GPUHardwareFault, status: "True", timeout: 0s}
  - {type: Ready, status: "False", timeout: 10m}
  - {type: Ready, status: Unknown, timeout: 10m}

Rollout and repair signals (PromQL sketches)

# Hardware-class Xids per node by driver version during a wave (dcgm-exporter reports the last Xid value)
count by (driver_version) (DCGM_FI_DEV_XID_ERRORS == 79 or DCGM_FI_DEV_XID_ERRORS == 64 or DCGM_FI_DEV_XID_ERRORS == 95)

# Row remap failures and uncorrectable remapped rows
sum by (Hostname) (DCGM_FI_DEV_ROW_REMAP_FAILURE) > 0
sum by (Hostname) (increase(DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS[24h])) > 0

# Repair budget: fraction of a pool not accepted
count(kube_node_labels{label_pool="h100-train", label_lifecycle_acme_io_state!="accepted"})
  / count(kube_node_labels{label_pool="h100-train"})

# Driver upgrade health
gpu_operator_nodes_upgrades_failed > 0

Failures & fixes

Security & hardening

The node lifecycle is a supply chain with physical steps. Firmware is code you run below the OS, BMCs can power-cycle and reflash your fleet, burn-in runs privileged, and a node shipped for RMA can leave the building with your model weights on its NVMe. Every stage needs the same rigour as the deploy path (L12–L14).

Firmware and BMC

Golden image and drivers

Privileged diagnostics

Model weights, checkpoints and deployments

Sanitisation and chain of custody before RMA or retirement

# Before a node leaves for RMA (sketch; run from the lifecycle workflow)
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --timeout=30m
kubectl delete node "$NODE"                                  # revoke kubelet cert in your PKI as well
ssh "$NODE" 'for d in /dev/nvme*n1; do sudo nvme sanitize "$d" --sanact=4; done'   # crypto erase, if supported
# Rotate this node's BMC credential in the vault; record serials and ticket id in the ledger

Interview soundbites

• "A GPU node is a state machine: racked, burn-in, accepted, suspect, repair, RMA. Only accepted counts as capacity, and each transition has exactly one owner."
• "Burn-in tests what production uses: long DCGM diag, hours of sustained stress, NVLink and fabric bandwidth against a baseline, and an SDC check. Slow is worse than dead."
• "Driver, fabric manager, firmware and kernel are one versioned tuple. Fabric manager has to match the driver, so they ship together."
• "I roll in waves: canary racks, five percent of a cell, then cell by cell, with a bake window and an automatic halt on Xid rate or NCCL bandwidth."
• "Training pools roll between jobs. I never let a running job straddle two driver versions."
• "I classify before I remediate. Xid 13 is usually the user. Xid 79 is usually the hardware."
• "The repair ladder is reset, reboot, reimage, reseat, RMA, and a second hardware failure on the same serial in 30 days skips straight to RMA."
• "Every remediation controller has a breaker. If more than a few percent of a pool looks broken, it's not the GPUs, so automation stops and a human looks."
• "A node going out for RMA gets its NVMe crypto-erased and its identity revoked. A node coming back is a new node and gets full burn-in."

How this maps to the JD

At 100+ clusters and 10K+ nodes (L01), the fleet is never fully healthy and never fully on one version. This lesson is the operational core of "run a fleet under partial failure": a lifecycle state machine, safe change systems for drivers and firmware (L15, L16), and guardrails that keep automation from amplifying incidents (L35). For a DevSecOps or Platform role, owning the firmware supply chain, BMC isolation and RMA sanitisation is the kind of security depth that separates you from candidates who only talk about container scanning.

Watch (talks & demos)

Enabling Fault Tolerance for GPU Accelerated AI Workloads in Kubernetes — Arpit Singh & Abhijit Paithankar, NVIDIA (KubeCon NA 2024)

CNCF. Walks through why GPU hardware is complex enough to fail constantly at scale, infrastructure bring-up and validation, and the four pillars of fault tolerance: monitoring, propagation (node and pod conditions, node-problem-detector), recovery and remediation, including hot spares and replacing faulty nodes. Operator takeaway: the vocabulary and building blocks behind the NodeCondition and ladder sections above.

Open on YouTube (opens in a new tab)

Operationalizing High-Performance GPU Clusters in Kubernetes: Lessons Learned from Training Databricks DBRX — Will Gleich & Wai Wu (KubeCon NA 2024)

CNCF. Lessons from running a 400-node, 3,072-GPU training cluster: GPU health detection with DCGM exporter and Prometheus, monitoring GPUDirect RDMA (which bypasses the CPU and is hard to observe), real failure cases during training, active health checks with auto-mitigation, and how GPU driver management through the GPU Operator should be tested before you trust auto-update. Operator takeaway: a real fleet's version of passive plus active detection.

Open on YouTube (opens in a new tab)

Also worth your time: Maintaining Large Scale AI Capacity @ Meta (opens in a new tab) (Systems @Scale 2024), on rolling kernel drivers and firmware continuously across hundreds of thousands of GPUs with deep health checks and a single orchestrator for disruptive work.

References

• NVIDIA GPU Operator: GPU driver upgrades (upgrade controller, state machine, NVIDIADriver upgradePolicy) (opens in a new tab)
• NVIDIA: DCGM diagnostics (opens in a new tab) · Xid errors catalogue (opens in a new tab) · GPU memory error management (row remapping) (opens in a new tab) · Fabric Manager user guide (opens in a new tab) · NVSentinel (opens in a new tab) · NVSentinel blog (opens in a new tab) · nccl-tests (opens in a new tab) · dcgm-exporter (opens in a new tab)
• Kubernetes: Node Problem Detector (opens in a new tab) · Cluster API MachineHealthCheck (opens in a new tab)
• AMD: Instinct GPU health check and 24 h burn-in reference design (opens in a new tab)
• Meta: CP-Bench (hardware failure, degradation and SDC test suite) (opens in a new tab) · Maintaining large scale AI capacity (opens in a new tab)
• Firmware: DMTF Redfish (opens in a new tab) (UpdateService, FirmwareInventory)
• Security: CVE-2024-0132 (NVIDIA Container Toolkit) (opens in a new tab) · NVIDIA product security (opens in a new tab)
• wafer-ai/gpu-perf-engineering-resources (opens in a new tab) — AI/ML platform fuel
• Prior: 38 — Capacity planning & procurement · 37 — Chaos / DR game days · 35 — Incident response · 29 — Training reliability · 27 — GPU supply chain / hydrate · 23 — DCGM / NCCL / roofline · 19 — Topology / NCCL · 16 — Temporal / Argo · 15 — Terraform / Atlantis · 12 — Node hardening · 03 — Cluster lifecycle
• Next: Lesson 40 — AI/ML platform: InfiniBand / RoCE fabric operations at scale — subnet management, congestion control, link flaps & fabric health for 10K+ GPU clusters

Quick self-check

Out loud, in under two minutes each:

1. 2,000 new H100 nodes arrive. List every state they pass through before a training job can land on them, what moves them between states, and what burn-in actually tests.
2. Why must fabric manager, driver and firmware be versioned as one tuple? What breaks first if you roll the driver alone on HGX systems?
3. A driver security bulletin drops while a 4,000-GPU run is mid-flight. Design the waves, the halt conditions, and how the training pool gets upgraded without killing the run.
4. Xid 13 on one node, Xid 79 on another, Xid 63 on a third. What do you do with each, and which one might be the user's fault?
5. At 2 a.m. automation marks 600 nodes unhealthy in ten minutes. What stops it from draining them, what do you suspect, and who gets paged?
6. A node is going out for RMA. What must happen before it leaves the building, and what happens when it comes back?