Lesson 29 — AI/ML platform: training job reliability at scale — checkpoint/restart, elastic training, straggler & failed-node detection

2026-10-07

What interviewers are really probing

Lessons 18–28 built the GPU fleet and then mostly served from it. This lesson covers the other big tenant: a synchronous training job that spans 1,000–16,000 GPUs for weeks. One bad GPU stalls every rank. Lesson 19 wired the NCCL fabric. Lesson 20 gang-admitted the job through Kueue. Lesson 23 gave you DCGM and nccl-tests. Lesson 26 priced preemption. Lesson 27 tiered and promoted checkpoints. The question underneath all of them: at 16k GPUs something breaks every couple of hours. How much of the cluster's time actually goes into training, and which knobs move that number?

What they check for (fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab), platform framing, not kernels):

• Failure math. MTTF shrinks as 1/N. You should be able to derive a checkpoint interval (Young/Daly) from MTTF and checkpoint cost, and explain why restart time R dominates past ~4k GPUs.
• Checkpoint mechanics. Sharded, async saves (PyTorch DCP async_save, staging, plan caching), commit markers, storage write bursts, and how δ (the GPU-blocking part) differs from total save time.
• Resume correctness. Optimizer, LR scheduler, grad scaler, dataloader position, and RNG. Interviewers love "the loss curve jumped after resume. Why?"
• Elasticity. torchrun c10d rendezvous, --nnodes=MIN:MAX, what really happens when a node leaves, and why elastic world size breaks your batch math. Then the next step: torchft / in-process restart.
• Hangs and stragglers. NCCL watchdog, timeouts, flight recorder, and why the straggler in a synchronous job is the rank that waits least.
• Failed-node handling. XID taxonomy, row remapping, NPD / NVSentinel auto-cordon, pre-flight burn-in, hot spares in the same topology block.
• Orchestration semantics. JobSet failurePolicy, Job podFailurePolicy, Kueue waitForPodsReady and TAS node replacement. Classify, then retry.
• The SLI. ETTR / goodput, not "the job is Running".
• Security. A checkpoint is the crown jewels and a code-execution vector on resume.

Interview sentence: "I treat a big training run as a reliability budget. MTTF falls as one over N, so I size checkpoint cadence with Young/Daly, get the GPU-blocking part of a checkpoint down to about ten seconds with async sharded saves, and then attack restart time: detect in seconds, cordon the bad node before restarting, swap in a pre-burned-in spare from the same rail block, and resume from a verified, committed checkpoint. I report ETTR, and I classify every failure so code bugs fail fast and hardware faults don't burn the restart budget."

Core diagram: Fault → Detect → Classify → Cordon/Replace → Restart → Resume (+ checkpoint pipeline, security floor)

How to read it: follow the solid line from the job down to storage first. That's the steady state: ranks train and DCP stages shards to host memory, then uploads them in the background to NVMe and then to object storage. A manifest + COMMITTED marker makes the checkpoint real. Then follow a fault. The NCCL watchdog catches hangs. DCGM and node-problem-detector catch sick hardware. Per-rank compute timing catches slow-but-alive GPUs. Every failure is classified (Job podFailurePolicy, then JobSet failurePolicy) before anything restarts. Bad nodes are tainted and replaced from a spare pool before the restart, so you don't land on the same broken GPU. The restart re-rendezvouses and loads the last verified committed checkpoint. The dotted lines: resume reads only signed manifests, and every restart's lost time goes into ETTR.

Idiosyncrasies / operator depth

1. The failure budget: MTTF ∝ 1/N, and the Young/Daly interval

Meta's RSC paper (arXiv 2410.21680 (opens in a new tab)) measured r_f = 6.50 failures per 1,000 node-days on RSC-1 and 2.34 on RSC-2, with month-to-month swings from ~2.5 up to ~17.5. The job-level model is MTTF = 1 / (N_nodes × r_f). It projects 1.8 h MTTF at 16,384 GPUs, and the paper observed 7.9 h for 1,024-GPU jobs (user and software failures make reality worse than the hardware-only model). Llama 3's 16k-H100 pretraining saw 466 interruptions in 54 days: 47 planned, 419 unexpected, ~78% hardware-attributed, GPU issues 58.7% of unexpected (arXiv 2407.21783 (opens in a new tab)), and still more than 90% effective training time.

• Young (1974): τ_opt ≈ √(2 · δ · M), where δ is the checkpoint cost that blocks training and M is the MTTF.
• Daly (2006) first order: τ_opt ≈ √(2δM) − δ (valid for δ much smaller than M; Daly also gives a higher-order form).
• Waste fraction: ≈ δ/τ + (τ/2 + R)/M. You lose the checkpoint cost each interval, plus on average half an interval of redone work, plus the restart cost R (detect + tear down + reschedule + rendezvous + reload) per failure.

Worked numbers with r_f = 6.5 per 1k node-days, 8 GPUs per node:

The punchline interviewers want: at 1k GPUs, synchronous five-minute checkpoints are fine. At 16k, even a 10 s δ leaves you at ~0.81 if restart takes 15 min, because R is now the biggest term. That's why the industry moved to in-memory/peer checkpoints, hot spares, and in-process restart (sections 6–7). Meta's paper makes the same point: hitting ETTR ≈ 0.9 at ~12k GPUs needs checkpoint overhead around 10 s or a ~6× lower failure rate.

Gotcha: don't hardcode the interval. r_f moves by an order of magnitude month to month (a driver bug, a bad batch of HBM). Recompute τ from the last 7–30 days of fleet r_f, per SKU, and let the trainer read it from config.

2. Checkpoint anatomy: δ is not "save time"

State size for mixed-precision Adam is ~16 bytes/param (bf16 weights 2 + fp32 master 4 + Adam m and v 8 + slack). A 70B model is roughly 1.1 TB per checkpoint, and 405B is ~6.5 TB. Writing 1.1 TB synchronously at 20 GB/s takes almost a minute. Spread across 2,048 nodes it's only ~3 GB per node, so the bottleneck is the storage burst, not the node.

Idiosyncrasies:

• One in-flight async save at a time. Kick off save N+1 before N finishes and you double host RAM (staged copies) and can OOM-kill the trainer. Wait on the previous future first.
• Host RAM is a capacity input. Staging needs pinned RAM ≥ the per-node shard (×2 if you overlap). On GPU nodes that run dataloader workers with big shuffle buffers, the async stager is what triggers the cgroup OOM.
• Object stores throttle per prefix. S3 handles ~3,500 PUT/s per partitioned prefix. 16k ranks writing ckpt/step-1200/rank-*.distcp under one prefix get 503 SlowDown, which surfaces as a slow or failed async save that nobody notices until resume. Spread keys (hash in the path) and pre-warm.
• Lustre / FSx: set the stripe count on the checkpoint dir (lfs setstripe -c -1) or one OST takes every write. Metadata servers hate millions of tiny files, so one file per rank, not per tensor.
• "Atomic" is your job. DCP writes many files. A crash mid-upload leaves a directory that looks like a checkpoint. The resume logic must pick the newest step with .metadata + COMMITTED + a verified manifest, never "the highest step number in the bucket".
• Retention: keep the last K (e.g., 3) for restart, every Nth (e.g., hourly/daily) for rollback after a loss spike, and lifecycle the rest. Rollback needs checkpoints from before the corruption, so "keep only latest" breaks SDC recovery (L27 covers the tiering and promotion pipeline).

3. Resume correctness: what silently diverges

Test it: run N steps, checkpoint, kill, resume, and assert the loss at step N+k matches an uninterrupted run within tolerance. Do this in CI for the training stack, not on the 16k job.

4. Elastic training: what torchrun actually does

• torchrun --nnodes=MIN:MAX --rdzv-backend=c10d --rdzv-endpoint=HOST:29400 --rdzv-id=JOB --max-restarts=K. The c10d backend hosts a TCPStore on the endpoint (no etcd needed).
• Any membership change restarts every worker. A node dies, so the survivors' agents tear down their local workers and re-rendezvous. A new node shows up waiting, and below MAX the agents also restart to admit it. Each round counts against --max-restarts. It's elastic membership, not live resizing.
• Rendezvous knobs (--rdzv-conf): join_timeout (default 600 s), last_call_timeout (30 s; after MIN joins, wait this long for more), close_timeout (30 s), heartbeat_timeout. At thousands of nodes, the default join_timeout is too short when image pulls or hydrate are slow.
• Rendezvous host is a SPOF. If the pod hosting the c10d store dies, everyone fails. Put the endpoint on a headless Service pointing at a stable rank-0 pod, or run a separate store.
• World size change breaks math. Global batch = per-rank batch × DP ranks × grad-accum. Shrink from 512 to 496 nodes and either hold global batch constant with grad-accum (not always an integer) or rescale the LR. Many teams refuse to shrink for pretraining (MIN = MAX) and use spares instead. TP/PP groups must stay whole, so elasticity happens in DP-replica units.
• Exit codes: torchrun reports a failure of its own (ChildFailedError), which hides the worker's code. Decorate the entrypoint with @record (torch.distributed.elastic.multiprocessing.errors) so the root-cause traceback is written to TORCHELASTIC_ERROR_FILE, and have your wrapper translate classes of failure into stable exit codes for podFailurePolicy.
• Beyond torchrun: torchft keeps a per-step quorum (Lighthouse) and lets an HSDP replica group drop out and rejoin by fetching peer state, with no global restart. NVIDIA Resiliency Extension (NVRx) offers in-process restart, hang detection, and straggler detection for Megatron/NeMo-style stacks. JobSet has an InPlaceRestart restart strategy behind a feature gate. All of these attack R.

5. Hangs: NCCL watchdog, timeouts, and the flight recorder

• There is no NCCL_TIMEOUT env var. The collective timeout is init_process_group(timeout=timedelta(...)). The NCCL default in recent PyTorch is 10 min, so a single hung rank idles 16k GPUs for 10 minutes plus restart. Large shops drop it to 2–5 min after checking that checkpoint saves and first-step compile don't exceed it, and they make sure no rank does slow I/O between collectives.
• TORCH_NCCL_ASYNC_ERROR_HANDLING decides what the watchdog does on error/timeout: 0 = nothing, 1 = abort the communicator and tear down the process, 2 = abort only, 3 = tear down without abort (the default). Tearing down is what you want. A process that hangs forever holds the GPUs and never triggers your restart policy.
• TORCH_NCCL_ENABLE_MONITORING=1 + TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC kills the process if the watchdog itself is stuck (e.g., a CUDA call hung on a dead GPU after XID 79).
• Flight recorder: TORCH_NCCL_TRACE_BUFFER_SIZE=2000 + TORCH_NCCL_DUMP_ON_TIMEOUT=1 + TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/ckpt-debug/nccl_trace_rank_. On timeout every rank dumps its ring buffer of collectives. Then torchfrtrace --prefix nccl_trace_rank_ DIR names the rank that never entered the collective or entered with mismatched sizes. Without it you're reading 16k log files.
• Desync vs hang: a rank that skipped a collective (code path diverged, e.g., a data-dependent if) looks exactly like a dead GPU. The flight recorder tells you which it was. Don't cordon nodes over a software desync.
• Transport-level: NCCL_IB_TIMEOUT (exponent, default 20, about 4 s per retry) and NCCL_IB_RETRY_CNT (default 7) control how long IB retries before NCCL errors out. Too low and link flaps kill jobs; too high and a dead peer takes minutes to surface. NCCL 2.24+ ships a RAS subsystem you can query (ncclras) to see which peers are unreachable while a job is hung.

6. Stragglers: slow-but-alive is worse than dead

A dead GPU triggers a restart. A GPU running 8% slow drags every rank down by 8% forever, and nothing alerts.

• The synchronous-training trap: every rank reports the same step time because everyone waits for the slowest at each all-reduce. Find the straggler by timing compute before the collective per rank, or time spent waiting in the collective. The straggler is the rank that waits least. NVRx's straggler detector scores ranks by CUDA kernel timings, relative to peers and to their own history. Megatron-LM exposes --log-straggler.
• Causes, with their tell:

• Act on stragglers with hysteresis. Flag a rank that's more than X% slower than the median for M consecutive windows, then mark the node for eviction at the next checkpoint boundary (don't kill a healthy job mid-interval for 5%), and swap in a spare on the restart.
• Synchronized power swings: 16k GPUs that idle together at a checkpoint barrier and then spike back together stress facility power. Llama 3 called this out. It's another reason to keep δ short and async.

7. Failed nodes: XIDs, health checks, auto-cordon, spares

XIDs you should be able to triage from memory (see NVIDIA's XID catalog):

• Passive checks (while running): DCGM health watches (dcgmi health -s a) and dcgm-exporter fields (XID, DCGM_FI_DEV_ROW_REMAP_PENDING, DCGM_FI_DEV_ROW_REMAP_FAILURE, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, DCGM_FI_DEV_PCIE_REPLAY_COUNTER) feed node-problem-detector custom plugins or NVIDIA's NVSentinel. Either way the result is a node condition, then a taint, cordon, and drain, then remediation (GPU reset, reboot, RMA ticket).
• Active checks (before admitting work): dcgmi diag -r 1 takes seconds (OK for every pod start), -r 2 takes minutes (OK for node bring-up and after remediation), and -r 3/4 are long (run on burn-in and RMA return). Pair with a multi-node nccl-tests all-reduce across the job's nodes and compare bus bandwidth to the SKU baseline (L23). A node can pass single-node DCGM and still have one bad IB rail.
• Don't restart onto the same bad node. This is the #1 waste pattern: JobSet recreates pods, the scheduler happily places them back on the node with the flaky GPU (it's still Ready), and you burn maxRestarts in a loop. The fix is ordering. Taint first, restart second (restartStrategy: BlockingRecreate helps because new pods aren't created until old ones are gone, giving the cordon controller time to act).
• Hot spares: for pretraining-scale jobs, keep ~1–3% extra nodes in the same topology block (rail group / spine, L19), already burned in, held by Kueue quota (or low-priority placeholder pods, as in L28). Kueue's TAS failed-node replacement (feature-gated TASFailedNodeReplacement and friends) can swap a failed node for one in the same topology domain instead of evicting the whole workload. Spares are pure idle cost, so price them against R × N_GPUs × failures/day (L26).
• Silent data corruption (SDC): no XID, just a loss spike or NaNs. You find it with loss/grad-norm anomaly detection, deterministic replay of the bad step on different nodes, and per-rank gradient checksums. Recovery means rolling back to a checkpoint before the corruption and quarantining the node. Both Llama 3 and Meta's RSC paper count SDC as a real hardware class.

8. Orchestration semantics: classify before you retry

• Job podFailurePolicy (needs restartPolicy: Never): FailJob on exit codes that mean "code/config bug" (retrying wastes 16k GPUs × R), Ignore on the DisruptionTarget condition (preemption, node drain, eviction; don't count it against backoffLimit), Count for everything else.
• JobSet failurePolicy: maxRestarts plus ordered rules. The first match wins, and with no match the default is RestartJobSet (counts). Actions: FailJobSet, RestartJobSet, RestartJobSetAndIgnoreMaxRestarts, RestartJob, RestartJobAndIgnoreMaxRestarts. Match on onJobFailureReasons (PodFailurePolicy, BackoffLimitExceeded, DeadlineExceeded) and targetReplicatedJobs. The standard pattern: infra disruptions restart without counting; real failures count; code bugs fail the JobSet.
• backoffLimit: 0 on child Jobs. One rank failing means the whole gang is broken, so let JobSet restart everything rather than the Job controller retrying one pod into a dead rendezvous.
• Kueue waitForPodsReady: if the gang doesn't become Ready within timeout (e.g., 10 min, because a node is broken or the image is slow), Kueue evicts and requeues it with exponential backoff (requeuingStrategy.backoffLimitCount, backoffBaseSeconds, backoffMaxSeconds) so it doesn't squat on a partial allocation. recoveryTimeout covers the case where a running workload loses readiness. blockAdmission: true serializes admission so two half-admitted gangs don't deadlock (L20).
• Kubeflow: Training Operator v1 PyTorchJob has elasticPolicy (minReplicas, maxReplicas, rdzvBackend, maxRestarts). Kubeflow Trainer v2 (TrainJob) is built on JobSet, so the JobSet semantics above are what you're really configuring.
• Spot/preemptible: preemption notices arrive as node events. Turn them into a checkpoint-now signal (a SIGTERM handler that triggers an emergency save if it fits in the notice window) and Ignore in podFailurePolicy (L26).

Concrete commands & config

torchrun launch — elastic membership, rendezvous tuning, NCCL watchdog + flight recorder

# Per-node entrypoint (pod command). MIN=MAX for pretraining (spares instead of shrinking).
# Endpoint = JobSet headless-Service DNS of the first pod (<jobset>-<rjob>-<job-idx>-<pod-idx>.<jobset>).
export TORCH_NCCL_ASYNC_ERROR_HANDLING=3          # default; tear down on watchdog error
export TORCH_NCCL_ENABLE_MONITORING=1
export TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=600       # kill if the watchdog thread itself is stuck
export TORCH_NCCL_TRACE_BUFFER_SIZE=2000          # flight recorder ring buffer
export TORCH_NCCL_DUMP_ON_TIMEOUT=1
export TORCH_NCCL_DEBUG_INFO_TEMP_FILE=/scratch/fr/nccl_trace_rank_
export NCCL_IB_TIMEOUT=22 NCCL_IB_RETRY_CNT=7     # tolerate brief link flaps
export TORCHELASTIC_ERROR_FILE=/scratch/err/error.json
 
torchrun \
  --nnodes=512:512 --nproc-per-node=8 \
  --rdzv-backend=c10d \
  --rdzv-endpoint=llama70b-pretrain-workers-0-0.llama70b-pretrain:29400 \
  --rdzv-id=llama70b-pretrain \
  --rdzv-conf=join_timeout=1800,last_call_timeout=60,close_timeout=60 \
  --max-restarts=3 --monitor-interval=5 \
  train.py --config /etc/train/llama70b.yaml

# train.py: timeouts, async sharded checkpoint, commit marker, full resume state
from datetime import timedelta
import torch, torch.distributed as dist, torch.distributed.checkpoint as dcp
from torch.distributed.checkpoint.state_dict import get_state_dict, set_state_dict
from torch.distributed.checkpoint.state_dict_saver import AsyncCheckpointerType
from torch.distributed.elastic.multiprocessing.errors import record
 
dist.init_process_group("nccl", timeout=timedelta(minutes=5))   # NOT an env var
 
def app_state(model, optim, sched, loader, step):
    msd, osd = get_state_dict(model, optim)                     # canonical, reshardable keys
    return {"model": msd, "optim": osd, "sched": sched.state_dict(),
            "loader": loader.state_dict(),                       # torchdata StatefulDataLoader
            "step": step, "rng": capture_rng_per_rank()}
 
@record                                                           # root-cause traceback -> error file
def main():
    ...
    pending = None
    for step in range(start_step, total_steps):
        train_step()
        if step % ckpt_every == 0:                                # ckpt_every from Young/Daly config
            if pending is not None:
                pending.result()                                  # one in-flight save at a time
                mark_committed_if_complete(prev_dir)              # .metadata present -> manifest -> COMMITTED
            prev_dir = f"{CKPT_ROOT}/step-{step:08d}"
            pending = dcp.async_save(app_state(model, optim, sched, loader, step),
                                     checkpoint_id=prev_dir,
                                     async_checkpointer_type=AsyncCheckpointerType.PROCESS)

JobSet — gang job with classify-then-restart failure policy and BlockingRecreate

apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: llama70b-pretrain
  namespace: train-t1
  labels:
    kueue.x-k8s.io/queue-name: pretrain-lq          # admitted as one gang by Kueue (L20)
spec:
  failurePolicy:
    maxRestarts: 6                                   # budget for REAL failures only
    restartStrategy: BlockingRecreate                # old pods gone before new ones -> cordon wins the race
    rules:
    - name: infra-disruption                         # drain / preemption / node failure: free restart
      action: RestartJobSetAndIgnoreMaxRestarts
      onJobFailureReasons: [PodFailurePolicy]
      onJobFailureMessagePatterns: [".*DisruptionTarget.*"]
    - name: user-bug                                 # non-retryable exit codes classified by the Job
      action: FailJobSet
      onJobFailureReasons: [PodFailurePolicy]
    # anything else -> default RestartJobSet (counts toward maxRestarts)
  replicatedJobs:
  - name: workers
    replicas: 64                                     # 64 Jobs x 8 pods = 512 nodes
    template:
      spec:
        parallelism: 8
        completions: 8
        backoffLimit: 0                              # one pod fails -> Job fails -> JobSet decides
        podFailurePolicy:
          rules:
          - action: FailJob                          # 42 = bad config / code bug; 43 = NaN guard tripped
            onExitCodes: { containerName: trainer, operator: In, values: [42, 43] }
          - action: FailJob                          # infra: still fail the Job, but via PodFailurePolicy
            onPodConditions: [ { type: DisruptionTarget } ]
        template:
          metadata:
            annotations:
              kueue.x-k8s.io/podset-required-topology: "cloud.provider.com/topology-block"   # TAS (L19/L20)
          spec:
            restartPolicy: Never
            serviceAccountName: trainer-llama70b     # workload identity; ckpt prefix only (Security)
            automountServiceAccountToken: false
            priorityClassName: pretrain-critical
            terminationGracePeriodSeconds: 120       # room for an emergency checkpoint on SIGTERM
            tolerations: [ { key: nvidia.com/gpu, operator: Exists, effect: NoSchedule } ]
            initContainers:
            - name: preflight                        # fail fast on a sick node BEFORE rendezvous
              image: registry.internal/gpu-preflight@sha256:PREFLIGHT_DIGEST
              command: ["/bin/sh", "-c", "dcgmi diag -r 1 && /opt/check/row-remap-clean.sh && /opt/check/ib-ports-up.sh"]
              resources: { limits: { nvidia.com/gpu: 8 } }
            containers:
            - name: trainer
              image: registry.internal/train/llama@sha256:TRAINER_DIGEST
              command: ["/opt/launch.sh"]
              resources:
                limits: { nvidia.com/gpu: 8, rdma/ib: 8 }
              securityContext:
                allowPrivilegeEscalation: false
                capabilities: { drop: [ALL], add: [IPC_LOCK] }   # RDMA memlock, not privileged
              volumeMounts:
              - { name: nvme, mountPath: /scratch }
            volumes:
            - name: nvme
              hostPath: { path: /mnt/nvme/train-t1, type: DirectoryOrCreate }

Note: rule order matters. Both classes reach JobSet with reason PodFailurePolicy. The Job controller's failure message names the matched condition ("…has condition DisruptionTarget matching FailJob rule…"), so the infra rule must come first. onJobFailureMessagePatterns exists only in newer JobSet releases. On older ones, drop the user-bug rule and set maxRestarts low, or route bug exit codes through a wrapper that never restarts.

Kueue — don't squat on a broken gang

# kueue-manager-config (Configuration)
waitForPodsReady:
  timeout: 10m                    # gang must be Ready within 10 min of admission
  recoveryTimeout: 5m             # running workload lost readiness (a pod died) -> evict + requeue
  blockAdmission: true
  requeuingStrategy:
    timestamp: Eviction
    backoffLimitCount: 5
    backoffBaseSeconds: 60
    backoffMaxSeconds: 1800
featureGates:
  TopologyAwareScheduling: true
  TASFailedNodeReplacement: true  # swap a failed node within the same topology domain

Node health: DCGM, nccl-tests, NPD auto-cordon

# Health watches + check (DCGM)
dcgmi group -c alljob -a 0,1,2,3,4,5,6,7
dcgmi health -g 1 -s a                        # watch PCIe, memory, inforom, thermal, power, NVLink
dcgmi health -g 1 -c -j                       # JSON verdict for automation
dcgmi diag -r 1                               # seconds: pod start / init container
dcgmi diag -r 2                               # minutes: node bring-up, after remediation
dcgmi diag -r 3                               # long: burn-in, RMA return
 
# Row remapping + ECC + clocks
nvidia-smi -q -d ROW_REMAPPER                 # "Pending: Yes" -> drain + reset; "Remapping Failure Occurred: Yes" -> RMA
nvidia-smi -q -d ECC,PERFORMANCE | grep -iE 'volatile|aggregate|slowdown|thermal|power brake'
nvidia-smi --query-gpu=index,pcie.link.gen.current,pcie.link.width.current,clocks.sm --format=csv
dmesg -T | grep -i 'NVRM: Xid'                # XID source of truth on the host
 
# Multi-node burn-in across the exact nodes of the gang (compare busbw to SKU baseline, L23)
mpirun -np 64 -N 8 --hostfile hosts \
  -x NCCL_DEBUG=WARN -x NCCL_IB_HCA=mlx5 \
  /opt/nccl-tests/build/all_reduce_perf -b 1G -e 8G -f 2 -g 1 -c 1
 
# IB fabric
ibstat | grep -E 'State|Rate'
perfquery -x | grep -E 'SymbolErrorCounter|LinkDownedCounter|PortRcvErrors'

// node-problem-detector custom plugin monitor (excerpt): GPU health -> node condition
{
  "plugin": "custom",
  "pluginConfig": { "invoke_interval": "30s", "timeout": "20s", "max_output_length": 200, "concurrency": 1 },
  "source": "gpu-health-monitor",
  "conditions": [
    { "type": "GPUUnhealthy", "reason": "GPUHealthy", "message": "all GPUs healthy" }
  ],
  "rules": [
    { "type": "permanent", "condition": "GPUUnhealthy", "reason": "XidFatal",
      "path": "/opt/npd/check_xid.sh", "timeout": "15s" },
    { "type": "permanent", "condition": "GPUUnhealthy", "reason": "RowRemapPending",
      "path": "/opt/npd/check_row_remap.sh", "timeout": "15s" }
  ]
}

A remediation controller (NVSentinel, a Medik8s NodeHealthCheck, or your own) watches GPUUnhealthy=True. It taints the node (gpu.health/unhealthy:NoSchedule plus NoExecute for fatal XIDs), cordons and drains it, runs the reset/reboot, re-runs dcgmi diag -r 2, and only then untaints. Rate-limit it: never remediate more than ~2–5% of a pool at once, or a bad health-check rollout cordons the fleet.

Debugging a stuck or flapping run

kubectl get jobset llama70b-pretrain -n train-t1 -o jsonpath='{.status.restarts}{"\n"}{.status.conditions}' | jq
kubectl get workloads -n train-t1 -o wide                          # Kueue admission / eviction / requeue state
kubectl describe workload -n train-t1 jobset-llama70b-pretrain-xxxxx | sed -n '/Conditions/,$p'
kubectl get pods -n train-t1 -l jobset.sigs.k8s.io/jobset-name=llama70b-pretrain \
  --field-selector=status.phase!=Running -o wide                    # which node keeps failing?
kubectl get pods -n train-t1 -o json | jq -r '.items[] | select(.status.containerStatuses[0].state.terminated)
  | [.spec.nodeName, .status.containerStatuses[0].state.terminated.exitCode, .status.conditions[]?.type] | @tsv' | sort | uniq -c
kubectl get nodes -l pool=train-h100 -o custom-columns=NAME:.metadata.name,UNHEALTHY:'.status.conditions[?(@.type=="GPUUnhealthy")].status',TAINTS:.spec.taints[*].key
 
# Hang: which rank never entered the collective?
torchfrtrace --prefix nccl_trace_rank_ /scratch/fr/ -o /tmp/fr_report.txt

PromQL — the SLI and the detectors

# Straggler: per-rank compute time vs job median (exported by the trainer), 10-min window
max by (job, rank, node) (avg_over_time(train_step_compute_seconds[10m]))
  / on (job) group_left
quantile by (job) (0.5, avg_over_time(train_step_compute_seconds[10m])) > 1.05

# Throttled GPUs (bitmask; anything beyond the GPU_IDLE bit = 1 is worth a look)
count by (Hostname) (DCGM_FI_DEV_CLOCKS_EVENT_REASONS > 1)

# Fatal hardware signals (page + auto-cordon)
max by (Hostname, gpu) (DCGM_FI_DEV_XID_ERRORS) == 79           # field holds the LAST XID seen
  or max by (Hostname, gpu) (DCGM_FI_DEV_ROW_REMAP_PENDING) > 0

# ETTR over 24h (trainer exports productive seconds; wall-clock since job admitted)
sum(increase(train_productive_seconds_total{job="llama70b"}[24h])) / (24 * 3600)

# Lost GPU-hours to restarts in 24h (launcher-exported metrics; feeds chargeback, L26)
sum(increase(train_restart_downtime_seconds_total{job="llama70b"}[24h])) * 4096 / 3600

Failures & fixes

Security & hardening

A training checkpoint is two things at once: the most valuable artifact the company produces (weights plus optimizer state, enough to keep training or fine-tune a stolen copy) and an executable input to the next resume. Anyone who can write to the checkpoint prefix can steer a multi-million-dollar run. They can poison weights or optimizer moments so a backdoor survives training, or plant a pickle that runs code on 16k GPUs the next time the job restarts, and restarts happen every couple of hours at this scale. The reliability machinery (auto-restart, spares, elastic rendezvous) is also an attack surface: it resumes automatically, from whatever is there, with privileged identities. L11, L12, L13, L14, and L27 defined the controls. Here they get applied to the checkpoint/restart loop.

Checkpoint & model-weight hardening (required)

Platform controls around the training job

# Kyverno: training pods must run signed images by digest and may only resume from their own prefix
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: training-supply-chain }
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 15
  rules:
  - name: verify-trainer-images
    match:
      any: [ { resources: { kinds: [Pod], namespaceSelector: { matchLabels: { workload: training } } } } ]
    verifyImages:
    - imageReferences: ["registry.internal/*"]
      mutateDigest: false
      verifyDigest: true                    # require @sha256 pins
      required: true
      attestors:
      - entries:
        - keyless:
            issuer: https://token.actions.githubusercontent.com
            subject: "https://github.com/acme/train-images/.github/workflows/release.yml@refs/heads/main"
            rekor: { url: https://rekor.sigstore.dev }
  - name: ckpt-prefix-matches-namespace
    match:
      any: [ { resources: { kinds: [Pod], namespaceSelector: { matchLabels: { workload: training } } } } ]
    validate:
      message: "CKPT_ROOT must be under s3://ckpt/<namespace>/"
      deny:
        conditions:
          any:
          - key: "{{ request.object.spec.containers[?name=='trainer'] | [0].env[?name=='CKPT_ROOT'] | [0].value || '' }}"
            operator: NotEquals
            value: "s3://ckpt/{{ request.namespace }}/*"

// S3 bucket policy excerpt: SSE-KMS mandatory, TLS only, VPC endpoint only, trainers confined to their namespace prefix
{
  "Statement": [
    { "Sid": "DenyNonTLS", "Effect": "Deny", "Principal": "*", "Action": "s3:*",
      "Resource": ["arn:aws:s3:::ckpt", "arn:aws:s3:::ckpt/*"],
      "Condition": { "Bool": { "aws:SecureTransport": "false" } } },
    { "Sid": "DenyWrongKMS", "Effect": "Deny", "Principal": "*", "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::ckpt/*",
      "Condition": { "StringNotEquals": { "s3:x-amz-server-side-encryption-aws-kms-key-id": "arn:aws:kms:us-east-1:111122223333:key/TRAIN-CMK" } } },
    { "Sid": "DenyOutsideVpce", "Effect": "Deny", "Principal": "*", "Action": "s3:*",
      "Resource": ["arn:aws:s3:::ckpt", "arn:aws:s3:::ckpt/*"],
      "Condition": { "StringNotEquals": { "aws:SourceVpce": "vpce-0abc123" } } },
    { "Sid": "TrainerOwnPrefixOnly", "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:role/eks-trainer" },
      "Action": ["s3:PutObject", "s3:GetObject"],
      "Resource": "arn:aws:s3:::ckpt/${aws:PrincipalTag/kubernetes-namespace}/*" }
  ]
}

# Sign at commit, verify at resume. Keyless with the trainer's workload identity needs a Fulcio that
# trusts your cluster's OIDC issuer (private Sigstore); otherwise sign with a KMS key (--key awskms://...).
cosign sign-blob --yes --bundle manifest.json.sigstore.json manifest.json
cosign verify-blob manifest.json --bundle manifest.json.sigstore.json \
  --certificate-identity "https://kubernetes.io/namespaces/train-t1/serviceaccounts/trainer-llama70b" \
  --certificate-oidc-issuer "https://oidc.eks.us-east-1.amazonaws.com/id/CLUSTER_ID" \
  && python verify_shards.py --manifest manifest.json --root "$CKPT_DIR" \
  && echo "resume OK"

Punchline: a hardened restart loop only resumes from checkpoints it can prove it wrote (prefix-scoped identity, KMS, signed hash manifest), never unpickles unverified bytes, keeps rendezvous and NCCL reachable only by its own gang (NetPol for TCP, PKeys for RDMA), and treats remediation and restart policies as privileged, rate-limited APIs. That's what makes aggressive auto-restart safe.

Interview soundbites

• "MTTF falls as one over N. At 16k GPUs it's under two hours, so checkpoint cadence is a formula, Young/Daly root of two δ M, recomputed from the live failure rate, not a vibe."
• "Past a few thousand GPUs, restart time beats checkpoint time. Async saves get δ to about ten seconds, and then all the ETTR left is in detect, replace, and reload."
• "In synchronous training every rank reports the same step time. The straggler is the rank that waits least in the all-reduce."
• "Taint first, restart second. Otherwise JobSet recreates the gang right back onto the bad GPU and burns the restart budget."
• "Classify before you retry: disruptions restart for free, code bugs fail fast, everything else counts against maxRestarts."
• "There's no NCCL_TIMEOUT env var. It's the process-group timeout, ten minutes by default, and with the flight recorder on, a hang names its own culprit rank."
• "A checkpoint is crown jewels plus a pickle. Prefix-scoped write identity, KMS, a signed hash manifest, and verify-before-load, because at this scale you resume every couple of hours."
• "The SLI is ETTR, not 'Running'. Llama 3 took 466 interruptions in 54 days and still kept more than 90% effective training time."
• "Next: distributed training storage and data loading at scale: dataset streaming, caching tiers, and I/O stalls."

How this maps to the JD

JDs that say "operate large-scale training infrastructure," "GPU fleet reliability," "maximize cluster goodput," or "secure the ML platform end to end" are asking whether you can keep a 10k-GPU synchronous job productive while hardware fails underneath it daily, and do it without letting the auto-restart path become an attack path. This lesson connects L19 (NCCL fabric, topology blocks for spares), L20 (gang admission, Kueue requeue), L23 (DCGM signals, nccl-tests baselines), L26 (preemption economics, the cost of spares and lost GPU-hours), and L27 (checkpoint tiering and promotion) with the security floor from L13/L14. Fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab). Next up, Lesson 30: distributed training storage & data loading at scale: dataset streaming, caching tiers, and I/O stalls.

Watch (talks & demos)

Real talks. Watch them after Failures & fixes.

Building Resilience for Large-Scale AI Training: GPU Management, Failure Detection, and Beyond

Ganeshkumar Ashokavardhanan (Microsoft AKS) & Ace Eldeib (Cohere), KubeCon NA 2024. This is the operator half of the lesson: the spectrum of hard and soft GPU failures, DCGM-based active and passive health checks, the GPU node-problem-detector, remedy controllers, and a demo of JobSet restart + checkpointing. It includes the "oops, the pod landed on the same bad node" failure that section 7 is about. Operator takeaway: layered health checks (init container, then NPD, then remediation) plus "prevent node reuse".

Open on YouTube (opens in a new tab)

Distributed Checkpoint

Iris Zhang & Chien-Chin Huang (Meta), PyTorch Conference 2023. How torch.distributed.checkpoint saves from every rank in parallel, why load-time resharding (save on one world size or parallelism, load on another) is hard, canonical state-dict keys, and the get_state_dict/set_state_dict helpers. Operator takeaway: the mechanics behind sections 2–4, and why elastic resume only works if the checkpoint format is reshardable.

Open on YouTube (opens in a new tab)

PyTorch Distributed and Fault Tolerance

Tristan Rice (Meta), PyTorch Conference 2025. The current PyTorch Distributed landscape with a focus on fault tolerance and torchft (per-step quorum, replica groups that drop out and rejoin without a global restart). Operator takeaway: where the restart ladder is heading. Shrinking R from minutes to seconds is the only way ETTR holds past ~10k GPUs.

Open on YouTube (opens in a new tab)

References

• Failure data & models: Revisiting Reliability in Large-Scale ML Research Clusters (Meta, ETTR, r_f, MTTF projections) (opens in a new tab) · The Llama 3 Herd of Models (§3.3.4 reliability: 466 interruptions / 54 days) (opens in a new tab)
• PyTorch: Distributed Checkpoint recipe (opens in a new tab) · Async saving with DCP (DefaultStager) (opens in a new tab) · 6x faster async checkpointing (process-based, plan caching) (opens in a new tab) · Reducing checkpointing times (opens in a new tab) · torchrun (elastic launch) (opens in a new tab) · ProcessGroupNCCL environment variables (opens in a new tab) · Flight Recorder tutorial (opens in a new tab) · StatefulDataLoader (opens in a new tab) · torchft (opens in a new tab)
• NVIDIA: XID errors catalog (opens in a new tab) · DCGM diagnostics (opens in a new tab) · nccl-tests (opens in a new tab) · NCCL RAS (opens in a new tab) · NVSentinel (opens in a new tab) · NVIDIA Resiliency Extension (opens in a new tab)
• Kubernetes: JobSet failure policy (opens in a new tab) · JobSet API (opens in a new tab) · Job pod failure policy (opens in a new tab) · Kueue waitForPodsReady (opens in a new tab) · Kueue topology-aware scheduling (opens in a new tab) · node-problem-detector (opens in a new tab) · Kubeflow Trainer (opens in a new tab)
• Security: CVE-2025-32434 (torch.load weights_only RCE) (opens in a new tab) · OpenSSF model signing (sigstore/model-transparency) (opens in a new tab) · Kyverno verifyImages (opens in a new tab) · EKS Pod Identity ABAC session tags (opens in a new tab) · S3 performance (per-prefix request rates) (opens in a new tab)
• Talk slides: KubeCon NA 2024 — Resilience for AI training (PDF) (opens in a new tab)
• wafer-ai/gpu-perf-engineering-resources (opens in a new tab) — AI/ML platform fuel
• Prior: 28 — Inference autoscaling · 27 — GPU supply chain / checkpoint lifecycle · 26 — GPU cost / reclaim / spot · 25 — Multi-tenant GPU isolation · 23 — GPU observability · 20 — Kueue / gang scheduling · 19 — GPU topology / NCCL · 18 — GPU node pools · 14 — ML deploy hardening · 13 — ML weights · 12 — Supply chain · 9 — NetworkPolicy / multi-NIC
• Next: Lesson 30 — AI/ML platform: distributed training storage & data loading at scale — dataset streaming, caching tiers, I/O stalls

Quick self-check

Out loud, in under two minutes each:

1. A 4,096-GPU job sees r_f = 6.5 failures per 1,000 node-days. Derive MTTF, the Young interval for δ = 30 s, and say which term dominates waste when restart takes 15 minutes. What would you fix first?
2. Walk an async DCP checkpoint end to end. Which phases block the GPUs, what makes a checkpoint "committed", and what goes wrong if you start a second save before the first finishes?
3. The loss jumped right after a resume, and a week later you notice duplicated data. List every piece of state you should have checkpointed and the one that breaks when the world size changes.
4. Every rank reports 1.42 s per step, but throughput is 6% below baseline. How do you find the straggler, and what do you do about it without killing the job mid-interval?
5. Write the JobSet failurePolicy + podFailurePolicy that restarts for free on node drains, fails fast on a config bug, and never lands back on the bad node. Then name three ways an attacker with write access to the checkpoint bucket could hurt the run, and the control that stops each one.