Lesson 19 — AI/ML platform: GPU topology — NVLink/NVSwitch, multi-NIC RDMA, GPUDirect, NCCL

2026-09-27

What interviewers are really probing

Lesson 18 productized GPUs as SKU pools. This lesson asks the next operator question: can you pack those GPUs without fabric awareness — or do you treat NVLink domains, RDMA rails, and NCCL topology as first-class capacity?

Interviewers at hyperscale AI platforms (fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab)) want evidence you have debugged silent 10× slowdowns, not only “8× nvidia.com/gpu Pending”:

• Packing without fabric awareness — scheduler sees GPUs; NCCL sees NVLink vs PCIe hops and which RDMA NIC is actually wired (L1, L4)
• NVLink vs PCIe — intra-node peer bandwidth; commodity PCIe boxes ≠ HGX/DGX NVSwitch domains
• NVSwitch domains — all-to-all GPU fabric inside a node (or multi-node NVLink / ComputeDomain on NVL racks)
• Multi-NIC RDMA (RoCE / InfiniBand) — rail-aligned NICs; why eth0-only CNI kills training (L9, L8)
• GPUDirect RDMA / Storage — NIC DMA into GPU HBM (GDR) and storage paths (GDS); host bounce buffers as the silent tax
• NCCL topology discovery — rings vs trees vs NVLS; NCCL_DEBUG, topology XML, NCCL_IB_HCA
• Topology-aware scheduling — labels, affinity, DRA / ComputeDomain; anti-affinity that destroys bandwidth; gang/queues preview → Lesson 20 upcoming
• Security floor on fast paths — RDMA privilege, fabric isolation, weight IAM on topology nodes (L11–L14)

Interview sentence: “I treat NVLink domains and RDMA rails as capacity — labels and affinity pack ranks into the fabric, NCCL env pins the right HCAs, GPUDirect avoids host bounce, and I never open node-wide weight IAM or hostNetwork just because the train fabric is fast.”

Core diagram: GPUs ↔ NVLink → host/PCIe → RDMA NICs → fabric → NCCL → security floor

Read it top-down: An NVLink/NVSwitch domain (HGX/DGX/NVL) gives GPUs a fast all-to-all path. Traffic that leaves the domain hits PCIe/host (NUMA hops) unless GPUDirect RDMA lets the NIC DMA GPU HBM directly. Multi-NIC RDMA (RoCE/IB rails) carries inter-node collectives; eth0 is for the pod CNI — not NCCL. The cluster fabric (rail alignment, PFC/ECN, jumbo MTU) either keeps rings local or silently stripes them across racks/AZs. NCCL discovers topology, builds rings/trees, and is pinned with env vars. Topology-aware scheduling (labels, affinity, DRA) decides whether packing respects the fabric. Everything sits on a security floor: RDMA device privilege lockdown, separate train vs inference fabrics, WI/IRSA + CMEK for weights on topology nodes, signed NCCL/CUDA/driver images, admission on privileged RDMA pods.

Idiosyncrasies / operator depth

1. NVLink / NVSwitch vs PCIe hops

Interview sentence: “I read nvidia-smi topo before I believe an 8-GPU SKU is one NVLink domain — PCIe hops inside a ‘GPU node’ are a packing bug waiting to page.”

2. Multi-NIC RDMA / GPUDirect — why eth0-only CNI kills training

Cloud fabric lessons (L5/L6) still apply: VPC MTU, sticky routes, and interconnect choice show up as NCCL timeouts if ignored.

Interview sentence: “Training pods get a dedicated RDMA interface via Multus/NAD; eth0 stays for kubelet/CNI — if NCCL lands on eth0, the job is already broken.”

3. NCCL env, topology XML, rings vs trees

Interview sentence: “I debug NCCL with DEBUG=INFO and a topo dump before I blame PyTorch — most ‘framework hangs’ are wrong NIC, GDR off, or ranks striped across racks.”

4. Topology labels, DRA, scheduler hints (preview L20)

Interview sentence: “Topology labels make the fabric schedulable; gang scheduling makes it atomic — affinity without gang still leaves you with half a ring on a slow rack.”

5. Stripe across racks / AZs = silent 10× slowdown

Interview sentence: “If step time regresses after a reschedule with the same GPU count, I assume topology first — racks and rails, not CUDA kernels.”

Concrete commands & config

Inventory: topology on the node

# Intra-node GPU / NIC topology matrix
nvidia-smi topo -m
nvidia-smi nvlink -s
nvidia-smi nvlink -e   # error counters — non-zero is a page
 
# RDMA devices & link
ibstatus 2>/dev/null || rdma link show
ibdev2netdev
ethtool -i eth1
ethtool -l eth1   # combined queues; often pinned per NUMA
 
# Is peermem available for GDR?
lsmod | grep -E 'nvidia_peermem|ib_core'
dmesg | grep -i -E 'peermem|gdr' | tail

kubectl: labels, RDMA plugin, NAD

# Topology labels you should see / enforce
kubectl get nodes -l sku=h100 \
  -L sku,rack,block,nvlink-domain,rdma-rail-count,nvidia.com/gpu.product
 
# RDMA device plugin / SR-IOV resources (illustrative names)
kubectl describe node GPU_NODE | sed -n '/Allocatable:/,/System Info:/p' | grep -iE 'rdma|mlx|pci'
 
# Multus NetworkAttachmentDefinition for train fabric
kubectl get net-attach-def -A
kubectl get pods -n model-training -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/networks}{"\n"}{end}'

# Sketch: secondary RDMA network for train pods (Multus)
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: rdma-train-rail
  namespace: model-training
spec:
  config: '{
    "cniVersion": "0.3.1",
    "type": "host-device",
    "device": "eth1",
    "ipam": { "type": "whereabouts", "range": "192.168.100.0/24" }
  }'
---
# RDMA shared-device plugin resource (pattern — pin to your plugin)
# Ads something like rdma/rdma_shared_device_a on nodes with mlx5_*

Job with topology constraints + NCCL env

apiVersion: v1
kind: ConfigMap
metadata:
  name: nccl-train-env
  namespace: model-training
data:
  NCCL_DEBUG: "INFO"
  NCCL_IB_HCA: "=mlx5_0,mlx5_1,mlx5_2,mlx5_3"
  NCCL_SOCKET_IFNAME: "eth1"
  NCCL_NET_GDR_LEVEL: "5"
  NCCL_P2P_LEVEL: "SYS"
  NCCL_IB_DISABLE: "0"
  NCCL_TOPO_DUMP_FILE: "/tmp/nccl_topo.xml"
---
apiVersion: batch/v1
kind: Job
metadata:
  name: train-topo-aware
  namespace: model-training
spec:
  parallelism: 4
  completions: 4
  template:
    metadata:
      annotations:
        k8s.v1.cni.cncf.io/networks: rdma-train-rail
      labels:
        app: train-topo
        trust-domain: train-prod
    spec:
      runtimeClassName: nvidia
      serviceAccountName: weight-writer
      restartPolicy: OnFailure
      nodeSelector:
        sku: h100
        pool: train-exclusive
        # Pack into one fabric island
        block: cell-a-block-3
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: train-topo
              topologyKey: topology.kubernetes.io/zone  # tighten to rack label in prod
        # Do NOT spread train ranks with podAntiAffinity on rack
      tolerations:
        - key: sku
          value: h100
          effect: NoSchedule
        - key: nvidia.com/gpu
          value: "true"
          effect: NoSchedule
      containers:
        - name: trainer
          image: registry.example.com/ml/trainer@sha256:REPLACE
          envFrom:
            - configMapRef:
                name: nccl-train-env
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              add: ["IPC_LOCK"]   # often required for RDMA reg; still not privileged
              drop: ["ALL"]
          resources:
            limits:
              nvidia.com/gpu: "8"
              # rdma/rdma_shared_device_a: "1"   # if using RDMA device plugin
          volumeMounts:
            - name: dshm
              mountPath: /dev/shm
      volumes:
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 16Gi

nccl-tests smoke (operator acceptance)

# Inside a topology-correct Job / debug pod with RDMA + GPUs
export NCCL_DEBUG=INFO NCCL_IB_HCA==mlx5_0,mlx5_1
./build/all_reduce_perf -b 8 -e 2G -f 2 -g 8
# Compare busbw to node SKU expectations; cliff = wrong NIC / no GDR / cross-rack
 
# Capture topo for the postmortem bundle
ls -l /tmp/nccl_topo.xml

Admission sketch: RDMA pods are not free privilege

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: rdma-train-floor
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: >
        !has(object.metadata.annotations) ||
        !has(object.metadata.annotations["k8s.v1.cni.cncf.io/networks"]) ||
        object.metadata.annotations["k8s.v1.cni.cncf.io/networks"] != "rdma-train-rail" ||
        (has(object.spec.serviceAccountName) &&
         object.spec.serviceAccountName != "default" &&
         has(object.spec.runtimeClassName) &&
         object.spec.runtimeClassName == "nvidia" &&
         !(has(object.spec.hostNetwork) && object.spec.hostNetwork) &&
         has(object.spec.nodeSelector) &&
         has(object.spec.nodeSelector.sku))
      message: "RDMA train pods require nvidia RuntimeClass, non-default SA, sku selector; hostNetwork forbidden"

Failures & fixes

Security & hardening

Fast fabrics amplify blast radius. GPUDirect and RDMA put device memory and NIC DMA on paths that traditionally needed privilege. Topology nodes also pull model weights — do not trade IAM for bandwidth.

AI/ML weight & registry hardening on topology nodes (required)

# Sketch: topology floor — fast path, narrow trust
apiVersion: policy.platform/v1
kind: GpuTopologyGate
metadata:
  name: topo-floor
spec:
  match:
    nodeSelectorKeys: ["nvlink-domain", "rdma-rail-count"]
  require:
    runtimeClass: nvidia
    imageDigestPin: true
    cosignVerify: true
    podSecurity: restricted
    hostNetwork: false
    rdmaViaNADOrDevicePlugin: true
    capabilitiesAllowed: ["IPC_LOCK"]
    networkPolicy: deny-by-default
    trainFabricNetPolIsolation: true
    weightBucketPublicAccess: false
    cmek: true
    workloadIdentity: true
    forbidNodeWideObjectIAM: true
    hfTokensInImages: false

Punchline: GPUDirect and NVLink are performance features. IAM, admission, NetPol, and signed images are what stop a topology node from becoming a weight exfil or DMA trampoline — fabric labels alone are not a security boundary.

Interview soundbites

• “GPUs without topology are half a capacity plan.”
• “Read nvidia-smi topo — NVSwitch domains aren’t marketing.”
• “eth0 is for the CNI; NCCL rides RDMA rails via Multus/NAD.”
• “GPUDirect skips the host bounce; peermem + GDR level make or break busbw.”
• “NCCL_DEBUG=INFO before you blame PyTorch.”
• “Anti-affinity that spreads train ranks across AZs is a self-inflicted 10×.”
• “Pack NVLink domains; gang-schedule so you don’t half-place a ring.”
• “RoCE needs PFC/ECN/MTU discipline — GPU ops owns the fabric story with netops.”
• “CAP_IPC_LOCK ≠ privileged ≠ hostNetwork.”
• “Never widen node IAM because the node has GDR.”
• “Train fabric and inference fabric are different trust zones.”
• “Next: gang, Kueue/Volcano, fair-share — topology-aware admission at queue time.”

How this maps to the JD

Hyperscale + AI/ML platform JDs that mention GPU fleets, distributed training, and high-speed interconnects are asking whether you can operate the fabric under the GPUs: NVLink/NVSwitch, multi-NIC RDMA, GPUDirect, NCCL, and topology-aware placement — with the same security bar as any other cell. This lesson extends L18 (pools) with L8/L9 (CNI, multi-NIC, NetPol), L5/L6 (cloud fabric), L11–L14 (admission, supply chain, weights, deploy isolation), and L1/L4 (fleet packing instincts). Fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab). Next up — Lesson 20: training/inference scheduling — gang, Kueue/Volcano, fair-share queues — so topology constraints become queueable contracts, not hope.

Watch (talks & demos)

Real talks — watch after Failures & fixes.

Lightning Talk: Mind the Topology — Smarter Scheduling for AI Workloads on Kubernetes (Roman Baron)

Why default scheduling that only sees GPU counts places distributed jobs across racks/blocks with unequal bandwidth — and how topology-aware placement keeps collectives on the fast path. Direct interview fuel for “silent 10× after reschedule.”

Open on YouTube (opens in a new tab)

NCCL Explained: How NVIDIA's GPU Communication Library Powers Distributed Deep Learning

Intra-node P2P over NVLink vs inter-node RDMA / GPUDirect, rings vs trees, and protocol choices — the mental model behind NCCL_DEBUG graphs and busbw cliffs.

Open on YouTube (opens in a new tab)

Keynote: Community-Driven Evolution of the Kubernetes Network Driver Model / DRA for AI Networking (Lionel Jouin & Antonio Ojea)

DRA as the common language for GPUs and secondary/high-speed networks — why Multus-era attachments are evolving toward topology-aware resource claims for AI fabrics.

Open on YouTube (opens in a new tab)

References

• NCCL: NVIDIA Collective Communications Library (opens in a new tab) · NCCL env variables (opens in a new tab) · NCCL tests (opens in a new tab)
• NCCL Deep Dive: Cross-DC & topology awareness (NVIDIA blog) (opens in a new tab)
• GPUDirect RDMA (opens in a new tab) · nvidia-peermem (opens in a new tab) · GPUDirect Storage (opens in a new tab)
• NVIDIA NVLink / NVSwitch (opens in a new tab) · IMEX for multi-node NVLink (opens in a new tab)
• NVIDIA DRA Driver for GPUs / ComputeDomain (opens in a new tab) · GPU Operator (opens in a new tab)
• Kubernetes Device Plugins (opens in a new tab) · Dynamic Resource Allocation (opens in a new tab) · Multus CNI (opens in a new tab)
• wafer-ai/gpu-perf-engineering-resources (opens in a new tab) — GPU fleets / topology / NCCL / serving fuel
• Prior: 18 — GPU node pools / MIG / time-slicing · 14 — ML deploy hardening · 13 — ML weights · 12 — Node/container hardening · 11 — Cluster security · 09 — NetworkPolicy / multi-NIC · 08 — CNI / Cilium / eBPF · 06 — Interconnect / NAT / BGP · 05 — Cloud networking · 04 — Borg/Mesos · 01 — Hyperscale mental model
• Next: Lesson 20 — training/inference scheduling (gang, Kueue/Volcano, fair-share queues)

Quick self-check

Out loud, in under two minutes:

1. How do you tell from the node whether eight GPUs are one NVSwitch domain or PCIe-bridged pairs?
2. Why does an eth0-only CNI attachment break multi-node NCCL — and what do you attach instead?
3. What does GPUDirect RDMA change in the data path versus a host bounce buffer?
4. Name three NCCL env vars you set before declaring a hang “framework bug.”
5. How can pod anti-affinity produce a silent multi-× slowdown for training?
6. List four security controls that must still hold on topology/RDMA nodes when jobs pull model weights.