Lesson 18 — AI/ML platform: GPU node pools, device plugins, MIG/time-slicing, SKU isolation

2026-09-26

What interviewers are really probing

Lesson 17 previewed heterogeneous SKUs and GPU tenancy as tradeoff axes. This lesson is the first deep AI/ML platform cut: can you operate GPU capacity as labeled, tainted, device-plugin-backed pools — not “nodes that happen to have accelerators”?

Interviewers at hyperscale (and AI platforms fueled by wafer-ai/gpu-perf-engineering-resources (opens in a new tab)) want evidence you have run GPU fleets under packing pressure and trust boundaries:

• SKU isolation as a pool contract — A100 vs H100 vs L4 are different products; labels + taints + separate autoscaler min/max (L3, L4, L17)
• Device plugin vs GPU Operator — who installs drivers, who advertises nvidia.com/gpu / mig-*, what breaks when the DaemonSet is unhealthy
• MIG vs time-slicing vs exclusive — hardware memory/fault isolation vs SW oversubscribe with no mem/fault isolation vs 1:1 ownership
• RuntimeClass + container toolkit — the nvidia runtime path; wrong RuntimeClass = silent CPU fallback or CrashLoop
• Scheduler packing reality — requests, tolerations, nodeSelectors; Pending forever is a pool-contract bug until proven otherwise
• Autoscaling gotchas — CA/NAP scales on pending GPU resources; MIG profile flips are drains; time-slice replicas fake headroom
• Security floor on shared GPUs — admission, NetPol, signed driver/workload images, weight IAM so a notebook on a shared L4 cannot read another tenant’s checkpoints (L11–L14)

Interview sentence: “I treat GPUs as SKU pools with taints and labels, let the GPU Operator own the device stack, choose exclusive / MIG / time-slice by isolation need — never time-slice across trust boundaries — and keep admission, NetPol, signed images, and per-pool weight IAM on the floor.”

Core diagram: SKU pools → plugin → resources → sharing modes → security floor

Read it top-down: SKU pools (labeled + tainted) feed the NVIDIA GPU Operator stack (driver, container toolkit, device plugin, GFD, DCGM). The plugin advertises nvidia.com/gpu, nvidia.com/mig-*, or nvidia.com/gpu.shared. You pick a sharing mode — exclusive (full GPU), MIG (HW memory/fault isolation), or time-slicing (SW replicas, no mem/fault isolation) — then the scheduler packs via nodeSelector/affinity + tolerations + RuntimeClass. Everything sits on a security floor: admission requiring SKU contracts, device-plugin RBAC, signed driver/workload images, NetworkPolicy on GPU namespaces, and WI/IRSA so weights never leak across tenants on shared nodes. Topology (NVLink/RDMA/NCCL) is Lesson 19 — preview only here.

Idiosyncrasies / operator depth

1. GPU node pools are products, not instance types

Interview sentence: “SKU isolation lives in labels, taints, and autoscaler pools — not in hoping the scheduler invents topology.”

2. NVIDIA device plugin vs GPU Operator

Prefer Operator-managed stacks at fleet scale (L3 lifecycle). Pin Operator/chart versions; treat driver upgrades as drain waves (Temporal saga territory from L16).

3. Exclusive vs MIG vs time-slicing

Operator truth from NVIDIA docs: time-slicing trades MIG’s memory/fault isolation for higher share counts. Never put hostile tenants on the same time-sliced GPU. GKE can combine MIG partitions with time-sharing inside a partition — still same-trust only (L14).

Interview sentence: “Exclusive for train/SLO; MIG when multi-tenant needs HW walls; time-slice only inside one trust boundary — never as a security control.”

4. Resource names and what the scheduler actually sees

Pods must request the exact resource name the pool ads. Requesting nvidia.com/gpu on a MIG-only node with strategy that hides whole GPUs → Pending.

5. RuntimeClass lockdown

# Workloads must opt into nvidia runtime — do not make it cluster default for all pods
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia

Admission should require runtimeClassName: nvidia (or your org’s name) for any pod requesting GPU resources, and forbid privileged + hostPID combos except the Operator’s own DaemonSets (L11).

6. Autoscaling gotchas (the pager at 2am)

7. Compose with prior security lessons

GPU pools are where L12–L14 become physical: signed images land on nodes that can read weight buckets. Shared pools need stricter IAM and NetPol, not looser — density is not an excuse to widen blast radius (L1).

Concrete commands & config

GKE-style GPU node pool (SKU + taint)

# Illustrative — separate pools per SKU
gcloud container node-pools create gpu-h100-exclusive \
  --cluster=cell-a --region=us-east1 \
  --machine-type=a3-highgpu-8g \
  --accelerator=type=nvidia-h100-80gb,count=8 \
  --num-nodes=0 --enable-autoscaling --min-nodes=0 --max-nodes=16 \
  --node-taints=sku=h100:NoSchedule,nvidia.com/gpu=true:NoSchedule \
  --node-labels=sku=h100,pool=train-exclusive,accelerator=nvidia-h100-80gb \
  --spot  # only if checkpointed; prefer on-demand for SLO train
 
gcloud container node-pools create gpu-l4-shared \
  --cluster=cell-a --region=us-east1 \
  --machine-type=g2-standard-8 \
  --accelerator=type=nvidia-l4,count=1,gpu-sharing-strategy=time-sharing,max-shared-clients-per-gpu=4 \
  --num-nodes=0 --enable-autoscaling --min-nodes=0 --max-nodes=32 \
  --node-taints=sku=l4-shared:NoSchedule \
  --node-labels=sku=l4,pool=interactive-shared,cloud.google.com/gke-gpu-sharing-strategy=time-sharing

Helm: NVIDIA GPU Operator (sketch)

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
helm upgrade --install gpu-operator nvidia/gpu-operator \
  -n gpu-operator --create-namespace \
  --set driver.enabled=true \
  --set mig.strategy=mixed \
  --set devicePlugin.config.name=time-slicing-config \
  --set operator.defaultRuntime=containerd

Device plugin time-slicing ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    sharing:
      timeSlicing:
        renameByDefault: true
        failRequestsGreaterThanOne: true
        resources:
          - name: nvidia.com/gpu
            replicas: 4

Label nodes that should apply it: nvidia.com/device-plugin.config=any. Keep MIG pools on different nodes with nvidia.com/mig.config=all-balanced (or your profile) — do not time-slice and MIG-chaos the same unlabeled pool.

Workload: exclusive H100 train

apiVersion: batch/v1
kind: Job
metadata:
  name: train-llama-cell-a
  namespace: model-training
spec:
  template:
    spec:
      runtimeClassName: nvidia
      serviceAccountName: weight-writer
      restartPolicy: OnFailure
      nodeSelector:
        sku: h100
        pool: train-exclusive
      tolerations:
        - key: sku
          value: h100
          effect: NoSchedule
        - key: nvidia.com/gpu
          value: "true"
          effect: NoSchedule
      containers:
        - name: trainer
          image: registry.example.com/ml/trainer@sha256:REPLACE
          resources:
            limits:
              nvidia.com/gpu: "8"

Workload: MIG inference slice

apiVersion: apps/v1
kind: Deployment
metadata:
  name: infer-small
  namespace: model-serving
spec:
  template:
    spec:
      runtimeClassName: nvidia
      serviceAccountName: weight-reader
      nodeSelector:
        sku: a100-mig
      tolerations:
        - key: sku
          value: a100-mig
          effect: NoSchedule
      containers:
        - name: server
          image: registry.example.com/ml/infer@sha256:REPLACE
          resources:
            limits:
              nvidia.com/mig-1g.5gb: "1"

kubectl inventory (operator morning checks)

# Pool contract
kubectl get nodes -L sku,pool,accelerator,nvidia.com/gpu.product,nvidia.com/mig.capable
 
# What the plugin ads
kubectl describe node -l sku=h100 | sed -n '/Allocatable:/,/System Info:/p'
kubectl get cm -n gpu-operator time-slicing-config -o yaml
 
# Operator health
kubectl get pods -n gpu-operator -o wide
kubectl get clusterpolicies.nvidia.com
kubectl get runtimeclass
 
# Pending GPU truth
kubectl get pods -A -o custom-columns=NS:.metadata.namespace,N:.metadata.name,ST:.status.phase \
  | grep Pending
kubectl describe pod -n model-training TRAIN_POD | sed -n '/Events/,$p'
 
# DCGM / GPU feature labels
kubectl get nodes -l nvidia.com/gpu.count -o json \
  | jq '.items[] | {name:.metadata.name, labels:.metadata.labels | with_entries(select(.key|startswith("nvidia.com/")))}'

Admission sketch: require SKU + RuntimeClass for GPU requests

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: gpu-pool-contract
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: >
        !has(object.spec.containers) ||
        !object.spec.containers.exists(c,
          has(c.resources) && has(c.resources.limits) &&
          (c.resources.limits.exists(k, k.startsWith("nvidia.com/")))) ||
        (has(object.spec.runtimeClassName) && object.spec.runtimeClassName == "nvidia" &&
         has(object.spec.nodeSelector) && has(object.spec.nodeSelector.sku))
      message: "GPU pods require runtimeClassName=nvidia and nodeSelector.sku"

Failures & fixes

Security & hardening

GPU nodes are high-value targets: they hold model weights in memory/disk cache, run privileged device stacks, and often share physical accelerators across teams. Soft density without a floor is how you get cross-tenant weight theft and driver-level escapes.

AI/ML weight & registry hardening on GPU pools (required)

# Sketch: shared GPU NS still gets the floor
apiVersion: policy.platform/v1
kind: GpuPoolGate
metadata:
  name: gpu-floor
spec:
  match:
    nodeSelectorKeys: ["sku"]
  require:
    runtimeClass: nvidia
    imageDigestPin: true
    cosignVerify: true
    podSecurity: restricted
    networkPolicy: deny-by-default
    weightBucketPublicAccess: false
    cmek: true
    workloadIdentity: true
    timeSliceSameTrustOnly: true
    forbidNodeWideObjectIAM: true

Punchline: MIG gives hardware walls; time-slicing does not. IAM, admission, NetPol, and signing are what stop a shared L4 notebook from becoming a weight exfil path — SKU taints alone are not enough.

Interview soundbites

• “GPUs are SKU pools with taints — not unlabeled capacity.”
• “Operator owns the device stack; the plugin only ads what the Operator configured.”
• “Exclusive for train/SLO; MIG for HW multi-tenant; time-slice only inside one trust domain.”
• “Time-slicing is density, not isolation — no mem, no fault walls.”
• “Pending GPU pods are a pool-contract bug until proven otherwise.”
• “MIG profile changes are drains, not ConfigMap daydreams.”
• “Replicas inflate allocatable; GPU memory is still finite.”
• “RuntimeClass nvidia is required, not decorative.”
• “Tenant NS stays PSS restricted — only gpu-operator may be privileged.”
• “Per-pod WI/IRSA for weights; never a node-wide models: role on shared pools.”*
• “Golden OS baseline; heterogeneity in labels and device plugins.”
• “Next: Lesson 19 — topology: NVLink, RDMA, NCCL — packing without fabric is half a design.”

How this maps to the JD

Hyperscale + AI/ML platform JDs that mention GPU fleets, inference/training, and 10K+ nodes are asking whether you can productize accelerators: pools, device plugins, sharing modes, and security. This lesson connects L1/L3/L4 (cells, lifecycle, Borg packing instincts), L11–L14 (admission, supply chain, weights, deploy isolation), L15 (pool IaC), and L17 (SKU vs golden image) into an operator-ready GPU pool story. Fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab). Next up — Lesson 19: GPU topology (NVLink/NVSwitch, multi-NIC RDMA, GPUDirect, NCCL) so packing respects the fabric, not just the resource name.

Watch (talks & demos)

Real talks — watch after Failures & fixes.

Mastering GPU Management in Kubernetes Using the Operator Pattern (Shiva Krishna Merla & Kevin Klues, NVIDIA / KubeCon)

End-to-end GPU Operator lifecycle: drivers, toolkit, device plugin, MIG/MPS profiles, monitoring — the operator depth behind ClusterPolicy and fleet GPU stacks.

Open on YouTube (opens in a new tab)

A Deep Dive on Supporting Multi-Instance GPUs in Containers and Kubernetes (Kevin Klues, NVIDIA)

How MIG is exposed to containers and Kubernetes, resource naming, and lifecycle of MIG devices on a node — essential before you promise “multi-tenant A100” in an interview.

Open on YouTube (opens in a new tab)

Scaling GPU Clusters Without Melting Down (Alay Patel & Ryan Hallisey, NVIDIA / KubeCon EU 2025)

Control-plane cost of large GPU fleets, scheduler/apiserver pressure, KWOK simulation, and DRA direction — why GPU scale is a platform problem, not only a device-plugin problem.

Open on YouTube (opens in a new tab)

References

• NVIDIA GPU Operator docs (opens in a new tab) · Time-slicing GPUs (opens in a new tab) · MIG with Kubernetes (opens in a new tab)
• NVIDIA k8s-device-plugin (opens in a new tab) · GPU Feature Discovery (opens in a new tab)
• GKE GPU time-sharing (opens in a new tab) · GKE GPU sharing concepts (opens in a new tab) · GKE MIG (opens in a new tab)
• Kubernetes Device Plugins (opens in a new tab) · RuntimeClass (opens in a new tab)
• wafer-ai/gpu-perf-engineering-resources (opens in a new tab) — GPU fleets / topology / serving fuel
• Prior: 17 — Systems design tradeoffs · 15 — Terraform / Atlantis · 14 — ML deploy hardening · 13 — ML weights · 12 — Node/container hardening · 11 — Cluster security · 04 — Borg/Mesos · 03 — Cluster provisioning · 01 — Hyperscale mental model
• Next: Lesson 19 — GPU topology (NVLink/NVSwitch, multi-NIC RDMA, GPUDirect, NCCL)

Quick self-check

Out loud, in under two minutes:

1. What labels and taints define a GPU SKU pool — and what happens if you skip the taint?
2. What does the GPU Operator manage that the device plugin alone does not?
3. Compare exclusive vs MIG vs time-slicing on memory isolation, fault isolation, and trust.
4. Why can time-slice “allocatable GPUs” look healthy while training OOMs?
5. Name three admission / IAM controls that must hold on a shared GPU pool when pods pull model weights.
6. What breaks if a GPU pod omits runtimeClassName: nvidia or the matching SKU toleration?