Lesson 20 — AI/ML platform: training/inference scheduling — gang, Kueue/Volcano, fair-share queues

2026-09-28

What interviewers are really probing

Lesson 18 gave you SKU pools. Lesson 19 made NVLink domains and RDMA rails schedulable. This lesson asks the operator question that kills GPU utilization in interviews and in production: can you admit multi-pod training without half-placing a gang — and share the fleet fairly without starving inference SLOs?

Interviewers at hyperscale AI platforms (fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab)) want evidence you have run batch queues under contention, not only “we use the default scheduler”:

• Vanilla kube-scheduler + Job parallelism fails for training — partial place → idle GPUs holding ranks → NCCL hang / watchdog abort (L1, L4)
• Gang scheduling semantics — all-or-nothing / minAvailable; topology from L19 is a packing constraint, not a hope
• Kueue — ClusterQueue, LocalQueue, ResourceFlavor, Cohort, borrowing/lending, fair-sharing, preemption, Workload CR; Job / JobSet / Kubeflow MPIJob / PyTorchJob admission
• Volcano — Queue, PodGroup, gang plugins, queue priority — when it shows up vs Kueue in interviews
• Fair-share / hierarchical queues — team quotas vs capacity planning; reclaim loaned capacity
• Training vs inference — exclusive gang train ≠ latency-SLO inference; don’t share without isolation (L14)
• Topology-aware + queue — flavors for NVLink/RDMA SKUs; don’t admit a gang that can’t pack into one fabric island
• Orchestration vs batch queue — Temporal/Argo (L16) own workflow DAG / retries; Kueue/Volcano own when GPU capacity is reserved
• Security floor on queues — queue admins ≠ weight writers; admission on CQ/LQ; signed controller images; preemption must not leave weight-writable sidecars (L11–L14)

Interview sentence: “I admit training as a gang through Kueue (or Volcano PodGroup) against topology-aware flavors, fair-share cohorts reclaim loaned GPUs, train and inference never share without isolation, and queue RBAC never implies weight-bucket IAM.”

Core diagram: Jobs → LocalQueue → ClusterQueue/Cohort → Flavors → schedule → nodes (+ Volcano path, security floor)

Read it left-to-right: A training Job / JobSet / MPIJob / PyTorchJob lands on a namespace LocalQueue (label kueue.x-k8s.io/queue-name). That queue points at a cluster-scoped ClusterQueue inside a Cohort that fair-shares and can borrow/lend with preemption on reclaim. Admission picks a ResourceFlavor (SKU + topology labels). Only then does kube-scheduler (optionally with Kueue TAS) place pods — ideally into one fabric island from L19. Volcano is the alternate path: its Queue + PodGroup gang-binds via plugins and replaces default scheduling for those pods. Inference is a different product: latency SLOs, not exclusive gangs — isolate pools/queues. Everything sits on a security floor: who can mutate queues, who can write weights, signed controller + train images, WI/IRSA, NetPol, and preemption that tears down the whole gang cleanly.

Idiosyncrasies / operator depth

1. Why vanilla kube-scheduler + Job parallelism fails for multi-pod training

Interview sentence: “Job parallelism is not gang scheduling — without all-or-nothing admission you donate GPUs to a hang.”

2. Gang scheduling semantics (all-or-nothing / minAvailable)

Interview sentence: “Gang is a capacity contract: admit only when the full ring fits — including topology, not just GPU count.”

3. Kueue building blocks

Integrations interviewers name-drop: plain batch/v1 Job, JobSet, Kubeflow MPIJob / PyTorchJob, RayJob, LeaderWorkerSet — kueue admits them as Workloads when the framework controller is wired.

Interview sentence: “LocalQueue is the team UX; ClusterQueue is the quota contract; ResourceFlavor is how SKU and topology become admit-time constraints.”

4. Volcano vs Kueue — when each shows up

Interview sentence: “Kueue is quota admission that keeps kube-scheduler; Volcano is a batch scheduler with PodGroup gang — I pick based on whether we own scheduling or only admission.”

5. Fair-share, hierarchical queues, team quotas vs capacity planning

Interview sentence: “Quotas without reclaim are lies; fair-share without preemption is a suggestion.”

6. Training vs inference scheduling (don’t share blindly)

Interview sentence: “Training buys atomic GPU sets; inference buys tail latency — one ClusterQueue for both is how you get both angry.”

7. Topology-aware flavors — admit only what packs

Interview sentence: “ResourceFlavor is where L18 SKUs and L19 topology become admission predicates — if the island can’t fit the gang, don’t Admit.”

8. Orchestration vs batch queue (L16)

Interview sentence: “Argo starts the Job; Kueue decides if the fleet can afford the gang today.”

Concrete commands & config

Inspect Kueue objects & Workload state

# Queues & flavors
kubectl get clusterqueues.kueue.x-k8s.io -o wide
kubectl get localqueues.kueue.x-k8s.io -A
kubectl get resourceflavors.kueue.x-k8s.io
kubectl describe clusterqueue team-a-cq
 
# Workloads: Admitted / Evicted / Finished
kubectl get workloads.kueue.x-k8s.io -A \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CQ:.status.admission.clusterQueue,STATE:.status.conditions[-1:].type
 
kubectl describe workload -n model-training JOB_NAME
kubectl get events -n model-training --field-selector reason=Admitted,reason=Evicted,reason=QuotaReserved
 
# Optional CLI (if installed)
kueuectl list workload -n model-training
kueuectl list clusterqueue

Volcano inspect

kubectl get queues.scheduling.volcano.sh
kubectl get podgroups.scheduling.volcano.sh -A
kubectl describe podgroup -n model-training train-gang
# volcano-cli where available:
# vcctl queue list && vcctl job list -n model-training
kubectl get pods -n model-training -o wide --field-selector=status.phase=Pending
kubectl describe pods -n model-training -l volcano.sh/job-name=train-gang | sed -n '/Events:/,$p'

YAML: ResourceFlavor + ClusterQueue + LocalQueue + Job

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: h100-nvlink
spec:
  nodeLabels:
    sku: h100
    pool: train-exclusive
    # Topology island — tighten to your inventory ([L19])
    block: cell-a-block-3
  # optional: nodeTaints + tolerations to match train taints
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: team-a-cq
spec:
  cohort: ml-platform
  queueingStrategy: BestEffortFIFO   # or StrictFIFO — know HOL blocking
  namespaceSelector: {}               # restrict in prod
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
      flavors:
        - name: h100-nvlink
          resources:
            - name: nvidia.com/gpu
              nominalQuota: 64
              borrowingLimit: 32
            - name: cpu
              nominalQuota: "512"
            - name: memory
              nominalQuota: 4Ti
  preemption:
    withinClusterQueue: LowerPriority
    reclaimWithinCohort: Any
  # fairSharing:
  #   weight: 25
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: team-a-train
  namespace: model-training
spec:
  clusterQueue: team-a-cq
---
apiVersion: batch/v1
kind: Job
metadata:
  name: train-gang-demo
  namespace: model-training
  labels:
    kueue.x-k8s.io/queue-name: team-a-train
  annotations:
    # Framework-specific: JobSet / MPIJob similarly labeled
spec:
  parallelism: 4
  completions: 4
  # kueue suspends until Admitted when managed
  template:
    metadata:
      labels:
        app: train-gang
        trust-domain: train-prod
      annotations:
        k8s.v1.cni.cncf.io/networks: rdma-train-rail
    spec:
      runtimeClassName: nvidia
      serviceAccountName: weight-writer
      restartPolicy: OnFailure
      # Flavor injects selectors; still pin topology intent
      nodeSelector:
        sku: h100
        pool: train-exclusive
      containers:
        - name: trainer
          image: registry.example.com/ml/trainer@sha256:REPLACE
          resources:
            limits:
              nvidia.com/gpu: "8"
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              add: ["IPC_LOCK"]
              drop: ["ALL"]

YAML: Volcano PodGroup sketch

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: ml-training
spec:
  weight: 1
  reclaimable: true
  capability:
    nvidia.com/gpu: 128
---
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: train-gang
  namespace: model-training
spec:
  minMember: 4
  minResources:
    nvidia.com/gpu: "32"
  queue: ml-training
  priorityClassName: train-high
---
# Pods / volcano Job set schedulerName: volcano and
# annotation scheduling.k8s.io/group-name: train-gang (pattern varies by version)

Admission sketch: queue mutation ≠ weight IAM

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kueue-queue-admin
rules:
  - apiGroups: ["kueue.x-k8s.io"]
    resources: ["clusterqueues", "resourceflavors", "localqueues"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  # Explicitly NOT: secrets, IAM bindings, bucket policies
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: kueue-job-floor
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["batch"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["jobs"]
  validations:
    - expression: >
        !has(object.metadata.labels) ||
        !has(object.metadata.labels["kueue.x-k8s.io/queue-name"]) ||
        (has(object.spec.template.spec.serviceAccountName) &&
         object.spec.template.spec.serviceAccountName != "default" &&
         has(object.spec.template.spec.runtimeClassName) &&
         object.spec.template.spec.runtimeClassName == "nvidia")
      message: "Kueue train Jobs require non-default SA + nvidia RuntimeClass"

Failures & fixes

Security & hardening

Queues and gangs concentrate power. Whoever admits a 64-GPU Workload can also become a weight exfil path if SA, images, and NetPol are sloppy. Preemption without a clean gang teardown leaves orphans that still mount checkpoints.

AI/ML weight & registry hardening on queued gangs (required)

# Sketch: queue + weight floor (policy-as-code shape)
apiVersion: policy.platform/v1
kind: MlQueueGate
metadata:
  name: kueue-weight-floor
spec:
  match:
    labels:
      kueue.x-k8s.io/queue-name: "*"
  require:
    imageDigestPin: true
    cosignVerify: true
    runtimeClass: nvidia
    serviceAccountNotDefault: true
    workloadIdentity: true
    weightBucketPublicAccess: false
    cmek: true
    forbidNodeWideObjectIAM: true
    networkPolicy: deny-by-default
    preemptibleSidecarWeightWrite: false
    kueueControllerImageSigned: true
    volcanoControllerImageSigned: true
    queueAdminImpliesWeightIAM: false

Punchline: Fair-share and gang admission are capacity features. RBAC splits, signed images (including controllers), WI/IRSA, NetPol, and whole-gang preemption are what stop a Cohort from becoming a multi-tenant weight breach.

Interview soundbites

• “Job parallelism is not a gang — partial place is how you waste a rack.”
• “Kueue admits quota; kube-scheduler places; Volcano does both for its pods.”
• “LocalQueue is UX; ClusterQueue is the contract; ResourceFlavor is SKU + topology.”
• “Borrow without reclaim is theft with good intentions.”
• “Fair-share needs preemption or it’s a story you tell finance.”
• “Don’t admit a gang that can’t pack into one fabric island.”
• “Train gangs and inference SLOs don’t share a ClusterQueue without isolation.”
• “Argo orchestrates steps; Kueue orchestrates GPU scarcity.”
• “Queue admin ≠ weight writer ≠ registry pusher.”
• “Preempt the whole Workload — orphan ranks still hold checkpoints.”
• “Sign kueue controllers like you sign trainers.”
• “Next: inference SLOs — TTFT/TPOT, goodput, KV-cache as capacity.”

How this maps to the JD

Hyperscale + AI/ML platform JDs that mention GPU fleets, multi-tenant training, and fair capacity are asking whether you can operate admission under contention: gang semantics, Kueue/Volcano, fair-share cohorts, topology-aware flavors, and the train vs inference split — with the same security bar as any other cell. This lesson extends L18/L19 (pools + topology) with L4/L1 (fleet packing / Borg instincts), L16 (orchestration ≠ batch queue), and L11–L14 (admission, supply chain, weights, deploy isolation). Fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab). Next up — Lesson 21: inference serving SLOs — TTFT/TPOT/goodput, continuous batching, KV-cache as capacity.

Watch (talks & demos)

Real talks — watch after Failures & fixes.

Multitenancy and Fairness at Scale with Kueue: A Case Study (Aldo Culquicondor & Rajat Phull)

Cohorts, borrowing, fair-share weights, and reclaim-via-preemption in production ML platforms — the interview core for “how do teams share GPUs without lying about quotas.”

Open on YouTube (opens in a new tab)

Building a Batch System for the Cloud with Kueue (Aldo Culquicondor & Kante Yin)

ClusterQueue / LocalQueue / ResourceFlavor, flavor injection, and preemption when reclaiming loaned capacity — how kueue sits in front of kube-scheduler without replacing it.

Open on YouTube (opens in a new tab)

Lightning Talk: Optimizing AI Workload Scheduling at Scale with Kueue (P. Matam)

Practical cluster-queue / local-queue / flavor / cohort vocabulary for AI training admission — tight interview flashcard after the two deeper talks.

Open on YouTube (opens in a new tab)

Volcano — Advanced Batch Scheduling for Kubernetes (CNCF Incubation series)

Gang scheduling, multi-queue fair-share, and topology-aware batch placement — the Volcano side of “Kueue vs Volcano” interview contrasts.

Open on YouTube (opens in a new tab)

References

• Kueue (opens in a new tab) · All-or-nothing scheduling (opens in a new tab) · Concepts: ClusterQueue / LocalQueue / ResourceFlavor (opens in a new tab)
• JobSet (opens in a new tab) · Kubeflow Trainer (opens in a new tab)
• Volcano (opens in a new tab) · PodGroup / gang (opens in a new tab) · CNCF Volcano (opens in a new tab)
• Kubernetes scheduling / DRA (opens in a new tab) · PriorityClass / preemption (opens in a new tab)
• wafer-ai/gpu-perf-engineering-resources (opens in a new tab) — GPU fleets / schedulers / serving fuel
• Prior: 19 — GPU topology · 18 — GPU node pools · 16 — Temporal / Argo · 14 — ML deploy hardening · 13 — ML weights · 12 — Node/container hardening · 11 — Cluster security · 09 — NetworkPolicy / multi-NIC · 04 — Borg/Mesos · 01 — Hyperscale mental model
• Next: Lesson 21 — inference serving SLOs (TTFT/TPOT/goodput, continuous batching, KV-cache as capacity)

Quick self-check

Out loud, in under two minutes:

1. Why does Job parallelism without gang semantics waste GPUs on multi-pod training?
2. Name Kueue’s LocalQueue, ClusterQueue, ResourceFlavor, Cohort — and what each decides.
3. How does Volcano’s PodGroup minMember differ from Kueue’s admit + waitForPodsReady?
4. What breaks if a team borrows cohort GPUs forever with no reclaim/preemption?
5. Why must train gangs and latency inference not share a ClusterQueue without isolation?
6. List four security controls that must hold when an admitted gang pulls model weights — including why queue admins must not be weight writers.