Lesson 25 — AI/ML platform: multi-tenant GPU isolation & quota economics across cells

2026-10-03

What interviewers are really probing

Lesson 18 gave you SKU pools and when MIG / time-slice / exclusive fit. Lesson 19 made NVLink / RDMA / NCCL schedulable — and fragile if you shatter domains for “fairness.” Lesson 20 admitted gangs with Kueue / Volcano fair-share. Lesson 21 made KV cache the capacity wallet. Lesson 22 split prefill vs decode. Lesson 23 gave you eyes. Lesson 24 routed across cells. This lesson is the tenant layer: how do you isolate orgs/teams on a shared GPU fleet, encode reserved vs burst economics, and charge back without silent sharing, cross-tenant weight leaks, or breaking topology — and without a train SA that can lateral-move into every tenant’s model bucket?

Interviewers at hyperscale AI platforms (fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab)) want evidence you have operated multi-tenant GPU quotas under contention, not only “we gave each team a namespace”:

• Soft vs hard multi-tenancy — when MIG / exclusive / time-slice / MPS fit (L18); what “isolation” actually means on silicon
• Tenant = org/team/product — ResourceQuota + LimitRange + PriorityClass + Kueue ClusterQueue / LocalQueue + cohort fair-share (L20)
• Isolation stack — namespaces, NetworkPolicy, RuntimeClass, device plugin / DRA, node taints/labels; SPOG vs per-cell quotas
• Cross-cell quota economics — reserved vs burst, oversubscribe, reclaim, chargeback/showback (GPU-hours, HBM-GB-hours, KV-seat-hours); idle waste vs fragmentation
• Capacity markets across cells — per-tenant admit ceilings; starved tenant vs wasted cell (L24)
• Topology constraints — don’t break NVLink / NCCL domains for fairness (L19)
• Inference multi-tenant — model-per-tenant vs shared model + request auth / rate limits; KV / cache isolation risks (L21)
• Training — gang + fair-share + preemption economics (L20)
• Gotchas — silent GPU sharing; quota that ignores MIG slices; ResourceQuota vs extended resources; cross-tenant weight bucket ACL leaks; train SA lateral movement
• Security floor — per-tenant signed weight digests, private registries, RO mounts, GetObject-only WI scoped per tenant prefix, no shared writable model cache, admission deny cross-tenant PVC/Secret, VAP for RuntimeClass + digests, NetPol default-deny, no flat RDMA trust, audit chargeback labels without PII/prompts (L13/L14)

Interview sentence: “I model each tenant as an org with ResourceQuota + Kueue ClusterQueue nominal quota and cohort fair-share; pick MIG / exclusive / time-slice by trust and SLO class; enforce NetPol + RuntimeClass + per-tenant signed digests and GetObject-only WI on weight prefixes; roll admit ceilings across cells without shattering NVLink domains; and charge back GPU-hours / HBM-GB-hours / KV-seat-hours with reserved-vs-borrowed split — never a flat shared writable model cache or a train SA that can reach every tenant’s bucket.”

Core diagram: Tenants → Quota CP → Cells (MIG / exclusive) → Isolation floor → Chargeback

Read it top-down, then the meters: Each tenant (org / team / product) hits a fleet quota & fair-share control plane — ResourceQuota / LimitRange / PriorityClass plus Kueue ClusterQueue / LocalQueue in a cohort that encodes reserved vs burst and reclaim (L20). That plane issues per-cell admit ceilings into independent cells, each with MIG and exclusive pools (L18) that must respect topology (L19). Everything sits on an isolation & security floor (NetPol, RuntimeClass, signed per-tenant weights, scoped IAM). Chargeback meters close the loop so finance and capacity planning see GPU-hours, HBM-GB-hours, and KV-seat-hours — not vanity SM util alone (L23/L24).

Idiosyncrasies / operator depth

1. Soft vs hard multi-tenancy on GPUs (pick by trust + SLO)

Gotcha: Time-slicing and MPS are utilization tools, not trust tools. Interviewers ding you if you say “we multi-tenant with MPS” when the threat model is cross-org weight theft.

2. Tenant = org/team/product — the control objects that matter

Interview trap: Equating “ResourceQuota of 8 GPUs” with “we have multi-tenant fairness.” Without cohort fair-share + preemption, the loudest submitter owns burst forever.

3. Isolation stack — layers you actually enforce

Operator stance: Per-cell hard ceilings + fleet fair-share policy that proposes ceilings. One global ResourceQuota across 100 cells without cell binding is a spreadsheet, not a control plane.

4. Cross-cell quota economics — reserved, burst, reclaim, waste

Formula interviewers like:
tenant_effective = min(cq_nominal + borrow_allowed, sum_over_cells(cell_admit_tenant), topology_fit)
If topology_fit is zero because you scattered a gang across NVLink domains for “fairness,” you manufactured waste (L19).

5. Capacity markets across cells (tie L24)

Default posture: Inference can follow capacity across cells (with digest pins). Multi-node train stays topology-local; quota must encode flavors (zone / rail / NVLink island), not raw GPU integers.

6. Topology — fairness that breaks NCCL is not fairness

7. Inference multi-tenant — model-per-tenant vs shared model

KV / cache isolation: Never key KV only by session_id without tenant_id. Shared prefix caches across trust domains are a data-exfil footgun. Prefer per-tenant cache partitions or separate replicas for high-trust tenants (L21).

8. Training — gang + fair-share + preemption economics

Gotcha: Preempting without checkpoint discipline “improves utilization” on slides and burns GPU-hours on restarts — chargeback should show useful GPU-hours vs restart waste.

9. Classic gotchas (say these out loud)

Concrete commands & config

Inventory — tenants, queues, GPU / MIG allocatable

# Tenant namespaces + quotas
kubectl get ns -l tenant.platform/org --show-labels
kubectl get resourcequota -A -l tenant.platform/org
 
# Extended resources on GPU nodes (parent + MIG)
kubectl get nodes -l nvidia.com/gpu.present=true \
  -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu,MIG1G:.status.allocatable.nvidia\.com/mig-1g\.10gb,MIG2G:.status.allocatable.nvidia\.com/mig-2g\.20gb
 
# Kueue queues
kubectl get clusterqueue,localqueue,resourceflavor -A
kubectl get clusterqueue -o custom-columns=NAME:.metadata.name,COHORT:.spec.cohort,PEND:.status.pendingWorkloads,ADM:.status.admittedWorkloads

ResourceQuota + LimitRange that include MIG

apiVersion: v1
kind: ResourceQuota
metadata:
  name: t1-gpu-quota
  namespace: tenant-t1
spec:
  hard:
    requests.nvidia.com/gpu: "8"
    requests.nvidia.com/mig-1g.10gb: "16"
    requests.nvidia.com/mig-2g.20gb: "8"
    requests.cpu: "200"
    requests.memory: 2Ti
    pods: "400"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: t1-gpu-limits
  namespace: tenant-t1
spec:
  limits:
    - type: Container
      max:
        nvidia.com/gpu: "8"
        nvidia.com/mig-1g.10gb: "4"
      defaultRequest:
        cpu: "1"
        memory: 4Gi

Kueue ClusterQueues in a cohort (reserved + burst)

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: h100-exclusive-rail-a
spec:
  nodeLabels:
    pool: train-exclusive
    topology.platform/rail: "a"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: t1-cq
spec:
  cohort: ml-org
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
      flavors:
        - name: h100-exclusive-rail-a
          resources:
            - name: nvidia.com/gpu
              nominalQuota: 64
              borrowingLimit: 64   # burst cap from cohort
            - name: cpu
              nominalQuota: 2048
            - name: memory
              nominalQuota: 16Ti
  preemption:
    withinClusterQueue: LowerPriority
    reclaimWithinCohort: Any
  fairSharing:
    weight: 1
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: t1-train
  namespace: tenant-t1
spec:
  clusterQueue: t1-cq

MIG pool vs exclusive — node labels / taints (shape)

# Exclusive train island
kubectl taint nodes -l pool=train-exclusive train-exclusive=true:NoSchedule
kubectl label nodes -l pool=train-exclusive \
  nvidia.com/gpu.product=H100 \
  topology.platform/rail=a \
  tenant.platform/trust=hard
 
# MIG inference pool (profiles via GPU Operator / MIG manager — [L18])
kubectl label nodes -l pool=infer-mig \
  nvidia.com/mig.config=all-1g.10gb \
  tenant.platform/trust=mig

Per-tenant admit ceiling across cells (sketch)

# Fleet CR / ConfigMap shape — ceilings feed L24 routers + Kueue flavors
kubectl --context fleet-admin get tenantquota t1 -o yaml
# status.cells:
#   cell-a: { reservedGpu: 32, burstGpu: 16, mig1g: 64, admit: green }
#   cell-b: { reservedGpu: 16, burstGpu: 8,  mig1g: 32, admit: yellow }
 
python3 - <<'PY'
# Binding admit for tenant on a cell
reserved, burst_used, cell_free, topology_ok = 32, 10, 40, 1
tenant_ceil = min(reserved + max(0, 16 - burst_used), cell_free) * topology_ok
print(f"tenant_cell_admit={tenant_ceil}")
PY

Chargeback / showback PromQL sketches

# Allocated GPU-hours by tenant (allocation truth — prefer over SM%)
sum by (tenant) (
  kube_pod_resource_request{resource="nvidia_com_gpu"}
  * on(namespace) group_left(tenant)
    kube_namespace_labels{label_tenant_platform_org!=""}
)

# Kueue queue join (allowlist kueue labels on kube-state-metrics)
sum by (label_kueue_x_k8s_io_cluster_queue_name) (
  rate(container_cpu_usage_seconds_total{container!=""}[5m])
  * on(namespace,pod) group_left(label_kueue_x_k8s_io_cluster_queue_name)
    kube_pod_labels{label_kueue_x_k8s_io_cluster_queue_name!=""}
)

# MIG profile-hours (exporter label mig_profile)
sum by (tenant, mig_profile) (rate(gpu_allocated_seconds_total[1h]))

# KV-seat-hours by tenant (inference)
sum by (tenant) (inference_kv_seats_inuse) * 1  # integrate over time in recording rule

NetPol default-deny between tenant namespaces

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: tenant-t1
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
---
# Allow only tenant-local + approved platform scrape / weight hydrate
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-platform-and-same-tenant
  namespace: tenant-t1
spec:
  podSelector: {}
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { tenant.platform/org: t1 }
        - namespaceSelector:
            matchLabels: { role: platform-metrics }
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { tenant.platform/org: t1 }
    - to:
        - namespaceSelector:
            matchLabels: { role: weight-hydrate }
      ports:
        - protocol: TCP
          port: 443

VAP — RuntimeClass + digest pins + no cross-tenant mounts (sketch)

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: tenant-gpu-floor
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["deployments", "jobs"]
  validations:
    - expression: >
        object.spec.template.spec.runtimeClassName in ["nvidia", "nvidia-mig", "kata-nvidia"]
      message: "GPU workloads require approved RuntimeClass"
    - expression: >
        !has(object.spec.template.metadata.annotations) ||
        object.spec.template.metadata.annotations["platform.weight/digest"].startsWith("sha256:")
      message: "weight digest pin required (sha256:...)"

Failures & fixes

Security & hardening

Multi-tenant GPU fleets concentrate every tenant’s weights, prompts, and train credentials on shared silicon. Soft sharing without a security floor turns “fair-share utilization” into cross-tenant exfil. Same spirit as L11–L14, applied to tenant quotas + cell pools.

AI/ML weight, model & deployment hardening for multi-tenant GPUs (required)

Multi-tenant quotas do not relax L13/L14. Each tenant hydrates signed, digest-pinned weights onto RO volumes with GetObject-only identity scoped to that tenant’s prefix. No shared writable cache across trust domains. Admission blocks cross-tenant PVC/Secret mounts and requires RuntimeClass + digests. NetPol default-deny between tenant namespaces; RDMA is not a free-for-all. Chargeback metrics carry tenant/queue/SKU — never prompts.

# Sketch: multi-tenant GPU security floor (platform CR shape)
apiVersion: policy.platform/v1
kind: MultiTenantGpuGate
metadata:
  name: gpu-tenant-floor
spec:
  match:
    labels:
      trust-domain: gpu-tenant
  require:
    imageDigestPin: true
    weightDigestPin: true
    cosignVerify: true
    privateRegistryOnly: true
    weightMount: "readOnly"
    sharedWritableModelCache: false
    workloadIdentity:
      objectGetOnly: true
      putObjectPrefixes: ["checkpoints/${tenant}/*"]
      getObjectPrefixes: ["models/${tenant}/*", "models/shared-base/*"]
      crossTenantPrefixAccess: false
    networkPolicy:
      defaultDeny: true
      allowSameTenantOnly: true
      flatRdmaTrust: false
    admission:
      runtimeClasses: ["nvidia", "nvidia-mig", "kata-nvidia"]
      denyCrossTenantPvc: true
      denyCrossTenantSecret: true
    quota:
      mustIncludeMigResources: true
      chargebackLabels: ["tenant", "queue", "sku", "cell"]
      forbidLabelKeys: ["prompt", "user_email", "session_raw"]
    queues:
      requireLocalQueue: true
      trainSaInInferNs: false

Punchline: Multi-tenant GPU platforms are security-critical capacity markets. Per-tenant digests, prefix-scoped GetObject-only WI, no shared writable caches, NetPol + VAP, topology-aware quotas, and chargeback without PII are what keep “fair-share utilization” from becoming cross-org compromise.

Interview soundbites

• “Soft sharing (time-slice/MPS) is utilization; MIG/exclusive/NetPol/RBAC is tenancy.”
• “Tenant = ns + ResourceQuota + PriorityClass + Kueue CQ/LQ in a cohort — not a spreadsheet.”
• “Nominal is reserved; borrow is burst with reclaim — fair-share weights stop burst monopolies.”
• “Quota that ignores MIG profiles is a fake ceiling.”
• “Don’t shatter NVLink/NCCL domains to look fair — topology-fit is part of admit.”
• “Per-cell hard ceilings + fleet fair-share proposals; SPOG alone oversubscribes reality.”
• “Bill GPU-hours / HBM-GB-hours / KV-seat-hours; show idle vs fragmentation separately.”
• “Shared-model inference needs tenant_id in auth, rate limits, and KV keys.”
• “Per-tenant signed digests, RO mounts, GetObject-only prefix WI, no shared writable model cache.”
• “Next: GPU cost attribution, reclaim & spot/preemptible economics on the fleet.”

How this maps to the JD

Hyperscale + AI/ML platform JDs that mention shared GPU fleets, multi-tenant platforms, and chargeback are asking whether you can isolate tenants and run quota economics under contention — not create three namespaces and hope. This lesson lifts L18–L24 into the tenant/market layer with L9/L11–L14 controls. Fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab). Next up — Lesson 26: GPU cost attribution, reclaim & spot/preemptible economics on the fleet.

Watch (talks & demos)

Real talks — watch after Failures & fixes.

Multitenancy and Fairness at Scale with Kueue: A Case Study

KubeCon / AI Day talk (Aldo Culquicondor & Rajat Phull) on Kueue nominal quotas, cohort borrowing, fair-share weights, reclaim preemption, and hierarchical cohorts for multi-team ML platforms. Operator takeaway: encode reserved vs burst in ClusterQueues, not in a side spreadsheet — and make reclaim/preemption explicit so burst tenants stay preemptible.

Open on YouTube (opens in a new tab)

Lessons learned orchestrating multi-tenant GPUs on OpenShift AI with NVIDIA KAI

Production-flavored walkthrough (Luca Berton) of bare-metal multi-tenant GPUs: GitOps tenant templates, NetPol/quotas, MIG vs time-slice vs full GPU by workload, DCGM visibility, and chargeback-oriented persona views. Operator takeaway: safety / fairness / efficiency are three different knobs — a working GPU cluster ≠ a shareable GPU platform.

Open on YouTube (opens in a new tab)

Mastering GPU Management in Kubernetes Using the Operator Pattern

NVIDIA talk (Shiva Krishna Merla & Kevin Klues) on GPU Operator lifecycle, MIG single vs mixed mode, time-slicing, and MPS configuration. Operator takeaway: admins choose sharing mode a priori today — map those modes to trust classes in your tenant model (L18); don’t sell soft replicas as hard isolation.

Open on YouTube (opens in a new tab)

References

• Kueue ClusterQueue (opens in a new tab) · Kueue Preemption / Fair Sharing (opens in a new tab) · Monitor resource usage by queue (opens in a new tab)
• NVIDIA MIG User Guide (opens in a new tab) · GPU Operator (opens in a new tab) · DRA (opens in a new tab)
• Kubernetes ResourceQuota (opens in a new tab) · NetworkPolicy (opens in a new tab) · ValidatingAdmissionPolicy (opens in a new tab)
• wafer-ai/gpu-perf-engineering-resources (opens in a new tab) — fleet / inference arithmetic fuel
• Prior: 24 — Multi-cluster inference routing / capacity · 23 — GPU observability · 21 — Inference serving SLOs · 20 — Training/inference scheduling · 19 — GPU topology · 18 — GPU node pools / MIG · 14 — ML deploy hardening · 13 — ML weights · 11 — Cluster security · 09 — NetworkPolicy
• Next: Lesson 26 — AI/ML platform: GPU cost attribution, reclaim & spot/preemptible economics on the fleet

Quick self-check

Out loud, in under two minutes:

1. Soft vs hard multi-tenancy on GPUs — when MIG, exclusive, time-slice, and MPS each fit (and what they do not guarantee).
2. Name the tenant control objects: ResourceQuota, LimitRange, PriorityClass, ClusterQueue, LocalQueue, cohort — and what “nominal vs borrow” means.
3. Explain why a ResourceQuota that only lists nvidia.com/gpu fails on a MIG pool — and how chargeback should meter MIG profile-hours.
4. Walk reserved vs burst reclaim when a starved owner returns; what fair-share weights change under contention.
5. List five security controls for multi-tenant weights/deploys — per-tenant digests, private registry, RO mounts, GetObject-only prefix WI, no shared writable cache, NetPol/VAP, no flat RDMA, chargeback labels without PII.