Lesson 26 — AI/ML platform: GPU cost attribution, reclaim & spot/preemptible economics on the fleet

2026-10-04

What interviewers are really probing

Lesson 18 gave you SKU pools. Lesson 19 made topology schedulable. Lesson 20 admitted gangs with Kueue fair-share and reclaim. Lesson 21 made KV cache the capacity wallet. Lesson 23 gave you eyes. Lesson 24 routed across cells. Lesson 25 encoded tenant quotas and reserved-vs-burst. This lesson is the FinOps layer: how do you attribute GPU cost truthfully, reclaim idle/borrowed capacity without burning restart waste, and run spot/preemptible vs on-demand/reserved economics — without breaking SLOs or relaxing weight/deploy hardening for “cheap” capacity?

Interviewers at hyperscale AI platforms (fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab)) want evidence you have operated GPU fleet economics under interruption and contention, not only “we turned on Kubecost”:

• Cost attribution meters — GPU-hours, MIG-profile-hours, HBM-GB-hours, KV-seat-hours (L21); allocation vs utilization vs reserved-idle vs fragmentation waste (L25)
• Showback vs chargeback — labels for tenant / queue / sku / cell / purchase-option; OpenCost / Kubecost / cloud CUR join patterns
• Reclaim economics — Kueue reclaimWithinCohort / fair-share preemption; owner returns → preempt borrowers; checkpoint discipline; PriorityClass & preemptible burst queues (L20)
• Spot / preemptible economics — ResourceFlavors for on-demand vs spot; flavor fungibility; Karpenter / CA interruption handling (2-min spot notice, cordon/drain); diversification; when not to put inference or non-checkpointed multi-node train on spot
• Committed use / reservations / savings plans vs spot mix; idle reserved = still bill reserved rate
• Cross-cell cost — route inference to cheaper green capacity when digest-pinned (L24) vs sticky train topology (L19)
• Gotchas — billing SM% only; double-counting parent GPU + MIG; spot interrupt mid-gang without gang-aware reclaim; cost labels with PII/prompts; hydrating weights onto ephemeral spot without RO digest pins; shared writable model cache on spot fleets; train SA that can PutObject anywhere after interrupt retry storms
• Security floor — signed digests, private registries, RO mounts, GetObject-only WI, no shared writable caches, admission/RBAC, NetPol — cheap capacity never relaxes L13/L14

Interview sentence: “I attribute GPU-hours, HBM-GB-hours, and KV-seat-hours by tenant + queue + sku + cell + purchase-option; encode reserved vs burst reclaim in Kueue; prefer spot ResourceFlavors for checkpointed batch with on-demand fallback for SLO-critical inference; join OpenCost/CUR without leaking prompts; and never relax signed digests, RO mounts, or scoped IAM just because capacity is cheap.”

Core diagram: Workloads → Attribution CP → Capacity markets → Reclaim → Cells → Security floor → FinOps ledger

Read it left-to-right, then the ledger: Workloads hit a cost attribution control plane that meters allocation truth (GPU / MIG / HBM / KV-seat hours) with labels that survive a CUR join. Those labels steer ResourceFlavors — reserved/CUD, on-demand, spot — into reclaim & preemption (cohort fair-share, PriorityClass, checkpoint discipline). Capacity lands in cells with purchase-option pools; inference may follow cheaper green cells when digests are pinned (L24), while train gangs stay topology-sticky (L19). Everything sits on a security floor that does not bend for spot savings. The FinOps ledger closes showback/chargeback without PII in labels.

Idiosyncrasies / operator depth

1. Meters that matter (stop billing vanity SM%)

Interview rule: Bill allocation + purchase-option; report utilization and restart waste as efficiency, not as the invoice.

2. Showback vs chargeback — and the label contract

Gotcha: High-cardinality labels from request IDs or prompt hashes blow cost pipelines and leak sensitive content. Chargeback cardinality belongs to tenant/queue/sku/cell/purchase-option — full stop.

3. Reclaim economics — Kueue as the market maker

Formula interviewers like:
effective_cost = reserved_rate × (nominal) + od_rate × (od_used) + spot_rate × (spot_used) + restart_waste_rate × (preempt_restarts)
If you optimize only spot_used and ignore restart_waste, you can “save” yourself into a more expensive week.

4. Spot / preemptible vs on-demand — flavor fungibility

Flavor fungibility: Prefer spot → on-demand → reserved for batch; prefer reserved → on-demand (never spot) for SLO infer. Encode as Kueue ResourceFlavors + admission, not tribal knowledge.

5. Interruption handling — Karpenter / Cluster Autoscaler

6. Committed use / reservations / savings plans

7. Cross-cell cost arbitration (tie L24)

8. Classic gotchas (say these out loud)

Concrete commands & config

Inventory — purchase-option labels, flavors, spot nodes

# Nodes by purchase-option / lifecycle
kubectl get nodes -L node.kubernetes.io/instance-type,karpenter.sh/capacity-type,purchase-option,nvidia.com/gpu.product
 
# Spot / preemptible candidates
kubectl get nodes -l karpenter.sh/capacity-type=spot -o wide
kubectl get nodes -l cloud.google.com/gke-preemptible=true -o wide   # GKE preemptible shape
 
# Kueue flavors + queues (reserved vs spot)
kubectl get resourceflavor,clusterqueue,localqueue -A
kubectl get clusterqueue -o yaml | rg -n "nominalQuota|borrowingLimit|flavors|preemption|reclaim"

ResourceFlavors — reserved, on-demand, spot

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: h100-reserved-rail-a
spec:
  nodeLabels:
    pool: train-reserved
    purchase-option: reserved
    topology.platform/rail: "a"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: h100-ondemand
spec:
  nodeLabels:
    pool: gpu-ondemand
    purchase-option: on-demand
    karpenter.sh/capacity-type: on-demand
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: h100-spot
spec:
  nodeLabels:
    pool: gpu-spot
    purchase-option: spot
    karpenter.sh/capacity-type: spot
  # Prefer spot; workloads must tolerate preempt + checkpoint

ClusterQueue — prefer spot for batch, reclaim reserved for owners

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: t1-batch-cq
spec:
  cohort: ml-org
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
      flavors:
        - name: h100-spot
          resources:
            - name: nvidia.com/gpu
              nominalQuota: 32
              borrowingLimit: 64
        - name: h100-ondemand   # fungible fallback when spot dry
          resources:
            - name: nvidia.com/gpu
              nominalQuota: 8
              borrowingLimit: 16
        - name: h100-reserved-rail-a  # last resort for this batch CQ
          resources:
            - name: nvidia.com/gpu
              nominalQuota: 0
              borrowingLimit: 8
  preemption:
    withinClusterQueue: LowerPriority
    reclaimWithinCohort: Any
  fairSharing:
    weight: 1
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: t1-infer-slo-cq
spec:
  cohort: ml-org
  resourceGroups:
    - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
      flavors:
        - name: h100-reserved-rail-a
          resources:
            - name: nvidia.com/gpu
              nominalQuota: 48
        - name: h100-ondemand
          resources:
            - name: nvidia.com/gpu
              nominalQuota: 16
              borrowingLimit: 16
        # intentionally NO spot flavor for SLO infer
  preemption:
    withinClusterQueue: LowerPriority
    reclaimWithinCohort: Any

PriorityClass + preemptible burst queue

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-reserved
value: 100000
globalDefault: false
description: Reserved / CUD seats — reclaim winners
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-ondemand
value: 50000
description: On-demand overflow
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-spot-burst
value: 10000
preemptionPolicy: PreemptLowerPriority
description: Spot / borrowed burst — must be checkpointed

Karpenter NodePools — spot diversification + OD fallback (sketch)

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-spot-batch
spec:
  template:
    metadata:
      labels:
        pool: gpu-spot
        purchase-option: spot
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["p5.48xlarge", "p5e.48xlarge", "p4d.24xlarge"]  # diversify
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a", "us-east-1b", "us-east-1c"]
      taints:
        - key: nvidia.com/gpu
          value: "true"
          effect: NoSchedule
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-spot
  disruption:
    consolidateAfter: 15m
    # Interruption handling: Karpenter drains on spot notice
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-ondemand-fallback
spec:
  template:
    metadata:
      labels:
        pool: gpu-ondemand
        purchase-option: on-demand
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
      taints:
        - key: nvidia.com/gpu
          value: "true"
          effect: NoSchedule
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-od

Spot notice → cordon/drain (operator loop)

# Watch interruption / Karpenter events
kubectl get events -A --field-selector reason=Interrupted -w
kubectl get nodeclaims -l karpenter.sh/capacity-type=spot -o wide
 
# Manual drain drill (chaos / game day)
NODE=$(kubectl get nodes -l karpenter.sh/capacity-type=spot -o jsonpath='{.items[0].metadata.name}')
kubectl cordon "$NODE"
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --grace-period=90 --timeout=110s
# Assert: checkpointed jobs resume; SLO infer replicas never lived here

Cost attribution — OpenCost + PromQL sketches

# OpenCost / Kubecost allocation API shape (cluster-local)
curl -s "http://opencost.opencost:9003/allocation/compute?window=24h&aggregate=label:tenant,label:purchase-option,label:sku"
 
# Ensure cost labels present on workloads
kubectl get pods -A -L tenant,queue,sku,cell,purchase-option | head

# Allocated GPU-hours by tenant + purchase-option (allocation truth)
sum by (tenant, purchase_option, sku) (
  kube_pod_resource_request{resource="nvidia_com_gpu"}
  * on(namespace,pod) group_left(tenant, purchase_option, sku)
    kube_pod_labels
)

# MIG profile-hours (avoid double-count with parent GPU)
sum by (tenant, mig_profile) (
  rate(gpu_mig_allocated_seconds_total[1h])
)

# KV-seat-hours by tenant (inference product meter)
sum by (tenant, cell) (inference_kv_seats_inuse)

# Reserved-idle: nominal reserved - allocated on reserved flavors
# (recording rule sketch — join CQ nominal vs allocated)

# Restart waste after preempt (jobs restarted due to interruption)
sum by (tenant, queue) (
  increase(training_job_restarts_total{reason="preempt_or_spot"}[1d])
) * on(tenant, queue) group_left avg(job_gpu_request)

CUR join pattern (sketch — no PII)

-- BigQuery / Athena shape: join K8s allocation export to CUR
SELECT
  a.tenant,
  a.queue,
  a.sku,
  a.cell,
  a.purchase_option,
  SUM(a.gpu_hours) AS alloc_gpu_hours,
  SUM(c.unblended_cost) AS cloud_cost,
  SUM(a.restart_waste_gpu_hours) AS restart_waste
FROM k8s_gpu_allocation a
JOIN cloud_cur c
  ON a.instance_id = c.line_item_resource_id
 AND a.hour = c.usage_hour
WHERE a.tenant IS NOT NULL
  -- never select prompt / user_email columns
GROUP BY 1,2,3,4,5;

Checkpoint + weight hydrate on spot (security-preserving)

# Job must: PriorityClass spot-burst, RO digest-pinned weights, scoped WI
apiVersion: batch/v1
kind: Job
metadata:
  name: t1-spot-train
  namespace: tenant-t1
  labels:
    tenant: t1
    queue: t1-batch
    sku: h100
    cell: cell-a
    purchase-option: spot
spec:
  template:
    metadata:
      annotations:
        platform.weight/digest: "sha256:REPLACE_ME"
      labels:
        tenant: t1
        purchase-option: spot
    spec:
      priorityClassName: gpu-spot-burst
      runtimeClassName: nvidia
      serviceAccountName: t1-train-sa
      nodeSelector:
        purchase-option: spot
      containers:
        - name: train
          image: registry.internal/train@sha256:REPLACE_ME
          resources:
            limits:
              nvidia.com/gpu: "8"
          volumeMounts:
            - name: weights
              mountPath: /models
              readOnly: true
            - name: scratch
              mountPath: /scratch
          env:
            - name: CHECKPOINT_URI
              value: s3://checkpoints/t1/jobs/t1-spot-train/
      volumes:
        - name: weights
          persistentVolumeClaim:
            claimName: t1-weights-ro
        - name: scratch
          emptyDir: {}
      restartPolicy: OnFailure

Failures & fixes

Security & hardening

Cheap capacity is not a license to weaken the model supply chain. Spot nodes, preemptible VMs, and aggressive reclaim amplify hydrate storms, retry amplification, and ephemeral-disk confusion — exactly when teams are tempted to “just cache weights writable on the node.” Same spirit as L11–L14, applied to FinOps + purchase-option pools.

AI/ML weight, model & deployment hardening for cost-optimized capacity (required)

Running spot / preemptible pools does not relax L13/L14. Every hydrate — especially after interrupt — must pull signed, digest-pinned weights onto RO volumes with GetObject-only identity. No shared writable cache “because spot disks die anyway.” Admission still requires RuntimeClass + digests; NetPol still default-denies; train SAs still cannot PutObject outside tenant checkpoint prefixes. Cross-cell cost routing only after digests are warm (L24).

# Sketch: FinOps GPU security floor (platform CR shape)
apiVersion: policy.platform/v1
kind: GpuFinOpsGate
metadata:
  name: gpu-cost-floor
spec:
  match:
    labels:
      trust-domain: gpu-tenant
  require:
    imageDigestPin: true
    weightDigestPin: true
    cosignVerify: true
    privateRegistryOnly: true
    weightMount: "readOnly"
    sharedWritableModelCache: false
    workloadIdentity:
      objectGetOnly: true
      putObjectPrefixes: ["checkpoints/${tenant}/*"]
      getObjectPrefixes: ["models/${tenant}/*", "models/shared-base/*"]
      crossTenantPrefixAccess: false
      # interrupt retries must not widen PutObject
      maxPutObjectQpsPerSa: 20
    networkPolicy:
      defaultDeny: true
      allowSameTenantOnly: true
    admission:
      runtimeClasses: ["nvidia", "nvidia-mig", "kata-nvidia"]
      requireCostLabels: ["tenant", "queue", "sku", "cell", "purchase-option"]
      forbidLabelKeys: ["prompt", "user_email", "session_raw"]
    flavors:
      sloInferAllowSpot: false
      batchPreferSpot: true
      requireCheckpointAnnotationForSpot: true
    reclaim:
      gangAware: true
      meterRestartWaste: true

Punchline: GPU FinOps is a security-critical capacity market. Attribute with honest meters and labels; reclaim with Kueue + checkpoints; prefer spot for checkpointed batch and on-demand/reserved for SLO inference; join CUR cleanly — and never trade signed digests, RO mounts, or scoped IAM for a lower line item.

Interview soundbites

• “Bill allocation + purchase-option; report util and restart waste as efficiency — not the invoice.”
• “Meters: GPU-hours, MIG-profile-hours, HBM-GB-hours, KV-seat-hours — never SM% alone.”
• “Labels: tenant + queue + sku + cell + purchase-option; never prompts or PII.”
• “Nominal is reserved; borrow is burst with reclaim — checkpoint or don’t burst.”
• “Spot for checkpointed batch; on-demand/reserved for SLO infer — encode in ResourceFlavors.”
• “Idle reserved still bills reserved rate — show reserved-idle separately.”
• “2-minute spot notice is not an SLO budget.”
• “Gang-aware reclaim or you donate GPU-hours to stranded ranks.”
• “Route infer to cheaper green cells only when digests are pinned — cost after security.”
• “Never hydrate unsigned weights onto spot to save money.”
• “Next: GPU supply chain — weight registries, checkpoint lifecycle & cold-start hydrate economics.”

How this maps to the JD

Hyperscale + AI/ML platform JDs that mention GPU FinOps, spot/preemptible fleets, chargeback, and utilization are asking whether you can run a capacity market under interruption — not paste a Kubecost screenshot. This lesson lifts L18–L25 into attribution + reclaim + purchase-option economics with L13/L14 as a hard floor. Fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab). Next up — Lesson 27: GPU supply chain — weight registries, checkpoint lifecycle & cold-start hydrate economics.

Watch (talks & demos)

Real talks — watch after Failures & fixes.

Divide and Conquer: Master GPU Partitioning and Visualize Savings with OpenCost

Kaysie Yu on GPU partitioning visibility via DCGM + OpenCost — how MIG / partitioning choices show up as real savings, not slideware. Operator takeaway: cost attribution needs GPU-aware meters (allocation by profile/SKU), not vanity SM% — wire OpenCost (or Kubecost) to the same labels you use for tenant/queue/purchase-option.

Open on YouTube (opens in a new tab)

Multitenancy and Fairness at Scale with Kueue

Aldo Culquicondor & Rajat Phull on nominal quotas, cohort borrowing, fair-share weights, and reclaim preemption. Operator takeaway: encode reserved vs burst reclaim in ClusterQueues — this is the control plane behind “owner returns → preempt borrowers,” and it is the right place to hang spot-burst PriorityClasses.

Open on YouTube (opens in a new tab)

Run workloads efficiently on EKS with Karpenter and EC2 Spot Instances

AWS re:Invent talk on Spot + Karpenter: interruption handling, diversification across pools, and when Spot fits. Operator takeaway: adapt the same pattern to GPU NodePools — capacity-type requirements, multi-family diversification, cordon/drain within the notice window — and keep SLO inference off Spot.

Open on YouTube (opens in a new tab)

References

• OpenCost (opens in a new tab) · Kubecost (opens in a new tab) · AWS CUR (opens in a new tab) · GCP BigQuery billing export (opens in a new tab)
• Kueue ClusterQueue (opens in a new tab) · Kueue Preemption / Fair Sharing (opens in a new tab) · Kueue ResourceFlavor (opens in a new tab)
• Karpenter concepts (opens in a new tab) · AWS EC2 Spot interruptions (opens in a new tab) · GKE Spot VMs (opens in a new tab)
• NVIDIA MIG User Guide (opens in a new tab) · DCGM (opens in a new tab)
• wafer-ai/gpu-perf-engineering-resources (opens in a new tab) — fleet / inference arithmetic fuel
• Prior: 25 — Multi-tenant GPU isolation / quota · 24 — Multi-cluster inference routing / capacity · 23 — GPU observability · 21 — Inference serving SLOs · 20 — Training/inference scheduling · 19 — GPU topology · 18 — GPU node pools / MIG · 14 — ML deploy hardening · 13 — ML weights · 11 — Cluster security
• Next: Lesson 27 — AI/ML platform: GPU supply chain — weight registries, checkpoint lifecycle & cold-start hydrate economics

Quick self-check

Out loud, in under two minutes:

1. Name four cost meters beyond SM% (GPU-hours, MIG-profile-hours, HBM-GB-hours, KV-seat-hours) and why allocation ≠ utilization ≠ reserved-idle ≠ fragmentation.
2. Explain showback vs chargeback and the five label keys you allow on cost series — plus two keys you forbid.
3. Walk reclaimWithinCohort when a reserved owner returns; why checkpoint discipline and gang-aware reclaim matter to the bill.
4. Prefer spot vs on-demand vs reserved for: checkpointed batch, SLO inference, non-checkpointed multi-node train — and how ResourceFlavors encode that.
5. List five security controls that still bind on spot capacity — signed digests, private registry, RO mounts, GetObject-only prefix WI, no shared writable cache, admission label contract, NetPol/RBAC.