Lesson 31 — AI/ML platform: evaluation & release gating at fleet scale — eval clusters, regression gates, canary rollouts of new weights

2026-10-09

What interviewers are really probing

Lesson 29 kept the training run alive and Lesson 30 kept it fed. Both end with a pile of checkpoints. This lesson covers how one of them becomes the weights that answer production traffic in every cell, and how you stop a bad one before it does. L27 moved signed weights to the GPUs. L21 and L28 defined serving SLOs and capacity. L24 gave you cells and weighted routing. Release gating ties those together: a new set of weights is a deploy, and an eval result is the test suite that has to pass before the deploy is allowed.

What they check for (fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab), platform framing, not research):

• Eval as infrastructure. Can you size and schedule an eval cluster: GPU-hours per suite, queue priority against training and serving, checkpoint-cadence evals versus full candidate evals, and sandboxes for agentic tasks?
• Reproducibility. Do you pin the weights digest, tokenizer, chat template, serving image, sampling parameters, dataset version, harness version and judge model? Do you evaluate on the same serving stack (quantization, TP layout, kernels) that production will run?
• Statistics, not vibes. Do you know the standard error on a 1,000-question benchmark, why paired comparisons beat two independent scores, and why 200 benchmarks at p=0.05 give you about ten false alarms per release?
• Gate design. Hard floors (safety, security, refusal), non-inferiority margins per slice, must-improve targets, and who is allowed to change the thresholds.
• Performance is part of quality. A model that writes 40% longer answers is a TPOT and KV-capacity regression (L21) even if every benchmark goes up.
• Online rollout. Shadow versus canary for weights, weighted routing between InferencePools, sticky sessions, statistical power at 1% traffic, per-cell canaries, bake time, and rollback as a routing change instead of a redeploy.
• Security. Eval sets are crown jewels (leak them and every future score is contaminated), the eval cluster runs untrusted candidate artifacts, shadow traffic carries real user data, and the gate decision itself has to be a signed artifact that admission enforces.

Interview sentence: "I treat new weights as a deploy with a test suite. Candidates run on a dedicated, Kueue-queued eval cluster with the exact production serving image and digest-pinned weights, tokenizer and judge. The gate is a signed attestation over paired, per-slice non-inferiority tests plus hard safety floors and a load test, and admission refuses any InferencePool whose weights lack it. Then I shadow, canary one cell at a time by HTTPRoute weight with automated analysis on errors, TTFT and TPOT, output length and an online judge score, and keep the old pool warm so rollback is a weight flip."

Core diagram: Checkpoint → Eval cluster → Gate → Shadow → Canary → Fleet (+ rollback, security floor)

How to read it: top to bottom is the life of a set of weights. Offline evals decide whether a candidate may be deployed at all; online stages decide whether it may take traffic, and how much. The attestation is the hinge: admission checks it, so skipping the eval cluster means the pods never start. The PNG adds the numbers: eval throughput math, error bars, the gate types, canary metrics and the security floor.

Idiosyncrasies / operator depth

1. Eval is a release gate, not a dashboard

Most teams start with evals as a leaderboard someone looks at. At fleet scale that breaks the first time a model ships because "the numbers looked fine" and nobody checked the slice that regressed. The fix is to treat promotion like any other deploy pipeline:

The output of the offline stages is not a chart. It is a decision record: candidate digest, baseline digest, every input that was pinned, every test with its interval, pass/fail per rule, and who approved any override. That record gets signed and attached to the weights (security section).

2. Eval clusters: capacity, queues and throughput math

Evals are a GPU workload with its own shape: bursty (every candidate, every checkpoint), latency-sensitive for the humans waiting on a release, and highly parallel. Run them on a dedicated ClusterQueue in the same Kueue cohort as training and batch (L20, L25):

• Reserved floor big enough for the fast checkpoint suite and one full candidate at a time; borrow the rest from training's idle quota.
• Priority: candidate evals for a pending release preempt exploratory evals; checkpoint evals never preempt the training gang that produced them.
• Use serving SKUs, not training SKUs, when you can. Evals are inference. An eval that ran on 8× H100 with TP=8 in bf16 while production serves FP8 with TP=4 on a different SKU is measuring a different system.

Throughput math (illustrative). A full candidate suite of 40,000 generative items averaging 600 output tokens is 24M output tokens. A 70B model on one 8-GPU vLLM replica at high batch might sustain on the order of 2,500–4,000 output tok/s, so call it 2–3 hours per replica. Eight replicas get it under 30 minutes. Add log-likelihood (multiple-choice) tasks, long-context tasks where prefill dominates, and agentic tasks that run tool sandboxes for minutes per item, and a full suite is commonly hundreds of GPU-hours for a large model, repeated per candidate and per seed. Put it in the cost model (L26) as its own line, because eval spend grows with model count and release cadence, not with traffic.

Checkpoint cadence. Evaluating every checkpoint with the full suite is the most common way to let eval spend outgrow training. Run a small, stable, low-variance suite on every checkpoint (or every Nth), and the full suite only on checkpoints nominated as candidates.

Agentic and code evals need sandboxes. Code execution, browser and tool-use tasks run model-generated code. That's a container per task, a hard timeout, no credentials and no network beyond the task's mock services (security section). Sandbox start time and capacity, not GPUs, often bound these suites.

3. Reproducibility: pin the whole system, then measure the noise you can't pin

A score is a function of far more than the weights. Pin all of it in the eval manifest:

Even fully pinned, LLM inference is not bitwise deterministic in practice: with dynamic batching, the same prompt can produce different outputs depending on what else is in the batch, because many kernels aren't batch-invariant (Thinking Machines' "Defeating Nondeterminism in LLM Inference" walks through why). Don't fight that in the gate; account for it. Run sampled tasks with several seeds or samples per item, report the variance, and make the gate statistical.

Evaluate the artifact you will ship. The most expensive gating bug I know is "we evaluated the bf16 checkpoint in a research harness and served an FP8 quantized build with a different chat template." Build the serving artifact first (the L27 OCI weights artifact plus serving image), then evaluate that through the same OpenAI-compatible endpoint the gateway will call.

4. Regression gates are statistics problems

Error bars. For a benchmark of n independent items with accuracy p, the standard error is sqrt(p(1-p)/n). At n = 1,000 and p = 0.80 that's about 1.26 points, so a 95% interval is about ±2.5 points. A "1.5-point drop" on that benchmark is noise. Items are rarely independent (many questions share a passage or a template), so use clustered standard errors by group; they are larger. See Miller, "Adding Error Bars to Evals" (arXiv 2411.00640).

Pair everything. Candidate and baseline answer the same questions, so test the per-item difference, not two independent scores. Pairing removes the variance from question difficulty and usually shrinks the interval a lot. Use a paired bootstrap or the paired-difference SE.

Multiple comparisons. A gate with 200 benchmark-slice checks at α = 0.05 will flag about ten regressions on an identical model. Options: fewer, larger, curated slices; non-inferiority margins chosen up front; a false-discovery-rate correction; and a small number of hard floors that never get statistical slack.

Three kinds of rule.

Contamination and Goodhart. Public benchmarks leak into web crawls and then into training data (L30 lineage is how you prove it didn't). Keep private held-out sets that never leave the eval enclave, embed canary strings to detect leakage, rotate items, and compare public versus private scores; a model that jumps on the public version and not the private one is memorizing. Any metric that becomes the only gate gets optimized; a balanced slate of slices plus human review of samples keeps it honest.

5. Judges are pinned dependencies

LLM-as-judge scales open-ended evaluation, but a judge is a model with known biases: position bias, verbosity bias, and self-preference (Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena"). Operator rules:

• Pin the judge (weights digest or provider model version, prompt, temperature 0) in the eval manifest. Changing the judge means re-baselining, never comparing across judge versions.
• Calibrate against humans. Keep a human-labeled golden set and track judge–human agreement (Cohen's κ) as a release metric of the eval system itself.
• Pairwise with swap. For A/B judging, run both orders and count only consistent verdicts.
• Cheaper judges where possible. Exact match, regex, unit tests and small classifiers are cheaper and less biased than a 70B judge.
• Never let a candidate judge itself or a sibling from the same training run.

Lesson 32 goes deep on judge calibration and keeping offline and online evals consistent; here the point is that the judge is a versioned dependency inside the gate.

6. Performance and capacity are part of the gate

New weights change serving behavior even at identical architecture:

• Output length distribution. A model that answers in 900 tokens instead of 600 costs 50% more decode time, KV seat-time and GPU-hours per request (L21, L26). Gate on p50/p95 output tokens on a production-shaped prompt set.
• Latency under load. Run a fixed-rate load test (GuideLLM, inference-perf) on a production-shaped pool and compare TTFT/TPOT p95 and max sustainable rate to the baseline at the same SLO.
• Memory. Different architecture, vocab size or context length changes KV blocks per GPU and therefore cell capacity (L23, L24). The autoscaler's per-replica capacity numbers (L28) must be updated before the rollout, or HPA will undershoot.
• Cold start. Bigger weights or a new format change hydrate time (L27); preheat the new digest into every cell's cache on promotion, before the canary.

7. Shadow, canary and progressive rollout for weights

Shadow (mirror). Copy a sample of real requests to the candidate, never return its answer, and compare offline. It finds crashes, latency, length drift, refusal-rate shifts and judge-score deltas on real distribution with zero user exposure. Two traps: shadowing doubles inference cost for the mirrored slice, and agentic requests must not execute side effects. A shadow model that calls send_email or a payments tool is no longer a shadow. Stub tools, or shadow only the first model turn.

Canary. Route a small weight of live traffic to a new InferencePool holding the candidate. Mechanics on Kubernetes:

• Gateway API Inference Extension: two InferencePools behind one HTTPRoute, shift backendRefs[].weight; the old pool stays up for rollback (GKE documents exactly this for base-model rollouts).
• KServe: canaryTrafficPercent on InferenceService (serverless mode), or LLMInferenceService group/weight; KServe recommends starting a new version at a small non-zero weight instead of weight 0 so the gateway has warmed the backend.
• Argo Rollouts: canary steps with the Gateway API traffic-router plugin and an AnalysisTemplate that aborts automatically on Prometheus queries.

Canary specifics for LLMs:

• Session stickiness. A multi-turn conversation that flips models mid-session behaves oddly and loses its prefix-cache hits. Hash on session or conversation ID, so a user stays on one version for the whole conversation.
• Statistical power. At 1% of 2,000 req/s you get 20 req/s, about 72k requests an hour. That's plenty for error rate and latency, but a 1-point change in a judge score sampled on 2% of those needs hours. Size bake time from the smallest effect you need to detect, not from habit.
• Compare canary to baseline over the same window, not to last week's numbers; traffic mix shifts by hour.
• One cell first. Canary in one cell (L24) with its own blast radius, then widen cell by cell with a bake per step. Tenants with strict contracts can be excluded from early waves by routing rule.
• Capacity for two versions. During rollout both pools hold GPUs. Reserve surge capacity or shrink the old pool as the new one grows; don't discover at 50% that the cell can't hold both (L28).

Canary metrics: request error rate and 5xx, TTFT/TPOT p95, output tokens p50/p95, refusal rate, stop-reason mix (length truncation spikes are a classic), tool-call parse failures, online judge score on sampled traffic, user feedback rate (thumbs down, regenerate), and business metrics where they exist.

Rollback is a routing change. Keep the old InferencePool warm for the bake period, so rollback is setting a weight to zero, measured in seconds. If the old pool scaled to zero, rollback becomes a cold start of a 140 GB model (L27) during an incident. Keep the old digest in each cell's cache until the new one has baked.

8. Ownership and separation of duties

• Training owns candidates. The eval platform owns the gate code and thresholds, and changing a threshold is a reviewed change in Git, not a flag in the release UI.
• An override ("ship despite the gate") is allowed, but it's a signed, logged exception with a named approver and an expiry, and admission sees it the same way it sees a passing attestation.
• The model registry stores state transitions (candidate, gated, canary, GA, deprecated, revoked) with the digests and attestation references. Revoking a digest has to reach every cell's admission policy.

Concrete commands & config

Kueue: a dedicated eval queue in the GPU cohort

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata: {name: eval-cq}
spec:
  cohort: gpu-fleet
  namespaceSelector: {matchLabels: {platform.example.com/role: eval}}
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
    flavors:
    - name: h100-serving-sku          # same SKU and flavor production serves on
      resources:
      - {name: nvidia.com/gpu, nominalQuota: 64, borrowingLimit: 192}
      - {name: cpu, nominalQuota: 1024}
      - {name: memory, nominalQuota: 8Ti}
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata: {name: candidate-evals, namespace: eval-llm}
spec: {clusterQueue: eval-cq}
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: {name: eval-release-candidate}
value: 50000
description: "Full suite for a pending release; preempts exploratory evals, not training gangs."

An eval Job: production serving image + harness client, everything pinned

apiVersion: batch/v1
kind: Job
metadata:
  name: eval-cand-2026-10-09-r1
  namespace: eval-llm
  labels:
    kueue.x-k8s.io/queue-name: candidate-evals
    kueue.x-k8s.io/priority-class: eval-release-candidate
spec:
  parallelism: 8
  completions: 8
  completionMode: Indexed          # shard the suite across 8 replicas
  template:
    spec:
      serviceAccountName: eval-runner        # RO on candidate + baseline prefixes, RW on results only
      restartPolicy: Never
      runtimeClassName: nvidia
      initContainers:
      - name: vllm
        restartPolicy: Always                  # native sidecar
        image: registry.corp/serving/vllm@sha256:3f1c...   # SAME digest prod serves
        args: ["--model=/weights", "--served-model-name=cand",
               "--tensor-parallel-size=4", "--quantization=fp8",
               "--max-model-len=32768", "--seed=1234"]
        resources: {limits: {nvidia.com/gpu: 4}}
        readinessProbe: {httpGet: {path: /health, port: 8000}, periodSeconds: 10}
        volumeMounts: [{name: weights, mountPath: /weights, readOnly: true}]
      containers:
      - name: harness
        image: registry.corp/eval/harness@sha256:9ab0...
        command: ["/bin/sh", "-c"]
        args:
        - |
          lm_eval --model local-completions \
            --model_args model=cand,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=64,max_retries=3,tokenizer=/weights \
            --tasks "$(cat /suite/shard-${JOB_COMPLETION_INDEX}.txt)" \
            --apply_chat_template --seed 1234 \
            --log_samples --output_path /results/${JOB_COMPLETION_INDEX}
        volumeMounts:
        - {name: weights, mountPath: /weights, readOnly: true}
        - {name: suite, mountPath: /suite, readOnly: true}
        - {name: results, mountPath: /results}
      volumes:
      - name: weights
        image:                                   # ImageVolume: the L27 OCI weights artifact, by digest
          reference: registry.corp/models/llm-70b@sha256:7d2e...
          pullPolicy: Always
      - name: suite
        configMap: {name: suite-v41}             # task list per shard, versioned
      - name: results
        ephemeral:
          volumeClaimTemplate:
            spec: {accessModes: [ReadWriteOnce], resources: {requests: {storage: 50Gi}}}

Per-item logs (--log_samples) matter more than the aggregate: the paired test below needs item-level results for both candidate and baseline.

The gate: paired deltas, margins and floors

# gate.py - item-level results for baseline and candidate on identical items
import json, numpy as np
 
rng = np.random.default_rng(0)
POLICY = json.load(open("gate-policy.json"))    # versioned in Git, owned by the eval platform
 
def paired_ci(base, cand, clusters, n_boot=10_000, alpha=0.05):
    """Cluster bootstrap of mean(cand - base) over clusters (e.g., shared passage IDs)."""
    d = cand - base
    ids = np.unique(clusters)
    by = {c: d[clusters == c] for c in ids}
    stats = []
    for _ in range(n_boot):
        pick = rng.choice(ids, size=len(ids), replace=True)
        stats.append(np.concatenate([by[c] for c in pick]).mean())
    lo, hi = np.quantile(stats, [alpha / 2, 1 - alpha / 2])
    return d.mean(), lo, hi
 
results, failed = [], False
for s in POLICY["slices"]:                       # e.g. code, math, multilingual, long_ctx, tool_use
    base, cand, cl = load_items(s["name"])        # arrays aligned by item id
    delta, lo, hi = paired_ci(base, cand, cl)
    if s["rule"] == "non_inferior":
        ok = lo > -s["margin"]                    # e.g. margin 0.01 = 1 point
    elif s["rule"] == "must_improve":
        ok = lo > s.get("target", 0.0)
    elif s["rule"] == "floor":                    # rates where lower is better (jailbreak, PII)
        rate = 1 - cand.mean()
        ub = rate + 1.96 * np.sqrt(rate * (1 - rate) / len(cand))
        ok = ub <= s["max_rate"]
    failed |= not ok
    results.append({"slice": s["name"], "rule": s["rule"], "delta": delta,
                    "ci": [lo, hi], "pass": bool(ok)})
 
json.dump({"candidate": CAND_DIGEST, "baseline": BASE_DIGEST, "policy": POLICY["version"],
           "manifest": EVAL_MANIFEST_DIGEST, "results": results, "pass": not failed},
          open("eval-attestation.json", "w"), indent=2)
raise SystemExit(1 if failed else 0)

Performance gate: same prompts, same rate, compare to baseline

# Production-shaped prompts (sampled, scrubbed), constant rate near the cell's planned per-replica load.
# GuideLLM CLI flags change between releases; check `guidellm run --help`.
for target in baseline candidate; do
  guidellm run \
    --backend kind=openai_http,target=http://$target.eval-llm.svc:8000 \
    --profile kind=constant,rate=12 \
    --constraint kind=max_duration,seconds=900 \
    --data kind=json_file,path=/suite/prod-shaped-prompts.jsonl \
    --output-path /results/perf-$target.json
done
# Gate: TTFT p95 and TPOT p95 within 5% of baseline at the same rate; output tokens p95 within 15%.

Sign the weights and attest the gate result

# Weights: OpenSSF model signing (sigstore/model-transparency), keyless with the promotion job's identity.
model_signing sign /models/llm-70b-cand --signature /models/llm-70b-cand/model.sig
 
# Gate result: an in-toto attestation bound to the weights artifact digest, signed by the EVAL CLUSTER's identity.
cosign attest --yes \
  --type https://corp.example/attestations/eval-gate/v1 \
  --predicate eval-attestation.json \
  registry.corp/models/llm-70b@sha256:7d2e...
 
# Anyone (admission, auditors) can verify who signed it:
cosign verify-attestation \
  --type https://corp.example/attestations/eval-gate/v1 \
  --certificate-identity-regexp '^https://kubernetes.io/namespaces/eval-llm/serviceaccounts/eval-gate$' \
  --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE \
  registry.corp/models/llm-70b@sha256:7d2e... | jq -r .payload | base64 -d | jq .predicate.pass

Canary: two InferencePools, one HTTPRoute

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: {name: chat-llm, namespace: serve-llm}
spec:
  parentRefs: [{name: inference-gw, kind: Gateway}]
  rules:
  - matches: [{headers: [{name: x-canary-optout, value: "true"}]}]   # strict-contract tenants
    backendRefs:
    - {group: inference.networking.k8s.io, kind: InferencePool, name: llm-70b-v41, weight: 1}
  - backendRefs:
    - {group: inference.networking.k8s.io, kind: InferencePool, name: llm-70b-v41, weight: 99}
    - {group: inference.networking.k8s.io, kind: InferencePool, name: llm-70b-v42, weight: 1}

# Ramp in Git (GitOps), one cell at a time: 1 -> 5 -> 25 -> 50 -> 100, bake between steps.
# Rollback during an incident: one patch, old pool still warm.
kubectl -n serve-llm patch httproute chat-llm --type=json -p='[
  {"op":"replace","path":"/spec/rules/1/backendRefs/0/weight","value":100},
  {"op":"replace","path":"/spec/rules/1/backendRefs/1/weight","value":0}]'
kubectl -n serve-llm get inferencepool,httproute

Or let Argo Rollouts drive the ramp and abort on analysis

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: {name: llm-70b, namespace: serve-llm}
spec:
  replicas: 16
  strategy:
    canary:
      canaryService: llm-70b-canary
      stableService: llm-70b-stable
      trafficRouting:
        plugins:
          argoproj-labs/gatewayAPI: {httpRoute: chat-llm, namespace: serve-llm}
      analysis:
        templates: [{templateName: llm-canary-health}]
        startingStep: 1
      steps:
      - setWeight: 1
      - pause: {duration: 30m}
      - setWeight: 5
      - pause: {duration: 2h}
      - setWeight: 25
      - pause: {duration: 6h}
      - setWeight: 50
      - pause: {}                      # human promotion to 100 after cross-cell bake
  # template.spec: serving pod pinned to the new weights digest (omitted)
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: {name: llm-canary-health, namespace: serve-llm}
spec:
  metrics:
  - name: error-rate-delta
    interval: 5m
    failureLimit: 2
    successCondition: result[0] <= 0.002
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          (sum(rate(llm_requests_total{version="canary",code=~"5.."}[10m])) / sum(rate(llm_requests_total{version="canary"}[10m])))
          - (sum(rate(llm_requests_total{version="stable",code=~"5.."}[10m])) / sum(rate(llm_requests_total{version="stable"}[10m])))
  - name: ttft-p95-ratio
    interval: 5m
    failureLimit: 2
    successCondition: result[0] <= 1.10
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          histogram_quantile(0.95, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket{version="canary"}[10m])))
          / histogram_quantile(0.95, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket{version="stable"}[10m])))
  - name: output-length-ratio
    interval: 10m
    failureLimit: 1
    successCondition: result[0] <= 1.15
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          (sum(rate(vllm:generation_tokens_total{version="canary"}[30m])) / sum(rate(vllm:request_success_total{version="canary"}[30m])))
          / (sum(rate(vllm:generation_tokens_total{version="stable"}[30m])) / sum(rate(vllm:request_success_total{version="stable"}[30m])))
  - name: online-judge-delta
    interval: 30m
    failureLimit: 1
    successCondition: result[0] >= -0.02
    provider:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: |
          avg_over_time(online_judge_score{version="canary"}[2h]) - avg_over_time(online_judge_score{version="stable"}[2h])

kubectl argo rollouts get rollout llm-70b -n serve-llm --watch
kubectl argo rollouts abort llm-70b -n serve-llm      # weight back to stable immediately
kubectl -n serve-llm get analysisrun -l rollouts-pod-template-hash --sort-by=.metadata.creationTimestamp

Metric names follow vLLM's Prometheus exporter; check the names your engine version exports, and add a version label at the pool or gateway level so canary and stable are separable.

Failures & fixes

Security & hardening

The gate is a security control, so it has to be tamper-evident and enforced. If anyone can ship weights without it, or fake its result, or quietly loosen it, the eval platform is theater. The eval cluster also runs the least-trusted artifacts in the company (unreviewed candidates, model-generated code, third-party judge calls) right next to the most sensitive data (private eval sets, mirrored user traffic). L11–L14, L27 and L30 built the primitives; here they are applied to promotion.

Model weights, checkpoints, images & deployment provenance (required)

Eval enclave, data and rollout controls

# Kyverno: serving pods in serve-* must reference weights by digest, signed by the release job,
# with a passing eval-gate attestation signed by the eval-gate identity.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: {name: require-eval-gated-weights}
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
  - name: weights-signed-and-gated
    match: {any: [{resources: {kinds: [Pod], namespaces: ["serve-*"]}}]}
    verifyImages:
    - imageReferences: ["registry.corp/models/*"]      # ImageVolume weights artifacts
      mutateDigest: false
      verifyDigest: true
      required: true
      attestors:
      - entries:
        - keyless:
            subject: "https://kubernetes.io/namespaces/release/serviceaccounts/model-release"
            issuer: "https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE"
      attestations:
      - type: https://corp.example/attestations/eval-gate/v1
        attestors:
        - entries:
          - keyless:
              subject: "https://kubernetes.io/namespaces/eval-llm/serviceaccounts/eval-gate"
              issuer: "https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE"
        conditions:
        - all:
          - key: "{{ pass }}"
            operator: Equals
            value: true

Check that your admission controller actually inspects image volumes (ImageVolume) and not just container images; if it doesn't, enforce on a models.corp/weights-digest annotation plus a validating policy that matches it against the pool's volume reference. Test the deny path in CI with an unsigned digest.

# Eval namespace: default-deny egress, allow registry, results endpoint, judge gateway, DNS.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: eval-egress, namespace: eval-llm}
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}, podSelector: {matchLabels: {k8s-app: kube-dns}}}]
    ports: [{protocol: UDP, port: 53}]
  - to: [{namespaceSelector: {matchLabels: {platform.example.com/role: judge}}}]
    ports: [{protocol: TCP, port: 8443}]
  - to: [{ipBlock: {cidr: 10.20.0.0/24}}]     # VPC endpoints: registry + results bucket
    ports: [{protocol: TCP, port: 443}]

Interview soundbites

• "New weights are a deploy. The eval suite is its test suite, and the gate result is a signed attestation that admission enforces, so skipping evals means the pods don't start."
• "Evaluate the artifact you'll ship: same serving image, quantization, TP layout, chat template and endpoint. A bf16 research score says nothing about an FP8 serving build."
• "On a thousand-question benchmark at 80%, the 95% interval is about plus or minus 2.5 points. I gate on paired, clustered intervals with non-inferiority margins, not on point estimates."
• "Two hundred checks at p equals 0.05 is ten false alarms per release. Fewer curated slices, margins set in advance, and hard floors only where there's no slack: safety and security."
• "Longer answers are a capacity regression. Output length and a fixed-rate TTFT/TPOT load test are part of the gate, and the autoscaler's per-replica numbers ship with the model."
• "Canary one cell by HTTPRoute weight between two InferencePools, sticky by conversation, analysis on errors, latency, output length and an online judge score, and keep the old pool warm so rollback is a weight flip, not a cold start."
• "The judge is a pinned dependency. Change it and you re-baseline; track its agreement with humans like any other SLI."
• "Eval sets are crown jewels: training identities are denied by bucket policy, and the eval cluster has no internet, because it runs the least-trusted artifacts next to the most sensitive data."
• "Next: Lesson 32 makes offline and online evals agree, and runs LLM-as-judge at fleet scale."

How this maps to the JD

JDs that say "own model release and deployment safety," "build evaluation infrastructure," or "ship new models to a global inference fleet reliably and securely" are asking whether you can turn research checkpoints into production releases with a real gate, and roll them out without a quality or capacity incident. This lesson connects L20/L25 (eval queues in the GPU cohort), L21/L28 (performance and capacity gates), L24 (cells and weighted routing), L26 (eval spend), L27 (signed weights and preheat), L29/L30 (checkpoints and data lineage), with the security floor from L11–L14. Fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab).

Watch (talks & demos)

Real talks. Watch them after Failures & fixes.

Open Source Canary Deployments with Application Metric Analysis

Hannah Troisi, CNCF lightning talk. How Argo Rollouts canary steps, AnalysisTemplates and automatic abort fit together, with a live demo of error-rate and latency regressions tripping the analysis. Operator takeaway: the mechanics behind section 7's AnalysisTemplate; swap HTTP success rate for TTFT, output length and judge-score deltas and it's an LLM weights canary.

Open on YouTube (opens in a new tab)

Exploring ML Model Serving with KServe (with fun drawings)

Alexa Griffith (Bloomberg), KubeCon. KServe's control and data plane, storage initializer, model serving runtimes, and a live canaryTrafficPercent demo: 10% to the new model revision, promote, or pin back. Operator takeaway: model versions as revisions with traffic splits, the predecessor of today's InferencePool and LLMInferenceService weight-based rollouts.

Open on YouTube (opens in a new tab)

Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work

Dat Ngo (Arize), AI Engineer World's Fair. Code checks versus classifiers versus LLM judges, calibrating a judge against a human golden set, conditional evals that skip downstream checks when control flow already failed, and in-path guardrails versus out-of-path evals. Operator takeaway: section 5, and why the judge is something you tune and version, not a black box. It's a vendor talk; take the concepts, not the product.

Open on YouTube (opens in a new tab)

References

• Eval harnesses & benchmarks: EleutherAI lm-evaluation-harness (opens in a new tab) · Stanford HELM (opens in a new tab) · UK AISI Inspect (opens in a new tab) · GuideLLM (opens in a new tab)
• Statistics & judges: Adding Error Bars to Evals (Miller, arXiv 2411.00640) (opens in a new tab) · Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al.) (opens in a new tab) · Defeating Nondeterminism in LLM Inference (Thinking Machines) (opens in a new tab)
• Rollout mechanics: Gateway API Inference Extension (opens in a new tab) · GKE Inference Gateway rollout operations (opens in a new tab) · KServe LLMInferenceService canary rollout (opens in a new tab) · Argo Rollouts analysis (opens in a new tab) · Argo Rollouts Gateway API plugin (opens in a new tab) · Kueue (opens in a new tab)
• Security & provenance: OpenSSF model signing (sigstore/model-transparency) (opens in a new tab) · cosign attestations (opens in a new tab) · SLSA provenance (opens in a new tab) · Kyverno policies (opens in a new tab)
• wafer-ai/gpu-perf-engineering-resources (opens in a new tab) — AI/ML platform fuel
• Prior: 30 — Training data loading / storage · 29 — Training reliability / checkpoint / elastic · 28 — Inference autoscaling · 27 — GPU supply chain / hydrate · 26 — GPU cost / reclaim / spot · 25 — Multi-tenant GPU isolation · 24 — Multi-cluster routing / capacity · 21 — Inference serving SLOs · 20 — Kueue / gang scheduling · 14 — ML deploy hardening · 13 — ML weights · 12 — Supply chain · 11 — PSS / admission / RBAC · 9 — NetworkPolicy / multi-NIC
• Next: Lesson 32 — AI/ML platform: offline/online eval consistency & LLM-as-judge at fleet scale

Quick self-check

Out loud, in under two minutes each:

1. Your candidate scores 78.9% versus the baseline's 80.1% on a 1,000-item benchmark. Is that a regression? What would you compute, and what changes if items come in groups of five sharing a passage?
2. Design the eval capacity plan for a team shipping a 70B model every two weeks with checkpoints every 2,000 steps: which suites run when, on which queue and SKU, and roughly how many GPU-hours per release?
3. List everything you'd pin in an eval manifest so a score from today can be reproduced in six months. Which of those changed the last time scores moved for "no reason"?
4. Walk through a canary of new weights in one cell: routing objects, stickiness, the metrics in the analysis, how long each step bakes and why, and the exact rollback command. What has to be true for rollback to take seconds?
5. Design the identities, buckets, attestations and admission rules so that (a) no pod can serve weights that didn't pass the gate, (b) a compromised eval job can't alter the candidate or forge a pass, and (c) a private eval set can never be read by a training job. Where does the override path live and who can use it?