Lesson 21 — AI/ML platform: inference serving SLOs — TTFT/TPOT/goodput, continuous batching, KV cache as capacity

2026-09-29

What interviewers are really probing

Lesson 18 gave you SKU pools. Lesson 19 made NVLink / RDMA schedulable. Lesson 20 admitted training gangs without half-placing them — and told you not to share those gangs with latency inference. This lesson is the inference half of that contract: can you operate serving capacity as KV-cache / HBM, not as “GPU %” or raw tok/s — and defend TTFT/TPOT SLOs under load?

Interviewers at hyperscale AI platforms (fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab)) want evidence you have run LLM serving under contention, not only “we deploy vLLM”:

• TTFT / TPOT / E2E / goodput — first-token vs inter-token vs end-to-end; goodput = tokens (or requests) delivered under SLO, not vanity throughput
• Continuous batching — iteration-level admit/evict vs static batching that leaves GPU slots idle
• KV cache as the real capacity constraint — GPU HBM; PagedAttention; prefix cache hit rate; free blocks = seats
• Admission control on KV / concurrency, not just QPS at the edge (L7 still rate-limits abuse)
• Autoscaling on queue depth / KV util / TTFT burn — naive CPU or even “GPU util” lies during decode-bound serving
• Multi-LoRA / model multiplexing — capacity tradeoffs when many adapters share one engine
• Isolation from training gangs (L20) — inference SLO pools, flavors, NetPol
• Observability — histograms for TTFT/TPOT, goodput, preemptions, KV fragmentation / cache hit rate
• Security floor on the serving path — signed serving images; read-only weight mounts / WI; no weight write from inference SA; private registry + provenance; bucket IAM for pulls; secrets; NetPol; admission/RBAC (L11–L14)

Interview sentence: “I size and admit inference on free KV blocks in HBM, schedule with continuous batching, SLO on TTFT/TPOT histograms and goodput — not tok/s vanity — autoscale on queue depth and KV util, keep train gangs off the inference pool, and the serve SA can only pull signed weights read-only.”

Core diagram: Client/Gateway → continuous batcher → KV/HBM → GPU nodes (+ metrics, security floor)

Read it left-to-right: Clients hit an edge gateway (auth, abuse rate limits from L7). An admission gate decides whether a new request may consume KV seats — not merely whether QPS < N. Admitted work lands on inference pods running a continuous batcher (vLLM / TGI / TensorRT-LLM style): between every decode step the scheduler can evict finished sequences and admit waiting ones. The scarce resource is KV cache in HBM (PagedAttention blocks, optional prefix cache). Placement stays on inference SKU pools isolated from train gangs (L20). Metrics (TTFT/TPOT histograms, goodput, KV util, queue depth) drive autoscaling and burn alerts. Everything sits on a security floor: signed serving images, read-only weight path via WI/IRSA, no write IAM on the inference SA, NetPol, secrets, admission.

Idiosyncrasies / operator depth

1. TTFT, TPOT/ITL, E2E, goodput vs throughput

Interview sentence: “Throughput without an SLO is a vanity metric; goodput is tokens that still met TTFT and TPOT.”

2. Continuous batching / iteration-level scheduling vs static batching

Interview sentence: “Continuous batching keeps the GPU full between tokens; chunked prefill stops one long prompt from freezing every decode stream.”

3. KV cache as the real capacity constraint

Interview sentence: “I don’t ask ‘how many GPUs?’ for serving — I ask ‘how many free KV blocks at p99 context?’”

4. Admission control & concurrency limits tied to KV

Interview sentence: “QPS caps abuse; KV admission protects the SLO.”

5. Autoscaling: queue depth / KV util / TTFT burn — not naive CPU

Wire KEDA / custom-metrics HPA (or fleet autoscaler) to those signals; scale the inference pool, not the train cohort (L20).

Interview sentence: “I autoscale inference on queue depth and KV util — CPU HPA is how you scale the wrong thing.”

6. Multi-LoRA / model multiplexing (high level)

Interview sentence: “Multi-LoRA densifies GPUs; it doesn’t repeal KV physics or noisy-neighbor SLOs.”

7. Isolation from training gangs — inference SLO pools (L20)

Interview sentence: “Train buys atomic GPU sets; inference buys tail latency — separate pools, separate NetPol, separate SAs.”

8. Observability that interviewers expect

Interview sentence: “If you only graph tok/s, you will green-dashboard your way into a TTFT incident.”

Concrete commands & config

Inspect inference Deployments / pods / GPU placement

# Inference pool nodes
kubectl get nodes -l pool=inference-serving -o wide
kubectl get pods -n model-inference -o wide \
  -l app=vllm-serve
 
# Resource + RuntimeClass sanity
kubectl get deploy -n model-inference vllm-serve -o yaml | sed -n '/resources:/,/runtimeClassName/p'
kubectl describe pod -n model-inference -l app=vllm-serve | sed -n '/Limits:/,/Events:/p'
 
# Waiting / Pending under pressure
kubectl get pods -n model-inference --field-selector=status.phase=Pending
kubectl top pods -n model-inference

vLLM-style engine flags (capacity & batching)

# Example container args — tune from load tests, not blog defaults
python -m vllm.entrypoints.openai.api_server \
  --model /models/llama-3-70b-instruct \
  --served-model-name llama-3-70b \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 128 \
  --max-num-batched-tokens 4096 \
  --enable-prefix-caching \
  --disable-log-requests \
  --kv-cache-dtype auto
# Related knobs operators discuss in interviews:
#   --enable-chunked-prefill  (when available for your version)
#   --block-size 16
#   --limit-mm-per-prompt ...

TGI / TensorRT-LLM style knobs (interview vocabulary)

# Hugging Face TGI (shape — flag names drift by version)
text-generation-launcher \
  --model-id /models/llama-3-70b-instruct \
  --num-shard 4 \
  --max-concurrent-requests 128 \
  --max-input-length 4096 \
  --max-total-tokens 8192 \
  --waiting-served-ratio 1.2 \
  --cuda-memory-fraction 0.90
 
# TensorRT-LLM / triton-style talking points: max_batch_size,
# max_num_tokens, kv_cache_free_gpu_mem_fraction, enable_chunked_context

Deployment sketch: inference pool + RO weights + WI

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-serve
  namespace: model-inference
  labels:
    app: vllm-serve
    trust-domain: inference-prod
spec:
  replicas: 4
  selector:
    matchLabels:
      app: vllm-serve
  template:
    metadata:
      labels:
        app: vllm-serve
        trust-domain: inference-prod
      annotations:
        # Optional: custom metrics scrape for TTFT/TPOT exporters
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
    spec:
      serviceAccountName: inference-readonly
      runtimeClassName: nvidia
      nodeSelector:
        sku: h100
        pool: inference-serving
      tolerations:
        - key: pool
          value: inference-serving
          effect: NoSchedule
      containers:
        - name: vllm
          image: registry.example.com/ml/vllm-openai@sha256:REPLACE
          args:
            - "--model=/models/llama-3-70b-instruct"
            - "--tensor-parallel-size=4"
            - "--gpu-memory-utilization=0.90"
            - "--max-num-seqs=128"
            - "--max-num-batched-tokens=4096"
            - "--enable-prefix-caching"
          ports:
            - containerPort: 8000
              name: http
          resources:
            limits:
              nvidia.com/gpu: "4"
              memory: 256Gi
            requests:
              nvidia.com/gpu: "4"
              cpu: "16"
              memory: 256Gi
          volumeMounts:
            - name: weights
              mountPath: /models
              readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          livenessProbe:
            httpGet:
              path: /health
              port: http
            initialDelaySeconds: 120
            periodSeconds: 20
          readinessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
      volumes:
        - name: weights
          # CSI / PVC populated by a *separate* pull Job with write IAM —
          # inference SA must not have PutObject ([L13])
          persistentVolumeClaim:
            claimName: llama-3-70b-weights-ro

HPA / KEDA sketch on queue depth or custom TTFT metric

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-serve
  namespace: model-inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-serve
  minReplicas: 4
  maxReplicas: 32
  metrics:
    - type: Pods
      pods:
        metric:
          name: inference_waiting_requests
        target:
          type: AverageValue
          averageValue: "8"
    - type: Pods
      pods:
        metric:
          name: kv_cache_usage_ratio
        target:
          type: AverageValue
          averageValue: "750m"   # ~0.75
  # Prefer KEDA ScaledObject with Prometheus trigger on
  # histogram_quantile(0.99, ttft_seconds) burn when available

NetworkPolicy sketch: edge → serve → registry / weight endpoint

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: inference-default-deny
  namespace: model-inference
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: inference-serve-allow
  namespace: model-inference
spec:
  podSelector:
    matchLabels:
      app: vllm-serve
  policyTypes: ["Ingress", "Egress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              role: edge-gateway
      ports:
        - protocol: TCP
          port: 8000
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              role: dns
    - to:
        # Private registry + weight VPC endpoint CIDRs — pin in real clusters
        - ipBlock:
            cidr: 10.10.0.0/16
      ports:
        - protocol: TCP
          port: 443

Admission floor sketch: digest + RO SA + inference pool

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: inference-serve-floor
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["deployments"]
  validations:
    - expression: >
        !has(object.metadata.labels) ||
        object.metadata.labels["trust-domain"] != "inference-prod" ||
        (has(object.spec.template.spec.serviceAccountName) &&
         object.spec.template.spec.serviceAccountName == "inference-readonly" &&
         has(object.spec.template.spec.runtimeClassName) &&
         object.spec.template.spec.runtimeClassName == "nvidia")
      message: "inference-prod requires inference-readonly SA + nvidia RuntimeClass"

Failures & fixes

Security & hardening

Serving concentrates customer traffic and model IP on the same path. A compromised inference pod with write IAM to the weight bucket is a supply-chain and exfil story. Unsigned serving images turn every scale-out into a fleet-wide trust failure. Edge without rate limits turns abuse into SLO burn for every tenant.

AI/ML weight & registry hardening on the serving path (required)

# Sketch: serving-path weight & image floor (policy-as-code shape)
apiVersion: policy.platform/v1
kind: MlServeGate
metadata:
  name: inference-weight-floor
spec:
  match:
    labels:
      trust-domain: inference-prod
  require:
    imageDigestPin: true
    cosignVerify: true
    runtimeClass: nvidia
    serviceAccount: inference-readonly
    workloadIdentity: true
    weightMountReadOnly: true
    weightBucketPublicAccess: false
    cmek: true
    inferenceSaPutObject: false
    trainSaOnServePods: false
    networkPolicy: deny-by-default
    edgeRateLimit: true
    apiKeyInImage: false

Punchline: Continuous batching and KV admission are capacity features. Signed serving images, read-only WI to CMEK weights, split SAs, NetPol, edge rate limits, and digest-pinned admission are what stop an inference scale-out from becoming a multi-tenant weight breach.

Interview soundbites

• “Throughput is vanity; goodput is tokens under TTFT/TPOT.”
• “KV free blocks are the seats — QPS is just the bouncer at the door.”
• “Continuous batching fills slots every decode step; static batching hosts a hostage situation.”
• “Chunked prefill stops one fat prompt from freezing every stream.”
• “Autoscale on queue depth and KV util — CPU HPA lies during decode.”
• “Prefix cache hit rate is a capacity lever, not a checkbox.”
• “Multi-LoRA densifies GPUs; it doesn’t repeal noisy-neighbor SLOs.”
• “Train gangs and inference SLOs don’t share a pool without isolation.”
• “Inference SA gets GetObject; never PutObject.”
• “Sign the vLLM image like you sign the weights.”
• “Edge rate limits abuse; KV admission protects the SLO.”
• “Next: prefill/decode disaggregation and MoE serving topology.”

How this maps to the JD

Hyperscale + AI/ML platform JDs that mention inference serving, GPU fleets, and latency SLOs are asking whether you can operate capacity under contention: TTFT/TPOT/goodput, continuous batching, KV-cache admission, sensible autoscaling, isolation from train gangs, and a security floor on the weight/image path. This lesson extends L20 (don’t share train gangs with latency inference) with L18/L19 (pools + topology), L7 (edge rate limits), and L11–L14 (admission, supply chain, weights, deploy isolation). Fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab). Next up — Lesson 22: prefill/decode disaggregation & MoE serving topology on the fleet.

Watch (talks & demos)

Real talks — watch after Failures & fixes.

Continuous Batching — How LLM Servers Keep the GPU Full

Iteration-level scheduling vs static batching: why short requests stop being hostages, and why “keep the batch full every decode step” is the operator mental model behind vLLM/TGI-class engines.

Open on YouTube (opens in a new tab)

Optimize LLM inference with vLLM (Michael Goin)

PagedAttention, continuous batching, and prefix caching as the three levers that show up in every serving interview — framed for production deployers, not CUDA kernel authors.

Open on YouTube (opens in a new tab)

Why Your vLLM p99 Latency Falls Apart in Production (and How to Fix It)

Chunked prefill, max_num_batched_tokens, and why PagedAttention alone does not fix tail latency — the TTFT/p99 incident story interviewers love.

Open on YouTube (opens in a new tab)

References

• vLLM (opens in a new tab) · PagedAttention blog (opens in a new tab) · Kwon et al., SOSP ’23 (arXiv:2309.06180) (opens in a new tab)
• Hugging Face TGI (opens in a new tab) · NVIDIA TensorRT-LLM (opens in a new tab)
• KEDA (opens in a new tab) · Kubernetes HPA custom metrics (opens in a new tab)
• wafer-ai/gpu-perf-engineering-resources (opens in a new tab) — GPU fleets / serving / SLO fuel
• Prior: 20 — Training/inference scheduling · 19 — GPU topology · 18 — GPU node pools · 14 — ML deploy hardening · 13 — ML weights · 12 — Node/container hardening · 11 — Cluster security · 09 — NetworkPolicy · 07 — Edge LB / DDoS
• Next: Lesson 22 — prefill/decode disaggregation & MoE serving topology on the fleet

Quick self-check

Out loud, in under two minutes:

1. Define TTFT, TPOT/ITL, E2E, and goodput — and why tok/s alone misleads.
2. How does continuous (iteration-level) batching differ from static batching?
3. Why is KV cache in HBM the real serving capacity constraint — and what does PagedAttention buy you?
4. Which signals should drive inference autoscaling, and why not CPU?
5. How do you isolate inference SLO pools from training gangs (L20)?
6. List four security controls on the serving path for model weights/images — including why the inference SA must not write checkpoints.