Lesson 17 — Systems design tradeoffs for rapidly evolving platforms

2026-09-25

What interviewers are really probing

Lesson 15 gated who may change the substrate. Lesson 16 owned what runs after the change. This lesson is the synthesis: under change velocity, can you pick a side, defend it, and name the footgun without collapsing into CAP textbook fluff?

Interviewers at hyperscale (100+ clusters / 10K+ nodes) are not asking “CAP theorem, go.” They want evidence you have operated under partial failure and product pressure:

• Blast radius vs leverage — cells vs shared control planes (L1 / L3)
• Consistency vs availability of the control plane — etcd quorum, regional failover, when stale reads are acceptable
• Homogeneous golden image vs heterogeneous SKUs — fleet simplicity vs GPU/ML hardware reality (L4, preview of the AI/ML track)
• Declarative desired-state vs imperative durable workflows — Atlantis/GitOps vs Temporal sagas (L15 / L16)
• Central platform vs per-cell forks — pave-the-road vs snowflake velocity
• Progressive delivery vs big-bang — canaries, flags, wave deploys vs freeze windows
• Strong contracts vs flexible extension — CRD/admission versioning as a product
• Multi-tenancy isolation levels — soft namespaces vs hard cells vs dedicated clusters; GPU pool sharing vs SKU isolation
• Build vs buy / managed vs DIY — control vs ops load at fleet scale
• Cost vs latency/reliability — overprovision, reserved capacity, inference KV-cache as a design choice (light preview)

Interview sentence: “I pick tradeoffs by blast radius and change velocity: cells when failure must not correlate; shared control when leverage beats the isolation tax; Atlantis for desired-state substrate, Temporal for durable sagas; progressive waves with metric gates — and I never trade away signing, weight IAM, admission, or GPU pool isolation for speed.”

Core diagram: velocity vs safety decision map

Read it top-down: Every platform decision radiates from a decision hub that balances blast radius × change velocity. The six major axes (cell vs shared, declarative vs imperative, progressive vs big-bang, soft vs hard tenancy, golden image vs SKUs, central vs forks) are not right/wrong — each side wins in a context. Under them sit consistency/availability of control planes, API contracts, build-vs-buy, and cost/SLO choices. The security floor is the non-negotiable strip: signing/provenance, admission/RBAC, weight/bucket IAM + CMEK, NetworkPolicy/GPU pool isolation, short-lived secrets. Velocity that punches through that floor is not “moving fast” — it is an incident waiting for a change ticket.

Idiosyncrasies / operator depth

1. Cell isolation vs shared control

Borg-like cells (L4) exist because correlated failure is the enemy of SLOs. Shared control exists because humans do not scale linearly with clusters. The interview move is naming which plane is shared (policy, observability, identity) vs which must stay cell-local (etcd, CNI, device plugins).

Interview sentence: “I share policy and identity; I isolate data planes and control-plane quorum — shared leverage above the floor, cell blast walls below it.”

2. Consistency vs availability of control planes

At scale: kube-apiserver HA is not the same as etcd HA. Regional failover of a management plane is an availability choice; accepting stale reads for a fleet scheduler is a different axis — document which verbs are quorum-required (create/update/delete) vs which can be eventually consistent (list/watch with RV lag alerts).

Interview sentence: “Writes stay quorum-consistent; destructive automation never acts on a known-stale cache without an explicit freshness gate.”

3. Homogeneous golden image vs heterogeneous SKUs

Operator truth: golden image for the OS/security baseline; SKU labels + taints + RuntimeClass/device plugins for hardware. Do not fork the entire node image per SKU unless driver/ABI forces it — fork the pool contract.

Interview sentence: “One hardened golden baseline; many labeled SKU pools — heterogeneity lives in topology and scheduling, not in snowflake AMIs.”

4. Declarative desired-state vs imperative durable workflows

This is the L15/L16 compose axis:

Compose rule: Atlantis applies the desired substrate; Temporal owns the durable saga (drain waves, canary promote, rollback); Argo runs K8s-native DAG pods when the step is a pod graph (L16).

Interview sentence: “Desired-state for what should exist; durable workflows for what must happen over time — never one plane pretending to be the other.”

5. Centralized platform service vs per-cell forks

Healthy platforms ship versioned escape hatches (annotated exceptions, time-boxed PolicyExceptions, scorecard waivers) — not infinite forks and not infinite “no.”

Interview sentence: “Default is the paved road; forks are time-boxed exceptions with an owner and an expiry — scorecards catch permanent snowflakes.”

6. Progressive delivery vs big-bang

At fleet scale prefer wave deploys across cells (1% → 10% → 50% → rest) with automatic halt on burn. Big-bang is a last resort with a rehearsed rollback and an incident commander.

Interview sentence: “Waves by default; big-bang only when dual-running is impossible — and then the rollback drill is the real deliverable.”

7. Strong contracts (API/schema) vs flexible extension

Treat API/schema like a public cloud product: additive changes preferred, conversion webhooks tested, admission version pins, deprecation windows published.

Interview sentence: “Platform APIs are products — additive by default, versioned when breaking, admission pins so velocity cannot skip the contract.”

8. Multi-tenancy isolation levels (incl. GPU preview)

GPU/ML preview (not the GPU lesson): shared GPU pools win for interactive/dev density; SKU-isolated pools win when NVLink topology, MIG partitions, or fair-share queues matter for training/inference SLOs (TTFT/TPOT, KV-cache capacity). Soft tenancy until contention or security boundary forces hard isolation (L14).

Interview sentence: “Namespaces until noisy-neighbor or SKU topology forces pool or cell isolation — GPU sharing is a tenancy decision, not just a scheduler flag.”

9. Build vs buy / managed vs DIY

Rule: buy the commodity, build the differentiation, but only with an exit + provenance story (SBOM, signed images, data egress, IaC that could recreate).

Interview sentence: “Managed for commodity control planes; DIY where topology and serving fabric differentiate — every buy needs an exit and a provenance path.”

10. Cost vs latency / reliability (light inference preview)

Inference goodput (TTFT/TPOT) is often a capacity design — KV-cache memory, replica count, and prefill/decode topology — not just “add HPA.” Flag it in interviews; deep dive lands in the upcoming AI/ML track.

Interview sentence: “Cost wins where work is deferrable and checkpointed; reserved capacity wins where SLO burn is the outage — KV-cache is capacity, not a free cache.”

Concrete commands & config

Architecture Decision Record (ADR) skeleton

# ADR-017: Cell-local etcd, shared policy plane
Status: Accepted
Context: 120 clusters; one shared OPA/Gatekeeper misconfig took 40% offline in staging rehearsal.
Decision: etcd + apiserver + CNI per cell; policy bundles + identity issuer shared with staged rollout.
Consequences: Upgrade waves via Temporal; policy changes progressive with scorecard gate.
Security: No standing cluster-admin on shared plane; short-lived tokens; signed policy bundles.

Platform scorecard (illustrative)

# platform-scorecard.yaml — CI / policy-as-code input
apiVersion: scorecard.platform/v1alpha1
kind: CellScore
metadata:
  name: cell-us-east-1-a
spec:
  required:
    - id: signed-images-only
      gate: admission.cosign.verify == true
    - id: pss-restricted
      gate: namespace.enforce == "restricted"
    - id: weight-bucket-private
      gate: ml.artifacts.publicAccess == false
    - id: gpu-pool-labeled
      gate: nodePools.gpu.taintsContain == "sku"
  progressiveDelivery:
    maxWavePercent: 10
    haltOn:
      - slo:burn_rate_1h > 2
      - error_ratio > 0.5%

Canary metric gate (Flagger-style sketch)

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: inference-frontend
  namespace: ml-serving
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: inference-frontend
  progressDeadlineSeconds: 600
  service:
    port: 8080
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
        interval: 1m
      - name: latency-p99
        thresholdRange:
          max: 500
        interval: 1m
    webhooks:
      - name: cosign-verify
        type: pre-rollout
        url: http://admission-hooks.platform/verify

Admission version pin + signed images

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-signed-digest
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["deployments"]
  validations:
    - expression: "object.spec.template.spec.containers.all(c, c.image.contains('@sha256:'))"
      message: "images must be digest-pinned"
---
# Pair with cluster image policy / cosign verify (L12) — progressive delivery must not skip this

Terraform module interface (strong contract)

# modules/cell/variables.tf — additive, versioned module
variable "cell_id" {
  type = string
}
variable "sku_pools" {
  type = map(object({
    machine_type = string
    min_count    = number
    max_count    = number
    taints       = list(object({ key = string, value = string, effect = string }))
    gpu          = optional(bool, false)
  }))
  description = "Heterogeneous pools under one golden baseline image"
}
variable "isolation_level" {
  type    = string
  default = "namespace" # namespace | node_pool | dedicated_cluster
  validation {
    condition     = contains(["namespace", "node_pool", "dedicated_cluster"], var.isolation_level)
    error_message = "isolation_level must be a known enum"
  }
}

Temporal wave deploy (imperative saga over declarative substrate)

# Illustrative — durable wave across cells after Atlantis apply
@workflow.defn
class CellAddonWave:
    @workflow.run
    async def run(self, change_ticket: str, cells: list[str]) -> str:
        for wave in chunk(cells, size=max(1, len(cells) // 10)):
            await workflow.execute_activity(
                cordon_and_drain, wave, start_to_close_timeout=timedelta(hours=2)
            )
            ok = await workflow.execute_activity(
                check_slo_gates, wave, start_to_close_timeout=timedelta(minutes=15)
            )
            if not ok:
                await workflow.execute_activity(rollback_wave, wave)
                return f"halted:{change_ticket}"
            await workflow.wait_condition(lambda: signal_received("wave_continue") or auto_advance())
        return f"done:{change_ticket}"

kubectl / policy quick checks

# Freshness / quorum mindset
kubectl get --raw /healthz/etcd | jq .
kubectl get ep -n kube-system kube-dns -o yaml  # never drain from a stale local cache alone
 
# Scorecard-ish inventory
kubectl get nodes -L sku,accelerator -l node.kubernetes.io/instance-type
kubectl get validatingadmissionpolicies.admissionregistration.k8s.io
kubectl get ns -o json | jq '.items[] | {n:.metadata.name, enforce:.metadata.labels["pod-security.kubernetes.io/enforce"]}'
 
# Progressive delivery objects
kubectl get canaries.flagger.app -A
kubectl get rollouts -A 2>/dev/null
 
# Multi-tenancy / GPU pool boundaries
kubectl get netpol -A
kubectl get resourcequota -A
kubectl describe node -l sku=gpu-h100 | head

Failures & fixes

Security & hardening

Every major tradeoff changes security posture. Velocity that skips the floor is how you get a signed-off outage with an unsigned image.

AI/ML angle (required)

Evolving platforms that host training/inference must not trade away:

# Sketch: even canary Serving must pass the floor
apiVersion: policy.platform/v1
kind: ServingGate
metadata:
  name: ml-canary-floor
spec:
  match:
    namespaces: ["ml-serving"]
    progressive: true
  require:
    imageDigestPin: true
    cosignVerify: true
    weightBucketPublicAccess: false
    cmek: true
    podSecurity: restricted
    gpuPoolLabel: "sku"
    networkPolicy: deny-by-default

Punchline: Shared platforms amplify mistakes; progressive delivery still ships code; soft multi-tenancy still shares a kernel and often a GPU. Signing, IAM, admission, NetPol, and short-lived secrets are not “phase 2.”

Interview soundbites

• “Blast radius first, leverage second — share policy, isolate quorum.”
• “Desired-state for substrate; durable workflows for time — compose L15 and L16.”
• “Waves by default; big-bang only when dual-run is impossible.”
• “Golden baseline, heterogeneous SKU pools — do not AMI-fork the universe.”
• “Platform APIs are products: additive, versioned, admission-pinned.”
• “Namespaces until noisy-neighbor or SKU topology forces hard pools.”
• “Buy commodity, build differentiation — every buy needs exit + provenance.”
• “Canaries without metric gates are change theater.”
• “Stale reads are fine for dashboards; never for mass cordon.”
• “KV-cache and reserved capacity are SLO design, not finance afterthoughts.”
• “Security floor: signing, weight IAM, admission, NetPol — velocity does not punch through.”
• “Forks expire; scorecards catch snowflakes; escape hatches still pass admission.”

How this maps to the JD

Hyperscale JDs that mention 100+ clusters / 10K+ nodes are probing whether you can design under change: cells, control-plane HA, IaC, orchestration, and security as one system — not as siloed buzzwords. This lesson stitches L1–L4 (fleet/cells/Borg instincts), L11–L14 (security floor including ML), and L15–L16 (change + saga planes) into defendable tradeoffs. The next curriculum arc goes deep on AI/ML platform engineering (GPU pools, topology/NCCL, gang/queues, inference SLOs, disaggregation, DCGM) fueled by wafer-ai/gpu-perf-engineering-resources (opens in a new tab) — this lesson only previewed the tenancy/SKU/SLO axes so those lessons land on prepared ground.

Watch (talks & demos)

Real talks that map to this lesson — watch after the Failures & fixes table.

Cluster Management at Google with Borg (John Wilkes, GOTO 2016)

Cells, packing, isolation, and why large-scale cluster management is a tradeoff machine — grounding for cell isolation vs shared leverage and Borg instincts from L4.

Open on YouTube (opens in a new tab)

Progressive Delivery with Flagger (Stefan Prodan, Weaveworks / CNCF)

Canaries, metrics gates, and GitOps-friendly progressive delivery — the operator depth behind progressive vs big-bang and why canaries without SLOs are theater.

Open on YouTube (opens in a new tab)

Control Plane, Service, or Both? Argo CD Multi-Cluster Architectures (Nicholas Morey, Akuity / CNCF)

Centralized management plane vs per-cluster instances — security, DX, and scale tradeoffs that mirror central platform vs per-cell forks and shared control blast radius.

Open on YouTube (opens in a new tab)

References

• Borg paper (Google) (opens in a new tab) · Site Reliability Engineering book (opens in a new tab)
• Flagger docs (opens in a new tab) · Argo Rollouts (opens in a new tab) · Progressive delivery overview (CNCF) (opens in a new tab)
• Kubernetes ValidatingAdmissionPolicy (opens in a new tab) · Pod Security Standards (opens in a new tab)
• SIG Multicluster (opens in a new tab) · Argo CD architectures (opens in a new tab)
• wafer-ai/gpu-perf-engineering-resources (opens in a new tab) — GPU fleets / topology / serving SLO fuel for upcoming lessons
• Prior: 16 — Temporal / Argo · 15 — Terraform / Atlantis · 14 — ML inference/training deploy hardening · 13 — ML weights / checkpoint hardening · 12 — Node/container hardening · 11 — Cluster security · 04 — Borg/Mesos · 03 — Cluster provisioning · 01 — Hyperscale mental model

Quick self-check

Out loud, in under two minutes:

1. When do you choose cell isolation over shared control — and what do you still share?
2. How do declarative Atlantis/GitOps and Temporal sagas compose without fighting?
3. Name two metric gates you require before a canary advances a wave.
4. How does soft vs hard multi-tenancy change GPU pool and NetworkPolicy design?
5. What never leaves the security floor when the business asks for more velocity?
6. Give one build-vs-buy decision at 100+ clusters and its exit/provenance story.