Lesson 11 — Cluster security: PSS, admission, RBAC, IAM least privilege & audit

2026-09-19

What interviewers are really probing

Lessons 08–09–10 secured the datapath, the packet firewall, and workload identity. This lesson is the control-plane / identity / admission layer those depend on: who can create a privileged pod, who can bind cluster-admin, which cloud principal a ServiceAccount becomes, and whether your audit trail will catch the escalation.

They are not asking you to recite every PSS field. They want:

• Pod Security Standards (Privileged / Baseline / Restricted) as namespace labels with enforce / audit / warn — and how that replaced deprecated PodSecurityPolicy
• Admission as a chain (mutating → validating) plus webhook availability as fleet risk (failurePolicy: Fail vs Ignore)
• RBAC least privilege: dangerous verbs (escalate, bind, impersonate, *), aggregation, kubectl auth can-i, BoundServiceAccountTokenVolume
• Cloud IAM mapped to K8s SAs (IRSA / Workload Identity) — why node instance roles are too broad
• Audit that detects secret reads and privilege escalation, not just “logging is on”

Interview sentence: “NetworkPolicy + mesh AuthZ + PSS/admission/RBAC is defense in depth. Any one alone is theater.”

Core diagram: API server chain, PSS, RBAC, IAM, audit

Read it top-down: Every write hits AuthN → AuthZ (RBAC) → mutating admission → validating admission (PSS built-in + optional VAP/Gatekeeper/Kyverno) → etcd. Audit observes the chain. Cloud IAM is a separate trust domain reached when a pod’s SA is federated (IRSA / GKE WI / Azure WI) — not “the node role can do everything.” Restricted app namespaces and privileged node-agent exceptions must be explicit, labeled, and audited.

Idiosyncrasies / operator depth

Pod Security Standards (PSS) & Pod Security Admission (PSA)

Apply via namespace labels (not a cluster CRD like old PSP):

PSP → PSA: PodSecurityPolicy was removed in 1.25. PSA is built-in and simpler — but not mutating. If old PSPs defaulted runAsUser / fsGroup, workloads can break the day you drop PSP unless those fields live in the PodSpec. Stage with warn + audit before enforce.

Gotcha: Anyone who can edit namespace labels can lower enforce from restricted → privileged. Guard label mutation with RBAC / admission; prefer cluster-wide defaults + per-NS raise, not free-for-all edits.

Admission: VAP, webhooks, Gatekeeper, Kyverno

Order matters:

1. MutatingAdmissionWebhook (inject sidecars, defaults)
2. Built-in validating (incl. Pod Security Admission)
3. ValidatingAdmissionPolicy (CEL in-API-server, no extra hop)
4. ValidatingAdmissionWebhook (Gatekeeper / Kyverno / custom)

failurePolicy: Fail vs Ignore: Fail = unavailable webhook blocks matching requests (safe for security, catastrophic for availability). Ignore = fail-open (availability first, attackers love blips). Hyperscale answer: HA replicas, PDBs, probe the webhook SLI, canary policy changes, and keep PSS enforce as a fail-closed backstop when the custom engine flaps.

RBAC: roles, bindings, dangerous verbs

• Role / ClusterRole = rules; RoleBinding / ClusterRoleBinding = who gets them
• Aggregation (aggregationRule) lets controllers compose ClusterRoles — audit the aggregators, not just the tip Role
• kubectl auth can-i / SubjectAccessReview = your interview demo tool

Dangerous permissions (treat as break-glass):

ServiceAccount tokens: Prefer BoundServiceAccountTokenVolume (projected, short-lived, audience-bound). Long-lived Secret-based SA tokens are legacy debt — disable automount where unused (automountServiceAccountToken: false).

Cloud IAM least privilege (IRSA / Workload Identity)

Say it: “The pod’s cloud power should come from its ServiceAccount, not from whatever the kubelet’s instance profile can do.”

Audit: what “on” actually means

Sink to SIEM; alert on: secrets get/list by unexpected SA, escalate/bind/impersonate, edits to webhook configs / ClusterRoles, PSA violations in audit mode, break-glass RoleBindings.

Hyperscale angles

• Multi-cluster policy drift: cell A restricted, cell B unlabeled = one breach path for the same app
• Admission webhook outage: Fail-closed deploys stop across all clusters sharing a broken engine upgrade
• Break-glass ClusterRoles: time-bound, ticketed, audited; never standing cluster-admin for humans
• Per-cell vs fleet-wide ClusterRoles: prefer namespaced Roles + narrow ClusterRoles; fleet-wide admin bindings are blast radius

Composition with L08–L10

Concrete commands & config

# --- PSS / PSA on a namespace ---
kubectl label ns payments \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted --overwrite
 
# Dry-run: would current pods survive Restricted?
kubectl label ns payments pod-security.kubernetes.io/enforce=restricted --dry-run=server -o yaml
 
# Who can weaken PSS labels?
kubectl auth can-i update namespaces --as=system:serviceaccount:ci:deployer
 
# --- RBAC introspection ---
kubectl auth can-i create pods -n payments --as=system:serviceaccount:payments:app
kubectl auth can-i '*' '*' --list --as=system:serviceaccount:ci:deployer
kubectl get clusterrolebinding -o wide | rg -i 'cluster-admin|edit|admin'
kubectl get rolebinding,clusterrolebinding -A -o yaml | rg -n 'escalate|impersonate|bind:|"*"'
 
# --- Admission webhooks ---
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
kubectl get validatingadmissionpolicies,validatingadmissionpolicybindings 2>/dev/null
# failurePolicy and timeoutSeconds are the availability knobs
kubectl get validatingwebhookconfiguration -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.webhooks[*].failurePolicy}{"\t"}{.webhooks[*].timeoutSeconds}{"\n"}{end}'
 
# --- Audit (apiserver flags / managed equivalent) ---
# Confirm audit policy is loaded; in managed K8s use console / logging config
kubectl get --raw /metrics | rg apiserver_audit || true

# Namespace: Restricted PSS for apps
apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted
---
# Privileged exception for a node agent (explicit, not default)
apiVersion: v1
kind: Namespace
metadata:
  name: node-agents
  labels:
    pod-security.kubernetes.io/enforce: privileged

# Least-privilege Role + Binding (no wildcards)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: payments-deployer
  namespace: payments
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services", "configmaps"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  # NOT secrets/*, NOT pods/exec, NOT escalate/bind
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payments-deployer
  namespace: payments
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: payments-deployer
subjects:
  - kind: ServiceAccount
    name: deployer
    namespace: ci

# IRSA-style SA annotation (EKS); analogous annotations exist for GKE/Azure WI
apiVersion: v1
kind: ServiceAccount
metadata:
  name: model-loader
  namespace: model-serving
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/model-loader-readonly
automountServiceAccountToken: true
---
# Projected bound token (audience + expiry) — prefer over legacy Secret tokens
# (often automatic on modern clusters for SA-mounted tokens)

# Sketch: Audit Policy — Metadata default, RequestResponse for sensitive APIs
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets", "serviceaccounts", "serviceaccounts/token"]
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
      - group: "admissionregistration.k8s.io"
        resources: ["validatingwebhookconfigurations", "mutatingwebhookconfigurations"]
  - level: Metadata
    omitStages: ["RequestReceived"]

Failures & fixes

Security & hardening

This lesson is the control-plane security story — treat it as the spine, then extend it to AI/ML platforms.

AI/ML angle (weights, registries, training/inference NS)

# Model NS posture checks
kubectl get ns model-serving -o yaml | rg pod-security
kubectl auth can-i get secrets -n model-serving --as=system:serviceaccount:model-serving:inferencer
kubectl auth can-i create pods --as=system:serviceaccount:model-serving:inferencer -n model-serving
# Expect: deploy path yes; raw secret list no; privileged profile no

AI/ML punchline: Restricted PSS + narrow SA + WI/IRSA on the weight bucket beats “the GPU node role can read prod checkpoints.” Mesh mTLS does not replace unsigned image admission.

Interview soundbites

• “PSS is namespace labels: Privileged / Baseline / Restricted with enforce, audit, warn — PSP is gone.”
• “PSA does not mutate; bake securityContext into the PodSpec.”
• “Webhook failurePolicy: Fail is a fleet availability risk — HA the engine and keep PSS as backstop.”
• “escalate, bind, and impersonate are privilege-escalation verbs — treat like root.”
• “kubectl auth can-i is how I prove least privilege in an interview.”
• “Cloud power comes from Workload Identity on the ServiceAccount, not the node instance role.”
• “Audit Metadata everywhere; RequestResponse on secrets, RBAC, and webhook configs.”
• “NetworkPolicy + mesh AuthZ + PSS/admission/RBAC is defense in depth — say all three.”
• “Privileged DaemonSets are explicit exceptions with owners and audit — not the cluster default.”
• “Inference NS: Restricted + signed images + WI-scoped weight reads; training RDMA never carries cloud keys.”

How this maps to the JD

Hyperscale roles own safe change on the API surface across 100+ clusters: PSS baselines that do not drift, admission that fails closed without freezing the fleet forever, RBAC without standing gods, cloud IAM tied to workload identity, and audit that catches escalation. Connect 08–10–11 in one breath: datapath → packets → identity → control-plane authorization & admission.

Next in the arc: Lesson 12 — Node/container hardening & supply-chain provenance (images, SBOM, signing) — the node and artifact side of what admission enforces.

Watch (talks & demos)

Real talks that map to this lesson — watch after the Failures & fixes table.

Migrating From PodSecurityPolicy — Tim Allclair & Sam Stoelinga (KubeCon NA 2022)

PSP removal, PSA profiles/modes, mutating-PSP footguns, and when to layer Gatekeeper/Kyverno/CEL on top of built-in PSS — the migration narrative interviewers still expect.

Open on YouTube (opens in a new tab)

Practical Challenges with Pod Security Admission — V Körbes & Christian Schlotter (KubeCon EU 2023)

Real-world Restricted rollouts, privileged exceptions, staging with audit/warn, and least-privilege component splits — operator depth beyond the docs table.

Open on YouTube (opens in a new tab)

Kyverno Introduction and Deep Dive — Charles-Edouard Brétéché & Jinhong Brejnholt (KubeCon EU 2023)

Policy-as-YAML admission (validate/mutate/generate) plus image signature verification — how a policy engine sits beside PSS without replacing it.

Open on YouTube (opens in a new tab)

Tools and Strategies for Making the Most of Kubernetes Access Control — Lucas Käldström & Micah Hausler (KubeCon NA 2025)

RBAC right-sizing, can-i / audit-driven least privilege, and the AuthN → AuthZ → admission mental model — pairs with the audit and dangerous-verbs sections above.

Open on YouTube (opens in a new tab)

References

• Pod Security Standards (opens in a new tab)
• Pod Security Admission (opens in a new tab)
• Enforce Pod Security Standards with Namespace Labels (opens in a new tab)
• Migrate from PodSecurityPolicy (opens in a new tab)
• ValidatingAdmissionPolicy (opens in a new tab)
• Dynamic Admission Control (webhooks) (opens in a new tab)
• RBAC authorization (opens in a new tab)
• Using RBAC wisely / privilege escalation (opens in a new tab)
• Auditing (opens in a new tab)
• EKS IRSA (opens in a new tab) · GKE Workload Identity (opens in a new tab)
• OPA Gatekeeper (opens in a new tab) · Kyverno (opens in a new tab)
• Prior lessons: 08 — CNI, 09 — NetworkPolicy, 10 — Service mesh / mTLS

Quick self-check

Out loud, in under two minutes:

1. How do you label a namespace for Restricted PSS, and why stage with warn/audit before enforce?
2. What breaks when a validating webhook is down and failurePolicy is Fail — and what is your backstop?
3. Name three RBAC verbs that enable privilege escalation and how you would detect their use in audit logs.
4. Why is a node instance role the wrong place for model-checkpoint S3 access, and what do you use instead?
5. How do NetworkPolicy, mesh AuthZ, and PSS/admission/RBAC compose into defense in depth?