Lesson 10 — Service mesh: Istio/Envoy/Linkerd & mTLS

2026-09-18

What interviewers are really probing

Lessons 08 and 09 gave you CNI datapath and L3/L4 NetworkPolicy. This lesson is what sits on top: a service mesh that terminates identity-aware mTLS, steers L7 traffic, and — if you misconfigure it — becomes the loudest outage and the softest “security theater” in the fleet.

They are not asking you to memorize every VirtualService field. They want:

• Mesh ≠ NetworkPolicy — NP is packet allow/deny on the CNI path; mTLS is cryptographic identity between workloads (SPIFFE). You need both stories.
• STRICT vs PERMISSIVE with a live migration failure mode (plaintext clients, half-injected namespaces, 503 cascades)
• Why sidecars are TCB, why ambient moves trust to ztunnel/waypoint, and what that does to blast radius
• Concrete operator tooling: istioctl, linkerd, Envoy admin, PeerAuthentication / AuthorizationPolicy, DestinationRule TLS modes

Cross-link: CNI/eBPF (08) → NetworkPolicy / multi-NIC (09) → this lesson. Mesh does not fix RDMA planes that bypass CNI; it also does not replace default-deny NetworkPolicy.

Core diagram: control plane, sidecar vs ambient, policy stack

Read it top-down: Control plane issues short-lived workload certs (SPIFFE) and pushes xDS. Classic path = per-pod proxy (Envoy in Istio, Linkerd-proxy in Linkerd) terminating mTLS beside the app. Ambient path = shared ztunnel for L4 mTLS (+ optional waypoint for L7). Underneath, 08 still owns pod networking and 09 still owns L3/L4 allow/deny. Encrypted free-for-all without AuthorizationPolicy is not zero-trust.

Idiosyncrasies / operator depth

Mesh sits on CNI — composition rules

Interview line: “NetworkPolicy is the packet firewall; mTLS is the passport check; AuthorizationPolicy is the guest list. I never claim mesh ‘replaces’ NetworkPolicy.”

STRICT vs PERMISSIVE (the migration footgun)

Istio PeerAuthentication modes (mesh / namespace / workload):

Linkerd’s default posture is closer to mTLS on for meshed↔meshed with less YAML ceremony — but you still own opaque ports, skip ports, and Server / ServerAuthorization (or newer policy CRDs) for who may connect.

PERMISSIVE forever in prod = you encrypted the compliant clients and left a plaintext hole for everything else. Attackers (and confused apps) take the hole.

Sidecar as TCB vs ambient trust shift

Sidecar model

• Init/redirect (iptables or equivalent) + proxy container share the pod network namespace
• App compromise ≠ automatic cert theft if keys stay in proxy memory — but proxy CVE / malicious config is still TCB
• Cost: inject churn, CPU/RAM tax, istio-proxy restart storms, init-container races with CNI

Ambient (Istio)

• ztunnel (per-node, typically Rust) does L4 HBONE/mTLS
• waypoint (Envoy) optional for L7 policies per namespace/service
• Trust moves to node agent + shared waypoint: fewer sidecars, different blast radius (node ztunnel down = many pods hurt)

Say out loud: “Ambient reduces sidecar tax; it does not delete the need for STRICT + AuthZ + locked-down control plane.”

SPIFFE / SPIRE

• Default Istio identity: spiffe://<trust-domain>/ns/<namespace>/sa/<serviceaccount>
• SPIRE when you need stronger attestation, multi-cluster federation, or non-K8s workloads joining the same trust domain
• Certs via SDS (Unix domain socket into the proxy) — private keys should not land as mounted Secret files in the app container

VirtualService / DestinationRule gotchas

• DestinationRule trafficPolicy.tls.mode: ISTIO_MUTUAL vs SIMPLE vs DISABLE — wrong mode = 503s that look like “mesh is broken”
• VirtualService without matching Gateway / host / subset labels = silent no-match (traffic falls through unexpectedly)
• Retries on non-idempotent POSTs = duplicate side effects; mesh retries are not free correctness
• Outlier detection ejecting the only healthy pod during a control-plane blip = self-DDoS
• Headless / raw IP / Multi-NIC: mesh interception often only on the primary path — same class of footgun as Multus in 09

Envoy admin & debugging instinct

When “Service A cannot talk to B after mesh”:

1. Is B injected / in ambient? (istioctl proxy-status, linkerd viz edges)
2. PeerAuthentication mode on B’s namespace?
3. AuthorizationPolicy deny?
4. DestinationRule TLS mode?
5. Envoy clusters/listeners: istioctl proxy-config or Envoy admin /:15000

Concrete commands & config

# --- Istio ---
istioctl version
istioctl proxy-status
istioctl analyze -n payments
istioctl pc listeners deploy/frontend -n payments
istioctl pc clusters deploy/frontend -n payments
istioctl pc routes deploy/frontend -n payments
istioctl pc secret deploy/frontend -n payments   # SDS certs present?
 
# Auth dumps / config
kubectl get peerauthentication,authorizationpolicy,destinationrule,virtualservice -A
kubectl -n istio-system get peerauthentication
istioctl x desc pod frontend-xxx -n payments
 
# Envoy admin (port-forward carefully — treat as sensitive)
kubectl -n payments port-forward deploy/frontend 15000:15000
# then: curl -s localhost:15000/certs | head
#       curl -s localhost:15000/clusters | rg payments
 
# --- Linkerd ---
linkerd check
linkerd viz edges -n payments
linkerd identity -n payments deploy/frontend
linkerd viz tap deploy/frontend -n payments --to deploy/backend

# Lock a namespace to mTLS-only (Istio)
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments
spec:
  mtls:
    mode: STRICT
---
# Identity allowlist: frontend SA may call backend on 8080
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: backend-allow-frontend
  namespace: payments
spec:
  selector:
    matchLabels:
      app: backend
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - "cluster.local/ns/payments/sa/frontend"
      to:
        - operation:
            methods: ["GET", "POST"]
            ports: ["8080"]
---
# Explicit ISTIO_MUTUAL to a subset (DestinationRule)
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: backend
  namespace: payments
spec:
  host: backend.payments.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
  subsets:
    - name: v2
      labels:
        version: v2

# Mesh-wide STRICT (put in istio root NS — high blast radius; stage carefully)
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

# Prove STRICT: plaintext from a non-mesh pod should fail
kubectl -n payments exec deploy/legacy-curl -c curl -- \
  curl -sS -o /dev/null -w "%{http_code}\n" http://backend.payments:8080/
# Expect failure / connection reset once STRICT + no plaintext path
 
# Prove AuthZ: wrong SA should get RBAC deny (HTTP 403 via Envoy)

Failures & fixes

Security & hardening

This lesson’s security story is identity that is enforced, not “TLS somewhere in the YAML.”

# Who can mutate mesh security objects?
kubectl auth can-i create peerauthentications -A \
  --as=system:serviceaccount:ci:deployer
kubectl auth can-i create authorizationpolicies -n model-serving \
  --as=system:serviceaccount:ci:deployer
 
# Model NS: unexpected plaintext or broad allow is a finding
kubectl get peerauthentication,authorizationpolicy -n model-serving -o yaml | rg -n "PERMISSIVE|DISABLE|{}|principals"

AI/ML punchline: meshing the inference Service stops casual east-west scrapes of the HTTP API; it does nothing for a world-readable weight bucket or an unsigned image. Split the trust domains on purpose.

Interview soundbites

• “Mesh sits on CNI — NetworkPolicy is L3/L4; mTLS is identity. I run both.”
• “PERMISSIVE is a migration mode, not a production posture.”
• “STRICT without AuthorizationPolicy is encrypted any-to-any.”
• “Sidecar proxy is TCB; ambient moves trust to ztunnel and waypoints — different blast radius, same need for AuthZ.”
• “SPIFFE ns/sa identity beats IP allowlists when pods churn.”
• “503 after mesh is usually inject, PeerAuthentication, DestinationRule TLS mode, or AuthZ — in that debug order.”
• “VirtualService retries are not free correctness.”
• “Envoy admin ports are sensitive — NetworkPolicy them like etcd.”
• “Inference APIs in mesh; model weights and registries hardened outside the mesh.”
• “RDMA/Multus often bypasses mesh intercept — same footgun as NetworkPolicy on eth1.”

How this maps to the JD

Hyperscale platform roles own east-west zero-trust without melting the fleet: mTLS that survives migration, AuthZ that matches SPIFFE identities, control planes that are not public, and an honest story about what the mesh cannot see (RDMA, weight registries). You should connect 08–09–10 into one sentence: datapath → packet policy → identity policy.

Next in the arc: Cluster security (PSS, admission, RBAC, IAM least privilege & audit) — hardening the control surface the mesh depends on.

Watch (talks & demos)

Real talks that map to this lesson — watch after the Failures & fixes table.

Is Istio Ambient Mesh Secure? — Christian Posta & John Howard (Istio Day / CNCF)

Threat model for sidecarless ambient: how ztunnel/waypoint preserve (or shift) mTLS and workload identity boundaries vs classic sidecars — the interview-grade ambient security narrative.

Open on YouTube (opens in a new tab)

Life of a Request in Istio Ambient Mesh — Gregory Hanson, Solo.io

Request path through ztunnel and waypoints (vs sidecar-to-sidecar) — how to debug ambient the way operators already debug Envoy 503s.

Open on YouTube (opens in a new tab)

Next-Level Security: Implementing mTLS in Istio Multi-Cluster Environments Using SPIRE — Eduardo Bonilla Rodriguez & Samuel Veloso (KubeCon EU 2024)

SPIFFE/SPIRE attestation + Istio SDS for multi-cluster mTLS — when default cluster trust domain is not enough.

Open on YouTube (opens in a new tab)

References

• Istio Security overview (opens in a new tab)
• Istio Mutual TLS migration (PERMISSIVE → STRICT) (opens in a new tab)
• Istio PeerAuthentication (opens in a new tab)
• Istio AuthorizationPolicy (opens in a new tab)
• Istio Ambient mesh (opens in a new tab)
• SPIFFE / SPIRE (opens in a new tab)
• Linkerd automatic mTLS (opens in a new tab)
• Linkerd policy (opens in a new tab)
• Envoy admin interface (opens in a new tab)
• Prior lessons: 08 — CNI / Cilium / eBPF, 09 — NetworkPolicy / multi-NIC / sFlow

Quick self-check

Out loud, in under two minutes:

1. How do NetworkPolicy and mesh mTLS compose — and why is “we have a mesh” not a segmentation answer?
2. What breaks when you flip PeerAuthentication to STRICT while some clients are still plaintext / unmeshed?
3. Sidecar vs ambient: where does the TCB live, and what is the blast radius of a ztunnel failure?
4. Why do inference API pods belong in the mesh while model weight buckets and image registries need a different hardening story?