Lesson 13 — AI/ML security: model weights & checkpoint hardening (registry, signing, bucket IAM, encryption)

2026-09-21

What interviewers are really probing

Lesson 12 closed the image half of the supply chain (digest, SBOM, cosign, admit-time verify). This lesson closes the weight / checkpoint half: the multi-gigabyte blobs that training jobs write and inference pods pull — often from object buckets or OCI artifact registries, and often with weaker controls than the container that loads them.

They want evidence you treat weights as first-class artifacts, not “files on S3”:

• Lifecycle — train → checkpoint → promote → serve, with a human/CI gate between staging and prod prefixes
• Immutability — content hashes, versioned keys / Object Lock; never a mutable latest.safetensors
• Encryption — SSE-KMS / CMEK (customer-managed keys); optional client-side for crown-jewel IP
• Bucket IAM least privilege — separate weight-writer (training SA) vs weight-reader (inference SA) via Workload Identity / IRSA — never broad GetObject on the node instance role
• Signing & provenance — cosign / OpenSSF model-signing on OCI model packages; in-toto/SLSA-style attestations tying checkpoint hash → training job identity → git commit → dataset digest
• Compose — signed serving image + unsigned weight = still broken (L12 punchline made operational here)
• Hyperscale — multi-cell mirrors of weight buckets + signature sidecars; CMEK rotation without breaking readers; audit logs for GetObject on model prefixes

Interview sentence: “I harden the serving image and the checkpoint plane — private store, CMEK, writer/reader SA split, signed digest, admit and audit the pull.”

Core diagram: train → checkpoint → encrypt/sign → private store → IAM gate → inference

Read it top-down (left-to-right on the PNG): the training job SA may write only to a staging prefix (or staging bucket) → checkpoints land under content-addressed / versioned keys with SSE-KMS/CMEK → a promote pipeline (CI + optional human gate) copies into prod, signs (cosign on ORAS/OCI packages, or model-signing / checksum manifests for directory-shaped models), and attaches provenance → inference uses a separate weight-reader identity → admission / runtime verify matches the digest you signed → CloudTrail / Data Access logs record GetObject on model prefixes. Preview Lesson 14: securing the inference/training deployments that consume these artifacts.

Idiosyncrasies / operator depth

Weight lifecycle (parallel to image promote)

Treat promote like an image release: immutable artifact ID, signature, provenance, and a record of who approved.

Storage planes: bucket vs OCI vs model hub

Many fleets are hybrid: base models from a controlled hub mirror into your bucket/registry; fine-tunes never leave your CMEK boundary.

Digest-like immutability for blobs

• Prefer keys like models/llama-ft/sha256-<hex>/model.safetensors or object versioning + Object Lock (WORM) on prod prefixes.
• Record the content hash in the promote ticket / SLSA predicate — same role digest plays for images.
• Ban mutable aliases (latest.safetensors, prod/current) in prod. Staging may use soft tags; promote resolves to hash.
• For multi-file directories: hash the manifest (or use OpenSSF model-signing, which is built for directory-shaped models).

Encryption: SSE-KMS / CMEK / client-side

Rotation without breaking readers: dual-encrypt window, grant both old and new key ARNs on the reader role, re-encrypt or rewrite objects on a schedule, then drop the old key grant. Hyperscale cells must share the same rotation playbook or DR pulls fail closed on KMS AccessDenied.

Bucket IAM: writer vs reader (the interview money question)

Training job SA  →  s3:PutObject on staging/*   (+ kms:GenerateDataKey)
Promote CI SA    →  staging Get + prod Put      (+ sign)
Inference SA     →  s3:GetObject on prod/*      (+ kms:Decrypt)
Nobody           →  s3:ListBucket on entire bucket from notebooks without need
Node instance role → NEVER broad models GetObject

• Workload Identity / IRSA / GKE WI / AKS WI bind the K8s SA to the cloud IAM role — compose with L11.
• Block Public Access account + bucket; disable ACLs; bucket policy deny Principal: *.
• Prefer VPC endpoints / Private Google Access so training/inference nodes never need public S3/GCS IPs for weight pulls.
• Prefix-level conditions (s3:prefix) beat “one role for all models.”

Signing & provenance for weights

Compose with L12: admission that only checks the serving image will happily schedule a signed image that loads a poisoned weight from an over-permissioned bucket. Verify both planes — or mount only from a verified OCI artifact volume / init-container that checks the signature before the main process starts.

Training identity split & exfiltration paths

Classic leaks operators actually see:

• Notebook namespace SA with s3:ListBucket + GetObject on models/*
• Debug sidecar / ephemeral pod using the node instance profile
• Unlogged egress to public HF / anonymous Git LFS
• Shared “ml-platform” role reused by train, promote, and serve
• Staging bucket with public ACL “just for the demos”

Mitigate with NetworkPolicy / egress proxies (L9 / mesh from L10), separate SAs, and Data Access audit alerts on unusual GetObject volume or principals.

Hyperscale idiosyncrasies

• Multi-cell mirrors: replicate objects and signatures (sidecar manifests, OCI referrers). A cell that has weights but not sigs fails verify; a cell with sigs but stale weights serves the wrong model.
• CMEK per region / per cell: document which key encrypts which prefix; avoid a single global key becoming a fleet SPOF without a rotation drill.
• Audit: enable object-level logging; alert on GetObject from non-inference principals on prod prefixes; retain for IR.
• Admission drift: same trust roots for model OCI artifacts across cells — fleet-manage Kyverno/Ratify CRs like L12 image policy.

Concrete commands & config

# --- AWS: bucket posture (illustrative) ---
aws s3api get-public-access-block --bucket models-prod
aws s3api get-bucket-encryption --bucket models-prod
aws s3api get-object-lock-configuration --bucket models-prod 2>/dev/null || true
 
# Put a content-addressed checkpoint (SSE-KMS)
HASH=$(sha256sum model.safetensors | awk '{print $1}')
KEY="models/llama-ft/sha256-${HASH}/model.safetensors"
aws s3api put-object \
  --bucket models-staging \
  --key "$KEY" \
  --body model.safetensors \
  --server-side-encryption aws:kms \
  --ssekms-key-id "$KMS_KEY_ARN"
 
# Promote: copy staging → prod (CI role), then sign
aws s3 cp "s3://models-staging/$KEY" "s3://models-prod/$KEY"
sha256sum model.safetensors | tee "model.safetensors.sha256"
cosign sign-blob --bundle model.sigbundle model.safetensors
# store bundle next to object or in sig sidecar prefix

# --- GCP-flavored sketch ---
gcloud storage buckets describe gs://models-prod --format='yaml(iamConfiguration,encryption)'
# CMEK on bucket / object; Uniform bucket-level access ON
gsutil hash -h model.safetensors
gcloud storage cp model.safetensors \
  "gs://models-staging/models/llama-ft/sha256-${HASH}/model.safetensors" \
  --encryption-key=projects/P/locations/L/keyRings/R/cryptoKeys/K

# --- ORAS: package weights as OCI artifact + cosign (ties to L12) ---
oras push registry.example.com/models/llama-ft:sha256-${HASH} \
  ./model.safetensors:application/vnd.example.safetensors \
  --artifact-type application/vnd.example.model
 
DIGEST=$(crane digest registry.example.com/models/llama-ft:sha256-${HASH})
REF="registry.example.com/models/llama-ft@$DIGEST"
cosign sign "$REF"
cosign verify "$REF"
cosign tree "$REF"   # signatures must mirror with the artifact

# IRSA / WI sketch: inference SA may GetObject prod prefix only
apiVersion: v1
kind: ServiceAccount
metadata:
  name: weight-reader
  namespace: model-serving
  annotations:
    # EKS example — swap for GKE/AKS WI annotation in your cloud
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/weight-reader
---
# Training SA — staging write only (separate role)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: weight-writer
  namespace: model-training
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/weight-writer

# IAM policy fragment — weight-reader (illustrative)
{
  "Effect": "Allow",
  "Action": ["s3:GetObject"],
  "Resource": "arn:aws:s3:::models-prod/models/*"
},
{
  "Effect": "Allow",
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "arn:aws:kms:REGION:ACCT:key/KEY-ID",
  "Condition": {
    "StringEquals": {
      "kms:ViaService": "s3.REGION.amazonaws.com"
    }
  }
}

# Kyverno: model-serving NS — require signed OCI model refs (compose L12)
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-model-oci-artifacts
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-signed-model-artifacts
      match:
        any:
          - resources:
              kinds: ["Pod"]
              namespaces: ["model-serving"]
      verifyImages:
        - imageReferences:
            - "registry.example.com/models/*"
          mutateDigest: true
          verifyDigest: true
          required: true
          attestors:
            - count: 1
              entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      # fleet-managed model-signing / cosign.pub
                      -----END PUBLIC KEY-----

# Authz / audit habits (L11 compose)
kubectl auth can-i get pods -n model-serving \
  --as=system:serviceaccount:model-serving:weight-reader
# Expect: pull weights yes; list secrets no; create privileged no
kubectl get ns model-serving -o yaml | rg pod-security
# CloudTrail / GCP Data Access: alert GetObject on models-prod from non-reader roles

Failures & fixes

Security & hardening

This lesson is the weight/checkpoint hardening story. Treat the tables below as the control checklist you walk in an interview — then compose with L11 (who may create what) and L12 (signed images).

Artifact & storage controls

Cluster & identity controls (compose L11 / L12)

AI/ML threat surfaces (operator view)

# Punchline checks before calling a model “prod-ready”
# 1) Image signed? (L12)  2) Weight hash signed + CMEK? (L13)
# 3) Reader SA only?      4) Audit on?
cosign verify "$SERVING_IMAGE"
cosign verify "$MODEL_OCI_REF"   # or verify-blob / model-signing verify
aws s3api get-bucket-encryption --bucket models-prod
kubectl get sa weight-reader -n model-serving -o yaml | rg role-arn

Punchline: A signed serving image pulling an unsigned, over-permissioned checkpoint is still a supply-chain fail. L12 closed images; L13 closes weights. Lesson 14 hardens the inference/training deployments that consume both.

Interview soundbites

• “Weights are artifacts: content hash, CMEK, sign, promote — same discipline as images, different storage plane.”
• “Training SA writes staging only; promote CI signs into prod; inference SA is GetObject + kms:Decrypt — never the node role.”
• “Mutable latest.safetensors in prod is the tag-moving problem for ML.”
• “Signed image + unsigned weight = still broken — verify both planes.”
• “ORAS + cosign lets model packages reuse the L12 admit path; raw buckets need sign-blob or model-signing + checksum sidecars.”
• “Block Public Access and CMEK are table stakes; the interview differentiator is writer/reader split and audit.”
• “Mirror cells must copy signatures with objects — cosign tree in DR.”
• “CMEK rotation: dual-grant window, then drop the old key — or serve dies on Decrypt.”
• “Notebooks that can ListBucket prod models are an exfil path — prefix IAM and NetworkPolicy.”
• “L11 who / L12 image / L13 weight / L14 deployment — full AI/ML hardening stack.”

How this maps to the JD

Hyperscale + AI/ML platform roles own safe model artifacts across 100+ clusters: encrypted private stores, least-privilege WI/IRSA for train vs serve, signed/provenanced checkpoints, and admit-time verification that composes with image supply chain. Connect L11 → L12 → 13: authorization → image provenance → weight provenance. Fuel for GPU-fleet context: wafer-ai/gpu-perf-engineering-resources (opens in a new tab) (operator framing). Next: Lesson 14 — AI/ML security: securing inference/training deployments.

Watch (talks & demos)

Real talks that map to this lesson — watch after the Failures & fixes table.

Inside the OpenSSF Model Signing Project: Securing the ML Supply Chain

Sigstore-based signing for directory-shaped models (not only container images) — the exact “weights are huge trees” problem this lesson centers.

Open on YouTube (opens in a new tab)

Manage and Run AI/ML Models as OCI Artifacts with ORAS

Package models as OCI artifacts, push to a registry, attach signatures via referrers — the ORAS path that reuses L12 cosign/admit patterns for weights.

Open on YouTube (opens in a new tab)

AWS re:Invent 2023 — Amazon S3 security and access control best practices (STG315)

Block Public Access, encryption defaults, IAM vs bucket policies, Access Analyzer — the bucket-IAM half of weight hardening at cloud scale.

Open on YouTube (opens in a new tab)

References

• OpenSSF Model Signing (opens in a new tab) · sigstore/model-transparency (opens in a new tab) · model-signing on PyPI (opens in a new tab)
• Sigstore (opens in a new tab) · cosign other types / OCI artifacts (opens in a new tab) · SLSA (opens in a new tab) · in-toto (opens in a new tab)
• ORAS (opens in a new tab) · OCI artifacts (opens in a new tab)
• AWS SSE-KMS (opens in a new tab) · S3 Block Public Access (opens in a new tab) · S3 Object Lock (opens in a new tab)
• GCS CMEK (opens in a new tab) · Uniform bucket-level access (opens in a new tab)
• EKS IRSA (opens in a new tab) · GKE Workload Identity (opens in a new tab)
• Kyverno verifyImages (opens in a new tab) · Ratify (opens in a new tab)
• GPU/platform fuel: wafer-ai/gpu-perf-engineering-resources (opens in a new tab)
• Prior: 12 — Node/container hardening & supply chain · 11 — Cluster security / PSS / RBAC

Quick self-check

Out loud, in under two minutes:

1. Why is a signed serving image still unsafe if the checkpoint bucket is world-readable or readable by the node role?
2. Draw the writer / promote / reader IAM split — who may Put vs Get, and on which prefixes?
3. How do content-hash keys + Object Lock beat a mutable latest.safetensors?
4. When do you pick ORAS+cosign vs OpenSSF model-signing vs cosign sign-blob?
5. What must multi-cell mirrors copy besides the weight bytes — and how do you verify?
6. Name three exfiltration paths from training/notebook namespaces and one control each.