Day 15 — Closed-loop learned policies under measured ceilings / sim-to-real residual validation

Day 14 owned LearnedResidual / ResidualCorrection and SoftSceneGraph as soft correctors and structured candidates into the affordance→schedule path: never_hard refuse cones / $F_z \ge 0$ / JointCommand / FOC / torque_scale / ContactState; residual/graph score $\neq$ MEASURED; inference at mid / host rates with age_ms. Day 13 owned AffordanceProposal as soft CLAIM candidates. Day 12 owned the ContactSchedule lifecycle and OBJECT kind. Day 11–Day 10 owned measured hand support and soft EE on FREE limbs. Day 09–Day 08 owned recovery abort and gait vs measured contact_set. Day 07 owned the feasible program. Day 06 owned the estimate. Day 05 owned sag honesty.

Today owns closed-loop learned policies (RL / imitation / behavior policies) that act only through soft refs and schedule / affordance candidates above the same measured hard set — still no learned-plant-as-constraint-truth — plus sim-to-real residual validation in the twin before free closed-loop learned multi-contact on hardware.

Policies are soft writers, not a second plant

A policy that writes friction cones, stamps ContactState membership, emits JointCommand beside the QP, or overrides torque_scale is detector-as-constraint-truth wearing a reward function. Closed-loop means the policy ticks on live EstimatorState + ContactState + PowerState (+ optional Day 14 residual / scene soft feeds) at mid / host rate with age_ms — not open-loop replay of a taped action sequence, and not inside µs / EtherCAT cyclic until jitter is measured and budgeted. MEASURE still admits contacts. Demotion is still mandatory when policy disagreement, residual / graph inconsistency, wrench disagreement, domain gap, or ceilings fail — including when the reward still looks busy.

Core thesis: a LearnedPolicy / PolicyAction / PolicyCommand is soft output only. Closed-loop means it consumes the same live EstimatorState + ContactState + PowerState stack (plus optional Day 14 residual / scene soft feeds) and publishes soft task refs, affordance ranking / generation, or CLAIM candidate admit — never hard friction cones / $F_z \ge 0$ / JointCommand / FOC $i_q$ / torque_scale / ContactState membership. Policy score $\neq$ MEASURED. MEASURE is still required before PROMOTE. There is still one QP face — not a “learned plant” that finishes a wall brace because the reward looked sure. The twin must keep residual bias / false edges / delay / disagreement / sag honesty and inject policy-domain gap (action bias, reward hacking into hard writes, latency, dropout, domain shift, false CLAIM success in sim) before free closed-loop learned multi-contact on hardware. Day 14 never_hard still binds. Inference / policy ticks run at mid / host rates with age_ms; never inside the current loop or EtherCAT cyclic write path until jitter is measured and budgeted.

Ownership rule: one writer for policy soft actions, one writer for affordance candidates, one writer for schedule intent, one writer for measured contact_set. When they disagree, measured contact wins for constraints. Policy confidence and reward never attach cones. Intent aborts, softens, retimes, or waits — it does not promote OBJECT because the policy scored a phantom rail first. Day 08 gait and Day 09 recovery still outrank a stuck schedule hold, even when the reward is loud. Day 14 residual / scene soft feeds remain soft when the policy consumes them.

How today fits the Days 05–14 stack

Do not bolt a “policy plant” that writes JointCommand or attaches cones beside the QP when the reward fires. That is Day 10’s second-plant anti-pattern wearing a GPU and a training curve.

Policy honesty / rate / domain gates

Day 13’s pre-CLAIM gate and Day 14’s residual-domain / rate honesty still hold. Day 15 adds policy-domain and sim-to-real honesty: closed-loop policies correct soft things at mid / host rates with explicit age — never as hard truth inside µs paths, never as open-loop replay labeled “closed-loop.”

Sequencing rules (all required):

1. Policy scores gate CLAIM entry and soft ranking / soft refs — never cone attach.
2. never_hard: true is not decoration — API and runtime must refuse policy writes into friction / unilaterality / JointCommand / FOC / torque_scale / ContactState (Day 14 bind still holds).
3. MEASURE still required before PROMOTE; policy score $\neq$ MEASURED; residual / graph score $\neq$ MEASURED.
4. Closed-loop means live state ticks at mid / host rates with age_ms — not open-loop replay; never inside the current loop or EtherCAT cyclic write path until jitter is measured and budgeted.
5. Policy disagreement with MEASURE / wrench, residual disagreement, graph inconsistency, or sim-to-real gate fail → fail gate or demote — never silently trust the reward.
6. Mid-sequence POWER_GATED / stale / recovery / inference dropout / reward-hack attempt into hard writes → freeze or abort the schedule; do not advance phases on stale tensors or hard-write attempts.
7. Twin must prove sim-to-real residual validation (action bias, reward hacking into hard writes, latency, dropout, domain shift, false CLAIM success in sim, plus Day 14 residual honesty) before free closed-loop learned multi-contact on hardware.

Demotion is success when the policy hallucinated support, the reward hacked toward a hard write, the sim CLAIMed on sticky contacts the hardware never saw, or the pack sagged. A robot that always finishes the policy-driven handoff into a collapsing bus is ignoring Day 05 with a prettier eval curve.

Ownership conflict: policy vs measure vs gait vs recovery vs power

Learned closed-loop policies are where teams accidentally create a sixth writer on support constraints. Freeze the split:

Hard vs soft: support constraints on measured FOOT / HAND / OBJECT stay hard. Soft EE and policy refs live only on FREE limbs and soft task refs. Policies (and Day 14 residual / scene feeds) are softer still — they feed ranking and CLAIM, they do not feed cones. If the QP goes infeasible under the expanded set, demote / abort / soften — do not silently drop an object cone so the policy “makes it.”

Mid-policy abort honesty

Closed-loop mid-rate paths are where teams disable safety “just until the episode ends.” Do not.

Abort, demote, and soften are success paths when the policy hallucinated, the reward hacked, the sim lied, or the pack sagged mid-handoff.

Shared API: LearnedPolicy into Day 13–14 shapes

Keep Day 02 / 04 wire honesty. Cyclic commands still carry BusHeader. Extend Day 14 residual / Day 13 AffordanceProposal / Day 12 ContactSchedule / WbcTaskSet — do not invent a parallel “policy bus” or second plant:

BusHeader:
  node_id, seq
  t_sample, clock_domain, age_budget_ms
  faults

# Upstream (read-only here; policy consumes, does not replace)
EstimatorState, PowerState  ⊇ BusHeader

ContactState ⊇ BusHeader:
  contact_set[]        # feet, hands, AND objects when measured
  SupportContact:
    link_id / frame
    kind               # FOOT | HAND | OBJECT
    mu, normal, wrench_est?
    confidence, age_ms
    flags              # settling / flicker / gated / grasp_lost
    parent_grasp_id?

# Optional Day 14 soft feeds the policy may consume
LearnedResidual / SoftSceneGraph  ⊇ BusHeader
  # still never_hard; still soft only — Day 14 unchanged

# Soft closed-loop policy (host / mid-rate) — NEVER hard
LearnedPolicy ⊇ BusHeader:
  mode                 # IDLE | CLOSED_LOOP | GATED | ABORT
  observes             # ESTIMATE | CONTACT | POWER | RESIDUAL? | SCENE?
  tick_rate_hint_hz    # mid / host — not µs cyclic
  age_ms               # expire stale policy ticks
  never_hard: true     # API + runtime refuse cones / Fz / JointCmd /
                       # FOC iq / torque_scale / ContactState writes
  relax_on_fault       # stale / dropout / domain_violation /
                       # reward_hack_attempt → expire / abort

PolicyAction / PolicyCommand ⊇ BusHeader:   # soft output only
  domain               # TASK_REF | AFFORDANCE_RANK | CLAIM_ADMIT
  delta_com_ref? / delta_ee_refs? / delta_capture?
  policy_rank?         # re-rank AffordanceProposal slots
  claim_candidates[]?  # soft CLAIM admit hints only
  confidence
  age_ms
  never_hard: true
  closed_loop: true    # must have consumed live state this tick
  relax_on_fault

# Soft perception feed — Day 13/14 + policy enrichment
AffordanceProposal ⊇ BusHeader:
  proposals[]:
    target_frame
    kind_hint          # FOOT | HAND | OBJECT
    pose_hint
    normal_hint?       # soft — not hard cone truth
    mu_hint?
    confidence         # gates CLAIM entry, NOT cone attach
    geometry_ok
    reachability_ok
    residual_rank?     # Day 14
    scene_ref?         # Day 14
    policy_rank?       # from PolicyAction AFFORDANCE_RANK
    policy_ref?        # which LearnedPolicy tick ranked this
    age_ms
    source             # VISION | TACTILE | FUSED | RESIDUAL | SCENE | POLICY | ...
  relax_on_fault

# Thin multi-contact face — Day 12–14 + policy admit
ContactSchedule ⊇ BusHeader:
  intent               # IDLE | MULTI_CONTACT | OBJECT_SUPPORT | ABORT
  slots[]:
    target_frame
    phase              # FREE | CLAIM | PROMOTE | HOLD | DEMOTE
    earliest_t / latest_t?
    priority_hint
    affordance_ref?
    residual_ref?
    scene_ref?
    policy_ref?        # which PolicyAction admitted this CLAIM
  yield_to             # GAIT | RECOVERY | POWER
  admit_policy         # min conf / geom / reach / residual_rank /
                       # graph consistency / policy_rank to open CLAIM
  relax_on_fault       # stale / POWER_GATED / flicker / grasp_lost /
                       # residual_disagree / graph_inconsistent /
                       # policy_disagree / inference_dropout /
                       # reward_hack_attempt → abort/demote

# Sim-to-real residual validation gate (twin + bring-up)
SimToRealGate ⊇ BusHeader:
  status               # RED | YELLOW | GREEN
  checks[]:
    action_bias
    reward_hack_to_hard_write
    latency_dropout
    domain_shift
    false_claim_success_in_sim
    residual_bias          # Day 14
    false_edges            # Day 14
    measure_disagreement   # Day 14
    power_sag              # Day 05
  age_ms
  block_free_closed_loop_until_green: true

ManipulationCommand ⊇ BusHeader:
  intent               # … | CLAIM_SUPPORT | LOCO_MANIP
                       # | OBJECT_SUPPORT | MULTI_CONTACT | ABORT
  ee_refs[] / hand_refs[]
  hand_role[] / support_claim[]
  object_claim[]
  schedule_ref?
  affordance_ref?
  residual_ref?
  scene_ref?
  policy_ref?
  relax_on_fault

# Extended task face — policy refs soft only
WbcTaskSet ⊇ BusHeader:
  mode                 # STANCE_DS | STANCE_SS | SWING | IMPACT | RECOVER_IP
                       # | MANIP | HAND_SUPPORT | LOCO_MANIP
                       # | OBJECT_SUPPORT | MULTI_CONTACT | ...
  com_ref / capture_ref?
  residual_hint?       # Day 14 soft additive
  policy_hint?         # soft additive on com_ref / ee_refs / capture
                       # NEVER hard constraints
  foot_refs[]
  hand_support_refs[]?
  object_support_refs[]?
  ee_refs[]            # soft EE — FREE limbs only
  hand_refs[]
  posture_bias?
  priority_order[]
  relax_on_fault

JointCommand ⊇ BusHeader:
  q_des? / qd_des? / tau_ff?
  iq_ceiling? / tau_ceiling?
  flags                # STO / soften / hold

Higher layers publish policy actions, residual corrections, scene structure, affordance candidates, and schedule intent. Measured ContactState owns support membership. Gait and recovery keep their writers. Rate loops enforce BMS ceilings. The twin owns SimToRealGate honesty. Nobody re-implements FOC, AHRS, or friction cones inside the policy net.

Physical ↔ digital pairing for policy / sim-to-real

If the twin always promotes when the policy scores a slot with sticky contacts and a full pack while hardware sees biased actions, reward hacks, delayed ticks, residual lies, and sag, you will tune learning rates forever and never ship a CLAIM that survives a tired bus or a sim-only success.

Bring-up sequence (before free closed-loop learned multi-contact)

Days 07–14 green (stance, step, recovery, FREE-hand reach, measured hand SUPPORT / loco-manip, scripted multi-contact / OBJECT handoffs, gated affordance CLAIM, residual / scene soft feeds with bias / false edges / delay / disagreement proven). Then:

1. Days 07–14 green — affordance admit + residual / scene soft paths still behave; no open recovery or gait fights; OBJECT / HAND promotes still require MEASURE; never_hard still enforced; detector-as-constraint-truth still impossible.
2. Offline policy replay — replay logged PolicyActions into soft refs and CLAIM admit only; confirm high policy_rank without MEASURE never attaches cones; confirm domain violation (never_hard) is refused; confirm stale age_ms expires candidates; confirm open-loop tape alone cannot promote.
3. Single live gated closed-loop tick — one known surface or fixed object; policy observes live state → soft action → gate → CLAIM → wait for MEASURED → PROMOTE → HOLD → DEMOTE; feet remain primary; confirm QP feasible; confirm policy_hint stays soft.
4. Sim-to-real residual validation — twin injects action bias, reward hacking into hard writes, latency / dropout, domain shift, false CLAIM success in sim, plus Day 14 residual bias / false edges / MEASURE disagreement / POWER_GATED mid-CLAIM; confirm drop / wait / demote / API reject; SimToRealGate must go GREEN before free hardware path.
5. Non-fight + mid-policy faults — force policy-CLAIM-during-swing, recovery-during-HOLD, stale / gated / flicker / residual disagreement / reward-hack attempt; confirm yield / abort / demote / reject.
6. Then add free closed-loop learned multi-contact that chooses ranking / soft refs above the QP — never replace cones, FOC, torque_scale, gait, recovery, or measured ContactState. Teleop / shared-autonomy arbitration under the same ceilings continues in Day 16.

Skip to “end-to-end closed-loop learned multi-contact in sim” and hardware’s first policy-ranked rail grasp becomes an unplanned tip / brownout experiment with better WandB charts.

Failure gallery

What to freeze this week

Extend frames and your control config with:

• LearnedPolicy / PolicyAction / PolicyCommand fields: domain, soft delta_* / policy_rank / claim_candidates, confidence, age_ms, closed_loop: true, never_hard: true
• SimToRealGate: RED/YELLOW/GREEN checks for action bias, reward-hack-to-hard-write, latency/dropout, domain shift, false CLAIM success in sim, plus Day 14 residual honesty + Day 05 sag
• AffordanceProposal extensions: policy_rank?, policy_ref?
• WbcTaskSet optional policy_hint — soft additive on FREE / soft refs only
• Who owns policy soft actions vs residual / scene soft feeds vs affordance candidates vs ContactSchedule intent vs measured contact_set vs gait vs recovery (disagree → wait/yield/demote; measured contact wins for cones; recovery and gait outrank schedule holds)
• Admit policy: confidence / geometry / reachability / residual_rank / graph consistency / policy_rank gate CLAIM, never cone attach
• Explicit rule: policy score $\neq$ MEASURED; residual / graph score $\neq$ MEASURED; learned layers never write JointCommand, friction cones, FOC $i_q$, or torque_scale
• Rate / jitter budget: policy ticks at mid / host rates with age_ms; forbidden inside µs / EtherCAT cyclic until jitter measured; closed-loop requires live state (not open-loop replay)
• Mid-policy abort / demote for stale, POWER_GATED, low torque_scale, flicker, residual disagreement, graph inconsistency, policy disagreement, reward-hack attempts, and inference dropout
• Twin hooks: action bias, reward hacking into hard writes, latency / dropout, domain shift, false CLAIM success in sim, residual bias / false edges / MEASURE disagreement, POWER_GATED mid-CLAIM, policy-vs-gait/recovery fights — same as hardware; SimToRealGate blocks free path until GREEN
• Bring-up gate: no free closed-loop learned multi-contact until steps 1–5 above are green

The lesson is not “drop a bigger policy on the robot until the eval reward looks busy.” It is make closed-loop learned policies the same contact-rich program in silicon and in the twin: soft refs and schedule / affordance candidates above the measured hard set, policy score gating CLAIM entry not cones, one estimate, honest power ceilings, measured FOOT/HAND/OBJECT in the hard support set, schedules that wait for measurement and yield to gait/recovery, soft tasks / policy_hints only where soft belongs, inference outside µs loops until jitter is honest, Day 14 never_hard still binding, and abort / demote / SimToRealGate paths that stay honest when the policy, the reward, or the sim lies mid-CLAIM.

Next

Day 16 — Teleop / shared-autonomy arbitration under the same measured ceilings: human teleop and learned policy as co-equal soft writers into refs / CLAIM admit — still no human-or-policy-as-constraint-truth; arbiter yields to gait / recovery / power; twin must keep mid-arbitration abort and sim-to-real honesty before free shared-autonomy multi-contact.