Update locking names backprop's sequential constraint: a layer cannot update until the full forward and backward passes complete

update-lockingbackpropagationsynthetic-gradientsparallelismdeepmind

The synthetic-gradients paper formalizes three locks backpropagation imposes, in its own words (§1): "(i) Forward Locking – no module can process its incoming data before the previous nodes in the directed forward graph have executed; (ii) Update Locking – no module can be updated before all dependent modules have executed in forwards mode; also, in many credit-assignment algorithms (including backpropagation (Rumelhart et al., 1986)) we have (iii) Backwards Locking – no module can be updated before all dependent modules have executed in both forwards mode and backwards mode." In that taxonomy the tightest lock — every weight waiting on the entire round trip through the backward pass — is backwards locking; update locking is the forward-only half of it. The paper nonetheless names the composite constraint "update locking" in its abstract ("must wait for the remainder of the network to execute forwards and propagate error backwards before they can be updated") and its stated goal ("The goal of this work is to remove update locking for neural networks. This is achieved by removing backpropagation."), which is the sense this note's title carries. (2026-09-11 audit: the promotion text had the two definitions swapped — "update locking (a module can't update until the full forward pass AND the backward pass that depends on it have run)… backward locking (it can't update until the backward pass reaches it). Update locking is the tightest" — corrected above against the paper's §1 text; the claim itself is unchanged.)

This is the computational cost of the backward pass, where claim-training-inference-compute-asymmetry-mechanism gave the arithmetic cost — the same backward pass that makes training ~3x inference is also what serializes it. Synthetic gradients break the lock by having each layer predict its own incoming gradient, decoupling updates. It sits alongside feedback alignment and Forward-Forward as attacks on backprop's structure, but from the systems/parallelism angle rather than the biological one — the lock is why the backward pass is hard to distribute, which connects the origins cluster to the inference-economics cluster's hardware concerns. See moc-backpropagation-origins, claim-hinton-biological-implausibility-four-objections.