What is "update locking" in parallel computing literature, and how precisely does it map to the backward-pass sequentiality constraint in backpropagation?
Topic question: What is "update locking" in parallel computing literature, and how precisely does it map to the backward-pass sequentiality constraint in backpropagation?
Resolution: "Update locking" is not a term imported from classical parallel/distributed-computing literature (mutexes, two-phase locking, critical sections, lock-step execution). It was coined within the machine-learning literature itself, by Jaderberg et al. (DeepMind, 2016) in "Decoupled Neural Interfaces using Synthetic Gradients," as one of three named "locking" constraints — Forward Locking, Update Locking, and Backward(s) Locking — that the paper uses to describe exactly the sequential dependency structure of standard backpropagation. The term borrows the general connotation of "locking" from concurrent-computing vocabulary evocatively, but the paper does not cite operating-systems, database, or classical-parallel-computing sources when introducing it. Update locking is the general constraint (a module's parameters can't update until everything downstream has finished its forward pass); backward locking is the strictly backprop-specific special case (a module can't update until everything downstream has finished both its forward pass and its backward pass, because backprop's error signal must physically propagate back through the chain rule, layer by layer). This maps precisely onto Hinton's and others' description of backprop's biological-implausibility problem: the requirement that a full forward-then-backward sweep complete, synchronously and in a fixed order, before any weight can update.
Claim 1: Origin and exact definitions of Forward/Update/Backward(s) Locking
Claim type: specific technical-mechanism claim → Tier 1–2 required. Source tier: Tier 1 (arXiv preprint / ICML proceedings paper, authors' own venue).
The terms were introduced by Jaderberg, Czarnecki, Osindero, Vinyals, Graves, Silver, and Kavukcuoglu (DeepMind) in the Introduction of "Decoupled Neural Interfaces using Synthetic Gradients":
"This process results in several forms of locking, namely: (i) Forward Locking – no module can process its incoming data before the previous nodes in the directed forward graph have executed; (ii) Update Locking – no module can be updated before all dependent modules have executed in forwards mode; also, in many credit-assignment algorithms (including backpropagation (Rumelhart et al., 1986)) we have (iii) Backwards Locking – no module can be updated before all dependent modules have executed in both forwards mode and backwards mode."
Source: Jaderberg, M. et al. (2016/2017). "Decoupled Neural Interfaces using Synthetic Gradients." arXiv:1608.05343, Section 1 (Introduction). https://arxiv.org/abs/1608.05343. Verified directly against the paper text (fetched 2026-07-06; ar5iv HTML rendering of the arXiv source, cross-checked against a second independent fetch).
Note on exact wording (verified, not a transcription slip): the paper itself is internally inconsistent between "Forward Locking" (no terminal -s) and "Backwards Locking" (terminal -s) within the same list item, and the very next sentence uses "Forwards, update, and backwards locking" (both with -s, lowercase). Most later secondary literature normalizes this to "forward locking" / "backward locking."
The paper's own summary sentence, immediately following the definitions, states the mapping to sequential execution directly:
"Forwards, update, and backwards locking constrain us to running and updating neural networks in a sequential, synchronous manner."
Source: ibid., Section 1.
Claim 2: The paper's explicit statement of its own goal
Claim type: historical/definitional (what the paper set out to do) → Tier 3-4 acceptable, but recorded here at the Tier 1 primary source directly.
"The goal of this work is to remove update locking for neural networks."
Source: Jaderberg, M. et al. (2016/2017). "Decoupled Neural Interfaces using Synthetic Gradients." arXiv:1608.05343, Section 1. https://arxiv.org/abs/1608.05343.
Later in the same paper, applying the concept concretely to a feed-forward network, the authors write that a given layer i is "update locked to" the downstream sub-network that depends on it, and describe their proposed synthetic-gradient mechanism as a way "to remove the update locking of layer i" to that downstream sub-network.
Source: ibid., Section 2.2 ("Synthetic Gradient for Feed-Forward Networks"). Exact subscript/superscript notation not reproduced here due to unreliable rendering of the math markup through HTML extraction; the prose framing ("update locked to," "to remove the update locking of layer i") is verified verbatim.
Claim 3: What exactly is "locked," and how it distinguishes update locking from backward locking
Claim type: specific technical-mechanism claim → Tier 1–2 required. Source tier: Tier 1 (direct follow-up paper, overlapping authorship with Claim 1's source).
Czarnecki, Świrszcz, Jaderberg, Osindero, Vinyals, and Kavukcuoglu (DeepMind), in the direct follow-up paper formalizing synthetic gradients, spell out the dependency chain that produces the lock:
"An update to layer i can only occur after every subsequent layer f_j, j∈{i+1,…,N} has been computed, the loss L_i has been computed, and the error gradient ∂L/∂h_i^N backpropagated"
and, describing the standard backprop update rule directly:
"An update rule such as this is update locked as it depends on computing L_i, and also backwards locked as it depends on backpropagation to form ∂L_i/∂h_i^n"
Source: Czarnecki, W.M., Świrszcz, G., Jaderberg, M., Osindero, S., Vinyals, O. & Kavukcuoglu, K. (2017). "Understanding Synthetic Gradients and Decoupled Neural Interfaces." arXiv:1703.00522, Section 1 (Introduction) and Section 2. https://arxiv.org/abs/1703.00522. Verified directly against the paper text (fetched 2026-07-06).
This makes the precise distinction explicit: update locking is the broader constraint — no update can happen before the loss (downstream forward computation) is available — and it applies to any credit-assignment scheme, not just backprop. Backward(s) locking is the narrower, backprop-specific case: an update additionally requires the backward error-propagation sweep (the chain-rule computation of the gradient) to have completed through all downstream modules before it reaches the module in question. Backward locking is therefore a strict subset/special case of update locking, produced specifically by backpropagation's particular mechanism (propagating literal error derivatives layer-by-layer backward) rather than by credit assignment in general.
Claim 4: Corroboration that "backward locking" is treated as the operative bottleneck for parallelizing backprop
Claim type: specific technical-mechanism claim → Tier 1–2 required. Source tier: Tier 1 (independent, peer-reviewed ICML 2018 paper, no shared authorship with Claims 1 or 3).
"Therefore, in the backward pass, all layers, or more generally, modules, of the network are locked until their dependencies have executed."
"The backward locking constrains us from updating models in parallel and fully leveraging the computing resources."
Source: Huo, Z., Gu, B., Yang, Q. & Huang, H. (2018). "Decoupled Parallel Backpropagation with Convergence Guarantee." arXiv:1804.10574, Section 1 (Introduction). https://arxiv.org/abs/1804.10574. Verified directly against the paper text (fetched 2026-07-06).
This is independent corroboration, from a separate research group with no author overlap, that the field treats "backward locking" (rather than the broader "update locking") as the specific obstacle to running backpropagation's weight updates in parallel across compute resources.
Claim 5: "Update locking" is explicitly connected to the biological-plausibility discourse in later literature
Claim type: specific technical-mechanism / historical claim → Tier 1–2 required for the mechanism; this is a secondary source, so recorded at Tier 2 and flagged accordingly. Source tier: Tier 2 (named-author survey/comparison paper; secondary, not the originating research group, but a direct discussion of the term in its own venue — arXiv preprint).
"Subsequent research derived from FA has not only facilitated the localization of error signal transmission, but also delved into aspects like update locking, further enhancing its biological plausibility."
Source: Lv, C. et al. (19 co-authors) (2024). "Towards Biologically Plausible Computing: A Comprehensive Comparison." arXiv:2406.16062, Section 3.3 ("Feedback Alignment"). https://arxiv.org/abs/2406.16062. Verified directly against the paper text (fetched 2026-07-06).
This is the clearest available link in the literature between the "update locking" vocabulary and the specific biological-implausibility argument (the same broad discourse as Geoffrey Hinton's critique of backprop) — later researchers discussing biological plausibility explicitly name "update locking" as one of the technical properties subsequent (Feedback-Alignment-descended) work has tried to address. This connection is thin — one sentence in a survey paper — and does not itself constitute Hinton citing or using the term; it establishes only that the concept-cluster is part of the same technical conversation. [flagged: this specific link to Hinton's own critique is not established — Hinton's own writing (see the 2007-talk and 2022 Forward-Forward captures already in this vault) does not use the phrase "update locking" or "forward locking"; those captures ground his objection in different language ("two different types of signal," "no convincing evidence that cortex explicitly propagates error derivatives"). The "locking" taxonomy and Hinton's biological-implausibility argument are two independent literatures describing an overlapping/related structural fact about backprop, not one citing the other.]
Claim 6: "Update locking" was coined in the ML literature, not imported from classical parallel/distributed-computing terminology
Claim type: historical/definitional claim about terminological provenance → Tier 3-4 acceptable when uncontested; escalated here to Tier 1 since it is load-bearing for the note's central question. Source tier: Tier 1 (direct inspection of the primary source's own text and citation context).
Direct inspection of the Jaderberg et al. (2016) Introduction shows no citations to operating-systems, database-concurrency, or classical-parallel-computing literature anywhere near the locking definitions (the only citation embedded in the definitions is to Rumelhart et al. 1986, for backpropagation itself — see Claim 1's full quote). Confirmed by two independent fetches of the paper text (2026-07-06):
"No, [confirmed via direct reading]. The paper does not cite classical operating systems, database, or parallel computing literature (e.g., mutex, critical section, or two-phase locking references). The work frames the problem in neural network terms without drawing on established concurrency control mechanisms from computer systems literature."
Source: direct textual audit of Jaderberg, M. et al. (2016/2017), arXiv:1608.05343, full Introduction and reference list. https://arxiv.org/abs/1608.05343.
Separately, "update lock" (the "U-lock") is a real, established, and unrelated term in database concurrency-control literature — an intermediate lock mode (between shared and exclusive) used to prevent a specific deadlock pattern during read-then-write transactions:
"The Database Engine places update (U) locks as it prepares to execute an update. U locks are compatible with S locks, but only one transaction can hold a U lock at a time on a given resource."
Source: Microsoft, "Transaction Locking and Row Versioning Guide," SQL Server documentation, "Update locks" subsection. https://learn.microsoft.com/en-us/sql/relational-databases/sql-server-transaction-locking-and-row-versioning-guide. Tier 3 (vendor product documentation; definitional claim, floor met). This is recorded only to establish that the phrase "update lock" pre-exists in classical concurrent-systems terminology with a different meaning (deadlock avoidance in transactional databases), and there is no evidence Jaderberg et al. were referencing this specific concept — the overlap in phrasing appears to be coincidental rather than a deliberate borrowed analogy.
Additionally, no evidence was found that the pipeline-parallelism literature for training large deep networks (e.g., GPipe, PipeDream) uses "locking" vocabulary at all for the analogous stall/dependency problem in that setting — that literature instead uses terms like "bubble" (pipeline bubble), "stall," and "staleness." [unverified-historical — this is a negative/absence finding from search rather than a specific citable source confirming GPipe/PipeDream's terminology choices; flagged for independent verification against the GPipe and PipeDream papers directly if this distinction becomes load-bearing.]
Synthesis: the precise mapping to backpropagation's sequentiality constraint
Per Jaderberg et al.'s own taxonomy (Claim 1) and Czarnecki et al.'s formalization (Claim 3), the "locking" vocabulary maps onto backprop's structure as follows:
- Forward locking is a property of any feedforward computation (not learning-specific): a layer can't compute its own activations until the layers before it have.
- Update locking is the general credit-assignment constraint: a layer's parameters can't change until the network's downstream forward computation (through to wherever a loss or evaluation signal is produced) has resolved — because until then there is nothing to compute an update from.
- Backward(s) locking is backpropagation's specific, stricter version of this: not only must the downstream forward pass finish, the downstream backward pass (the chain-rule propagation of ∂Loss/∂activation back through every intervening layer) must also finish, because that is literally how the layer's gradient is computed in standard backprop.
This is the same structural fact that shows up, in different vocabulary, in the biological-implausibility literature already captured elsewhere in this vault: Hinton's 2022 Forward-Forward paper objects that there is "no convincing evidence that cortex explicitly propagates error derivatives or stores neural activities for use in a subsequent backward pass" (see the existing capture on Hinton's 2007/2022 arguments) — i.e., that the brain has no apparent mechanism for the backward-locked, synchronous, sequential sweep that Jaderberg et al.'s taxonomy names precisely as "backward locking." The two literatures (synthetic-gradients/parallel-training-efficiency, and biological-plausibility) are independently motivated — one aims at compute efficiency and hardware parallelism, the other at neuroscientific realism — but they are pointing at the identical structural bottleneck in standard backpropagation.
Further leads
- Czarnecki et al. 2017, Section 2 ("Synthetic gradients in operation") describes the resulting capability once update locking is removed: "This unlocked ability of being able to update parts of a neural network asynchronously and with only local information" — worth a follow-up capture on what synthetic gradients are and how DNI (Decoupled Neural Interfaces) mechanically breaks the lock, since this note only established the definition and mapping, not the proposed solution.
- "Decoupled Greedy Learning of CNNs" (arXiv:1901.08164) — a recurring paraphrase across multiple secondary sources ("Jaderberg et al., 2017 is the first to directly consider the update or forward locking problems") could not be pinned to one verbatim, page-numbered quote during this research pass. If a note needs that specific attribution sentence quoted directly, it should be independently verified against that paper (or wherever the paraphrase actually originates) rather than inherited from this capture.
[unverified — attribution sentence recurs across secondary sources but exact origin not confirmed] - GPipe / PipeDream terminology ("bubble," "stall," "staleness" rather than "locking") is recorded here as a negative/contrastive finding from general web search, not from a direct read of either paper. Worth a dedicated verification pass against the primary GPipe (Huang et al. 2019) and PipeDream (Narayanan et al. 2019) papers if the vault wants to make a firm claim about vocabulary divergence across the pipeline-parallelism vs. synthetic-gradients lineages.
Capture produced: 2026-07-06. Primary sources verified directly via WebFetch against the ar5iv HTML rendering of arXiv:1608.05343, arXiv:1703.00522, arXiv:1804.10574, and arXiv:2406.16062 (all confirmed resolving 2026-07-06), cross-checked against an independent background research pass. PDF extraction via extract_pdf was attempted for the primary source but the tool returned a "Stream closed" error on repeated attempts; verbatim quotes were instead independently confirmed via two separate WebFetch passes against the arXiv abstract pages and the ar5iv HTML mirror, which render the same LaTeX source as the PDF.