---
id: "20260706-0217-what-is-the-wengert"
title: "Capture: What is the Wengert list (tape), and how does it underlie the autograd graph implementations in PyTorch, JAX, and TensorFlow?"
type: "capture"
status: "promoted"
promoted_to: ["30-notes/claim-wengert-list-named-for-forward-mode-inventor.md (claims 1–2; further-lead resolved from queen cache: Griewank 2012 never mentions Wengert, naming point rests on Baydin/Innes)","30-notes/claim-autograd-three-reifications-of-the-tape.md (claims 4–6 + synthesis; claim 3's storage-scaling folded in as the cost note)"]
not_promoted: ["ADOL-C self-description upgrade and TF-eager (Agrawal 2019) direct read — remain further leads in this capture; promote if either becomes load-bearing","Innes/Zygote as a standalone source note — real lead, but out of tonight's safelist scope; flagged for a future queue topic"]
promotion_note: "Queen cycle 19, 2026-07-07."
origin: "batch"
model: "claude-sonnet-5"
date_created: "2026-07-06T00:00:00.000Z"
provenance: "batch run 2026-07-06; harvested from 2026-06-29-what-is-the-reverse-mode-automatic-differentiation-backpropagation-correspondence-stated-precisely, whose 'Further leads' section explicitly flagged this as 'a good candidate for a dedicated note connecting historical reverse-mode AD to current ML framework implementation.' Researched via WebSearch + WebFetch; the two load-bearing Tier 1 sources (Baydin et al. 2018 JMLR survey; Innes 2020 MLSys 'Sense & Sensitivities' paper) were both obtained as full PDFs and read directly, not relayed through a WebFetch summarizer. PyTorch, TensorFlow, and JAX quotes are from each project's own official documentation, fetched directly."
derived_from: []
tags: ["automatic-differentiation","wengert-list","tape","autograd","computational-graph","pytorch","tensorflow","jax","jaxpr","reverse-mode-ad","dynamic-computational-graph","differentiable-programming"]
source_url: "https://jmlr.org/papers/volume18/17-468/17-468.pdf"
source_author: "Atılım Güneş Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, Jeffrey Mark Siskind"
source_date: "2018 (JMLR 18(153):1-43); arXiv:1502.05767 first posted 2015"
source_tier: 1
other_sources: [{"url":"https://proceedings.mlsys.org/paper_files/paper/2020/file/b422680f3db0986ddd7f8f126baaf0fa-Paper.pdf","author":"Michael J. Innes","date":"2020, Proceedings of the 3rd MLSys Conference","note":"'Sense & Sensitivities: The Path to General-Purpose Algorithmic Differentiation' (the Zygote/Julia AD paper). Read directly as PDF, not via WebFetch summarization. Load-bearing for the tape/Wengert-list/graph equivalence claim and for the JAX-specific staging discussion.","tier":1},{"url":"https://docs.pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html","note":"Official PyTorch tutorial, 'A Gentle Introduction to torch.autograd,' fetched directly. Accessed 2026-07-06.","tier":1},{"url":"https://docs.pytorch.org/docs/2.12/notes/autograd.html","note":"Official PyTorch docs, 'Autograd mechanics,' fetched directly. Accessed 2026-07-06.","tier":1},{"url":"https://www.tensorflow.org/guide/autodiff","note":"Official TensorFlow guide, 'Introduction to gradients and automatic differentiation,' fetched directly. Accessed 2026-07-06.","tier":1},{"url":"https://docs.jax.dev/en/latest/jaxpr.html","note":"Official JAX docs, 'JAX internals: The jaxpr language,' fetched directly. Accessed 2026-07-06.","tier":1},{"url":"https://docs.jax.dev/en/latest/autodidax.html","note":"Official JAX docs, 'Autodidax: JAX core from scratch' — searched in full for the literal string 'tape'; zero occurrences found. Accessed 2026-07-06.","tier":1},{"url":"https://en.wikipedia.org/wiki/Automatic_differentiation","note":"corroborating/pointer source for the historical naming mismatch (Wengert = forward mode, 'Wengert list' colloquially attached to reverse-mode tape structures), accessed 2026-07-06. Per the 2026-07-06 Cali ruling in 00-meta/sources.md, treated strictly as a pointer to primary sources, not as evidence in its own right — the load-bearing version of this claim below rests on Baydin et al. and Innes directly.","tier":4}]
---


This capture answers a question flagged as a follow-on from [[claim-linnainmaa-reverse-mode-single-pass]] and the reverse-mode-AD/backpropagation correspondence work: what exactly is a "Wengert list" or "tape," and how does that one historical idea show up — in three visibly different engineering forms — inside PyTorch, TensorFlow, and JAX.

---

## Core question: What is a Wengert list (tape), and how does it underlie autograd in PyTorch, JAX, and TensorFlow?

**Short answer, established below claim by claim:** A Wengert list is the recorded sequence of elementary operations (an "evaluation trace") produced when a composite function is broken into intermediate steps — the substrate both forward- and reverse-mode automatic differentiation operate over. The term traces to Robert Wengert's 1964 paper, though Wengert's own method was forward-mode; the "tape" terminology and its specific use for storing a forward pass to replay backward is a later attachment to his name. All three major deep-learning frameworks implement some version of "record the forward computation, then walk it backward," but they reify the idea as three structurally different objects: PyTorch's dynamically rebuilt object graph, TensorFlow's explicitly named `tf.GradientTape`, and JAX's `jaxpr` — a functional intermediate representation produced by tracing, which is a "tape" only in a loose conceptual sense and not in JAX's own vocabulary.

---

## Claim 1: A Wengert list is the evaluation trace of elementary operations produced by decomposing a function into intermediate variables — the basic representation AD techniques operate on

**Claim type:** definitional
**Sourcing floor:** Tier 3–4 acceptable; achieved Tier 1 (read directly from the primary PDF)

Baydin, Pearlmutter, Radul & Siskind's JMLR survey introduces the term directly, describing the computation y = f(x₁,x₂) = ln(x₁) + x₁x₂ − sin(x₂) broken into intermediate variables vᵢ:

> "On the left hand side of Table 2 we see the representation of the computation y = f(x1, x2) = ln(x1) + x1x2 − sin(x2) as an evaluation trace of elementary operations—also called a Wengert list (Wengert, 1964). We adopt the three-part notation used by Griewank and Walther (2008), where a function f : Rⁿ → Rᵐ is constructed using intermediate variables vi such that... variables vi−n = xi... are the input variables, variables vi... are the working (intermediate) variables, and variables ym−i = vl−i... are the output variables."

Innes's Zygote paper (MLSys 2020) gives the same object a slightly broader label, explicitly equating three names for it:

> "This form can be viewed as a simple programming language; it is often referred to as a Wengert list, tape or graph (Bartholomew-Biggs et al., 2000)."

And states directly that this structure is the shared substrate of all reverse-mode AD implementations:

> "This section outlines our proposed derivative transform for SSA form code, building on the Wengert list transformation that all reverse-mode ADs use at a minimum."

**Provenance:**
- source: Baydin, Pearlmutter, Radul & Siskind, "Automatic Differentiation in Machine Learning: a Survey," JMLR 18 (2018), pp. 1–43. https://jmlr.org/papers/volume18/17-468/17-468.pdf. Tier 1. Read directly as PDF.
- corroborating source: Innes, "Sense & Sensitivities: The Path to General-Purpose Algorithmic Differentiation," Proceedings of the 3rd MLSys Conference (2020). https://proceedings.mlsys.org/paper_files/paper/2020/file/b422680f3db0986ddd7f8f126baaf0fa-Paper.pdf. Tier 1. Read directly as PDF.
- related existing notes: [[claim-linnainmaa-reverse-mode-single-pass]]

---

## Claim 2: The term traces to Robert Wengert's 1964 paper, which described forward-mode differentiation — the "tape" data structure that reverse-mode AD tools carry the name "Wengert list" for a mode Wengert himself did not publish

**Claim type:** historical — escalated to Tier 1–2 given the specificity of the attribution (which technique Wengert actually published); achieved Tier 1
**Sourcing floor:** Tier 3–4 acceptable for uncontested historical claims, but this one is escalated per the rubric because it is a load-bearing, slightly counterintuitive point (the list is named for someone who published the *other* mode of AD)

Baydin et al. state the origin and the mode directly, in their dedicated history section:

> "Ideas underlying AD date back to the 1950s (Nolan, 1953; Beda et al., 1959). Forward mode AD as a general method for evaluating partial derivatives was essentially discovered by Wengert (1964). It was followed by a period of relatively low activity, until interest in the field was revived in the 1980s mostly through the work of Griewank (1989)..."

They separately identify reverse mode's automatic-implementation origin as later and distinct:

> "Speelpenning (1980) subsequently introduced reverse mode AD as we know it, in the sense that he gave the first implementation that was actually automatic, accepting a specification of a computational process written in a general-purpose programming language and automatically performing the reverse mode transformation."

Innes's Zygote paper states the same forward/reverse split in its opening sentence:

> "Its forward (Wengert, 1964) and later reverse (Speelpenning, 1980) modes were first developed for scientific computing, in languages like Fortran."

Wikipedia's "Automatic differentiation" article states the naming-mismatch point more explicitly than either Tier 1 source does — worth recording as a pointer, not as the evidentiary basis, per the sourcing floor: "a data structure known as a 'tape' or a Wengert list (however, Wengert published forward accumulation, not reverse accumulation..." This is consistent with, but not needed to establish, the Tier 1 finding above.

**Provenance:**
- source: Baydin et al. (2018), Section 3.3 "Origins of AD and Backpropagation," pp. 13–14. Tier 1. Read directly as PDF.
- corroborating source: Innes (2020), p. 1 (Introduction). Tier 1. Read directly as PDF.
- pointer/corroborating source (not load-bearing): Wikipedia, "Automatic differentiation," https://en.wikipedia.org/wiki/Automatic_differentiation, accessed 2026-07-06. Tier 4.
- bibliographic identity of the original paper (uncontested, definitional — Tier 3–4 sufficient): Robert E. Wengert, "A simple automatic derivative evaluation program," *Communications of the ACM* 7(8):463–464, 1964 — confirmed directly in both Tier 1 sources' own reference lists.

---

## Claim 3: The Wengert list/tape operates as a two-phase mechanism — forward pass records the computation and its dependencies; backward pass replays that record in reverse, propagating adjoints via the chain rule — and its storage cost scales with the number of recorded operations

**Claim type:** technical-mechanism (with an embedded quantitative/scaling claim)
**Sourcing floor:** Tier 1–2 required; achieved Tier 1

Baydin et al. describe the two-phase structure directly:

> "In reverse mode AD, derivatives are computed in the second phase of a two-phase process. In the first phase, the original function code is run forward, populating intermediate variables vi and recording the dependencies in the computational graph through a bookkeeping procedure. In the second phase, derivatives are calculated by propagating adjoints v̄i in reverse, from the outputs to the inputs."

They state the storage-scaling cost of keeping this record directly:

> "The advantages of reverse mode AD, however, come with the cost of increased storage requirements growing (in the worst case) in proportion to the number of operations in the evaluated function."

Innes's paper independently states the same scaling point when contrasting reverse mode with forward mode:

> "...Julia's forward-mode AD (Revels et al., 2016) has constant memory overhead (compared to reverse mode's tape, linear in the number of instructions executed) and has minimal time overhead..."

As a concrete illustration of what "recording onto a tape, then playing it back" means in an actual implementation (not one of the three target frameworks, but the generic mechanism they all descend from), Baydin et al. describe ADOL-C, a C++ AD tool built on operator overloading:

> "ADOL-C requires the use of AD-enabled types for variables, and records arithmetic operations on variables in tape data structures, which can subsequently be 'played back' during reverse mode AD computations."

**Provenance:**
- source: Baydin et al. (2018), Section 3.2 "Reverse Mode" (p. 12) and Section 5.3 "Operator Overloading" (p. 26). Tier 1. Read directly as PDF.
- corroborating source: Innes (2020), Section 3.4 "Mixed-Mode AD," p. 6–7. Tier 1. Read directly as PDF.
- related existing notes: [[claim-linnainmaa-reverse-mode-single-pass]]

---

## Claim 4: PyTorch implements the tape/Wengert-list idea as a dynamic, "define-by-run" computational graph — a DAG of tensors and gradient functions built by operator overloading as the program actually executes, and rebuilt from scratch on every forward pass

**Claim type:** technical-mechanism
**Sourcing floor:** Tier 1–2 required; achieved Tier 1

Baydin et al. place PyTorch (alongside autograd and Chainer) in the "define-by-run"/"dynamic computational graph" lineage, contrasted with the older "define-and-run" graph-building style:

> "...the terms define-by-run and dynamic computational graph refer to the general-purpose AD capability available in newer PyTorch-like systems where a model is a regular program in the host programming language, whose execution dynamically constructs a computational graph on-the-fly that can freely change in each iteration."

PyTorch's own official documentation describes the mechanism in matching terms. From the official tutorial "A Gentle Introduction to torch.autograd":

> "autograd keeps a record of data (tensors) & all executed operations (along with the resulting new tensors) in a directed acyclic graph (DAG)... In a forward pass, autograd does two things simultaneously: run the requested operation to compute a resulting tensor, and maintain the operation's gradient function in the DAG."

From the official "Autograd mechanics" notes:

> "autograd records a graph recording all of the operations that created the data as you execute operations, giving you a directed acyclic graph whose leaves are the input tensors and roots are the output tensors... By tracing this graph from roots to leaves, you can automatically compute the gradients using the chain rule."

> "DAGs are dynamic in PyTorch... the graph is recreated from scratch at every iteration, and this is exactly what allows for using arbitrary Python control flow statements."

**Provenance:**
- source: Baydin et al. (2018), Section 4.2, p. 18. Tier 1. Read directly as PDF.
- corroborating sources: PyTorch official docs — "A Gentle Introduction to torch.autograd," https://docs.pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html, and "Autograd mechanics," https://docs.pytorch.org/docs/2.12/notes/autograd.html. Both Tier 1 (official project documentation), fetched directly, accessed 2026-07-06.

---

## Claim 5: TensorFlow implements the tape concept explicitly and by name — `tf.GradientTape` "records" operations executed in its context during the forward pass onto a tape object, then the backward pass traverses that recorded tape in reverse to compute gradients

**Claim type:** technical-mechanism
**Sourcing floor:** Tier 1–2 required; achieved Tier 1

TensorFlow's official guide, "Introduction to gradients and automatic differentiation," states the mechanism directly and is the one of the three frameworks that uses the word "tape" as its own vocabulary, not just as a description supplied by outside commentators:

> "TensorFlow 'records' relevant operations executed inside the context of a tf.GradientTape onto a 'tape.'"

> "To differentiate automatically, TensorFlow needs to remember what operations happen in what order during the forward pass."

> "TensorFlow then uses that tape to compute the gradients of a 'recorded' computation using reverse mode differentiation."

> "Then, during the backward pass, TensorFlow traverses this list of operations in reverse order to compute gradients."

**Historical nuance worth flagging:** Baydin et al.'s 2018 survey characterizes TensorFlow, at the time of that paper's writing, as belonging to the *other* camp — the "define-and-run"/"static computational graph" systems (grouped with Theano, Caffe, and CNTK), contrasted directly with PyTorch's dynamic-graph approach: "In mainstream frameworks including Theano (Bastien et al., 2012), TensorFlow (Abadi et al., 2016), Caffe (Jia et al., 2014), and CNTK (Seide and Agarwal, 2016) the user first constructs a model as a computational graph using a domain-specific mini language, which then gets interpreted by the framework during execution." This reflects TensorFlow's original (TF1.x) graph-then-session execution model. The `tf.GradientTape` mechanism quoted above is TensorFlow's later, eager-execution-era, explicitly tape-named successor — the two descriptions are not in tension, they describe the same project at different points in its history, roughly bridged by the "TensorFlow eager" transition that Baydin et al. cite in passing elsewhere in the same paper (Agrawal et al., 2019, "Tensorflow eager: A multi-stage, python-embedded dsl for machine learning").

**Provenance:**
- source: TensorFlow official guide, "Introduction to gradients and automatic differentiation," https://www.tensorflow.org/guide/autodiff. Tier 1 (official project documentation), fetched directly, accessed 2026-07-06.
- corroborating/contextual source: Baydin et al. (2018), Section 4.2, p. 17–18. Tier 1. Read directly as PDF.

---

## Claim 6: JAX does not implement a literal "tape" object; it traces the Python function into a `jaxpr` — a functional, statically-typed intermediate representation — and defines its forward- and reverse-mode transformations over that IR rather than over a mutable recorded operation list

**Claim type:** technical-mechanism
**Sourcing floor:** Tier 1–2 required; achieved Tier 1

JAX's own internals documentation defines the object its transformations actually operate on:

> "Jaxprs are JAX's internal intermediate representation (IR) of programs. They are explicitly typed, functional, first-order, and in algebraic normal form (ANF)."

> "Conceptually, one can think of JAX transformations, such as jax.jit() or jax.grad(), as first trace-specializing the Python function to be transformed into a small and well-behaved intermediate form that is then interpreted with transformation-specific interpretation rules."

As a direct check on whether JAX's own documentation uses "tape" language at all, "Autodidax: JAX core from scratch" — JAX's own from-first-principles walkthrough of how its autodiff is implemented — was searched in full for the literal word "tape." It does not appear anywhere in that document, across sections covering both forward-mode (JVP) and reverse-mode (VJP/partial evaluation into a jaxpr) differentiation.

Innes's Zygote paper independently and directly discusses JAX's design in these terms, describing it as a "staged" approach distinct from tape-recording:

> "In a recent system like JAX (Frostig et al., 2018), the Python interpreter can be viewed as taking on the role of compiler. Like Julia's abstract interpreter, it evaluates code with partial information (types) in order to statically resolve polymorphic methods, and then takes a back seat for program runtime."

> "But the trace need not be a pure Wengert list, and 'staging' more of the host language's constructs (for example, control flow) into the trace makes this look increasingly like source-to-source AD. Many state-of-the-art ML systems, which originally took very different approaches to AD, are converging towards these staged programming approaches (Agrawal et al., 2019; PyTorch Team, 2018; Frostig et al., 2018)."

**Provenance:**
- source: JAX official docs, "JAX internals: The jaxpr language," https://docs.jax.dev/en/latest/jaxpr.html. Tier 1 (official project documentation), fetched directly, accessed 2026-07-06.
- corroborating source (absence-of-term check): JAX official docs, "Autodidax: JAX core from scratch," https://docs.jax.dev/en/latest/autodidax.html. Tier 1, fetched directly and searched in full, accessed 2026-07-06.
- corroborating source: Innes (2020), Section 1.1 "Convergence in AD Design," pp. 1–2. Tier 1. Read directly as PDF.

---

## Synthesis: one idea, three different reifications

Across all three frameworks, the same underlying idea persists: record the sequence of elementary operations performed on the way to computing an output (a Wengert list, in Wengert's and Baydin et al.'s terminology), then traverse that record backward, applying the chain rule at each step to propagate adjoints. What differs is what kind of object each framework actually builds to hold that record:

- **PyTorch** builds a mutable, object-based DAG of tensors and `grad_fn` gradient functions, assembled by operator overloading as ordinary Python code executes, and discarded and rebuilt from scratch on every forward pass ([[claim-linnainmaa-reverse-mode-single-pass]]'s single-backward-sweep structure, reified at Python-object granularity).
- **TensorFlow** builds an explicitly named `tf.GradientTape` object — the one of the three that uses "tape" as its own vocabulary, not an outside description — which records operations during a `with` block and is played back in reverse when a gradient is requested.
- **JAX** builds no literal tape at all in its own documentation's own words; it traces a function into a `jaxpr`, a functional, immutable intermediate representation, and defines `jvp`/`vjp` as transformations over that representation. Per Innes's independent, direct description, this can be understood as "forward tape + backward player" *in conceptual structure* — but that framing is a secondary gloss (Tier 1, but not JAX's own language), not JAX's own self-description, which never uses the word "tape."

None of the three claims above required stretching any single source past what it actually says; the interesting finding is closer to a gap than a discovery — the "Wengert list" label is doing real historical work (it names the shared ancestor concept) while doing comparatively little descriptive work for at least one of the three frameworks it is sometimes casually applied to.

---

## Further leads

- Griewank's 2012 Documenta Mathematica history paper ("Who Invented the Reverse Mode of Differentiation?"), already used elsewhere in the vault for the Linnainmaa cluster, was not re-consulted here for the Wengert-specific naming question; it would be worth checking directly whether Griewank himself comments on the "Wengert list" label being retroactively attached to a reverse-mode data structure, since this capture currently rests that specific framing on Wikipedia (Tier 4, non-load-bearing) rather than a primary source.
- ADOL-C's own documentation/papers (Walther and Griewank, 2012) were not read directly here — only described via Baydin et al.'s secondary summary — and would let a future capture upgrade the "tape data structures... played back" quote to a source about ADOL-C's own self-description rather than a survey's description of it.
- The TensorFlow "eager execution" transition (Agrawal et al., 2019, "TensorFlow Eager: A multi-stage, Python-embedded DSL for machine learning") is cited only in passing here via Baydin et al.'s bibliography; reading that paper directly would sharpen exactly when and how TensorFlow moved from the "define-and-run" graph style Baydin et al. describe to the tape-based `tf.GradientTape` style documented today.
- Innes's Zygote/MLSys paper is a substantial, previously-unused-in-this-vault Tier 1 primary source on AD implementation design generally (SSA-form differentiation, control-flow handling, checkpointing, complex/higher-order differentiation) — worth a dedicated note or two beyond the Wengert-list-specific material extracted here.

> [!note] Seek's commentary:
> The neatest part of this chain is the JAX result, and it's a negative result: searching JAX's own from-scratch explanation of its own autodiff for the word "tape" and finding it zero times. That's a small, checkable fact that cuts against the loose popular habit (visible even in this vault's own earlier captures and in casual ML writing generally) of saying "PyTorch, TensorFlow, and JAX all use a tape" as if it were one uniform mechanism with three names. Two of them do, in different senses — PyTorch's dynamic DAG and TensorFlow's literally-named GradientTape both really are a record-then-replay structure. JAX's jaxpr is a different kind of object (immutable, functional, produced by tracing into an IR that also feeds `jit`), and the "tape" framing for it comes entirely from outside commentary (Innes, here) rather than from JAX's own documentation. Worth being precise about which claim is actually resting on a name and which is resting on a mechanism.