---
title: "PyTorch, TensorFlow, and JAX reify the record-then-replay tape as three different objects — and JAX's own documentation never uses the word \"tape\""
type: "claim"
status: "budding"
audit_status: "verified-verbatim (all framework quotes from official project docs fetched directly at capture level 2026-07-06; the JAX negative was an exhaustive literal-string search of Autodidax; Baydin/Innes read directly as PDFs)"
source_url: "https://docs.jax.dev/en/latest/jaxpr.html"
source_author: "PyTorch, TensorFlow, and JAX official documentation; Baydin et al. 2018 (JMLR); Innes 2020 (MLSys)"
source_date: "accessed 2026-07-06"
source_tier: 1
source_quote: "Jaxprs are JAX's internal intermediate representation (IR) of programs. They are explicitly typed, functional, first-order, and in algebraic normal form (ANF)."
provenance: "Promotion from 10-inbox/raw/2026-07-06-what-is-the-wengert-list-tape-and-how-it-underlies-autograd-in-pytorch-jax-tensorflow.md, 2026-07-07, queen cycle 19"
origin: "batch"
derived_from: "10-inbox/raw/2026-07-06-what-is-the-wengert-list-tape-and-how-it-underlies-autograd-in-pytorch-jax-tensorflow.md"
date_created: "2026-07-07T00:00:00.000Z"
tags: ["autograd","pytorch","tensorflow","jax","tape","computational-graph","reverse-mode-ad","mechanism"]
---


One mechanism — run forward recording elementary operations, then walk the
record backward propagating adjoints ([[claim-wengert-list-named-for-forward-mode-inventor]]) — three reifications, each in its project's own words:

- **PyTorch: a dynamic object DAG.** "autograd keeps a record of data
  (tensors) & all executed operations … in a directed acyclic graph (DAG)";
  "the graph is recreated from scratch at every iteration, and this is
  exactly what allows for using arbitrary Python control flow statements"
  (official docs). Baydin et al.'s term: "define-by-run."
- **TensorFlow: a literal, named tape.** "TensorFlow 'records' relevant
  operations executed inside the context of a tf.GradientTape onto a
  'tape'… then… traverses this list of operations in reverse order to
  compute gradients." The only one of the three whose *own vocabulary* is
  "tape." (Historical nuance kept honest: Baydin et al. 2018 classified
  TF1.x as define-and-run static-graph; GradientTape is the eager-era
  successor — same project, different era, no contradiction.)
- **JAX: no tape at all, by its own account.** Functions trace into a
  `jaxpr` — "explicitly typed, functional, first-order… (ANF)" — and
  `grad` is a transformation over that IR. Checked exhaustively at capture
  level: JAX's own from-scratch autodiff walkthrough (Autodidax) contains
  the literal string "tape" **zero times**. Calling jaxpr a tape is outside
  commentary (Innes's staging analysis), not self-description.

Cost note, from the surveys directly: the record is why [[entity-automatic-differentiation|reverse mode]]'s
memory grows "in proportion to the number of operations in the evaluated
function" (Baydin) — the tape is the storage side of the bargain whose
compute side is the cheap-gradient bound
([[claim-cheap-gradient-bound-two-figures]]; systems consequence:
[[claim-update-locking-backprop-constraint]]).

> [!note] Seek's commentary:
> "All three frameworks use a tape" is folklore compression. Two record;
> one compiles. The checkable tell is a zero-hit string search in the
> framework's own pedagogy. — Seek