PyTorch, TensorFlow, and JAX reify the record-then-replay tape as three different objects — and JAX's own documentation never uses the word "tape"

One mechanism — run forward recording elementary operations, then walk the record backward propagating adjoints (claim-wengert-list-named-for-forward-mode-inventor) — three reifications, each in its project's own words:

• PyTorch: a dynamic object DAG. "autograd keeps a record of data (tensors) & all executed operations … in a directed acyclic graph (DAG)"; "the graph is recreated from scratch at every iteration, and this is exactly what allows for using arbitrary Python control flow statements" (official docs). Baydin et al.'s term: "define-by-run."
• TensorFlow: a literal, named tape. "TensorFlow 'records' relevant operations executed inside the context of a tf.GradientTape onto a 'tape'… then… traverses this list of operations in reverse order to compute gradients." The only one of the three whose own vocabulary is "tape." (Historical nuance kept honest: Baydin et al. 2018 classified TF1.x as define-and-run static-graph; GradientTape is the eager-era successor — same project, different era, no contradiction.)
• JAX: no tape at all, by its own account. Functions trace into a jaxpr — "explicitly typed, functional, first-order… (ANF)" — and grad is a transformation over that IR. Checked exhaustively at capture level: JAX's own from-scratch autodiff walkthrough (Autodidax) contains the literal string "tape" zero times. Calling jaxpr a tape is outside commentary (Innes's staging analysis), not self-description.

Cost note, from the surveys directly: the record is why reverse mode's memory grows "in proportion to the number of operations in the evaluated function" (Baydin) — the tape is the storage side of the bargain whose compute side is the cheap-gradient bound (claim-cheap-gradient-bound-two-figures; systems consequence: claim-update-locking-backprop-constraint).