---
id: "20260713-0234-is-catastrophic-vs-graceful"
title: "Is 'catastrophic vs. graceful (power-law) forgetting' a real regime distinction determined by task geometry, and is Kline's MNIST drop-8 setup correctly classed as the graceful regime?"
type: "capture"
status: "promoted"
origin: "batch"
writer_model: "claude-sonnet-5"
date_created: "2026-07-13T00:00:00.000Z"
provenance: "batch run, 2026-07-13"
derived_from: []
routed_from: "50-questions/question-catastrophic-vs-graceful-forgetting-regime.md"
tags: ["catastrophic-forgetting","continual-learning","forgetting-curve","power-law","neuro-AI-parallel","representational-overlap","sourcing-floor"]
source_urls: ["https://arxiv.org/pdf/2506.12034","https://www.cs.swarthmore.edu/~meeden/DevelopmentalRobotics/cat_forget.pdf"]
source_authors: ["Dylan Kline (University of Rochester)","Robert M. French (University of Liège)"]
source_dates: ["2025-06-19 (Kline, v2 on arXiv)","1999 (French, Trends in Cognitive Sciences 3(4), 128-135)"]
source_tiers: ["Tier 1 — Kline's own paper, own experiments, read in full via extract_pdf (tls: verified)","Tier 1 — French's own review article, exact published text (TICS 1999), mirrored as a course PDF at cs.swarthmore.edu but verbatim against the citation header; read in full via extract_pdf (tls: verified)"]
promoted: "2026-07-18 by claude-opus-4-8"
promoted_to: ["[[claim-catastrophic-forgetting-is-a-structure-dependent-regime-not-universal]] — claim 1 (the regime distinction is established connectionist literature, not a universal property); budding, French 1999 Tier 1","[[claim-representational-overlap-determines-catastrophic-vs-graceful-forgetting]] — claim 2 (representational overlap is the proposed determinant); seedling, French 1999 Tier 1","[[claim-klines-mnist-drop-8-is-single-task-drift-not-a-disjoint-task-switch]] — claim 3 (Kline's setup is single-task drift, correctly classed graceful); seedling, Kline 2025 Tier 1. Claim 4 (Kline's feature-overlap explanation for the ~0.18 floor) folded in as the note's floor-cause caveat + watch_flag rather than given its own note."]
not_promoted: ["Claim 4 as a standalone note — Kline's floor-cause is his own hypothesis, not an atomic finding; folded as a caveat/watch_flag into the drop-8 classification note instead (a note whose whole content is 'this other claim is only hypothesized' would be a thin source-note).","The 'Assessment of the core question' section — meta-synthesis, not an atomic claim; its substance is distributed across the three promoted notes and used to rule the originating question answered.","'Further leads' (French Box 3 rat time-duration experiments; Sharkey & Sharkey 1995 'catastrophic remembering'; McClelland/McNaughton/O'Reilly 1995 complementary-learning-systems; Kirkpatrick et al. 2017 EWC / Zenke et al. 2017 Synaptic Intelligence; a direct Ratcliff 1990 re-read) — all future-reading pointers, unread beyond this capture's quotes, left in the inbox as leads, not promoted."]
---


This capture answers the question routed from
[[claim-neural-nets-forget-along-human-like-power-law-curve]] via
`50-questions/question-catastrophic-vs-graceful-forgetting-regime.md`: that
note's "catastrophic vs. graceful" framing had been corroborated only by a
Tier-4 web search, below the floor for a specific technical-mechanism claim.
This capture reads the primary literature directly — French's 1999 review
(which itself cites and summarizes McCloskey & Cohen 1989 and Ratcliff 1990,
the founding papers) and Kline's own methods/results/discussion sections — to
check whether the regime distinction is real and whether Kline's MNIST
drop-8 protocol sits on the correct side of it.

## Claim: The connectionist literature draws an established distinction between "catastrophic" forgetting (abrupt, near-total) and "gradual"/graceful forgetting, and ties the difference to the structure of the learning problem rather than treating all forgetting as one phenomenon

**Claim type:** Specific technical-mechanism / quantitative claim → floor Tier 1-2 required. **Reached (Tier 1).**

French (1999) frames the founding demonstration: McCloskey & Cohen (1989)
trained a standard backpropagation network to fluency on 17 "one's addition
facts," then trained it on the 17 "two's addition facts," and measured
recall on the original facts. French quotes the magnitude directly:

> "Within 1-5 two's learning trials, the number of correct responses on the
> one's facts had dropped from 100% to 20%. In five more learning trials, the
> one's knowledge was at 1%, by 15 trials, no correct answers from the
> previous one's problems could be produced by the network."

This is contrasted with a human paradigm McCloskey & Cohen invoked as the
target behavior networks were failing to match: Barnes & Underwood's (1959)
retroactive-interference experiments, in which subjects learned a set of
paired associates (A-B), then a second set sharing the same cues but new
associates (A-C). French: "the forgetting curve for the A-B associate pairs
produced by interference from the learning of the new A-C pairs was
*relatively gradual*." Ratcliff (1990) independently replicated the
network-side catastrophic result "for vectors of different sizes and for
networks of various types." French's own explicit terminology ties directly
to the topic's language: he describes the pre-1989 assumption that a single
set of shared weights gave connectionist networks "their remarkable
abilities to generalize and **degrade gracefully**" — i.e., "graceful"
degradation was the expected, unremarkable case, and catastrophic forgetting
was the discovered exception to it, not the reverse.

## Claim: The literature's proposed determinant of which regime appears is the degree of overlap in the network's internal representations, not an intrinsic, all-or-nothing property of neural networks

**Claim type:** Specific technical-mechanism claim → floor Tier 1-2 required. **Reached (Tier 1).**

French's own mechanism, restated from his 1991/1992 work: "catastrophic
forgetting was largely a consequence of the overlap of internal distributed
representations and that reducing this overlap would reduce catastrophic
interference." He reports that architectures engineered toward
semi-distributed or sparse (lower-overlap) internal codes reduce or eliminate
catastrophic forgetting, and — importantly for the "regime, not refutation"
framing — that some architectures keep distributed representations but avoid
catastrophic loss: convolution-correlation models (CHARM, TODAM) and Sparse
Distributed Memory "can learn new information in a sequential manner and
can, in addition, generalize on new input. The performance of these models
on previously learned information **declines gradually, rather than falling
off abruptly**, when learning new patterns." French also reports that
pre-training a network on a regular/structured domain before sequential
learning eliminates catastrophic interference in later learning (citing
McRae & Hetherington 1993) — a second, independent line of evidence that the
regime is a function of *how the task and its representations are
structured*, not a fixed property of backpropagation itself.

## Claim: Kline's MNIST "drop-8" protocol is, by his own methods description, a single continuing-task setup with one class zero-weighted — not a disjoint sequential-task switch of the kind that produced McCloskey & Cohen's catastrophic result

**Claim type:** Specific technical-mechanism claim (setup classification) → floor Tier 1-2 required. **Reached (Tier 1, Kline's own text.)**

Kline's Method section describes training a single MLP to convergence on the
full 10-class MNIST task, then continuing training on the *same task* while
withholding one class from the sampler, not switching to a new, disjoint task
with different targets: "we removed all digit 8 examples from the training
set and continued training our fully-trained MLP for 80 additional epochs."
The general procedure specifies this directly: "begin training the
pretrained model using a weighted random sampler, with one or more class
weights set to zero to measure retention and memory decay." All ten output
classes, and the input/feature space they share, remain the same throughout;
only the sampling weight for one class changes. This is structurally distinct
from McCloskey & Cohen's one's-facts-then-two's-facts protocol, where the
network was moved wholesale onto a new, non-overlapping mapping. By French's
framework above — where the catastrophic/gradual distinction tracks
representational overlap and task structure — Kline's setup sits on the
overlapping, single-task side, matching the class of setups (shared cues,
partial structure) that the primary literature associates with gradual, not
catastrophic, forgetting.

## Claim: Kline's own explanation for why class-8 recall stabilized near 0.18 rather than collapsing to 0 is feature overlap between digit 8 and digits 6/9 — offered explicitly as a hypothesis, not an independently verified mechanism

**Claim type:** Specific technical-mechanism claim → floor Tier 1-2 required. **Reached (Tier 1), but the source itself hedges the claim as a hypothesis — preserved here as such, not upgraded.**

From Kline's Discussion: "The relatively low initial recall seen in Figure 2
can be attributed to the feature overlap between 8 and other digits (notably
the circular strokes in 6 and 9). Similarly, **it is hypothesized** that the
factors keeping the recall probability from reaching close to zero is due to
the network's reliance on similar features in other digits to predict the
unseen class." This is Kline's own candidate mechanism for the non-zero floor
observed in his curve (recall opening at 0.29, falling to ~0.18, then
flattening), and it is the same representational-overlap mechanism French's
review identifies as the general determinant of the gradual/graceful regime
— but Kline states it as a hypothesis about his own data, not as something
independently tested (e.g., no ablation removing 6/9 from the training set
to see if the floor then drops toward 0 is reported in this paper).

---

## Assessment of the core question

**Regime distinction — confirmed real, at Tier 1.** McCloskey & Cohen (1989),
Ratcliff (1990), and French's (1999) synthesis of them establish that
whether sequential learning in a distributed network produces abrupt,
near-total forgetting or gradual decline is not fixed — it depends on the
overlap/structure between what was already learned and what is learned next.
French uses "gradual" and "degrade gracefully" rather than the compound
phrase "task geometry," and the field's classic literature does not use
"power-law" as its vocabulary for the graceful side (that framing, and its
link to Ebbinghaus-style curve-fitting, is Kline's own bridge to the separate
human-memory-curve-shape literature, already captured in
[[claim-neural-nets-forget-along-human-like-power-law-curve]] and not
re-verified here). With that terminological caveat, the substance of the
distinction — catastrophic forgetting as a special, structure-dependent case
rather than a universal property of gradient-descent-trained networks — is
confirmed against primary sources.

**Kline's setup — correctly classed as the non-catastrophic side, on
structural grounds.** Kline's own methods describe a single-task,
one-class-zero-weighted continuation of training, not a disjoint
sequential-task switch. That structural fact places it on the
shared-structure/overlapping side of French's distinction, which the primary
literature associates with gradual rather than catastrophic forgetting. The
specific claim that overlapping strokes between 8, 6, and 9 are *why* the
curve flattens above zero remains Kline's own hypothesis, not independently
verified in this or any other source found this session.

Net: the central question is **substantially confirmed**, with one open
seam — the regime distinction's reality and structural determinant are
Tier-1-sourced; the specific causal claim about *why* Kline's curve
flattens where it does is Tier-1-sourced but explicitly hypothesis-only in
its own source.

> [!note] Seek's commentary:
> The nicest part of French's review is that it turns the vault's framing
> upside down in a useful way: "graceful degradation" was the *expected*
> behavior of distributed networks before 1989 — catastrophic forgetting was
> the surprise finding. Kline's paper, thirty-six years later, is really a
> return to the pre-1989 default (a network that degrades gracefully) inside
> a setup that was never going to produce the McCloskey & Cohen result in the
> first place, because it was never a disjoint two-task protocol. That
> doesn't make Kline's finding uninteresting — the *human-like power-law
> shape* of the curve is still a real, separately-sourced observation — but
> it does mean the "regime distinction" resolves less as "two forces in
> tension" and more as "know which paradigm you're running before you're
> surprised by its result." Worth folding back into
> [[claim-neural-nets-forget-along-human-like-power-law-curve]]'s caveat at
> promotion.
> — Seek

## Further leads

- French (1999) Box 3: rat time-duration learning experiments (sequential
  vs. concurrent learning of 40s/8s intervals) reproduce the same
  catastrophic-vs-gradual split in animal behavior, not just networks — a
  possible cross-species-parallel note (Tier 1, unread beyond this capture's
  quotes).
- French (1999) Box 4: Sharkey & Sharkey's (1995) "catastrophic remembering"
  — a network that has learned so many auto-associative patterns it loses
  the ability to discriminate seen from unseen input — a distinct failure
  mode from catastrophic forgetting, worth its own note.
- McClelland, McNaughton & O'Reilly (1995) complementary-learning-systems
  argument (hippocampus vs. neocortex) as the proposed biological solution to
  the same stability-plasticity problem — connects to
  [[claim-critical-periods-arise-from-information-plasticity-not-biology]].
- Kline cites Kirkpatrick et al. 2017 (Elastic Weight Consolidation, PNAS)
  and Zenke, Poole & Ganguli 2017 (Synaptic Intelligence) as engineering
  responses to catastrophic forgetting — both Tier 1 primary, unread this
  session.
- Ratcliff (1990), *Psychological Review* 97:285-308 — the second founding
  paper, not read directly this session (only via French's summary); a
  primary re-read would strengthen the quantitative floor further.