---
title: "Does the Information Bottleneck ('learning is forgetting') actually explain deep-network critical periods, and is its compression phase even universal?"
type: "question"
status: "answered"
date_raised: "2026-07-12T00:00:00.000Z"
writer_model: "claude-opus-4-8"
answered_log: "2026-07-20 — answered no on both counts by [[claim-ib-compression-phase-is-nonlinearity-dependent-not-universal]], [[claim-ib-compression-may-be-a-binning-artifact-not-real-mutual-information]], and [[claim-critical-periods-fim-signal-does-not-correlate-with-ib-compression-signal]]. What settled it: primary sources now on file (Shwartz-Ziv & Tishby 2017, Saxe et al. 2018, Goldfeld et al. 2019, Achille et al. 2019, all Tier 1) show the compression phase is nonlinearity-dependent rather than universal, its measurement is independently disputed as a possible binning artifact, and the founding critical-periods paper's own Fisher-Information signal does not correlate with it — the IB link is borrowed vocabulary, not a demonstrated mechanism."
tags: ["information-bottleneck","critical-periods","tishby","saxe","information-plasticity","deep-learning","verification"]
---


Achille & Soatto frame their "Information Plasticity" account of deep-network
critical periods
([[claim-critical-periods-arise-from-information-plasticity-not-biology]]) as the
Information-Bottleneck (IB) view applied to critical periods: Tishby's claim that
training has two phases — fit, then compress/forget input detail to generalize —
summarized as "the most important part of learning is actually forgetting." If
that connection is real and load-bearing, the vault should link IB to critical
periods directly. But two things block promoting any IB note today:

1. **Sourcing.** The only source the capture carries for the IB mechanism is a
   Quanta Magazine feature (Tier 3). A specific technical-mechanism claim needs a
   Tier 1–2 primary. The IB-of-deep-learning primaries are Tishby & Zaslavsky
   2015 ("Deep Learning and the Information Bottleneck Principle") and
   Shwartz-Ziv & Tishby 2017 ("Opening the Black Box of Deep Neural Networks via
   Information").
2. **Contested.** Andrew Saxe et al. ("On the Information Bottleneck Theory of
   Deep Learning," ICLR 2018) showed some large networks generalize well
   *without* a drawn-out compression phase — the two-phase story may be an
   artifact of saturating nonlinearities, not a universal law. So even a
   well-sourced IB note would have to carry that dispute.

## Why it matters

If IB genuinely underwrites critical periods, it unifies three vault threads —
critical periods, Fisher-Information/Information-Plasticity, and the
"forgetting-to-generalize" story — under one mechanism. If Saxe is right that
compression is not universal, the Information-Plasticity result may hold for a
narrower reason than the IB framing implies, and the vault should keep the two
claims separate.

## What I'd need to answer it

- Read Shwartz-Ziv & Tishby 2017 and Tishby & Zaslavsky 2015 primaries; extract
  whether they claim compression is necessary for generalization.
- Read Saxe et al. 2018 (ICLR); extract the exact scope of the counterexample.
- Check whether Achille & Soatto's own papers derive critical periods *from* the
  IB objective or merely borrow its vocabulary.

## Candidate next moves

- If confirmed and bounded, write a claim-note linking IB to critical periods and
  demote the Quanta source to a pointer.
- Either way, record the Saxe dispute as its own note so the vault doesn't inherit
  the "compression is settled" framing the Quanta piece implies.


## Progress log

- 2026-07-20 — answered no on both counts by [[claim-ib-compression-phase-is-nonlinearity-dependent-not-universal]], [[claim-ib-compression-may-be-a-binning-artifact-not-real-mutual-information]], and [[claim-critical-periods-fim-signal-does-not-correlate-with-ib-compression-signal]]. What settled it: primary sources now on file (Shwartz-Ziv & Tishby 2017, Saxe et al. 2018, Goldfeld et al. 2019, Achille et al. 2019, all Tier 1) show the compression phase is nonlinearity-dependent rather than universal, its measurement is independently disputed as a possible binning artifact, and the founding critical-periods paper's own Fisher-Information signal does not correlate with it — the IB link is borrowed vocabulary, not a demonstrated mechanism.