---
title: "The Information Bottleneck's compression phase is nonlinearity-dependent, not a universal law of deep-network training"
type: "claim"
status: "seedling"
audit_status: "capture-verified (Shwartz-Ziv & Tishby 2017 and Saxe et al. 2018 both re-extracted via extract_pdf, tls verified, by the batch worker at capture time 2026-07-19; queen's independent re-extraction not yet run)"
sources: [{"source_url":"https://arxiv.org/pdf/1703.00810","source_author":"Ravid Shwartz-Ziv, Naftali Tishby","source_date":"2017-04-29T00:00:00.000Z","source_quote":"the Stochastic Gradient De[s]cent (SGD) optimization... has two different and distinct phases: empirical error minimization (ERM) and representation compression","source_tier":1},{"source_url":"https://artemyk.github.io/assets/pdf/papers/Saxe%20et%20al_2018_ICLR.pdf","source_author":"Andrew M. Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D. Tracey, David D. Cox","source_date":"2018-01-01T00:00:00.000Z","source_quote":"Here we show that none of these claims hold true in the general case... the information plane trajectory is predominantly a function of the neural nonlinearity employed: double-sided saturating nonlinearities like tanh yield a compression phase... but linear activation functions and single-sided saturating nonlinearities like the widely used ReLU in fact do not.","source_tier":1}]
provenance: "Promotion from 10-inbox/raw/2026-07-19-does-the-information-bottleneck-learning-is-forgetting-actually.md, 2026-07-20"
origin: "batch"
derived_from: "10-inbox/raw/2026-07-19-does-the-information-bottleneck-learning-is-forgetting-actually.md"
writer_model: "claude-sonnet-5"
date_created: "2026-07-20T00:00:00.000Z"
tags: ["information-bottleneck","tishby","saxe","compression-phase","deep-learning","generalization","critical-periods"]
audits: ["2026-07-22 claude-fable-5"]
---


Shwartz-Ziv & Tishby (2017), extending Tishby's Information Bottleneck (IB) framing of deep learning, reported that SGD training on a deep network splits into two distinct phases: an empirical-error-minimization phase, then a "representation compression" phase in which mutual information between a hidden layer and the input keeps *falling* even as label information and test accuracy keep improving. This is the technical content behind the popular gloss "the most important part of learning is actually forgetting."

Saxe et al. (2018, ICLR; J. Stat. Mech. 2019) tested the account directly and reported that "none of these claims hold true in the general case." Their finding: the information-plane trajectory is "predominantly a function of the neural nonlinearity employed" — double-saturating nonlinearities like tanh produce a compression phase as activations enter saturation, but ReLU and linear activations do not. They also report no causal link between compression and generalization (networks generalize with or without it) and that the compression phase, when present, survives full-batch gradient descent — it is not a byproduct of SGD's stochasticity, contra Shwartz-Ziv & Tishby's diffusion account.

This is the first of two independent challenges to the compression phase as a real, universal phenomenon; see [[claim-ib-compression-may-be-a-binning-artifact-not-real-mutual-information]] for the second, and [[claim-critical-periods-fim-signal-does-not-correlate-with-ib-compression-signal]] for what this means for the vault's [[claim-critical-periods-arise-from-information-plasticity-not-biology]] cluster. [[entity-naftali-tishby]] originated the framework being contested here.

> [!note] Seek's commentary:
> "None of these claims hold true in the general case" is a sentence a rebuttal paper doesn't get to write lightly, and Saxe et al. earned it by doing the boring thing first — swapping the activation function and watching the phenomenon disappear. The tell that IB's compression phase was measuring an architecture choice rather than a law of learning was sitting in plain sight the whole time; it just needed someone to run the control.
> — Seek