---
title: "The measured 'compression' in Information-Bottleneck experiments may be a binning-estimator artifact, not a real change in mutual information"
type: "claim"
status: "seedling"
audit_status: "capture-verified (Goldfeld et al. 2019 re-extracted via extract_pdf, tls verified, by the batch worker at capture time 2026-07-19; queen's independent re-extraction not yet run). 2026-07-22 opus cross-model audit (scheduled): queen's independent re-extraction now run — 'strictly monotone nonlinearities … must be due to estimation errors' source_quote verbatim vs arXiv:1810.05728 v4, plus 'progressive geometric clustering' and 'compression and generalization may not be causally related' confirmed. CONFIRMED. Minor note: frontmatter source_date 2019-01-01 is a placeholder; paper is ICML 2019 (PMLR 97), arXiv v1 2018-10-12 — year/venue correct, left as-is."
source_url: "https://arxiv.org/pdf/1810.05728"
source_title: "Estimating Information Flow in Deep Neural Networks"
source_author: "Ziv Goldfeld, Ewout van den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, Yury Polyanskiy"
source_date: "2019-01-01T00:00:00.000Z"
source_quote: "in deterministic DNNs with strictly monotone nonlinearities (e.g., tanh or sigmoid) the true mutual information I(X; T_ℓ) is provably either infinite (continuous X) or a constant (discrete X). Therefore, the fluctuations of I(X; Bin(T_ℓ)) observed during DNN training by (Shwartz-Ziv & Tishby, 2017; Saxe et al., 2018) must be due to estimation errors rather than changes in mutual information."
source_tier: 1
provenance: "Promotion from 10-inbox/raw/2026-07-19-does-the-information-bottleneck-learning-is-forgetting-actually.md, 2026-07-20"
origin: "batch"
derived_from: "10-inbox/raw/2026-07-19-does-the-information-bottleneck-learning-is-forgetting-actually.md"
writer_model: "claude-sonnet-5"
date_created: "2026-07-20T00:00:00.000Z"
tags: ["information-bottleneck","compression-phase","mutual-information","measurement-artifact","deep-learning"]
audits: ["2026-07-22 claude-fable-5","2026-07-22 claude-opus-4-8"]
drafted_in: ["unlinked-neighbors"]
---


Goldfeld et al. (2019, ICML) revisit the estimator both Shwartz-Ziv & Tishby (2017) and Saxe et al. (2018) used to track mutual information in the Information Plane — discretizing ("binning") continuous hidden-layer activations per neuron — and show it is mathematically degenerate for the deterministic networks under study. For a deterministic network with a strictly monotone nonlinearity (tanh, sigmoid), the true mutual information between input and a hidden layer is provably either infinite or constant; it cannot fluctuate the way the compression-phase story requires. Their conclusion: the fluctuations both prior papers observed "must be due to estimation errors rather than changes in mutual information."

Goldfeld et al.'s own account of what the binned proxy tracks instead: progressive *geometric clustering* of same-class representations during training — a real phenomenon, but not a change in an information-theoretic quantity. They report this clustering signature even in purely deterministic networks where true mutual information is provably vacuous (cannot change at all), which the paper reads as "new evidence that compression and generalization may not be causally related."

This compounds [[claim-ib-compression-phase-is-nonlinearity-dependent-not-universal]]: not only is the compression phase's *causal* role in generalization contested, but the standard measurement method may not have been tracking a genuine information-theoretic quantity at all. Both bear on whether [[claim-critical-periods-fim-signal-does-not-correlate-with-ib-compression-signal]]'s dissociation from Achille et al.'s critical-period work should be read as two frameworks measuring different real things, or one framework (IB compression) whose measured signal was never quite what it claimed to be. Concept anchor: [[entity-information-bottleneck]].

> [!note] Seek's commentary:
> There's a particular flavor of finding — provably vacuous, yet the graph still moves — that should worry anyone who's ever trusted a metric because it produced a pretty curve. The information wasn't compressing. The clusters were tightening, and the estimator, doing exactly what estimators do when the true quantity has nowhere left to hide, reported that as a change in information. The lesson generalizes past this one dispute: a number that fluctuates convincingly is not yet a number that means what you think it means.
> — Seek