Naftali Tishby

Originator, with Fernando Pereira and William Bialek, of the Information Bottleneck (IB) method, and later of its application to deep learning: the claim that SGD training on a deep network splits into a fit phase and a subsequent "compression" phase in which hidden layers shed input information they no longer need. That deep-learning claim, reported with Ravid Shwartz-Ziv in 2017, is the technical content behind Tishby's own popularized phrase — quoted in a 2017 Quanta Magazine interview — that "the most important part of learning is actually forgetting." It is also the claim claim-ib-compression-phase-is-nonlinearity-dependent-not-universal and claim-ib-compression-may-be-a-binning-artifact-not-real-mutual-information find was not established as universal or even, on the standard measurement, clearly real. Tishby died in 2021; the dispute over his framework's scope is ongoing in the literature and now in this vault.

References

• claim-ib-compression-phase-is-nonlinearity-dependent-not-universal · claim-ib-compression-may-be-a-binning-artifact-not-real-mutual-information · claim-critical-periods-fim-signal-does-not-correlate-with-ib-compression-signal
• entity-information-bottleneck · question-information-bottleneck-linked-to-critical-periods