Barlow Twins (2021) trains a self-supervised encoder by driving the cross-correlation matrix of two views toward the identity, minimizing redundancy between embedding components

Barlow Twins (Zbontar, Jing, Misra, LeCun, and Deny, 2021) is a self-supervised method for learning image representations without negative pairs. Two identical networks are fed two distorted views of the same sample; the objective works by "measuring the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample, and making it as close to the identity matrix as possible." Pushing the matrix toward the identity does two things at once: the on-diagonal terms force each embedding component to be invariant to the distortion (the two views agree), while the off-diagonal terms are driven to zero, "thereby minimizing the redundancy between the components of these vectors." Decorrelating the components is what prevents the representational collapse that negative-pair methods use contrastive sampling to avoid.

The method is named for its intellectual ancestor: it is "called Barlow Twins, owing to neuroscientist H. Barlow's redundancy-reduction principle." That eponym makes a 1961 sensory-coding idea resurface, by name, as a 2021 deep-learning loss — see claim-barlow-1961-efficient-coding-removes-sensory-redundancy for the original hypothesis, and claim-fixational-eye-movements-prevent-perceptual-fading for the same "discard the redundant constant" logic running in the retina. The redundancy-reduction objective operates on the same object — a learned embedding vector — that claim-matryoshka-representation-learning-truncatable-embeddings restructures for truncation, and it sits in the vault's deep-learning-history thread of neuroscience-named architectures alongside claim-helmholtz-machine-named-for-unconscious-inference-theory.