Contrastive Learning

“Mimics” the way that humans learn about the world.

Uses the principle of contrasting unlabeled samples/data points against each other on vision tasks to learn attributes that are common between data classes and attributes that set apart a data class from another. Those that belong to the same distribution are “pushed towards” each other in the embedding space, and those in different distributions are “pulled apart.”

Originally, it started as a form of self-supervised learning, as “positive” and “negative” samples are used to guide the model. It has now evolved to be used for fully supervised learning and semi-supervised learning settings, as well.

The basic contrastive learning framework consists of selecting a data sample, called “anchor,” a data point belonging to the same distribution as the anchor, called the “positive” sample, and another data point belonging to a different distribution called the “negative” sample.

The SSL model tries to minimize the distance between the anchor and positive samples, i.e., the samples belonging to the same distribution, in the latent space, and at the same time maximize the distance between the anchor and the negative samples.

Images in same class lie closer together in the embedding space.

Dimensionality Reduction by Learning an Invariant Mapping — Chopra, Hadsell, LeCun

Contrastive loss was first introduced in 2005, with its original application being in dimensionality reduction. In general, the goal of dimensionality reduction can be formulated as transforming a D-dimensional vector into a d-dimensional vector, where d << D, while preserving as much information as possible. By adding the constraints of learning a parametric mapping function (from D to d dimensions) such that the mapping:

1. preserves neighborhood relationships between the data (e.g. if two data points were similar before the transformation, they should remain similar or close to each other after)
2. mapping should generalize to new, unseen data

The contrastive loss function introduced is:

$${\mathcal{L}}_{cont}({\text{x}}_i,{\text{x}}_j;\theta )=1[y_i=y_j]\Vert f_{\theta }({\text{x}}_i)-f_{\theta }({\text{x}}_j){\Vert }_2^2+1[y_i\ne y_j]\text{max}(0,ϵ-\Vert f_{\theta }({\text{x}}_i)-f_{\theta }({\text{x}}_j){\Vert }_2{)}^2$$

It takes input pairs $(x_i,x_j)$ and labels $y$ and minimizes the embedding distance when they are from the same class, and maximizes the embedding distance otherwise (via negating the euclidian distance) up to some margin $ϵ$. When the distance between points of different classes is greater than some margin , the loss is 0, since the points are large enough apart. This ensures that the positive pairs are still in some margin $ϵ$, and we don’t force dissimilar points further away than necessary.

The general form of the contrastive loss function has the same structure as cross-entropy loss.

Further advances have been made on this foundational contrastive loss function, including triplet loss, n-pair loss, noise-contrastive estimation (NCE), information noise contrastive estimation (InfoNCE), and normalized temperature-scaled cross-entropy (NT-Xent) aka SimCLR.

References

The Beginner’s Guide to Contrastive Learning
Losses explained: Contrastive Loss