Information Noise Contrastive Estimation (InfoNCE)

Representation Learning with Contrastive Predictive Coding — van den Oord, Li, VInyals

Information noise contrastive estimation (InfoNCE) loss builds on the noise-contrastive estimation (NCE) framework. It generalizes the binary classification problem (true data vs. noise distribution) to a multiclass classification problem, where the goal is to predict which of the N samples (1 positive and N-1 negatives) come from the true data distribution. It uses categorical cross-entropy loss to identify the positive sample.

Here, our true data is drawn from a conditional distribution with “context” c: $p(x|c)$. And our N-1 negative samples are drawn from the marginal distribution (independent of context c): $p(x)$. Let $X$ be all N samples, only one of which is $x_{\text{pos}}$ , and $C$ be the position where the positive sample is located.

The probability of detecting the correct positive sample is:

$$p(C=pos|X,c)=\frac{p(X,C=pos|c)}{p(X|c)}=\frac{p(x_{\text{pos}}|c){\prod }_{i=1,...N;i\ne pos}p(x_i)}{{\sum }_{j=1}^N[p(x_{\text{j}}|c){\prod }_{i=1,...N;i\ne j}p(x_i)]}$$

Here, the denominator is summing over all the N possible positions for the positive sample.

Dividing the numerator and denominator by $\prod p(x)$ helps us cancel some terms:

$$=\frac{p(x_{\text{pos}}|c){\prod }_{i=1,...N;i\ne pos}p(x_i)}{{\sum }_{j=1}^N[p(x_{\text{j}}|c){\prod }_{i=1,...N;i\ne j}p(x_i)]}\ast \frac{1/\prod p(x)}{1/\prod p(x)}=\frac{\frac{p(x_{pos}|c)}{p(x_{pos})}}{{\sum }_{j=1}^N\frac{p(x_j|c)}{p(x_j)}}$$

Taking a scoring function $f(x,c)\propto \frac{p(x|c)}{p(x)}$, we get

$$=\frac{f(x_{pos},c)}{{\sum }_{i=1}^{N}f(x_i,c)}$$

The InfoNCE loss optimizes the negative log probability (aka cross-entropy loss) of classifying the positive example correctly:

$${\mathcal{L}}_{InfoNCE}=-\mathbb{E}[\log \frac{f(x,c)}{{\sum }_{i=1}^{N}f(x_i,c)}]$$

In this way, InfoNCE is described as a probabilistic contrastive loss that retains information most helpful for predicting future samples. InfoNCE loss is commonly used in self-supervised learning for contrastive learning.

Showing consistency of InfoNCE and ranking NCE loss

$f(x,c)\propto \frac{p(x|c)}{p(x)}$

• In InfoNCE, the N-1 negative samples are drawn from the marginal p(x), so p(x) = q(x)

$f_{\text{score}}(x,u)\propto \frac{p(x|u)}{q(x)}\propto \frac{f(x,u)/Z_f(u)}{q(x)}\propto \frac{f(x,u)}{q(x)}$

References