Noise-Contrastive Estimation (NCE)

Noise-contrastive estimation, A new estimation principle for unnormalized statistical models — Gutmann, Hyvärinen

The idea of noise contrastive estimation, or NCE, is to run a logistic regression to differentiate the target data $x$ from noise $\tilde{x}$. Note that a logit (log-odds) can be used to say the logit of a sample $u$ being from the target data distribution $p_{\theta }$ instead of the noise distribution $q$: $l_{\theta }(u)=\log \frac{p_{\theta }(u)}{q(u)}=\log p_{\theta }(u)-q(u)$

From here, we can convert logits into probabilities with sigmoid, and apply cross-entropy loss, giving us the loss function:

$${\mathcal{L}}_{\text{NCE}}=-\frac{1}{N}\sum _{i=1}^N\left[\log \sigma ({\ell }_{\theta }(x_i))+\log (1-\sigma ({\ell }_{\theta }({\tilde{x}}_i)))\right]$$ $$\text{where }\sigma (\ell )=\frac{1}{1+\exp (-\ell )}=\frac{p_{\theta }}{p_{\theta }+q}$$

You will notice this loss is defined a little differently than the others, where it is defined in terms of probabilities rather than distances. NCE is used to estimate the parameters $\theta$ of the modeled data distribution $p_{\theta }$ that distinguishes true samples from artificially generated noise samples $q$. When the classifier is optimized, the corresponding ${\theta }^{\ast }$ gives the desired data distribution.

Follow-up work has incorporated multiple negative samples, instead of only one positive and one negative pair.

Ranking NCE

• Multi-class extension of distinguishing real data from generated nosie
• for a conditioned model $p(x|u)=\frac{f(x,u)}{Z_f(u)}$, where $Z_f(u)=\int f(x,u)dx$

$$P(\text{sample is real}|x,u)=\frac{p(x|u)}{p(x|u)+Kq(x)}$$ $$=\frac{f(x,u)/Z_f(u)}{f(x,u)/Z_f(u)+Kq(x)}$$ $$=\frac{f(x,u)/Z_f(u)}{f(x,u)/Z_f(u)+Kq(x)}\cdot \frac{Z_f(u)/q(x)}{Z_f(u)/q(x)}$$ $$=\frac{f(x,u)/q(x)}{f(x,u)/q(x)+K{Z}_f(u)}$$

But this now has the intractable partition function in it

$$Z_f(u)=\int f(x,u)dx=\frac{f(x,u)}{q(x)}q(x)dx={\mathbb{E}}_{x\sim q(x)}[\frac{f(x,u)}{q(x)}]$$

We replace the expectation with the average over the K available negative samples $\{y_{i,j}{\}}_{j=1}^K$ which are drawn from $q(x)$.

$$Z_f(u)=\frac{1}{K}\sum _{j=1}^K\frac{f(y_{i,j},u)}{q(y_{i,j})}$$

Plugging that into our probability calculation:

$$P(\text{sample is real}|x,u)=\frac{f(x,u)/q(x)}{f(x,u)/q(x)+{\sum }_{i=1}^K\frac{f(y_{i,j},u)}{q(y_{i,j})}}$$

Final Ranking NCE loss is the average estimated log-probability over set of all observed true samples.

$${\mathcal{L}}_{RankingNCE}=\frac{1}{n}\sum _{i=1}^n\log \frac{f(x,u)/q(x)}{f(x,u)/q(x)+{\sum }_{i=1}^K\frac{f(y_{i,j},u)}{q(y_{i,j})}}$$

References

Noise-contrastive estimation, A new estimation principle for unnormalized statistical models