Paper
Neural Bayes: A Generic Parameterization Method for Unsupervised\n Representation Learning
We introduce a parameterization method called Neural Bayes which allows\ncomputing statistical quantities that are in general difficult to compute and\nopens avenues for formulating new objectives for unsupervised representation\nlearning. Specifically, given an observed random variable $\\mathbf{x}$ and a\nlatent discrete variable $z$, we can express $p(\\mathbf{x}|z)$,\n$p(z|\\mathbf{x})$ and $p(z)$ in closed form in terms of a sufficiently\nexpressive function (Eg. neural network) using our parameterization without\nrestricting the class of these distributions. To demonstrate its usefulness, we\ndevelop two independent use cases for this parameterization:\n 1. Mutual Information Maximization (MIM): MIM has become a popular means for\nself-supervised representation learning. Neural Bayes allows us to compute\nmutual information between observed random variables $\\mathbf{x}$ and latent\ndiscrete random variables $z$ in closed form. We use this for learning image\nrepresentations and show its usefulness on downstream classification tasks.\n 2. Disjoint Manifold Labeling: Neural Bayes allows us to formulate an\nobjective which can optimally label samples from disjoint manifolds present in\nthe support of a continuous distribution. This can be seen as a specific form\nof clustering where each disjoint manifold in the support is a separate\ncluster. We design clustering tasks that obey this formulation and empirically\nshow that the model optimally labels the disjoint manifolds. Our code is\navailable at \\url{https://github.com/salesforce/NeuralBayes}\n
Authors: Arpit, Devansh · Wang, Huan · Xiong, Caiming · Socher, Richard · Bengio, Yoshua