Paper
Modeling High-Dimensional Discrete Data with Multi-Layer Neural Networks
The curse of dimensionality is severe when modeling high-dimensional discrete data: the number of possible combinations of the variables ex-plodes exponentially. In this paper we propose a new architecture for modeling high-dimensional data that requires resources (parameters and computations) that grow only at most as the square of the number of vari-ables, using a multi-layer neural network to represent the joint distribu-tion of the variables as the product of conditional distributions. The neu-ral network can be interpreted as a graphical model without hidden ran-dom variables, but in which the conditional distributions are tied through the hidden units. The connectivity of the neural network can be pruned by using dependency tests between the variables. Experiments on modeling the distribution of several discrete data sets show statistically significant improvements over other methods such as naive Bayes and comparable Bayesian networks, and show that significant improvements can be ob-tained by pruning the network. 1
Authors: Yoshua Bengio · Samy Bengio