Paper
Neural Function Modules with Sparse Arguments: A Dynamic Approach to\n Integrating Information across Layers
Feed-forward neural networks consist of a sequence of layers, in which each\nlayer performs some processing on the information from the previous layer. A\ndownside to this approach is that each layer (or module, as multiple modules\ncan operate in parallel) is tasked with processing the entire hidden state,\nrather than a particular part of the state which is most relevant for that\nmodule. Methods which only operate on a small number of input variables are an\nessential part of most programming languages, and they allow for improved\nmodularity and code re-usability. Our proposed method, Neural Function Modules\n(NFM), aims to introduce the same structural capability into deep learning.\nMost of the work in the context of feed-forward networks combining top-down and\nbottom-up feedback is limited to classification problems. The key contribution\nof our work is to combine attention, sparsity, top-down and bottom-up feedback,\nin a flexible algorithm which, as we show, improves the results in standard\nclassification, out-of-domain generalization, generative modeling, and learning\nrepresentations in the context of reinforcement learning.\n
Authors: Lamb, Alex · Goyal, Anirudh · Słowik, Agnieszka · Mozer, Michael · Beaudoin, Philippe · Bengio, Yoshua