Paper
Distributed Representation Prediction for Generalization to New Words
Learning distributed representations of symbols (e.g. words) has been used in several Natural Language Processing systems. Such representations can capture semantic or syntactic similarities between words, which permit to fight the curse of dimensionality when considering sequences of such words. Unfortunately, because these representations are learned only for a previously determined vocabulary of words, it is not clear how to obtain representations for new words. We present here an approach which gets around this problem by considering the distributed representations as predictions from low-level or domain-knowledge features of words. We report experiments on a Part Of Speech tagging task, which demonstrates the success of this approach in learning meaningful representations and in providing improved accuracy, especially for new words. 1
Authors: Hugo Larochelle · Yoshua Bengio · Département D’informatique Et Recherche Opérationnelle