Paper
Continual Weight Updates and Convolutional Architectures for Equilibrium\n Propagation
Equilibrium Propagation (EP) is a biologically inspired alternative algorithm\nto backpropagation (BP) for training neural networks. It applies to RNNs fed by\na static input x that settle to a steady state, such as Hopfield networks. EP\nis similar to BP in that in the second phase of training, an error signal\npropagates backwards in the layers of the network, but contrary to BP, the\nlearning rule of EP is spatially local. Nonetheless, EP suffers from two major\nlimitations. On the one hand, due to its formulation in terms of real-time\ndynamics, EP entails long simulation times, which limits its applicability to\npractical tasks. On the other hand, the biological plausibility of EP is\nlimited by the fact that its learning rule is not local in time: the synapse\nupdate is performed after the dynamics of the second phase have converged and\nrequires information of the first phase that is no longer available physically.\nOur work addresses these two issues and aims at widening the spectrum of EP\nfrom standard machine learning models to more bio-realistic neural networks.\nFirst, we propose a discrete-time formulation of EP which enables to simplify\nequations, speed up training and extend EP to CNNs. Our CNN model achieves the\nbest performance ever reported on MNIST with EP. Using the same discrete-time\nformulation, we introduce Continual Equilibrium Propagation (C-EP): the weights\nof the network are adjusted continually in the second phase of training using\nlocal information in space and time. We show that in the limit of slow changes\nof synaptic strengths and small nudging, C-EP is equivalent to BPTT (Theorem\n1). We numerically demonstrate Theorem 1 and C-EP training on MNIST and\ngeneralize it to the bio-realistic situation of a neural network with\nasymmetric connections between neurons.\n
Authors: Ernoult, Maxence · Grollier, Julie · Querlioz, Damien · Bengio, Yoshua · Scellier, Benjamin