Paper
Updates of Equilibrium Prop Match Gradients of Backprop Through Time in\n an RNN with Static Input
Equilibrium Propagation (EP) is a biologically inspired learning algorithm\nfor convergent recurrent neural networks, i.e. RNNs that are fed by a static\ninput x and settle to a steady state. Training convergent RNNs consists in\nadjusting the weights until the steady state of output neurons coincides with a\ntarget y. Convergent RNNs can also be trained with the more conventional\nBackpropagation Through Time (BPTT) algorithm. In its original formulation EP\nwas described in the case of real-time neuronal dynamics, which is\ncomputationally costly. In this work, we introduce a discrete-time version of\nEP with simplified equations and with reduced simulation time, bringing EP\ncloser to practical machine learning tasks. We first prove theoretically, as\nwell as numerically that the neural and weight updates of EP, computed by\nforward-time dynamics, are step-by-step equal to the ones obtained by BPTT,\nwith gradients computed backward in time. The equality is strict when the\ntransition function of the dynamics derives from a primitive function and the\nsteady state is maintained long enough. We then show for more standard\ndiscrete-time neural network dynamics that the same property is approximately\nrespected and we subsequently demonstrate training with EP with equivalent\nperformance to BPTT. In particular, we define the first convolutional\narchitecture trained with EP achieving ~ 1% test error on MNIST, which is the\nlowest error reported with EP. These results can guide the development of deep\nneural networks trained with EP.\n
Authors: Ernoult, Maxence · Grollier, Julie · Querlioz, Damien · Bengio, Yoshua · Scellier, Benjamin