Paper
Reply to Huszár: The elastic weight consolidation penalty is empirically valid
In our recent work on elastic weight consolidation (EWC) (1) we show that forgetting in neural networks can be alleviated by using a quadratic penalty whose derivation was inspired by Bayesian evidence accumulation. In his letter (2), Dr. Huszar provides an alternative form for this penalty by following the standard work on expectation propagation using the Laplace approximation (3). He correctly argues that in cases when more than two tasks are undertaken the two forms of the penalty are different. Dr. Huszar also shows that for a toy linear regression problem his expression appears to be better. We would like to thank Dr. Huszar for pointing out … [↵][1]1To whom correspondence should be addressed. Email: [email protected]. [1]: #xref-corresp-1-1
Authors: James Kirkpatrick · Razvan Pascanu · Neil Rabinowitz · Joel Veness · Guillaume Desjardins · Andrei A. Rusu · Kieran Milan · John Quan · Tiago Ramalho · Agnieszka Grabska-Barwinska · Demis Hassabis · Claudia Clopath · Dharshan Kumaran · Raia Hadsell