Repository logo
 

Overcoming catastrophic interference in connectionist networks using Gram-Schmidt orthogonalization.


Type

Article

Change log

Authors

Srivastava, Vipin 
Sampath, Suchitra 
Parker, David J 

Abstract

Connectionist models of memory storage have been studied for many years, and aim to provide insight into potential mechanisms of memory storage by the brain. A problem faced by these systems is that as the number of items to be stored increases across a finite set of neurons/synapses, the cumulative changes in synaptic weight eventually lead to a sudden and dramatic loss of the stored information (catastrophic interference, CI) as the previous changes in synaptic weight are effectively lost. This effect does not occur in the brain, where information loss is gradual. Various attempts have been made to overcome the effects of CI, but these generally use schemes that impose restrictions on the system or its inputs rather than allowing the system to intrinsically cope with increasing storage demands. We show here that catastrophic interference occurs as a result of interference among patterns that lead to catastrophic effects when the number of patterns stored exceeds a critical limit. However, when Gram-Schmidt orthogonalization is combined with the Hebb-Hopfield model, the model attains the ability to eliminate CI. This approach differs from previous orthogonalisation schemes used in connectionist networks which essentially reflect sparse coding of the input. Here CI is avoided in a network of a fixed size without setting limits on the rate or number of patterns encoded, and without separating encoding and retrieval, thus offering the advantage of allowing associations between incoming and stored patterns. PACS Nos.: 87.10.+e, 87.18.Bb, 87.18.Sn, 87.19.La.

Description

Keywords

Algorithms, Computer Simulation, Models, Neurological, Nerve Net, Neurons, Synaptic Potentials

Journal Title

PLoS One

Conference Name

Journal ISSN

1932-6203
1932-6203

Volume Title

9

Publisher

Public Library of Science (PLoS)
Sponsorship
Royal Society (2013/R1)
The Royal Society; The Leverhulme Foundation (UK); National Initiative of Research in Cognitive Science by the Department of Science and Technology, Government of India.