All components of the CNS are surrounded by a diffuse extracellular matrix (ECM) containing chondroitin sulphate proteoglycans (CSPGs), heparan sulphate proteoglycans (HSPGs), hyaluronan, various glycoproteins including tenascins and thrombospondin, and many other molecules that are secreted into the ECM and bind to ECM components. In addition, some neurons, particularly inhibitory GABAergic parvalbumin-positive (PV) interneurons, are surrounded by a more condensed cartilage-like ECM called perineuronal nets (PNNs). PNNs surround the soma and proximal dendrites as net-like structures that surround the synapses. Attention has focused on the role of PNNs in the control of plasticity, but it is now clear that PNNs also play an important part in the modulation of memory. In this review we summarize the role of the ECM, particularly the PNNs, in the control of various types of memory and their participation in memory pathology. PNNs are now being considered as a target for the treatment of impaired memory. There are many potential treatment targets in PNNs, mainly through modulation of the sulphation, binding, and production of the various CSPGs that they contain or through digestion of their sulphated glycosaminoglycans.