Oligomeric assemblies of the amyloid β peptide (Aβ) have been investigated for over two decades as possible neurotoxic agents in Alzheimer's disease. However, due to their heterogeneous and transient nature, it is not yet fully established which of the structural features of these oligomers may generate cellular damage. Here, we study distinct oligomer species formed by Aβ40 (the 40-residue form of Aβ) in the presence of four different metal ions (Al3+, Cu2+, Fe2+, and Zn2+) and show that they differ in their structure and toxicity in human neuroblastoma cells. We then describe a correlation between the size of the oligomers and their neurotoxic activity, which provides a type of structure-toxicity relationship for these Aβ40 oligomer species. These results provide insight into the possible role of metal ions in Alzheimer's disease by the stabilization of Aβ oligomers.