Several studies assumed that the analysis of numerical information happens in a fast and automatic manner in the human brain. Utilizing the high temporal resolution of electroencephalography (EEG) in a passive oddball adaptation paradigm, we compared event-related brain potentials (ERPs) evoked by unattended shape changes and unattended numerosity changes. We controlled visual stimulus properties in a stringent manner. Unattended changes in shape elicited significant, gradual adaptation effects in the range of early visual components, indicating the fast and automatic processing of shapes. Changes in numerosity did not elicit significant changes in these early ERP components. The lack of early number-specific effects was qualified by a significant interaction between Shape and Number conditions. Number change elicited gradual ERP effects only on late ERP components. We conclude that numerosity is a higher-level property assembled from naturally correlating perceptual cues and hence, it is identified later in the cognitive processing stream.