Reduced gaze following has been associated previously with lower language scores in children with autism spectrum disorder (ASD). Here, we use eye-tracking in a controlled experimental setting to investigate whether gaze following and attention distribution during a word learning task associate with later developmental and clinical outcomes in a population of infants at familial risk for ASD. Fifteen-month-old infants (n = 124; n = 101 with familial risk) watched an actress repeatedly gaze toward and label one of two objects present in front of her. We show that infants who later developed ASD followed gaze as frequently as typically developing peers but spent less time engaged with either object. Moreover, more time spent on faces and less on objects was associated with lower concurrent or later verbal abilities, but not with later symptom severity. No outcome group showed evidence for word learning. Thus, atypical distribution of attention rather than poor gaze following is a limiting factor for language development in infants at familial risk for ASD.