Human beings have remarkable social attention skills. From the initial processing of cues, such as eye gaze, head direction, and body orientation, we perceive where other people are attending, allowing us to draw inferences about the intentions, desires, and dispositions of others. But before we can infer why someone is attending to something in the world we must first accurately represent where they are attending. Here we investigate the "where" of social attention perception, and employ adaptation paradigms to ascertain how head and body orientation are visually represented in the human brain. Across two experiments we show that the representation of two cues to social attention (head and body orientation) exists at the category-specific level. This suggests that aftereffects do not arise from "social attention cells" discovered in macaques or from abstract representations of "leftness" or "rightness."