Size is an important visuo-spatial characteristic of the physical world. In language processing, previous research has demonstrated a processing advantage for words denoting semantically "big" (e.g., jungle) versus "small" (e.g., needle) concrete objects. We investigated whether semantic size plays a role in the recognition of words expressing abstract concepts (e.g., truth). Semantically "big" and "small" concrete and abstract words were presented in a lexical decision task. Responses to "big" words, regardless of their concreteness, were faster than those to "small" words. Critically, we explored the relationship between semantic size and affective characteristics of words as well as their influence on lexical access. Although a word's semantic size was correlated with its emotional arousal, the temporal locus of arousal effects may depend on the level of concreteness. That is, arousal seemed to have an earlier (lexical) effect on abstract words, but a later (post-lexical) effect on concrete words. Our findings provide novel insights into the semantic representations of size in abstract concepts and highlight that affective attributes of words may not always index lexical access.