Patients with multimodal semantic deficits following stroke ('semantic aphasia') have largely intact knowledge, yet difficulty controlling conceptual retrieval to suit the circumstances. Although conceptual representations are thought to be largely distinct from episodic representations of recent events, controlled retrieval processes may overlap across semantic and episodic memory domains. We investigated this possibility by examining item familiarity and source memory for recent events in semantic aphasia following infarcts affecting left inferior frontal gyrus. We tested the hypothesis that the nature of impairment in episodic judgements reflects the need for control over retrieval: item familiarity might be relatively intact, given it is driven by strong cues (re-presentation of the item), while source recollection might be more impaired since this task involves resolving competition between several potential sources. This pattern was observed most strongly when the degree of competition between sources was higher, i.e., when non-meaningful sources had similar perceptual features, and existing knowledge was incongruent with the source. In contrast, when (i) spatial location acted as a strong cue for retrieval; (ii) existing knowledge was congruent with episodic memory and (iii) distinctiveness of sources was increased by means of self-referential processing, source memory reached normal levels. These findings confirm the association between deregulated control of semantic and episodic memory in patients with semantic aphasia and delineate circumstances that ameliorate or aggravate these deficits.