Built upon a wealth of neuroimaging, neurostimulation, and neuropsychology data, a recent proposal set forth a framework termed controlled semantic cognition (CSC) to account for how the brain underpins the ability to flexibly use semantic knowledge (Lambon Ralph et al., 2017; Nature Reviews Neuroscience). In CSC, the 'semantic control' system, underpinned predominantly by the prefrontal cortex, dynamically monitors and modulates the 'semantic representation' system that consists of a 'hub' (anterior temporal lobe, ATL) and multiple 'spokes' (modality-specific areas). CSC predicts that unfamiliar and exacting semantic tasks should intensify communication between the 'control' and 'representation' systems, relative to familiar and less taxing tasks. In the present study, we used functional magnetic resonance imaging (fMRI) to test this hypothesis. Participants paired unrelated concepts by canonical colours (a less accustomed task - e.g., pairing ketchup with fire-extinguishers due to both being red) or paired well-related concepts by semantic relationship (a typical task - e.g., ketchup is related to mustard). We found the 'control' system was more engaged by atypical than typical pairing. While both tasks activated the ATL 'hub', colour pairing additionally involved occipitotemporal 'spoke' regions abutting areas of hue perception. Furthermore, we uncovered a gradient along the ventral temporal cortex, transitioning from the caudal 'spoke' zones preferring canonical colour processing to the rostral 'hub' zones preferring semantic relationship. Functional connectivity also differed between the tasks: Compared with semantic pairing, colour pairing relied more upon the inferior frontal gyrus, a key node of the control system, driving enhanced connectivity with occipitotemporal 'spoke'. Together, our findings characterise the interaction within the neural architecture of semantic cognition - the control system dynamically heightens its connectivity with relevant components of the representation system, in response to different semantic contents and difficulty levels.