We examined how attention modulates the neural encoding of continuous speech under different types of interference. In an EEG experiment, participants attended to a narrative in English while ignoring a competing stream in the other ear. Four different types of interference were presented to the unattended ear: a different English narrative, a narrative in a language unknown to the listener (Spanish), a well-matched nonlinguistic acoustic interference (Musical Rain), and no interference. Neural encoding of attended and unattended signals was assessed by calculating cross-correlations between their respective envelopes and the EEG recordings. Findings revealed more robust neural encoding for the attended envelopes compared with the ignored ones. Critically, however, the type of the interfering stream significantly modulated this process, with the fully intelligible distractor (English) causing the strongest encoding of both attended and unattended streams and latest dissociation between them and nonintelligible distractors causing weaker encoding and early dissociation between attended and unattended streams. The results were consistent over the time course of the spoken narrative. These findings suggest that attended and unattended information can be differentiated at different depths of processing analysis, with the locus of selective attention determined by the nature of the competing stream. They provide strong support to flexible accounts of auditory selective attention.