Systems that protect enclaves from privileged software must consider software-based side-channel attacks. Our system isolates enclaves on separate secure cores to stop attackers from running on the same core as the victim, which mitigates intra-core side-channel attacks. Redesigning the memory hierarchy based on enclave ownership protects enclaves against inter-core side-channel attacks. We implement this system and evaluate it in terms of communication performance, memory overhead and hardware area. Combining physical isolation and a redesigned memory hierarchy protects enclaves against all known software-based side-channel attacks.