We gave 40 participants a task in which they needed to select target objects from an array according to the instructions of either an informed director (who shared their perspective of the array) or an ignorant director (whose view of the array was restricted due to barriers). Importantly, sometimes only one of the directors was visible, and on some trials when both directors were present, participants were required to switch between perspectives. We found that participants were faster to select items from the informed director’s perspective than the ignorant director’s perspective, but that they slowed when there was a visible but inactive second director. Crucially, relative to nonswitch trials where the same perspective was taken twice consecutively, participants exhibited a significant cost of switching between perspectives when returning to take their own perspective, but not when switching to the other point of view. We interpret these results as evidence that participants inhibit their more salient perspective in order to adopt another’s, and then incur an asymmetric switch cost as a result. This suggests that although we are egocentric by default, our egocentricity is effectively, albeit temporarily, eliminated if we have just adopted an alternative frame of reference.