The conformal loop ensemble (CLE) is the canonical conformally invariant probability measure on non-crossing loops in a simply connected domain in $C$ and is indexed by a parameter $\kappa \in (8/3,8)$. We consider CLE on the whole-plane in the regime in which the loops are self-intersecting ($\kappa \in (4,8)$) and show that it is invariant under the inversion map $z\mapsto 1/z$. This shows that whole-plane CLE for $\kappa \in (4,8)$ defines a conformally invariant measure on loops on the Riemann sphere. The analogous statement in the regime in which the loops are simple ($\kappa \in (8/3,4]$) was proven by Kemppainen and Werner and together with the present work covers the entire range $\kappa \in (8/3,8)$ for which CLE is defined. As an intermediate step in the proof, we show that CLE for $\kappa \in (4,8)$ on an annulus, with any specified number of inner-boundary-surrounding loops, is well-defined and conformally invariant.