The dense packing of interacting particles on spheres has proved to be a useful model for virus capsids and colloidosomes. Indeed, icosahedral symmetry observed in virus capsids corresponds to potential energy minima that occur for magic numbers of, e.g., 12, 32 and 72 identical Lennard-Jones particles, for which the packing has exactly the minimum number of twelve five-fold defects. It is unclear, however, how stable these structures are against thermal agitation. We investigate this property by means of basin-hopping global optimisation and Langevin dynamics for particle numbers between ten and one hundred. An important measure is the number and type of point defects, that is, particles that do not have six nearest neighbours. We find that small icosahedral structures are the most robust against thermal fluctuations, exhibiting fewer excess defects and rearrangements for a wide temperature range. Furthermore, we provide evidence that excess defects appearing at low non-zero temperatures lower the potential energy at the expense of entropy. At higher temperatures defects are, as expected, thermally excited and thus entropically stabilised. If we replace the Lennard-Jones potential by a very short-ranged (Morse) potential, which is arguably more appropriate for colloids and virus capsid proteins, we find that the same particle numbers give a minimum in the potential energy, although for larger particle numbers these minima correspond to different packings. Furthermore, defects are more difficult to excite thermally for the short-ranged potential, suggesting that the short-ranged interaction further stabilises equilibrium structures.