Calcium silicate perovskite (CaSiO$3$) is one of the major mineral components of the lower mantle, but has been the subject of relatively little work compared to the more abundant Mg-based materials. One of the major problems related to CaSiO$3$ that is still the subject of research is its crystal structure under lower mantle conditions - a cubic Pm$3¯$m structure is accepted in general, but some have suggested that lower-symmetry structures may be relevant. In this paper, we use a fully first-principles vibrational self-consistent field method to perform high accuracy anharmonic vibrational calculations on several candidate structures at a variety of points along the geotherm near the base of the lower mantle to investigate the stability of the cubic structure and related distorted structures. Our results show that the cubic structure is the most stable throughout the lower mantle, and that this result is robust against the effects of thermal expansion.