Understanding the internal structures of planets with a large H$2$O component is important for the characterisation of sub-Neptune planets. The finding that the mini-Neptune K2-18b could host a liquid water ocean beneath a mostly hydrogen envelope motivates a detailed examination of the phase structures of water-rich planets. To this end, we present new internal structure models for super-Earths and mini-Neptunes that enable detailed characterisation of a planet's water component. We use our models to explore the possible phase structures of water worlds and find that a diverse range of interiors are possible, from oceans sandwiched between two layers of ice to supercritical interiors beneath steam atmospheres. We determine how the bulk properties and surface conditions of a water world affect its ocean depth, finding that oceans can be up to hundreds of times deeper than on Earth. For example, a planet with a 300 K surface can possess H$2$O oceans with depths from 30-500 km, depending on its mass and composition. We also constrain the region of mass-radius space in which planets with H/He envelopes could host liquid H$2$O, noting that the liquid phase can persist at temperatures up to 647 K at high pressures of $218$-$7\times 104$ bar. Such H/He envelopes could contribute significantly to the planet radius while retaining liquid water at the surface, depending on the planet mass and temperature profile. Our findings highlight the exciting possibility that habitable conditions may be present on planets much larger than Earth.