As a field of study that investigates how microscopic interactions affect macroscopic material properties, condensed matter physics takes great interest in understanding how charge carriers respond to stimuli. A fundamental distinction has therefore existed between conductors and insulators, where charge carriers' ability to roam freely separates the two phases of matter. However, even this seemingly immutable distinction has recently begun to blur. The recent discovery of a bulk Fermi surface in the Kondo insulator SmB6 has raised serious questions about how a fundamentally metallic property could be intrinsically realized in a robust insulator. This thesis describes my work to further our understanding of this surprising behavior by searching for other examples of unconventional quantum oscillatory insulators.

Following SmB6, two additional strongly correlated insulators have been found to display insulating quantum oscillations: YbB12 and FeSb2. YbB12 is an f-electron Kondo insulator, much like SmB6, and exhibits similar electrical and thermal properties to SmB6. On the other hand, FeSb2 is a d-electron insulator that had previously attracted attention for its unusual thermal properties. While united in exhibiting intrinsic insulating phase quantum oscillations that originate from the sample bulk, the two materials separately illuminate important insights into insulating phase quantum oscillations.

Despite its similar physical properties to SmB6, YbB12 was found to exhibit slow and heavy quantum oscillations, in stark contrast to the fast and light quantum oscillations in SmB6. YbB12 has also been found to display sizable quantum oscillations in electrical resistivity, a feature which has not yet been observed in SmB6 and remains the topic of much theoretical debate. Furthermore, an investigation of quantum oscillations in the field-induced metallic phase of YbB12 has revealed more of the same Fermi surface seen in the insulating phase. Taken together, the discoveries in YbB12 suggest a rich variety of insulating phase Fermi surfaces which are shaped by material band structure, much like metallic phase Fermi surfaces.

While discoveries in YbB12 hint at the breadth of insulating Fermi surfaces, the investigation of FeSb2 takes us one step closer to pinpointing the mechanism that underlies insulating phase quantum oscillations. Quantum oscillations in FeSb2 have shown a strong coupling to a metamagnetic-like transition in the magnetic torque, which reflects anisotropic magnetization. The first-order transition signals the onset of a novel high field phase with enhanced quantum oscillation amplitude and frequency spectrum. The temperature dependence of this onset has also shown a tentative link to a dramatic increase in quantum oscillation amplitudes at low temperatures beyond the description of the Lifshitz-Kosevich formalism, a behavior also seen in SmB6. The discoveries in FeSb2 therefore strengthens the proposition of Fermi surfaces arising from novel quasiparticles that do not participate in longitudinal charge transport.

The discovery of insulating phase quantum oscillations in the correlated insulators YbB12 and FeSb2 has helped accelerate a rapidly growing field of study around the unexpected finding of unconventional quantum oscillatory insulators. The new discoveries show a rich landscape of insulating Fermi surfaces, and provide hints towards the origin of this surprising behavior. These studies further motivate efforts to uncover additional unconventional quantum oscillatory insulators and devise a theoretical description to capture the rich variety of unconventional insulating Fermi surfaces.