Groups of eukaryotic cilia and flagella are capable of coordinating their beating over large scales, routinely exhibiting collective dynamics in the form of metachronal waves. The origin of this behavior--possibly influenced by both mechanical interactions and direct biological regulation--is poorly understood, in large part due to a lack of quantitative experimental studies. Here we characterize in detail flagellar coordination on the surface of the multicellular alga Volvox carteri, an emerging model organism for flagellar dynamics. Our studies reveal for the first time that the average metachronal coordination observed is punctuated by periodic phase defects during which synchrony is partial and limited to specific groups of cells. A minimal model of hydrodynamically coupled oscillators can reproduce semi-quantitatively the characteristics of the average metachronal dynamics, and the emergence of defects. We systematically study the model's behaviour by assessing the effect of changing intrinsic rotor characteristics, including oscillator stiffness and the nature of their internal driving force, as well as their geometric properties and spatial arrangement. Our results suggest that metachronal coordination follows from deformations in the oscillators' limit cycles induced by hydrodynamic stresses, and that defects result from sufficiently steep local biases in the oscillators' intrinsic frequencies. Additionally, we find that random variations in the intrinsic rotor frequencies increase the robustness of the average properties of the emergent metachronal waves.