In spite of their importance for understanding phonon transport phenomena in thin films and polycrystalline solids, the effects of boundary roughness scattering on phonon specularity and co- herence are poorly understood because there is no general method for predicting their dependence on phonon momentum, frequency, branch and boundary morphology. Using the recently formulated atomistic S-matrix method, we develop a theory of boundary roughness scattering to determine the mode-resolved phonon coherence and specularity parameters from the scattering amplitudes. To illustrate the theory, we apply it to phonon scattering in realistic nonsymmetric graphene grain boundary (GB) models derived from atomic structure predictions. The method is validated by comparing its predictions with frequency-resolved results from lattice dynamics-based calculations. We prove that incoherent scattering is almost perfectly diffusive. We show that phonon scattering at the graphene GB is not diffuse although coherence and specularity are significantly reduced for long-wavelength flexural acoustic phonons. Our approach can be generalized to other atomistic boundary models.