The emergence of a novel pathogen in a susceptible population can cause rapid spread of infection. High prevalence of SARS-CoV-2 infection in white-tailed deer (Odocoileus virginianus) has been reported in multiple locations, likely resulting from several human-to-deer spillover events followed by deer-to-deer transmission. Knowledge of the risk and direction of SARS-CoV-2 transmission between humans and potential reservoir hosts is essential for effective disease control and prioritisation of interventions. Using genomic data, we reconstruct the transmission history of SARS-CoV-2 in humans and deer, estimate the case finding rate and attempt to infer relative rates of transmission between species. We found no evidence of direct or indirect transmission from deer to human. However, with an estimated case finding rate of only 4.2%, spillback to humans cannot be ruled out. The extensive transmission of SARS-CoV-2 within deer populations and the large number of unsampled cases highlights the need for active surveillance at the human-animal interface.