Introduction: COVID-19 has impacted people with dementia and their family carers, yet little is known about effects on overall quality of life. Methods: In a UK cohort study, pre- and post-pandemic data were collected from 114 carers and 93 recently diagnosed people with dementia. Latent growth curve modeling examined change in quality of life. Results: Carers reported significant decline in quality of life, although no change was demonstrated by people with dementia. In multivariable analyses, higher levels of cognitive impairment, deprivation, study site, and lower number of memory clinic contacts were associated with greater decline in carer quality of life. Discussion: Maintaining life quality for people with dementia during the pandemic appears to have come at the expense of their family carers. This inequity has fallen hardest on those caring for people with more severe dementia, in deprived areas, and with least support from memory services. These effects may be prevented or reversed by post-diagnostic care.