BACKGROUND: The COVID-19 pandemic imposed major changes on daily-life routine worldwide. To the best of our knowledge, no study quantified the changes on moderate to vigorous physical activity (MVPA) and sedentary behaviors (SB) and its correlates in Brazilians. This study aimed to (i) evaluate the changes (pre versus during pandemic) in time spent in MVPA and SB in self-isolating Brazilians during the COVID-19 pandemic, and (ii) to explore correlates. METHODS: A cross-sectional, retrospective, self-report online web survey, evaluating the time spent in MVPA and SB pre and during the COVID-19 pandemic in self-isolating people in Brazil. Sociodemographic, behavioral, and clinical measures, and time in self-isolation were also obtained. Changes in MVPA and SB and their correlates were explored using generalized estimating equations (GEE). Models were adjusted for covariates. RESULTS: A total of 877 participants (72.7% women, 53.7% young adults [18-34 years]) were included. Overall, participants reported a 59.7% reduction (95% CI 35.6-82.2) in time spent on MVPA during the pandemic, equivalent to 64.28 (95% CI 36.06-83.33) minutes per day. Time spent in SB increased 42.0% (95% CI 31.7-52.5), corresponding to an increase of 152.3 (95% CI 111.9-192.7) minutes per day. Greater reductions in MVPA and increases in SB were seen in younger adults, those not married, those employed, and those with a self-reported previous diagnosis of a mental disorder. CONCLUSIONS: People in self-isolation significantly reduced MVPA levels and increased SB. Public health strategies are needed to mitigate the impact of self-isolation on MVPA and SB. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s11332-021-00788-x.