Since the emergence of Coronavirus Disease 2019 (COVID-19), researchers in machine learning and radiology have rushed to develop algorithms that could assist with diagnosis, triage and management of the disease (1). As a result, thousands of diagnostic and prognostic models using chest radiographs and computed tomography (CT) have been developed. However, with no standardised approach to development or evaluation, it is difficult, even for experts, to determine which models may be of most clinical benefit. In this work, we share our main concerns and present some possible solutions.