As smartphone and wearable device ownership increase, interest in their utility to monitor physical activity has risen concurrently. Numerous examples of the application of wearables in clinical and epidemiological research settings already exist. However, whether these devices are all suitable for physical activity surveillance is open for debate. In this article, we discuss four key issues specifically relevant to surveillance that we believe need to be tackled before consumer wearables can be considered for this measurement purpose: representative sampling, representative wear time, validity and reliability, and compatibility between devices. A recurring theme is how to deal with systematic biases by demographic groups. We suggest some potential solutions to the issues of potential concern such as providing individuals with standardised devices, considering summary metrics of physical activity less prone to wear time biases, and the development of a framework to harmonise estimates between device types and their inbuilt algorithms. We encourage collaborative efforts from researchers and consumer wearable manufacturers in this area. In the meantime, we caution against the use of consumer wearable device data for inference of population level activity without the consideration of these issues.