Recent developments in sensory and communication technologies have made the development of portable air-quality (AQ) micro-sensing units (MSUs) feasible. These MSUs allow AQ measurements in many new applications, such as ambulatory exposure analyses and citizen science. Typically, the performance of these devices is assessed using the mean error or correlation coefficients with respect to a laboratory equipment. However, these criteria do not represent how such sensors perform outside of laboratory conditions in large-scale field applications, and do not cover all aspects of possible differences in performance between the sensor-based and standardized equipment, or changes in performance over time. This paper presents a comprehensive Sensor Evaluation Toolbox (SET) for evaluating AQ MSUs by a range of criteria, to better assess their performance in varied applications and environments. Within the SET are included four new schemes for evaluating sensors' capability to: locate pollution sources; represent the pollution level on a coarse scale; capture the high temporal variability of the observed pollutant and their reliability. Each of the evaluation criteria allows for assessing sensors' performance in a different way, together constituting a holistic evaluation of the suitability and usability of the sensors in a wide range of applications. Application of the SET on measurements acquired by 25 MSUs deployed in eight cities across Europe showed that the suggested schemes facilitates a comprehensive cross platform analysis that can be used to determine and compare the sensors' performance. The SET was implemented in R and the code is available on the first author's website.