Repository logo

Psychometric scaling of TID2013 dataset

Accepted version


Conference Object

Change log


Mikhailiuk, Aliaksei  ORCID logo
Perez-Ortiz, M 


TID2013 is a subjective image quality assessment dataset with a wide range of distortion types and over 3000 images. The dataset has proven to be a challenging test for objective quality metrics. The dataset mean opinion scores were obtained by collecting pairwise comparison judgments using the Swiss tournament system, and averaging votes of observers. However, this approach differs from the usual analysis of multiple pairwise comparisons, which involves psychometric scaling of the comparison data using either Thurstone or Bradley-Terry mod- els. In this paper we investigate how quality scores change when they are computed using such psychometric scaling instead of averaging vote counts. In order to properly scale TID2013 quality scores, we conduct four additional experiments of two different types, which we found necessary to produce a common quality scale: comparisons with reference images, and cross-content comparisons. We demonstrate on a fifth validation experiment that the two additional types of comparisons are necessary and in conjunction with psychometric scaling improve the consistency of quality scores, especially across images depicting different contents.



46 Information and Computing Sciences, 4603 Computer Vision and Multimedia Computation

Journal Title

2018 10th International Conference on Quality of Multimedia Experience, QoMEX 2018

Conference Name

2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX)

Journal ISSN

Volume Title


Engineering and Physical Sciences Research Council (EP/P007902/1)
European Research Council (725253)