Reliability of Crowdsourcing for Subjective Quality Evaluation of Tone Mapping Operators - Ecole Centrale de Nantes
Conference Papers Year : 2021

Reliability of Crowdsourcing for Subjective Quality Evaluation of Tone Mapping Operators

Abstract

Tone mapping operators (TMO) are functions which map high dynamic range (HDR) images to limited dynamic media while aiming to preserve the perceptual cues of the scene that govern its aesthetic quality. Evaluating aesthetic quality of TMOs is non-trivial due to the high subjectivity of preference involved. Traditionally, TMO aesthetic quality has been evaluated via subjective experiments in a controlled laboratory environment. However, the last decade has brought a surge in popularity of crowdsourcing as an alternative methodology to conduct subjective experiments. However, uncontrolled experiment conditions and unreliability of participant behaviour puts doubts on the trustworthiness of the collected data. In this study, we explore the possibility of using crowdsourcing platforms for subjective quality evaluation of TMOs. We have conducted three experiments with systematic changes to investigate the effect of experiment conditions and participant recruitment methods on the collected subjective data. Our results show that subjective evaluation of TMO aesthetic quality can be conducted on Prolific crowdsourcing platform with negligible differences in comparison to laboratory experiments. Furthermore, we provide objective conclusions about the effect of number of observers on the certainty of the pairwise comparison results.
Fichier principal
Vignette du fichier
MMSP_2021 Goswami_Ak.pdf (2.12 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03298957 , version 1 (12-08-2021)
hal-03298957 , version 2 (22-08-2021)

Identifiers

Cite

Abhishek Goswami, Ali Ak, Wolf Hauser, Frédéric Dufaux, Patrick Le Callet. Reliability of Crowdsourcing for Subjective Quality Evaluation of Tone Mapping Operators. IEEE International Workshop on Multimedia Signal Processing (MMSP'2021), Oct 2021, Tampere, Finland. ⟨10.1109/MMSP53017.2021.9733707⟩. ⟨hal-03298957v2⟩
192 View
197 Download

Altmetric

Share

More