Detecting low-quality crowdtesting workers

Ricky K.P. Mok, Weichao Li, Kow Chuen Chang

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

7 Citations (Scopus)

Abstract

QoE crowdtesting is increasingly popular among researchers to conduct subjective assessments of different services. Experimenters can easily access to a huge pool of human subjects through crowdsourcing platforms. A fundamental problem threatening the integrity of crowdtesting is to detect cheating from the workers who work without any supervision. One of the approaches in classifying the quality of workers is analyzing their behavior during the experiments. A major challenge is to systematically analyze the mouse cursor trajectory. However, existing works usually analyze the trajectory coarsely, which cannot fully extract the information imbedded in the trajectory. In this paper, we propose to use finer-grained cursor trajectory analysis, including submovement analysis, to identify low quality workers. Our approach is to define a set of ten worker behavior metrics to quantify different types of worker behavior. A jQuery-based library was implemented to collect the worker behavior. Moreover, four different 5-point Likert scale rating methods were employed. A number of methods, including question design, instructions, and human inspections, are used to label workers into three categories. We then apply multiclass Naive Bayes classifier to construct different models using all or some of the metrics and the workers' category. Our results show that the error rates of the model trained from four metrics is equal or less than 30% for four rating methods. By combining the predictions from the four rating methods, the successful rate in detecting low-quality workers is around 80%.
Original languageEnglish
Title of host publication2015 IEEE 23rd International Symposium on Quality of Service, IWQoS 2015
PublisherIEEE
Pages201-206
Number of pages6
ISBN (Electronic)9781467371131
DOIs
Publication statusPublished - 10 Feb 2016
Event23rd IEEE International Symposium on Quality of Service, IWQoS 2015 - Portland, United States
Duration: 15 Jun 201516 Jun 2015

Conference

Conference23rd IEEE International Symposium on Quality of Service, IWQoS 2015
Country/TerritoryUnited States
CityPortland
Period15/06/1516/06/15

ASJC Scopus subject areas

  • Safety, Risk, Reliability and Quality
  • Media Technology
  • Management of Technology and Innovation
  • Computer Networks and Communications

Cite this