Mechanical Turk-based experiment vs laboratory-based experiment: A case study on the comparison of semantic transparency rating data

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

9 Citations (Scopus)

Abstract

In this paper, we conducted semantic transparency rating experiments using both the traditional laboratory-based method and the crowdsourcing-based method. Then we compared the rating data obtained from these two experiments. We observed very strong correlation coefficients for both overall semantic transparency rating data and constituent semantic transparency data (rho > 0:9) which means the two experiments may yield comparable data and crowdsourcing-based experiment is a feasible alternative to the laboratorybased experiment in linguistic studies. We also observed a scale shrinkage phenomenon in both experiments: the actual scale of the rating results cannot cover the ideal scale [0, 1], both ends of the actual scale shrink towards the center. However, the scale shrinkage of the crowdsourcing-based experiment is stronger than that of the laboratory-based experiment, this makes the rating results obtained in these two experiments not directly comparable. In order to make the results directly comparable, we explored two data transformation algorithms, z-score transformation and adjusted normalization to unify the scales. We also investigated the uncertainty of semantic transparency judgment among raters, we found that it had a regular relation with semantic transparency magnitude and this may further reveal a general cognitive mechanism of human judgment.
Original languageEnglish
Title of host publication29th Pacific Asia Conference on Language, Information and Computation, PACLIC 2015
PublisherShanghai Jiao Tong University
Pages53-62
Number of pages10
Publication statusPublished - 1 Jan 2015
Event29th Pacific Asia Conference on Language, Information and Computation, PACLIC 2015 - Shanghai, China
Duration: 30 Oct 20151 Nov 2015

Conference

Conference29th Pacific Asia Conference on Language, Information and Computation, PACLIC 2015
Country/TerritoryChina
CityShanghai
Period30/10/151/11/15

ASJC Scopus subject areas

  • Artificial Intelligence
  • Human-Computer Interaction
  • Linguistics and Language

Cite this