Recognizing Textual Entailment by Hierarchical Crowdsourcing with Diverse Labor Costs

With the rapid advancement of supervised learning and the rise of large language models, the demand for high-quality labeled datasets has surged. However, crowdsourced datasets often suffer from label noise. To address this challenge, we present a hierarchical crowdsourcing framework with diverse labor costs that aims to mitigate label noise. Our framework models crowdsourcing workers with varying labor costs and leverages hierarchical crowdsourcing under limited budget constraints to enhance data quality. We establish a loop for label selection and checking, strategically selecting checkers from different cost levels, including perfect workers (Oracles) and regular experts, to optimize the checking process. Additionally, we tackle an NP-hard problem in label selection. Experimental evaluation on a real-world dataset for the Recognizing Textual Entailment (RTE) task demonstrates a significant improvement in labeled dataset quality, leading to state-of-the-art performance in downstream tasks.