TY - JOUR A2 - Yu, F. A2 - Li, L. A2 - Sanchez, L. AU - Samimi, Parnia AU - Ravana, Sri Devi PY - 2014 DA - 2014/05/19 TI - Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review SP - 135641 VL - 2014 AB - Test collection is used to evaluate the information retrieval systems in laboratory-based evaluation experimentation. In a classic setting, generating relevance judgments involves human assessors and is a costly and time consuming task. Researchers and practitioners are still being challenged in performing reliable and low-cost evaluation of retrieval systems. Crowdsourcing as a novel method of data acquisition is broadly used in many research fields. It has been proven that crowdsourcing is an inexpensive and quick solution as well as a reliable alternative for creating relevance judgments. One of the crowdsourcing applications in IR is to judge relevancy of query document pair. In order to have a successful crowdsourcing experiment, the relevance judgment tasks should be designed precisely to emphasize quality control. This paper is intended to explore different factors that have an influence on the accuracy of relevance judgments accomplished by workers and how to intensify the reliability of judgments in crowdsourcing experiment. SN - 2356-6140 UR - https://doi.org/10.1155/2014/135641 DO - 10.1155/2014/135641 JF - The Scientific World Journal PB - Hindawi Publishing Corporation KW - ER -