Review Article

Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review

Table 2

Different applications of crowdsourcing.

ApplicationDescription

Natural language processingCrowdsourcing technology was used to investigate linguistic theory and language processing [7]

Machine learningAutomatic translation by using active learning and crowdsourcing was suggested to reduce the cost of language experts [8, 9]

Software engineeringThe use of crowdsourcing was investigated to solve the problem of recruiting the right type and number of subjects to evaluate a software engineering technique [10]

Network event monitoringUsing crowdsourcing to detect, isolate, and report service-level network events was explored which was called CEM (crowdsourcing event monitoring) [11]

Sentiment classificationThe issues in training a sentiment analysis system using data collected through crowdsourcing were analysed [12]

CataloguingThe application of crowdsourcing for libraries and archives was assessed [13]

Transportation planUse of crowdsourcing was argued to enable the citizen participation process in public planning projects [14]

Information retrievalTo create relevance judgments, crowdsourcing was suggested as a feasible alternative [15]