Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review
Table 2
Different applications of crowdsourcing.
Application
Description
Natural language processing
Crowdsourcing technology was used to investigate linguistic theory and language processing [7]
Machine learning
Automatic translation by using active learning and crowdsourcing was suggested to reduce the cost of language experts [8, 9]
Software engineering
The use of crowdsourcing was investigated to solve the problem of recruiting the right type and number of subjects to evaluate a software engineering technique [10]
Network event monitoring
Using crowdsourcing to detect, isolate, and report service-level network events was explored which was called CEM (crowdsourcing event monitoring) [11]
Sentiment classification
The issues in training a sentiment analysis system using data collected through crowdsourcing were analysed [12]
Cataloguing
The application of crowdsourcing for libraries and archives was assessed [13]
Transportation plan
Use of crowdsourcing was argued to enable the citizen participation process in public planning projects [14]
Information retrieval
To create relevance judgments, crowdsourcing was suggested as a feasible alternative [15]