Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review

Table 4

Worker types based on their behavioral observation.

Workers	Description
Diligent	Completed tasks precisely with a high accuracy and longer time spent on tasks
Competent	Skilled workers with high accuracy and fast work
Sloppy	Completed tasks quickly without considering quality
Incompetent	Completed tasks in a time with low accuracy
Spammer	Did not deliver useful works