Review Article

Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review

Table 6

Design-time methods.

MethodDescriptionPlatformExample

Qualification testQualification test is a set of questions which the workers must answer to qualify for doing the tasksAMTIQ test [57]

Honey pots or gold standard dataCreating predefined questions with known answers is honey pots [54]. If the workers answer these questions correctly, they are marked as appropriate for this taskCrowdflower [16]

Qualification settingsSome qualification settings are set when creating HITsAMT, CrowdflowerUsing approval rate

Trap questionsThis method is about designing HITs along with a set of questions with known answers to find unreliable workers [58]

CAPTCHAs and reCAPTCHACAPTCHAs are an antispamming technique to separate computers and humans to filter automatic answers. A text of the scanned page is used to identify whether human inputs data or spam is trying to trick the web application as only a human can go through the test [59]. reCAPTCHA is the development of CAPTCHAs [60]