Amazon Mechanical Turk's Data Quality Crisis: New Insights into Detection

Insights from the Field

Mechanical Turk

VPNs

data quality

fraud detection

The Shape of and Solutions to the MTurk Quality Crisis was authored by Ryan Kennedy, Scott Clifford, Tyler Burleigh, Philip Waggoner, Ryan Jewell and Nicholas Winter. It was published by Cambridge in PSR&M in 2020.

New research reveals a growing problem of fraud on Amazon's Mechanical Turk platform. Using VPNs, respondents are gaining unauthorized access to studies and submitting low-quality data.

🔍 What We Did

We analyzed 38 existing MTurk studies to understand the scope of this issue.

📊 Our Findings

* Data quality is declining due to fraudulent access via VPNs, a problem that has worsened recently.

* Fraudulent respondents provide significantly lower-quality data than legitimate users.

* This low-quality input can weaken treatment effects in experiments.

🛠️ How We Help

Researchers now have two tools: an application to identify fraud in existing MTurk datasets and a method to block fraudulent access during Qualtrics surveys.