|
|
 |
Listserv Message Center
 |
Mechanical Turk As a Source of Online Participants |
 |
|
|
|
| Posted by: | Sam Gosling |
| Title/Position: | Professor |
| School/Organization: | University of Texas, Austin |
| Sent to listserv of: | SPSP, SESP, SPSSI |
| Date posted: | May 27th, 2010 |
|
|
|
|
 |
 |
|
Hi All,
I know many of you are already beginning to use Amazon’s Mechanical Turk to collect data. But for those of you who haven’t discovered it yet, it’s worth taking a look.
Mechanical Turk is a website (www.mturk.com) administered by Amazon.com that matches “requesters,” who have tasks that need completing (e.g., an online study) with an enormous pool of “workers” who do the tasks, often for astonishingly small amounts of money. What makes the service so great for collecting data is that it is easy to use, it collects data rapidly, it’s amazingly inexpensive, it’s versatile (just about anything that can be done online can be done on MTurk), and the samples are demographically diverse (at least compared with the viable alternatives like student samples). We’ve had great success using the site for scale validation and replicating lab studies – in just a few months we’ve collected thousands of responses, usually paying just 5-10 cents for 5-10 minutes of peoples’ time. So, for example, many of the people sending requests to this list for participants for their online studies could use MTurk to finish their studies in a day for around $10.
Naturally, we initially had some concerns about the quality of the data and the diversity of the sample. We ran a few studies to test these concerns and to examine the effects of different payment amounts on data quality and response rates. Our findings were very encouraging. The analyses show that the site’s participants are considerably more diverse than typical US college samples and are slightly more diverse than other Internet samples. More important, the quality of the data was surprisingly good – internal consistencies and test-retest reliabilities on several measures were just as good as those from other traditional and online samples. Of course, like all methods, MTurk has its drawbacks. But having used the site a few times and having run these studies, we are finding that the pros far outweigh the cons.
In case you’re interested in seeing the findings for yourself (or want to justify the use of MTurk to editors/reviewers), we can send you our “in press” paper describing what we found (to be published in Perspectives on Psychological Science). You can download the paper here (http://tinyurl.com/289p7ky) or shoot an email to my co-author Michael Buhrmester (Buhrmester@gmail.com).
We’d be interested to hear about your experiences using MTurk.
Best regards,
Sam Gosling
|
|
 |
|
|
|
|
|
|
|
|
Forum Home Page
