IA Summit 12: Crowdsourced Unmoderated Usability Testing

I choose to attend a couple of the 20 minute dual session blocks to see what I thought about the smaller sessions. I have to admin I was impressed with the amount of information people were able to fit in 20 minutes. The first one I attended was Crowdsourced Unmoderated Usability Testing by Inge De Bleecker. This was a nice overview of some of the tools currently available, and some of the pro’s and con’s of outsourced recruitment and unmoderated testing.

Crowdsourcing is how we go about…

  • …outsourcing to a large group of people in an open call.
  • …enabling convenient on demand access to a pool of human workers .
There are many different types of tests that can be conducted with crowdsourcing:
  • Screenshot click tests.
    • Usabilla
    • UserZoom
    • UsabilityHub
  • Screenshot timed tests.
    • UserZoom
    • UsabilityHub
  • Tasked based usability study with online survey.
    • UserTesting
    • Loop11
    • UserZoom
    • DIY
These types of tests are good for:
  • Personal device use.
  • Testing in their own environment.
  • Fast turnaround.
  • Cheap(er) testing.
On the other hand, they are not good for:
  • Observing your users while completing tasks.
  • Asking ad hoc questions.
  • Subject video recordings.
While you generally use the same process as you would for on-site testing, except that you’ll need to take extra steps in writing your task plan:
  • Participants can’t get blocked or they won’t be able to continue.
  • You only get one shot with each participant.
  • You have to guide them to the right task without influencing them.
  • All questions need to be required, so you’ll need to omit the extras, and be sure to add an “other” option.
  • Be sure to encourage participants to write down their thoughts at the end.
  • Remember that the task and questions types will depend on the tool being used.
She recommends a minimum of 10 participants and usually tries for 15-20 with a $35 stipend based on how the recruiting is done.
“If you pay peanuts, you will get monkeys”