IA Summit 12: Crowdsourced Unmoderated Usability Testing
I choose to attend a couple of the 20 minute dual session blocks to see what I thought about the smaller sessions. I have to admin I was impressed with the amount of information people were able to fit in 20 minutes. The first one I attended was Crowdsourced Unmoderated Usability Testing by Inge De Bleecker. This was a nice overview of some of the tools currently available, and some of the pro’s and con’s of outsourced recruitment and unmoderated testing.
Crowdsourcing is how we go about…
- …outsourcing to a large group of people in an open call.
- …enabling convenient on demand access to a pool of human workers .
There are many different types of tests that can be conducted with crowdsourcing:
- Screenshot click tests.
- Usabilla
- UserZoom
- UsabilityHub
- Screenshot timed tests.
- UserZoom
- UsabilityHub
- Tasked based usability study with online survey.
- UserTesting
- Loop11
- UserZoom
- DIY
These types of tests are good for:
- Personal device use.
- Testing in their own environment.
- Fast turnaround.
- Cheap(er) testing.
On the other hand, they are not good for:
- Observing your users while completing tasks.
- Asking ad hoc questions.
- Subject video recordings.
While you generally use the same process as you would for on-site testing, except that you’ll need to take extra steps in writing your task plan:
- Participants can’t get blocked or they won’t be able to continue.
- You only get one shot with each participant.
- You have to guide them to the right task without influencing them.
- All questions need to be required, so you’ll need to omit the extras, and be sure to add an “other” option.
- Be sure to encourage participants to write down their thoughts at the end.
- Remember that the task and questions types will depend on the tool being used.
She recommends a minimum of 10 participants and usually tries for 15-20 with a $35 stipend based on how the recruiting is done.
“If you pay peanuts, you will get monkeys”
Slides
Hi, an important addition to the above list is userlytics.com, the only one of the above that includes a recorded webcam view of participants in a “Video-in-Video” session (what they do, see and hear, together with their facial expressions and they say).
Also, with userlytics you can observe and ask add hoc questions (using the moderation functionality), and as mentioned above you are video capturing a 360 degree view of the participant and their interactions