After describing to a few people how I used MTurk for quick “usability” testing, I got enough positive responses that I thought it was worth writing up a quick report on it.

One of my favorite things about interaction design is putting things in front of users. Actually talking to real people about my project has revealed the most surprises and provided the most insights. Sure, you can try to find precedent elsewhere or track down a colleague who knows something about it, but oftentimes it’s just easier, faster and cheaper to round up some people and talk to them.

But what if you can’t do that for whatever reason? What if you need to answer a question that, as far as you can find out from the internet, has not been answered publicly before? What if the possibility of getting some live bodies in front of you is right out?

Faced with this problem, I remembered something that a teammate of mine did once, and this post from Udi’s Spot, and tried something a little different. I mishmashed user testing with A/B testing, deployed to 100 users. I got results back for analysis in 3 hours, with a total cost of $5.

How? Amazon Mechanical Turk. I wanted to know whether there was a good reason why dynamic filtering sidebars (as seen on Kayak) are always on the left. Is there a reason or is it just convention? Will it confuse or frustrate users if it’s on the right? Will they just not see it? Will they hate it? So I deployed two very simple questionnaires to two different groups. Group A got pictures of websites that had filtering sidebars on the left. Group B got photoshopped versions of those same websites with filtering sidebars on the right. The users were just asked to rate the appeal of the designs from 1-5, and they had a textarea if they wanted to provide additional feedback.

After I got the first round of results back, I speculated on the results: users preferred the sidebar on the left…unless the sidebar on the right really popped. Of course, it was still speculation, so I deployed another round to address that question. Wowza! That was quick, easy and cheap. And fun! There’s a progress indicator in MTurk so you can watch your little human robot army chew up your tests in real-time.

And now, the caveat: no, this isn’t scientifcologically sound. This is no substitute for proper user testing, and this should only be used in situations that a very simple and specific question is being addressed. And of course, with such simple questions, your analysis is especially subjective and speculative. If it’s a complex question, you’d be better off either splitting the test into iterative deployments, or just finding yourself some flesh-and-blood users. But this is an interesting way to get a quick sanity check on a design puzzle before you jump off that cliff.

And yes, amazingly enough, there are people on MTurk who are bored enough to do this for five cents.