Our First User Study

Recently, I have been struggling to find the right balance between developing 10Pens and marketing Keeples. (10Pens, by the way, is another project that I am working on with Sidharth, which you can read about here). After pounding my way through at least 30 hours of Flex tutorials last week, I have been rigorously coding up 10Pens — all the while forgetting to execute our marketing plans for Keeples. It is not a coincidence that our new trickle of users into Keeples has recently diminished to a near standstill. Our goal was to reach 100 unique questions asked on Keeples between January 20 and February 13, and up till yesterday, the count was upto a grand total of 10.

Thus, I decided to make a new marketing effort for Keeples! Today, I dressed up, gathered my laptop and a bag of candy, and drove over to a gas station that my parents manage. I set up shop next to the cashier and fastened a sign to the counter with some scotch-tape: “FREE CANDY FOR USER STUDY”.

The Results:

The results were quite positive. During the two hours I was there, I asked eleven people to participate, and six of them agreed. The feedback I got from the users was pretty informative. For example, I asked a couple of people for their opinions about our weekly email system:

Me: “Suppose we picked out 5 questions for you each week to answer, based on your expertise and interests. Would you think of the weekly email as a nuisance? Or would you actually answer the questions?”
User: “If I regularly used the site, then of course I wouldn’t mind.”

Another person from the study remarked that he would only answer questions which came directly from his friends. Strangers, he did not care for. Our user participation model, by the way, is a huge topic of interest for us, so we would appreciate any and all feedback from our readers about it.

The Challenges:

There were two main challenges I faced while performing the study:

1) Customer demographic. A good portion of the people coming in to the station did not speak much English and used the internet rarely if at all. Many people did not even have an email address. I could only ask a handful of people to participate, but fortunately the few people who I did ask were very helpful.

2) Question Quality. Many people were having difficulty coming up with a question on the spot. A couple of them did ask good specific questions which require human knowledge, such as “Where can I learn Bollywood dancing in San Jose, CA?”. However, most of the users asked really generic questions such as, “What is the capital of Greenland?” Keeples is certainly useful for the former question, because we will send out that question to everyone in our community who lives in San Jose and is interested in dance. However, you don’t need to use Keeples to figure out the capital of Greenland…
To solve this problem in the future, I will try to find ways to encourage users to ask high-quality questions.

What I have learned:

1) The sign helps a lot! It seems to make your user study look much more professional. I tried doing the same user study yesterday without a sign, and users were significantly less receptive.
2) Start out be telling them what they get out of it. “Hey, we’re giving out free candy today … if you help us out with a user study.”
3) Explicitly tell them how long it is going to take: “It will only take 30 seconds to a minute”. It does sound obvious, but it’s easy to forget how important that little piece of information is from the users’ point of view. I got this advice directly from one of the users in our study, who happened to be a researcher himself.
4) Have a visible basket of candy which you can point to.
5) A user was much more likely to agree to the user study if he saw someone else agree to it. Three of the user studies happened back-to-back-to-back. One person agreed; then the person standing directly behind him agreed; then the person standing directly behind HIM agreed. Nice.

The key here seems to be to give people something to LOOK at instead of simply verbally telling them something. Also, reassuring them that it is a low commitment on their part and reminding them that there is a juicy candy at the end of the study seems to help. Quick, simple, and clear was the goal. In fact, during the user study, I gained a new appreciation for all the effort we had put in to making the website as simple as possible to use. The fact that there was no registration/signup required for Keeples was absolutely essential to getting these users to participate, as many of them were rather impatient about the time investment.

Overall, the user studies seem to be quite effective. Tomorrow I will go back to the station, this time for six hours. Hopefully, I can recruit another 15 unique questions that way. I’ll post the results here after I have performed a few more user studies — and burned through a few more boxes of candy.