Paid Website Usability Testers
Kazi Ziaul Islam
Strayer University
Human Computer Interaction
CIS375
Prof. Kourosh Samia
December 8, 2014

Paid Website Usability Testers
Two attributes determine if a website is useful and they are usability and utility. Utility is whether or not the website provides the required information or features. Whereas, “usability is a quality attribute that assesses how easy user interfaces are to use.” (Nielsen, 2012, para. 1) You cannot have one without the other. Suppose your site is brimming to the top with lots of information and has all the bells and whistles, but if the Interface design (ID) is poor and as a result the User Experience (UX) suffers then no one is ever going to spend the time and effort to discover those great features. This is the main reason why companies must perform usability evaluation. Although the heuristic evaluation method may take less time and is usually less expensive, the “heuristic evaluators could not play the role of real users and could not predict actual problems users might face while interacting with the sites” (Hasan, Morris, & Proberts, 2012, p. 726) User Testing is essential prior to going live in a production environment and can done on a budget. Katie Sherwin of the Nielsen Norman Group reiterates that “testing doesn’t have to be expensive and by testing with just 5 users, you can uncover 85% of the issues.” (Sherwin, 2014, para. 8) After all it is an iterative process and the law of diminishing return comes into play in user testing as well as seen in Figure 1 below.
Figure 1. from “Why You Only Need to Test with 5 Users” by Jakob Nielsen (March 19, 2000) retrieved from http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

The point Nielsen makes is not that you stop testing after 5 users. Rather he says that it would be a better use of your resources to perform your initial test, uncover the issue, resolve it and then test again with 5 different users. This way you can test quickly and be more effective.
Usertesting.com is a website designed to recruit usability study participant for website owners. They give the opportunity for anyone capable of meeting some basic criteria to earn money to perform web testing or mobile testing. Currently, testers can earn $10 for web testing and $15 for mobile testing. While it’s typical for companies to provide monetary incentive to test participants, the rates they pay out can vary greatly depending on the demography and suitability of the user profile for the study. According to a report generated by the Nielsen Norman Group, The “average incentive paid to external users was $64 per hour of test time.” (Sova & Nielsen, n.d., p. 6) Typically high-end professionals can earn almost four times as much as non-professional users. One of the question that inevitably arises when considering paying website usability testers is how reliable is the information that is gathered from them. “The reliability or consistency of a method is how well it produces the same results on separate occasions under the same circumstances.” (Rogers, Sharp, & Preece, 2011, p. 471) There are numerous studies that been done to discover this issue. The results are still inconclusive. While there are well-known cases that some of data may be skewed based on numerous factors, following good usability evaluation guidelines can mitigate these factors. For example, The National Institute of Standards and Technology (NIST) is an agency of the U.S. Department of Commerce, which provides such guidelines. There is the trend that some users tend to be more negative when online, however this can also mean that if you design the activity or questions properly you can typically elicit a more honest answer. Contrary to what people might think about in the wild studies, testing done through usertesting.com where multiple data points are being captured and analyzed and there can be fine granular control of the demography and profile of testers, the information can be considered pretty reliable.
Usability Evaluation Methods or UEMs were classified info four general categories by Nelsen and Mack: automatic, empirical, formal and informal. (Hasan, Morris, & Proberts, 2012, p. 707) The evaluation method being used by usertesting.com is a user-based UEM, where the user interaction is recorded via a screen recorder while the think-aloud protocol is used to capture user’s verbal comments. Mouse movements, clicks and keystrokes as well as spoken comments while the user is using the website is captured. Once the recording session has been completed, an online questionnaire is filled out by the user to gather reaction to the website and evaluate the user experience. Typically the four questions asked are as follows: 1. What frustrated you the most about this site? 2. If you had a magic wand, how would you improve this site? 3. What did you like about the site? 4. How likely are you to recommend this site to a friend or colleague (0=Not at all likely, and 10=Very Likely)? These three data streams, on-screen recording, voice recoding and post-test evaluation via questionnaire are combined and analyzed. One of the potential problems that may occur when using the think-aloud protocol is when the user encounters a problem; they will tend to go silent as they try to think about how to resolve the matter. This is a known problem and with proper instruction and training this can be overcome.
There are two product lifecycle evaluations that are usually performed. To check if a product continues to meet user’s needs during the design phase are known as formative evaluation. “Evaluations that are done to assess the success of a finished product are known as summative evaluations.” (Rogers et al., 2011, p. 437). Companies can choose at which stage they want to test but it is recommended that both of these evaluations are done with small number of test users to keep costs down. 5 to 12 users is the standard but as we have seen staying toward the lower end of that spectrum gives you the most bang-for-the-buck.
Evaluations can be classified into three broad categories: Controlled setting, Natural setting, and Non-user involved setting. The natural settings for of the test environment for web users, where users are in their natural environment like on their own computer in their home, is used to determine how the product is used in the real world. “Remote studies of online behavior”, such as the one conducted by usertesting.com, “can be conducted to evaluate natural interactions for a lot of participants in their own homes, cheaply and quickly. (Rogers et al., 2011, p. 436) “In studies in natural settings, the onus is on the evaluators to fit in with the participants and to cause as little disturbance to participants and their activities as possible.” (Rogers et al., 2011, p. 461) Since user never interacts with the evaluators directly in this case there is no likelihood of corrupting the data. Furthermore, “a benefit of uncontrolled settings is that unexpected data can be obtained that provides quite different insights into people’s perceptions and their experiences of using, interacting, or communicating through the new technologies in the context of their every day and working lives.” (Rogers et al., 2011, p. 442)
Since the Testing company selects their tester from a large pool of potential users it is likely they can get the proper demographic mixture and sampling needed by their customers. “Validity is concerned with whether the evaluation method measures what it is intended to measure. This encompasses both the method itself and the way it is performed.” (Rogers et al., 2011, p. 471) The incentive for providing accurate information is high as the compensation they are offering is rewarding. Once the three data stream are collected and analyzed it can be compared against other studies and cases for check for validity.
Useresting.com is great for companies who are either not setup for recruiting study participants or do not have the resources to hire specialized recruiting agencies. Since recruiting participants to test a University website is typically not a problem, we can save money by doing everything in-house. “It’s an empirical fact derived from observing many prospective students using many university sites that these users are often frustrated or thwarted by the frequent usability problems on university sites.” (Sherwin, 2014, para. 1) A non-monetary incentive such as a small-gift or a voucher to the school bookstore would be helpful and keeps costs minimal. We have to ensure that we not only get a good mixture of current and prospective students, we should also gather good cross-section of the population by doing selective sampling. Particularly the use of diversity sampling, where you are specifically looking for differences such as age, gender, previous schooling, would be a good goal to keep in mind when recruiting. To avoid biases I would opt to perform a remote observation in a controlled lab environment where I would provide the participant with set tasks to perform on the University website, while recording their interaction, followed by an online questionnaire. I would not use the think-aloud protocol as it may distract the user from performing the tasks at hand if they are uncomfortable speaking to the computer, which may feel unnatural to them. Once I collect the On-screen recording and questionnaire I would have the passive observers evaluate each user test and analyze the data. I would use a group of five users at a time per Nielsen’s suggestion. The information gleaned from this test would then drive us to improve our question and formulate better testing methods.

References
Hasan, L., Morris, A., & Proberts, S. (2012, July). A comparison of usability evaluation methods for evaluating e-commerce websites. Behaviour & Information Technology, 31(7), 707-737. http://dx.doi.org/10.1080/0144929X.2011.596996
Nielsen, J. (2012). Usability 101: Introduction to Usability. Retrieved from http://www.nngroup.com/articles/usability-101-introduction-to-usability/
Rogers, Y., Sharp, H., & Preece, J. (2011). Interaction design (3rd ed.). Chichester, U.K.: Wiley.
Sherwin, K. (2014). University Websites: Top 10 Design Guidelines. Retrieved from http://www.nngroup.com/articles/university-sites/
Sova, D. H., & Nielsen, J. (n.d.). 234 Tips and Tricks for Recruiting Users as Participants in Usability Studies. Retrieved from http://www.nngroup.com/reports/how-to-recruit-participants-usability-studies/