Remote Usability Testing

Buyers Guide 2013

by Matthew Niederberger

Opening Thoughts

Foreword
Remote usability testing has been around for some time now. Slowly but surely, as company budgets get tighter and the thirst for insights grows, innovation will keep nding new opportunities to create necessary solutions to help us excel as professionals. Remote usability testing is such a solution. With laboratory usability studies often starting with 5-gure costs, remote usability testing oers an aordable alternative for companies keen on learning from their customers and website visitors.

With lower costs to getting valuable insights, remote usability testing inarguably opens up the market for all companies to start researching. There are less rocks to hide under, less excuses to convey. Companies can simply no longer deny the fact that remote usability research helps improve their companys service, and ultimately their companys bottom line. This guide wont tell you which provider is the best. It is my goal to remain objective. All the service providers mentioned oer dierent solutions, with dierent features, at dierent prices. This guide is an attempt to point out to you, from a practitioners and business point of view, whats important to know about a service provider prior to doing business with them. I have a lot of respect for the service providers who oer us these

Yes, I agree, remote usability testing is not comparable to regular ol laboratory usability testing, but that does not mean that it cannot function as a surrogate, or additional research method. Many usability specialists argue that moderated tests can lead to more eective and valuable insights, than (unmoderated) remote usability testing. Although I understand the arguments, consideration must be given to the quality/price ratio. Even with a professional performing the analysis a ve person remote usability study can be up to 10 times more cost eective than a laboratory usability study.

amazing solutions for performing remote usability research. Even though some would say that a market leader has been established, new companies are already working hard to challenge that status quo. I hope that the providers of remote usability testing tools will continue to innovate and excel. Lets face it... in the end, it is the customer who benets, and that is all that really counts. Matthew Niederberger
Founder actualinsights*

www.actualinsights.com
ii

People say I make strange choices, but they're not strange for me. My sickness is that I'm fascinated by human behavior, by what's underneath the surface, by the worlds inside people.
- Johnny Depp

Chapter 1

What I do.

actualinsights* is a one-stop solution company geared towards supplying small, medium, and enterprise size companies with pragmatic insights into conversion and user experience opimization.

Section 1

What I do
HOW I CAN HELP YOU SUCCEED 1. Services for every budget 2. Receive actionable insights 3. Pragmatic research approach 4. Vast experience in multiple verticals 5. Internationally focused

A single goal
I started actualinsights* with one goal:

To help companies gain aordable and eective insights to help them increase revenue and customer satisfaction online. Whats in it for you?
With so many tools in the market today, it is hard to nd the right match for you and your company. My goal is to help you nd the right tool for your research needs. Even if you are just starting, or if you are an Enterprise company looking to leverage other forms of data such as web analytics.

How to nd me?
e: matthew@actualinsights.com t: +31648227889

By working closely with you and your company in a pragmatic fashion I can help you gain insights into customer attitudes and behavior. These insights will help you prioritize your optimization eorts, helping you x or improve those online features and functionalities, and increase revenue and customer satisfaction.

LinkedIN

How my experience ts in with your goals

My experiences range from online user research studies to web monitoring activities such as interaction, performance, voice of customer, and analytics. With many years of experience under my belt from working with various major companies I am condent that I can help you succeed, too.
5

Twitter FACEBO

Section 2

Services
Conversion Optimization
Take action from insight, but if you are not sure how, then let me help you. From mockups to prototypes, and projects plans to success. Lets get started today.

Website Performance Research

Find out how your websites performance in terms of page load and server response times are aecting your visitors experience.

Website Usability Testing

Test your own website and see how real visitors interact and think about it. Or test a competitor, and see what they like there.

Web Analytics
Are you looking beyond visits and page views? Make sure you really make the most of web analytics today.

Mobile Usability Testing

Have a mobile app or website? Perform tests a various mobile and tablet devices and learn what visitors think of it.

Workshops
Tired of constantly hiring expensive external consultants? *wink* I am more than happy to train you and your team on various topics.

Customer Feedback Analysis

Listen, analyze, and optimize. There is nothing more powerful than the voice of the customer to help you improve your business.

Public Speaking
From Google to Travelport, Finland to Spain, I will speak and share my insights and motivations during your exclusive event.
6

Section 3

Testimonials
Theres something fresh about Matthew Ive always appreciated as hes willing to share knowledge and skills through his blog as well as through informal chats and internationally recognized public speaking events, at which he excels. His community focused efforts reect his no-nonsense approach to gaining and sharing insights. Matthew is able to focus on business and customer values, drawing on his broad knowledge of qualitative and quantitative analytics tools hes gained through the many years in our industry. His
Aurlie Pols Mind Your Privacy

unique blend of digital analytics knowledge and user experience make him an ideal, pragmatic candidate for any company aiming to take their online presence and data driven decision making processes one step further.

Matthew is one of a kind; few are as dedicated, focused and knowledgeable as him. And make no doubt, Matthew knows what he wants. But above all, Matthew gets things done with contagious warmth and passion that Ive seldom experienced. Matthew Niederberger is no doubt one of the fastest rising stars on the European ecommerce arena.

Tony Markovski MassRelevance

7

Matthew is on top of new developments in UX and analytics and very eager to try out new methods and tools. He has a very clear picture of developments in the UX landscape. The variety of tools in his toolbox help him to gain a very good understanding of his users and the way they behave and think. Matthew is eager to learn, a smart guy, professional, and I look forward working with him in the future.

Paul Veugen Founder Usabilla

Matthew is a thought leader in his niche, Ive enjoyed his keynotes several times. We have regular talks about the tools in the market place and share our experiences in testing, analytics and travel in general. I always learn something new!

Wouter Blok Google

Chapter 2

Remote Usability Testing

What exactly is Remote Usability Testing and how can your business benet from it? Being a relative new technology can be benecial for some, confusing for others. With a handful of service providers in todays market, how can you best choose who to deal with?

Section 1

What is it?
Remote Usability Testing, also known as unmoderated usability testing, is a technique used by usability and user experience professionals to conduct usability tests without a lab, and not in the presence of the participant. Simply put, it is an alternative to laboratory usability testing where tests are: setup and launched via an online portal completed by participants sitting in the comfort of their own home, in theory, anywhere in the world analyzed by the test owner**, online, back behind his/her own desk (possibly at home as well) As in any usability test, the participants screen and audio (voice) are recorded for later analysis. In some cases, the participant is recorded as well, giving the test owner a complete view of the participants behavior online, thoughts spoken allowed, and possible facial expressions.
** Some providers oer enhanced services where they will analyze the videos for you saving you time (it can take an hour to fully analyze a 20 minute video), but also leveraging their expertise.

The latter is often overlooked, but often used as an argument in favor of laboratory usability testing. Recent technological breakthroughs, however, have proven that even facial expressions in the form of confusion, surprise, frustration, and more, can be analyzed with remote usability testing. Personally, I think the biggest dierentiator between the oine vs. online forms of usability testing is the communication between the host and the participant. Remote usability testing allows not only enterprise size companies, but even a oneman businesses, to discover critical insights in web/mobile user behaviors. Where tools like iPerceptions, Qualaroo, Usabilla, and Mouseow give us insights through customer feedback forms and interaction analysis, remote usability testing tools will give you a more richer experience in looking over the shoulder and into the mind of the participant. And to think, we are still in early phases of it all.
10

Section 2

How can it benet you?

TOP 5 BENEFITS FOR YOU 1. Perform aordable usability research 2. Fast study turn around times 3. Uses less internal resources 4. Helps gain valuable insights 5. Innovative technology

In a recent study* conducted among user experience specialists, in house and in agencies, the question was asked: What one thing do you think would help most improve user experience for your users? The majority of the respondents on both sides answered that user testing would help the most. It would help in not only improving the user experience but also in improving return on investments. A better understanding, proof of visitors struggling online, will help ght the common myths that: user experience is simply a buzz word companies know best what customers want user testing costs too much To help you better understand how remote usability testing can benet you, I have listed several of them here. The benets have helped me over the years, so again from a practitioners point of view, they should help you, too.

Better Understanding
Over the past years my biggest observation with usability testing in general is its unique power to create empathy among colleagues, but more importantly among stakeholders.
* eConsultancy User Experience Survey Report 2013 11

Empathy often leads to a willingness to better understand the users experience on your website or mobile app. This alone pushes your knowledge and advantage over competitors to a higher level.

Cost Eectiveness
If you have any experience at all with usability research, then I am sure that you at one point in time have seen the bill for one of these studies. I have never seen a study cost less than a signicant 5-gure total. Without alternatives, these kind of numbers would seem normal. Lets do the math for a remote usability research study, using an external consultant to perform the analysis: 10 participants $49 = $490 8 hours consultancy services $125/hour = $1000 - 2 hours for test preparation and setup - 5 hours for video analysis (30 minutes per video) - 1 hour post study consultation So for $1490 you can, in theory, conduct a remote usability research study. Yes, yes, I am underselling the consultancy hours, but in its basic form the numbers match up. Dont forget, it is not only the time that your are saving with setup and deployment of the research study. Laboratory usability research studies require even more time to prepare, especially with regards to recruiting panel members.

Panel Recruitment
Alright, great technology, but like other online user research tool, such as customer feedback surveys, I will need a huge panel to get some signicant insights. The above statement is what withholds a lot of people/ companies from expanding user research eorts. A medium level survey tool can cost anywhere between $49 to $99 a month, but without visitors giving feedback it wont do you much use. Remote usability testing tools like those mentioned in this guide dont suer from this. All the service providers mentioned will allow you to make use of their panel members for testing. The time consuming task of recruiting your own panel members is removed from the equation. Furthermore, all service providers give you control over segmentation options for their panel. From age to income, education to online skill levels.
12

Need more? It is quite common to see the feature allowing you to invite your own panel members for your study. TIP The service providers often guarantee a high level of quality with regards to the videos, so keep in mind that when using your own panel members this guarantee is voided.

Mobile remote usability testing can no longer be ignored. Not only can you test mobile websites, but also apps (paid & free). Some service providers even oer capabilities for testing unreleased iOS apps, perfect for those moments where you want to start early with your optimization cycle. On both the technical and logistical spectrum the service providers will arrange everything allowing you, as a business owner, focus on setting up the test and reaping the benets from the insights.

Mobile Usability
Talking about mobile adoption rates these days is like talking about the weather. Everybody does it and everyone still acts surprised as if it is an unexpected thunderstorm that will ruin their weekend plans.

Setup and Analysis

Has usability test engineering become a thing of the past? To a certain extent, yes it has. With todays remote usability testing tools there is denitely a lot that you can do yourself. It doesnt matter if you are a freelancer testing for a client, or an in-house specialist at some large company, the tools oer solutions that will help anyone get started today. Setting up a test is in all cases nothing more than a process of lling in forms. Even with test templates to help you get started in under a minute. If you are testing a specic hypothesis then some oine work will be required to gure out how to setup the test, but even this is a one-time thing. The more you setup tests, the more you learn what capabilities there are, the more ecient you as a practitioner will become, saving yourself, or your company, valuable time and money.

Mobile usability testing using remote research tools is a fact. Testing can be performed on both Android and iOS devices without the use of special software. The most common solution is by using a webcam connected to a computer, pointed towards the mobile device (often top-down). Using this method allows you not only to see and hear the basics, ie. the participants device screen and voice, but also to see the movements of the hand.

13

I often compare it to baking. Simple cupcakes dont require much preparation and can be prepared in a breeze, but a multi-layer cake with all the trimmings will take more time and eort. There is no getting around it, neither in baking, nor remote usability test setups. Analyzing the results is what gives you the insights. Luckily most service providers oer a collection of online analysis tools to make your life easy. Using these online analysis tools will help you keep note of any interesting behaviors and comments simply by adding annotations to the video. Not stopping there, some service providers allow you to create small clips from the video. These clips can later be viewed in a highlight video. The highlight videos look and feel like compilation videos. Time is money, so spend it wisely and by using these form of compilation videos will allow you to see several insightful moments from all participants in a single video clip. Are you not sure if you have the skills to analyze remote usability tests? I can understand. In the study I mentioned earlier, many companies indicated that a lack of resources and/or specialist knowledge was holding them back from starting with user testing. Luckily for them professional services are available, not only from myself, but also from the service providers themselves. The biggest benet to you is, is that if you want to analyze yourself, you can.

Sharing Insights
After performing your analysis online you need to take action. Action in the sense that your insights are worthless if no one acts upon them. Maybe you found a technical problem during your analysis, or maybe the majority of the participants had trouble nding your hero products call-to-action button. Sharing these insights with people responsible for the design/interaction of the website or mobile app is essential. Sharing can be done in several dierent ways, depending on the service provider you choose. The most widely available feature is downloading the videos. Now although downloading sounds great, how many of your colleagues will actually sit and watch several 15 minute videos? Not many, believe me.
14

The two best options for sharing video are: sharing an individual clip from a single participant creating a compilation video from multiple clips Because of the compilation options that several service providers oer, you can easily create videos with highlights from all the sessions, or highlights on a specic insights, and share these. The last important sharing option, is the option to share your annotations. As youve just read, annotating the video with your insights during the analysis is a common feature. Downloading those annotations, strangely, is not. Some service providers oer it and leverages your capability to sharing insights with your team to a stellar level.

To help you get an idea of the participants thoughts after having participated in a study, service providers oer you the opportunity to ask questions. These questions are often asked at the end of the video/audio recording part, and participants must type their answers in a form. Ranging from 3 to 100 post test questions, there is a dierence among service providers in terms of limits. Either way it is an extra chance for you to pick the participants brain. Question types range from open ended questions to radio button form elds, from checkboxes to likert scales. The most common question you see among these surveys are NPS related questions. In some cases scores from some of the survey questions, or custom questions provided by the service providers themselves can be benchmarked against other companies in your industry. Preparation, and some caution, should be observed with the survey questions. Survey questions can be a powerful form of insights, but asking the proper question in an unambiguous manner is key to gaining signicant insights from survey responses.

Post Test Surveys

Collecting user experience data is what it is all about with remote usability research. This qualitative form of data is immensely valuable but costs a lot of time and eort to collect, sort, and report.

Time To Insights
Did I save the best for last? Maybe, that is for you to judge. When I talk about remote usability testing I do seem to talk about turn around times a lot. With that, I mean the time it takes for a study

15

to be completed, from when a study is launched, until all the participants have completed and uploaded it. I put on a smirk every time I hear marketeers talk about their biannual usability research studies. Studies that take weeks to prepare, and weeks to analyze. Yes, there are benets to laboratory research over the remote kind, but every day the distinction between the two is gradually disappearing. Why wait 2 weeks, a month, or even more to get your study results. When looking at todays remote usability research providers, the longest turn around time is still only a mere 72 hours. Of course, you still need to analyze the results or have them analyzed for you by a providers professional services, but like I mentioned early, through the use of online analysis tools even this becomes less time consuming, giving you the time to focus more on your business.

16

Chapter 3

Service Providers
With three main players in the market, it can get confusing who is the right match for you. Although the technology seems similar, there is quite a dierence between features, pricing, quality, and even turn around times.

Section 1

UserTesting.com
TOP 5 KEY FEATURES 1. Video, audio, and written feedback within an hour 2. Test any website or mobile app, including your competitors 3. Observe participants in their natural environment 4. Large panel ensures that participants match your target audience 5. Improve your conversion rate or your money back

UserTesting.com provides the fastest and most aordable web, desktop and mobile app testing in the market. In less than one hour, the company gives marketers, product managers and UX designers, on-demand access to users in their target audience, who deliver audio, video and written feedback on websites or apps. UserTesting.com is also the only service that allows customers to test unreleased apps and get detailed feedback within a day. Enterprise solutions are also available for clients who need a more robust testing solution. In addition to UserTesting.coms standard service, enterprise clients have access to a dedicated research team, an expanded recruiting pool with millions of real people, user intercepts, moderated testing, and access to industry experts like Steve Krug, Brian Eisenberg, Tim Ash and more.

UserTesting.com customers include

VIEW SAMPLE VIDEO

LINK LINK

18

Used by the top 10 web properties in the U.S., UserTesting.com has run hundreds of thousands of usability tests. UserTesting.com is headquartered in Mountain View, CA. User panels are currently available in the U.S., Canada, and the UK. For more information, visitwww.usertesting.com or Phone:1-888-877-1882 Email: sales@usertesting.com Twitter: @UserTesting

19

Section 2

Whatusersdo.com
TOP 5 KEY FEATURES 1. International panel 2. SmartPhone & Tablet Testing 3. Enterprise Plans & Agency Packs 4. Recruitment options 5. Analysis by WhatUsersDo

Based in Europe, WhatUsersDo is one of the most established providers in the Remote Usability market. Serving over 1,500 Digital Professionals from their London HQ they have been operating for more than ve years - regularly testing across 20+ countries in multiple languages. WhatUsersDo clients are as diverse as EasyJet, Telefonica, Tesco, Virgin Money and Sage as well as thousands of SMEs any organisation that wants to put user insight at the heart of their decision-making. WhatUsersDo are transforming the working practices of organisations who recognise that basing design decisions on user insight, rather than internal opinions or hunches, makes for a better experience for their customers and more successful digital properties. They do this by answering the why of user behaviour (e.g. Why do people leave my website or App?) that completes the partial picture that analytics packages provide (e.g. the what and how many of Google Analytics).

VIEW SAMPLE VIDEO

Whatusersdo customers include

LINK LINK

20

WhatUsersDo oers two levels of service: 1. Pay As You Go self-serve testing for SMEs who need actionable feedback from users a few times per year. 2. Managed Service for larger clients and agencies who want to embed UX Testing in their processes and test throughout the Development Lifecycle. The Managed Service plan oers an enhanced service level that includes expert advice in test design, analysis of results and advanced testing allowing clients to truly embed UX Testing in their design and development processes. On these enhanced plans customers benet from: - enhanced panel options: selecting users from the WhatUsersDo international consumer panel, invite their own customers or recruit on-demand - full-lifecycle testing: from concept through to wireframe to live site benchmarking - multi-channel testing: including desktop, tablet and SmartPhone (or even all three) - international testing: run multi-lingual tests with analysis (and participants spoken thoughts) presented in their own language. For more information on service levels contact sales@whatusersdo.com or call +44 (0)207 036 1950 or visit www.whatusersdo.com Twitter: @whatusersdo
21

Section 3

YouEye.com
TOP 5 KEY FEATURES 1. Emotion Recognition technology 2. Webcam recording 3. Dedicated research team for study creation and analysis 4. Custom tester demographics 5. Human review of each result forguaranteedquality

YouEye is an online research platform that features remote usability testing. YouEye empowers companies to quickly identify the biggest opportunities to improve their products, websites or services. With webcam-based usability testing, study analytics and customized feedback communities, it's never been easier or more aordable to build amazing experiences. YouEye has worked with leading companies including Unilever, Walmart Labs, SapientNitro, Airbnb and others, and oers customized oerings to enterprise and agency customers based on their specic needs and business goals. YouEye also oers a touchless solution for rapid feedback. Participants for these studies start at $39, and most results are available within 48 hours.

YouEye customers include

VIEW SAMPLE VIDEO

LINK LINK

22

By implementing the YouEye solution into your organization, you can gain a huge edge on competitors, delight your customers and increase your conversions. You can learn more about YouEye athttp://www.youeye.com. Reach out to us at hi@youeye.com, andfollow @YouEyeTests on Twitterto keep up with the latest updates.

23

Chapter 4

The Comparison
How do these service providers weigh up against each other? On the following pages critical features, from a practitioners point of view, will be compared. From Pricing Plans to Test Setup, and from Analysis to Customer Service.

Section 1

Plans & Pricing

Whatusersdo Start price per participant Offers free trial Offers bulk discount Offers Premium plans $39 30 (approx $45) $39

25

Section 2

Test & Recording Types

Whatusersdo Browser testing Prototype websites Upload prototype les Mobile/Tablet apps Unreleased apps Screen & Audio Recording Webcam Recording Emotion Registration Max recording time per test *** ** 15 mins ** ** 20 mins ** 15 mins
= Innovative and/or Unique Feature

Legend * iOS only ** Premium accounts only (ie. Enterprise level plans) *** Suggested limits 26

Section 3

Panel Recruitment

Whatusersdo Panel Members Included Panel Member Nationalities Panel Segmentation Use own panel members Screen Participants Rate participants US, CA, UK ** US, UK, AUS, FR, DE, IE, IT, NL, ES, PL *** US **

= Innovative and/or Unique Feature

Legend ** Premium accounts only (ie. Enterprise level plans) *** UK panel members only 27

Section 4

Test Setup & Analysis

Whatusersdo Setup wizard/walkthrough Test templates Task library Multi language interface Post Task Survey NPS Add video markers Add annotations Create clips Create Highlight Video

Legend ** Premium accounts only (ie. Enterprise level plans) 28

Section 5

Customer Service & Sharing

Whatusersdo Money back guarantee Quality Assurance Project Management Analysis Service Average test turn around time Email links to clips/notes Download Videos ** ** < 1 hr ** ** <48hrs ** ** < 48hrs

= Innovative and/or Unique Feature

Legend ** Premium accounts only (ie. Enterprise level plans) 29

Chapter 5

Thoughts for 2014

Whats in store for 2014? Where will new technological advances take the eld of remote usability testing? Let the service providers and myself share our thoughts with you.

Co-Founder Dave Garr Computing is moving from one screen a computer accessing the web to four screens: a computer, a tablet, a phone, and a smart TV. The next big UX trend is understanding that user experience does not mean just web site experience or mobile experience or phone experience or store experience. Companies will stop designing each experience in a vacuum. Theyll start putting together the holistic understanding of the entire user experience. I also see a lot of work being done on sentiment analysis -- software that mines text for meaning and insight. Its often used to extract opinions and emotions from social media to help companies determine how people feel about their products. Sentiment analysis is an extremely dicult problem, but its a problem that will eventually be solved. When the problem is solved, itll cull through a massive amount of text and automatically call attention to the biggest, most frequently mentioned issues. This will increase the value of qualitative tools like open-ended survey questions and transcripts of user testing.

CEO Kyle Henderson The future of usability testing will look and feel like the analytics products we use today. We will see big data meet qualitative feedback. New technologies and forms of measurement will appear. Cross discipline collaboration for discovering insights will also move to the web. We'll have teams engaging around UX quality and improvements through online social interactions. The future of usability is in the cloud.

Founder Lee Duddell In many ways, were only just starting to see remote UX testing become widely adopted. A survey we ran with Econsultancy at the end of 2012 showed that although 78% of brands were committed to providing the best possible UX, over 60% of them were still basing digital design decisions on hunches and opinions. Its clear from this that, although available for many years, lab-based approaches do not provide the answer to closing this gap. In just the past 12 months weve seen remote UX testing becoming truly embedded with our clients who now see it as an essential part of both their development process as well as a means to conducting user research to inform strategic decisions. In the coming months and years we will be developing WhatUsersDo to support even better decision-making and ultimately improved return from digital channels. While I cant reveal all of the details, expect to see: - increased multi-channel capabilities - more support for beyond usability insight and testing - integration with third party tools - an emphasis on user-generated, rather than purely task-based, insights to better understand users natural behaviour.

Founder Matthew Niederberger As a practitioner I have been lucky enough to have tested all the tools in this guide. One thing to keep in mind when considering one of the fabulous tools is that no tool is truly 100% alike. Each tool has its own set of useful features, each suited for specic needs, your needs. In 2014 I expect many of these dierences to grow. The feature gap between the tools is what makes them unique, and I think that every single service provider will continue to strive for that uniqueness, even though the basics will remain the same. A trend that I am starting to see and one that I hope continues to grow is more quantiable data, more usability metrics. Analyzing user videos takes a lot of time and eort and remains subjective to the professional analyzing it. By incorporating a small amount of metrics into the tools will help speed up the understanding and adoption of the solutions within companies. I think that 2014 will see more expansion of these solutions into the European arena oering more multi-lingual panels. The service companies, mainly the US based ones, will have to look for alternative recruiting methods, beyond the often used Mechanical Turk. There are challenges ahead, but also plenty of opportunities.

Copyright

2013 Matthew Niederberger/actualinsights*

xxxv