Michael Conklin is MarketTools’ Chief Methodologist. He is a marketing research veteran and analytical
methodology pioneer with 24 years experience. He came to MarketTools after serving for 10 years as a
Senior Vice President and Team Leader of Analytic Services for GfK Custom Research Inc. in Minneapolis.
He is a frequent speaker at the American Marketing Association's Annual Advanced Research Techniques
Forum as well as at the American Statistical Association's Joint Statistical Meetings conference . In
addition, Conklin has been published in key marketing publications, such as Marketing News and
Marketing Research Magazine, as well as numerous statistical journals. He received his education at The
Wharton School.
About Zoomerang
Zoomerang ( www.zoomerang.com ) offers an easy way for businesses to get fast answers to
important questions. With more than 100 million surveys sent and counting, Zoomerang has grown
rapidly and has been adopted worldwide. The number of subscribers using Zoomerang surveys
has doubled every year since 1999, and Zoomerang has been featured in Fortune magazine, on
NBC News, and on the NBC Today show. Zoomerang is part of the MarketTools family, a leading
provider of technology and advisory services for conducting Web-based market research.
This White Paper is provided to our Zoomerang zPro subscribers as an added benefit to the zPro survey
service. Zoomerang provides best practices and research methodology that our subscribers find useful to
increase both the quality and impact of their surveys.
2
White Paper Measuring and Tracking Customer Satisfaction
Best Practices in
Customer Satisfaction Research
How satisfied are your customers? Are they more or less pleased with your company than they
were last year? How does your customer satisfaction level compare with that of your competitors?
What impact does it have on your company’s profits?
Guesswork and hunches won’t cut it. You need hard data. You need to measure and monitor
customer satisfaction—consistently and regularly. You need a customer satisfaction research
strategy and program.
You can do it yourself, but there are a few things you need to consider. This paper gives you the
benefit of the experience and lessons learned by industry experts to help you design, distribute,
and analyze your customer satisfaction research. We want you to act on insight, not instinct.
$80
$0
Existing Customer New Customer
Customer Type
Moreover, one bad experience can outweigh a whole lot of good experiences. Because of e-mail
and instant messaging, that bad experience can quickly be broadcast to dozens, hundreds, or
thousands of other customers, magnifying its impact. So if your business is doing something that
frustrates customers, you need to know right away.
It is critical to give customers the opportunity to provide feedback about their overall satisfaction
level and specific likes and dislikes. It is equally important to consistently measure and monitor that
input. Without an effective customer satisfaction research program in place, your company will be
losing business, missing opportunities, and putting itself at a competitive disadvantage.
3
White Paper Measuring and Tracking Customer Satisfaction
> Transactional surveys solicit feedback directly from the product or service user about that
particular encounter. They are conducted immediately after each customer transaction.
For example, a survey may be administered after a call center experience.
> Relationship surveys collect input from people who have an ongoing relationship with the
company and have had multiple transactions. They are regularly scheduled surveys-
often quarterly. The respondents typically are responsible for deciding whether to continue
the working relationship.
Transactional surveys are sent out at the discretion of whoever has contact with customers—but
the data should be collected immediately after the interaction while the experience is still fresh in
the customer’s mind.
Relationship surveys should be spread out over the course of a year. Here are a couple of points to
keep in mind:
> If there is only one data point for each year, a single event could have a large impact on
results.
> Research in the hotel industry reveals that satisfaction ratings are consistently higher in
the spring than in the fall.
> For companies with a sufficiently large customer base—at least 10,000—relationship
surveys can be conducted on a continuous basis but reported on a monthly basis to
address the effects of seasonality or single events.
In many cases conducting both transactional and relationship surveys may be appropriate. For
example, a company may conduct transactional research for customer service purposes and
relationship research for routine checkups on the health of the business.
4
White Paper Measuring and Tracking Customer Satisfaction
> Random selection. First, it is essential that a random selection of customers be contacted
to avoid introducing bias into how the customers were selected—and to help ensure that
the sample of customers is representative of the entire customer base. For example, if a
company surveyed only customers who contacted customer service, those individuals
may be very different from customers who have not contacted customer service.
> Margin of error. Next you need to establish the level of precision you want to have in the
accuracy of the results. The margin of error (also known as a confidence interval) is an
indicator of sample accuracy for random samples. It is the plus-or-minus figure that is
commonly reported with news polls, such as “plus or minus four points.” In a customer
satisfaction example, if the reported result is 80% satisfaction with a 4-point margin of
error, that means the true answer is somewhere between 76% (80 – 4) and 84% (80 + 4)
assuming the whole population of customers had been asked. The larger the sample size,
the more accurate the results (or the smaller the confidence interval).
> Confidence level. Then you need to determine how certain you want to be that the survey
results are within the margin of error. The confidence level indicates the level of certainty
that the survey results are within the confidence interval for random samples. Typically,
researchers use the 95% confidence level.
Together, the confidence level and margin of error together describe the certainty you have in the
precision of the data. For example, for a reported result of 80% satisfaction at the 95% confidence
level with a 4-point margin of error, you can say that you are 95% certain that percentage of
satisfied customers is between 76% and 84%.
The table below outlines the sample sizes needed for different customer bases at varying levels of
accuracy at the 95% confidence level.
Another important consideration in determining the sample size is how much analysis will need to
be done on groups of customers. To analyze customers in particular industries or regions, the
sample size should be adjusted to at least 75 people in each group.
A number of online sample-size calculators are available that will provide the sample size neces-
sary for a particular confidence interval, or the margin of error for a particular sample size. Just
search “sample size calculator” with your search engine of choice on the Internet.
Online data collection offers significant advantages over other modes of interviewing customers,
and you should use it whenever possible. The advantages include:
5
White Paper Measuring and Tracking Customer Satisfaction
> Speed. The Internet offers instantaneous distribution of survey and real-time accumulation
and tabulation of results. This allows for immediate data analysis, even while the survey is
still in progress. Because customer satisfaction results are used to identify problems and
fix them, the faster responses arrive, the faster they can be addressed. In contrast, mail
surveys suffer from long lag times and low response rates, as low as 5%. Telephone
surveys take longer because of declining response rates. Refusal rates for phone
interviews have reached 60% (AC Nielsen, 2004). With the ease of answering online
surveys, they can be completed faster and a broader segment of the customer base can
be reached. Depending on a number of variables—the relationship with the survey
recipients, the length of the survey, whether a reminder is sent, and whether an incentive
is offered—response rates for online surveys can be upward of 35%. Even for online
surveys in which there is no prior relationship with recipients, response rates can be 23%
to 31% (Quirk’s Marketing Research Review, 2005).
> Candor. People are more honest when their answers are not filtered through someone on
the phone. This is essential for research on sensitive subject matter where studies indicate
people are more likely to answer questions on the Web than they are on the phone or in
personal interviews. (Quirk’s Marketing Research Review, 2003). The removal of
interviewer bias and the elimination of the wait time for an interviewer to record results
also yields more candid and complete responses to open-ended questions. This is
particularly important when customers volunteer additional information to explain their
satisfaction ratings. Such responses provide insight into what a company is doing well
and frequently provide warning signs about the health of the business relationship.
> Cost. The Internet eliminates many of the costs associated with traditional marketing
research. Online surveys avoid postage and telephone costs as well as basic materials like
paper, staples, envelopes, and printing. Because it is self-directed, there is no interviewer
cost. Finally, it’s more convenient so the cost of offering incentives can be reduced.
Online and offline methods of data collection can also be combined. If offline methods are neces-
sary for part of the customer base, the data for customers who can be reached only via mail, in
person, or by telephone can be input to an online survey tool. That way all survey results can be
captured, reviewed, and analyzed together.
Consider the example of a bank’s customer satisfaction survey. The first question should be
“Overall, how satisfied are you with Bank X?” Then ask for specifics: their teller experiences, the
availability of ATMs, etc. Otherwise, the customers’ answers to the overall satisfaction question will
be influenced by their satisfaction with specific attributes of that relationship that they may not
have thought about until they were asked.
6
White Paper Measuring and Tracking Customer Satisfaction
An insightful alternative to the 5-point satisfaction scale for customer service is a 5-point expecta-
tions scale. It provides clear direction and allows customers a polite way to suggest that a
company has not done a great job.
7
White Paper Measuring and Tracking Customer Satisfaction
Similarly, question content and scales should remain consistent over time. Customer satisfaction
scores are typically evaluated on a quarterly basis or at regular intervals; and if the questions and
the scales change, evaluating performance over time is compromised.
Note: Scales can have a dramatic effect on satisfaction ratings, so be wary of writing questions to
drive up the numbers—especially if results are being used for compensation purposes. Be
cautious about interpreting competitors’ claims of satisfaction scores. Know how the question was
asked before jumping to any conclusions about the health of the business.
If you build your own survey be sure to include all the key questions:
Think of the customer satisfaction survey as part of an ongoing conversation with your customers.
Rather than cramming every question you have into a single survey, do multiple surveys throughout
the year on different topics.
8
White Paper Measuring and Tracking Customer Satisfaction
Interpreting
Survey Results
When you’re analyzing customer satisfaction survey results, the most important goals are
minimizing the low scores and improving the top scores. It is important to monitor the “top
two-box” satisfaction number, which is the combined percentage of those saying they are very or
somewhat satisfied. It is essential, however, to call out to management the proportion of customers
who are dissatisfied and to reduce those percentages. Insights into how to do that are found by
learning what those who provide high ratings have to say and reviewing the results of those who
are dissatisfied with performance.
100%
90%
25% Top two-box = 78%
30%
35% 38%
80%
70%
Very Satisfied
60%
Somewhat Satisfied
39%
35%
50%
Neither Satisfied nor
40%
40% Dissatisfied
40%
Somewhat Dissatisfied
5%
30%
14% Very Dissatisfied
18% 8%
20% 7%
10%
9%
10% 9%
13% Bottom two-box = 15%
11% 8% 6%
0%
Q1 Q2 Q3 Q4
Q1 Q2 Q3 Q4
Satisfaction Differential 33 44 58 63
(Top two-box – Bottom two-box)
9
White Paper Measuring and Tracking Customer Satisfaction
Competitive benchmarks for customer satisfaction can be difficult to come by because this is
typically proprietary information. Moreover, competitors’ results cannot be accurately interpreted
without knowing the specifics of question wording and the scales used. The satisfaction ratings for
the recipients of the Malcolm Baldrige Quality Award given by the U.S. Department of Commerce,
however, provide some insight into those companies that have been recognized for achievements in
quality and performance.
> Review the open-ended comments. Reading the volunteered comments following up on
the rationale for the satisfaction rating is essential. The remarks among the most satisfied
customers will shed light on what a company is doing well. They may also provide early
warning signs if satisfied customers voice concerns with a service, product, or customer
support experience. Dissatisfied customers will be vocal about the rationale for their
satisfaction rating and will provide actionable suggestions for how to improve.
> Cross-tabulate by satisfaction rating. A key analysis includes looking at the people who
are really happy and those who are not happy. Cross-tabulation features make compari-
sons between satisfied and unsatisfied customers fast and easy. The tendency in analyz-
ing customer satisfaction results is to focus on those company, product, or service
attributes that elicit the greatest dissatisfaction overall—price, for example. Most custom-
ers will want a lower price regardless of their satisfaction level. It is more revealing to
compare how satisfied and dissatisfied customers feel about various aspects of the
product or service relationship. The gap in satisfaction ratings for each attribute will help
prioritize areas for improvement—the areas with the greatest difference should have the
highest priority. In the example cross-tabulation table below, price has the lowest top
two-box satisfaction score among all customers. The gap between satisfied and dissatis-
fied customers, however, is biggest for customer service, so that is the area with the most
impact on overall satisfaction.
10
White Paper Measuring and Tracking Customer Satisfaction
4. Contact Customers
If possible, dissatisfied customers should be personally contacted to see if there is something that
can be done to improve their perception of the business. This is important not simply to increase
the odds of keeping that customer but also to prevent negative word of mouth. The personal
contact itself may succeed in doing that. Additionally, a discount or free product should be
considered. If policies or products are changed based on customers’ feedback, those customers
should be contacted to let them know about those changes and that their feedback was taken
seriously.
Additional Resources
Zoomerang recommends the following resources to learn more about customer satisfaction
research:
> Lester, Tom. “The Cost of Not Caring for Your Customers,” Financial Times,
January 30, 2006.
> McGregor, Jena. “Would You Recommend Us?” Business Week, January 30, 2006.
> Reichheld, Frederick F. Loyalty Rules! How Today’s Leaders Build Lasting Relationships
(Harvard Business School Press, 2001).
> For more information about online survey research, please visit the MarketTools Resource
Library.
11
MarketTools, Inc.
Zoomerang
150 Spear Street, Suite 600
San Francisco, CA 94105
U.S.A.
800-316-0662
sales@zoomerang.com
Copyright 2006 by MarketTools, Inc. This white paper can not be copied or distributed without
MarketTools permission. This whitepaper may be used by Zoomerang Subscribers only.