Measuring customer satisfaction of service based on an analysis of the user

generated contents: sentiment analysis and aggregating function based MCDM

approach
Daekook Kang1, Yongtae Park2
1

Department of Industrial Engineering, Seoul National University, Daehak-dong, Gwanak-gu, Seoul, 151-744, Republic of
Korea (phone: 82-10-3152-2537; fax: 82-2-878-3511; e-mail: daekooky@snu.ac.kr).
2
Department of Industrial Engineering, Seoul National University, Daehak-dong, Gwanak-gu, Seoul, 151-744, Republic of
Korea (phone: 82-10-3152-2537; fax: 82-2-878-3511; e-mail: parkyt1@snu.ac.kr).
opinion mining. Some studies classified sentiment of
words extracted from review document [5,6,7]. Other
research focused on subjective expressions [8,9],
subjective sentences [10], and topics [11,12]. However,
previous research did not provide practical information
for evaluating customer satisfaction based on the
customer review because they just focused on the
technique itself. Thus, there is a need for using sentiment
analysis appropriately to analyze customer satisfaction
based on the customer review.
Multi Criteria Decision Making (MCDM) approach
can be the most suitable method to support an analysis of
customer review with sentiment analysis. This is because
service attributes that affect customer satisfaction are
much different among various categories compared with
those of product. Also multiple attributes affect customer
satisfaction of service simultaneously and weights of
service attributes are different. Because of this, sentiment
analysis cant cover all of these aspects single-handed. It
is reasonable to use MCDM approach to support an
analysis of customer review with sentiment analysis
effectively because MCDM approach can deal with multi
attributes for evaluating service simultaneously.
Accordingly, this study proposes a framework for
measuring customer satisfaction of service based on an
analysis of the customer review by using sentiment
analysis and MCDM approach. Especially, because of the
ease of usage and interpretation, aggregating function
based MCDM approach is used in this study. The
proposed framework mainly consists of two parts: data
collection and preprocessing and measurement of
customer satisfaction. In the stage of data collection and
preprocessing, dictionaries of service attributes and
sentimental words are constructed based on the review
data by using text mining. Then, keyword vectors of
customers opinions are constructed by using sentiment
analysis. In the stage of measurement of customer
satisfaction, measuring of customer satisfaction is
conducted by using three aggregating function based
MCDM methods. Lastly, validity test is conducted with
various results derived from different MCDM methods.
The proposed framework can provide useful information
for diagnosing customer satisfaction and propose
directions for improvement of service through review data
of service.
The rest of this paper is organized as follows. In
section II, methodological background of this study is
provided. The section of first introduces sentiment

Abstract User generated content in the Web is a main

channel to monitor customer satisfaction of service.
Especially, customer review is recognized as the core of user
generated content. Even though previous studies analyzed
customer review using sentiment analysis, these studies have
limitations in providing practical information for measuring
customer satisfaction of service. Multi Criteria Decision
Making (MCDM) approach is suitable for supporting an
analysis of customer review with sentiment analysis because
in service, multiple attributes affect customer satisfaction
simultaneously. In response, this study proposes a
framework for measuring customer satisfaction of service
based on the user generated contents using sentiment
analysis and MCDM approach. The proposed framework
can provide practical information about customer
satisfaction and directions for improvement of service.
Keywords Customer satisfaction, user generated
contents, sentiment analysis, MCDM approach

I. INTRODUCTION
With the rapid technological advance especially in
internet, e-business and information sharing has been
activated [1]. The fact that many business activities are
occurred through the computer and internet reveals the
core issue of how e-business can make high customer
satisfaction [2]. Especially in the service, this issue has
significant importance because customers want to get
much customized services in e-business [3]. Thus it has
become important to be able to monitor and enhance
customer satisfaction in e-business.
User generated contents can be recognized as a major
channel to listen and monitor a voice of customer in the
service [4]. This user generated contents mean all
materials on the website that is uploaded, viewed, and
downloaded by the users. Among many kinds of user
generated contents, customer review is considered as the
core of user generated contents because it contains direct
experiences of customers. Also, customer review has the
merit in terms of reliability since customers write review
voluntarily, whereas survey method that is generally used
to investigate customer satisfaction can lead to inaccurate
results because of problems in contents of survey and
attitude of respondent.
Some previous research have been focused on their
efforts to extract useful information automatically from
the customer review by using sentiment analysis and

978-1-4673-0110-7/12/$31.00 2012 IEEE

244

Proceedings of the 2012 IEEE ICMIT

analysis. Also, three aggregating function based MCDM

methods, Technique for Order Preference by Similarity to
an Ideal Solution (TOPSIS), Serbian: ViseKriterijumsa
Optimizacija I Kompromisno Resenje (VIKOR), and
Grey Relational Analysis (GRA), are briefly explained in
this section. In Section III, the proposed framework is
introduced with the overall process and detailed
procedures. Lastly, section V concludes the paper with
some contributions and observations about future research.

TOPSIS is presented by [14] with reference to [15].

The basic concept of TOPSIS is simple. The chosen
alternative should be as close to the ideal solution as
possible and as far from the negative solution as possible.
Ideal solution is constructed as a composite of the best
performance values in terms of each criterion in decision
matrix while the negative solution is the composite of the
worst performance values. In TOPSIS, the n-dimensional
Euclidean distance is used to calculate distance from ideal
solution or negative solution.

II. METHODOLOGICAL BACKGROUND

(2) VIKOR

A. Sentiment analysis

VIKOR is a compromise ranking method to optimize

the multi-response process [16]. This method is used for
ranking and selecting from a set of alternatives in the
presence of conflicting criteria based on the closeness to
the ideal solution. In VIKOR, each alternative is
evaluated according to each criterion and ranked by
comparing the closeness to ideal alternative.
Compromising ranking index that represents preference of
each alternative is derived by considering both maximum
group utility and minimum individual regret of the
opponent.

Sentiment analysis is the technique for identifying

how sentiments are expressed in texts and whether the
expressions represent positive negative opinions toward
specific subject [13]. This method includes several
identifications such as sentiment expressions, polarity and
strength of the expression, and their relationship to the
subject. Among them, most of work in the field of
sentiment analysis focuses on identification of polarities
of sentiment expressions.
A general approach of sentiment analysis is to find
sentiment expression for a given subject and distinguish a
lexicon of positive or negative words. There are three
types of lexicons. First one is a lexicon that has a positive
polarity such as beautiful. Second, words such as horrible
are classified as a lexicon that has a negative polarity.
Lastly, there is a lexicon that has a contextual polarity
which means that a sentiment of the word varies
depending on the context.
In this study, customer satisfaction of service is
measured based on the review data by using sentiment
analysis. Polarity of words in review data to the service
attribute is analyzed. Then, rating index is measured by
using polarity of words with respect to each service
attribute for a given service alternative.

(3) GRA
Grey relational space is the concept that is proposed
by combining system theory and space theory [17]. The
concept of grey relations emphasizes the greyness that
means incomplete information. As a method for analyzing
these grey relations, GRA is proposed to understand the
uncertain relations between things, components of
systems. GRA assumes that relationships can be identified
among complex factors in a system. This method provides
the correlations between the reference (desired) factors
and other compared (alternative) factors of system based
on the degree of similarity [18].
III. RESEARCH FRAMEWORK

B. Aggregating function based MCDM approach

A. Overview of the proposed framework

MCDM are well-known acronyms for multiple

criteria decision making that is method for solving
decision and planning problems involving multiple
criteria. There are many MCDM methods. Among them,
aggregating function based MCDM methods such as
TOPSIS, VIKOR, and GRA provide simple value to rank
each alternative by considering the weight of different
criteria. Therefore, it is easy to determine preferences of
alternatives. Also, aggregating function based MCDM has
the merit on the easiness of mathematical calculation. For
these reasons, in this study, three aggregating function
based MCDM methods, TOPSIS, VIKOR, and GRA, are
used to measure customer satisfaction of service. Brief
explanations of each MCDM method are as follows.
(1) TOPSIS

245

Proceedings of the 2012 IEEE ICMIT

derive information about customer satisfactions for the

service.
C. Construction of dictionaries of service attributes and
sentimental words
In this phase, two dictionaries are constructed with
reference to the review data. Firstly, dictionary of service
attributes are constructed. To do this, as an ex-ante work
for analyzing customer satisfaction, general service
attributes that customers usually consider are extracted
based on the literature review. Then, service attributes are
hierarchically structured with parent components and subcomponents. Parent components consist of general service
attributes derived from the literature review and subcomponent is extracted from review data by using text
mining.
By extracting sub-component for making hierarchical
structure of service attributes, dictionary of service
attributes are constructed. Generally, service attributes in
the review data are expressed as a noun phrase. Thus, text
mining is conducted to extract service attributes apropos
of noun phrase in review data by using Term Frequency
Inverse Document Frequency (TF-IDF). Lastly, derived
service attributes are corresponded to the sub-component
of dictionary of service attributes.
Secondly, sentimental words in review data are also
extracted to construct dictionary of sentimental words.
Since sentimental words are typically expressed as a verb
phase, adjective phase and adverbial phase, sentimental
words are extracted from these phases with reference to
level of TF-IDF. Then, polarity of derived sentiment
words in the dictionary are classified into three groups:
positive polarity, negative polarity, and contextual
polarity. Sentimental word that has contextual polarity is
classified into neutral position and afterwards, this word is
manually classified into positive or negative position
based on relationships of service attributes.

Fig. 1. Overall research framework

The proposed framework for mesuring customer

satisfaction based on user generated contents, as shown in
Fig. 1, mainly consists of two parts from an overall
perspective: data collection and preprocessing and
measurement of customer satisfaction. In the stage of data
collection and preprocessing, firstly, review data of
service is collected from the Web. Then, dictionaries of
service attributes and sentimental words are constructed
by using text-mining. With reference to dictionaries,
review documents are transformed into keyword vectors
of customers opinions to use in next stage. In the stage of
measurement of customer satisfaction, based on the
keyword vectors of customers opinions, rating of service
is calculated with respect to each service attribute and
weights of service attributes are also measured. Next,
evaluation of final level of customer satisfaction are
conducted by using aggregating function based MCDM.
In this step, various results of evaluation derived by using
three aggregating function based MCDM, TOPSIS,
VIKOR, and GRA, are compared. Lastly, with reference
to real overall rating of service, validity test is conducted
to determine which MCDM approach has the highest
performance for measuring customer satisfaction of
service.

D. Construction of keyword vectors of customers

opinions
After dictionaries of service attributes and sentimental
words are constructed, review documents are transformed
into keyword vectors of customers opinions. For each
review document, if keyword of service attribute and
keyword of sentimental word are appeared at the same
time, it is considered as information of customers
opinions. In this case, keyword vectors of customers
opinions are constructed as (1).

B. Collection of review data

In this phase, review data of service is collected with
reference to Website. Review data includes overall rating
for service and customers subjective opinions for service.
Especially, customers subjective opinions for the service
represent function of service and its merits and demerits.
Therefore, we can derive important service attributes that
customers carefully consider by analyzing customers
subjective opinions for the service.
In this study, review data of some similar services are
collected to measure and compare customer satisfactions
of services. Because similar services may be belonged in
same categories, important service attributes are no great
difference. Thus, common service attributes are extracted
from review data of similar services and analyzed to


(1)
  
 : Review of customer i
 : Overall rating of customer i
  : Polarities for m attributes of

service

246

Proceedings of the 2012 IEEE ICMIT

w: Weights of service attributes

 : Customer is polarity for service attribute

Keyword vectors of customers opinions consist of

two components: Overall rating and polarity for service
attribute. Overall rating can be expressed as 5 point likert
scale. Polarity for service attribute is expressed as an
integer number between -2 and 2. If polarity for a certain
service attribute is strongly positive, value of  has 2.
If polarity for a certain service attribute is positive, value
of  has 1. Meanwhile, if polarity for a certain service
attribute is strongly negative, value of  has -2 and if
polarity for a certain service attribute is negative, value of
 has -1. Lastly, if there is no polarity of a certain
service attribute, it is express as a value 0. For example, if
a customer writes a review of service such as Overall
rating: 4, a is the worst but b is good, keyword vectors
of customers opinions is constructed as (4, (-2,1,0,)).
Fig.2 shows examples of construction of keyword vectors.

On the other hand, to measure level of customer

satisfaction for each service attribute, summing up score
for the polarity of service attribute is needed. After total
score of the polarity of service is derived, it should be
normalized into a number between 0 and 10 to compare
level of customer satisfaction among different services as
shown in Fig. 3.

Fig. 2. Examples of construction of keyword vectors

For each customer, keyword vector is constructed and

total score is derived by summing up score of the polarity
for a certain service attribute.

Fig. 3. Examples of normalization of score for the polarity of service

attribute

E. Measurement of customer satisfaction with respect to

each service attribute
For applying aggregating function based MCDM
approach, in this phase, weights of service attributes and
level of customer satisfaction for each service attribute
should be derived. To measure weights of service
attributes, some assumptions should be established. This
is because a polaritys impact of the service attribute into
overall rating can be different among customers even if
customers express same polarity for the same service
attribute. Thus, it should be assumed that positive
(negative) polarity for the service attribute contribute to
increase (decrease) overall rating of service. Using this
assumption, weights of service attributes can be derived
based on (2). In this step, multiple regression analysis
with least square method is used to measure weights of
service attributes
   

Fig. 4. Examples of decision matrix

Using derived weights of service attributes and level

of customer satisfaction for each service attribute
(criterion of evaluation), decision matrix is established to
use in the next stage as shown in Fig. 4.
F. Evaluation of final level of customer satisfaction
For evaluating final level of customer satisfaction in
each service, derived decision matrix is used as a data
source for the aggregating function based MCDM
approach. In this study, as mentioned earlier, three
aggregating function based MCDM approaches, TOPSIS,

(2)

 : Customer is overall rating of service

247

Proceedings of the 2012 IEEE ICMIT

VIKOR, and GRA, are used and compared with each

other.
First, in TOPSIS, derived level of customer
satisfaction for each service attribute is used to choose
service alternative that is closest to the positive ideal
solution and farthest from the negative ideal solution.
Thus, TOPSIS method provides information of customer
satisfaction that focuses on distances from best
performance and worst performance in each service.
Second, in VIKOR, service alternative is evaluated by
measuring only closeness to ideal solution. Also, ranking
index in VIKOR is derived by considering both maximum
group utility and minimum individual regret of the
opponent. Third, GRA uses level of the correlations
between the desired factors and other alternative factors
of system by considering the degree of similarity, not the
closeness to ideal solution.
Since each MCDM approach focuses on different
perspectives to measure preference of alternatives, results
of evaluation can be different among three approaches.
For this reason, in this study, different MCDM
approaches are used together to provide reliable results.

derive information of customer satisfaction. Second, this

study uses MCDM approach to evaluate customer
satisfaction. Therefore, compared with other service
alternatives, we can know which service attributes should
be improved. Also, through validity test, we can find the
most suitable MCDM approach for a certain category of
service. Third, this framework for analyzing customer
satisfaction can provide valuable information to the
practitioner because developer of certain service can take
results of evaluation into consideration at the stage of the
launch of beta version.
Future research of this study is as follows. First, more
advanced sentiment analysis is needed to increase the
accuracy of the analysis especially for issue of rating of
polarity. Second, previous MCDM approaches can be
modified to analyze customer satisfaction of service well.
Lastly, appropriate case study should be conducted to
apply the proposed framework.

ACKNOWLEDGMENT
This work was supported by the National Research
Foundation of Korea(NRF) grant funded by the Korea
government (MEST) (No.2011-0030814)

G. Validity test
To know how reliable results are derived, validity test
is conducted. Preference results of three MCDM
approaches are compared with real preference results by
using real overall rating of service. Hit ratio of three
MCDM approaches are calculated based on the accuracy
of prediction in evaluating preference. Since high hit ratio
implies that preference of service alternatives are well
predicted, we can conclude that the MCDM approach that
has high hit ratio are more suitable for measuring
customer satisfaction in certain category of service.
According to categories of service, MCDMs performance
for evaluating customer satisfaction can be changed. Thus,
validity test is needed to find appropriate approach.

REFERENCES
[1] I. M. Klopping and E. McKinney, Extending the
technology acceptance model and the task-technology fit
model to consumer e-commerce, Information technology,
learning, and performance journal, vol. 22 no. 1 , pp.35
48 , 2004
[2] C.C. Sun and G.T.R. Lin, Using fuzzy TOPSIS method for
evaluating the competitive advantages of shopping websites,
Expert Systems with Applications, vol. 36, no. 9, pp. 1176411771, 2009
[3] G. Buyukozkan and G. Cifci, A combined fuzzy AHP and
fuzzy TOPSIS based strategic analysis of electronic service
quality in healthcare industry, Expert systems with
applications, vol.39, no.3, pp.2341 2354, 2012
[4] E. Constantinides and S. J. Fountain, Web 2.0: Conceptual
foundations and marketing issues, Journal of Direct, Data
and Digital Marketing Practice, vol.9, no.3, pp.231 - 244,
2008
[5] V. Hatzivassiloglou and K. R. McKeown, Predicting the
semantic orientation of adjectives In Proceedings of the
8th conference on european chapter of the association for
computational linguistics, pp. 174181, Madrid, Spain,
1997
[6] P. D. Turney, Thumbs up or thumbs down? Semantic
orientation applied to unsupervised classification of
reviews, , In Proceedings of the 40th Annual Meeting of
the Association for Computational Linguistics (ACL-2002),
pp. 417-424, 2002
[7] L-W. Ku and H-H. Chen, Mining opinions from the web:
beyond relevance retrieval, Journal of American Society
for Information Science and Technology, vol. 58, no. 12,
pp.1838-1850, 2007
[8] S-M. Kim and E. Hovy, Determining the sentiment of
opinions, In Proceedings of the 20th international

V. CONCLUSION
This study proposed a framework for measuring
customer satisfaction of service based on user generated
contents using sentiment analysis and aggregating
function based MCDM approach. Specifically, first,
review data is collected to construct dictionaries of
service attribute and sentimental words. Then, keyword
vectors of customers opinions are derived by using
sentiment analysis. Based on keyword vectors, customer
satisfaction with respect to each service attribute is
measured. Lastly, evaluation of final level of customer
satisfaction is conducted by using three aggregating
function based MCDM methods and validity test is
conducted to compare the performance of each method.
This study has three major contributions. First, since
this study proposes framework for analyzing customer
satisfaction of service based on customer review, it can be
the basis of other research that deal with review data to

248

Proceedings of the 2012 IEEE ICMIT

conference on computational linguistics (COLING 2004),

August 23 27, pp. 13671373. Geneva, Switzerland, 2004
[9] T. Wilson, J. Wiebe, and P. Hoffmann, Recognizing
contextual polarity in phrase-level sentiment analysis, In
Proceeding of the conference on empirical methods in
natural language processing (EMNLP 2005), October 68,
pp. 347354, Vancouver, B.C., Canada, 2005
[10] B. Pang and L. Lee, A sentimental education: sentiment
analysis using subjectivity summarization based on
minimum cuts, In Proceedings of the 42nd annual meeting
of the Association for Computational Linguistics (ACL),
July 2126, pp. 271278, Barcelona, Spain, 2004
[11] J. Yi, T. Nasukawa, W. Niblack, and R. Bunescu,
Sentiment analyzer: extracting sentiments about a given
topic using natural language processing techniques, In
Proceedings of the 3rd IEEE international conference on
data mining (ICDM 2003), November 1922, pp. 427434,
Florida, USA, 2003
[12] K. Hiroshi, N. Tetsuya,W. Hideo, Deeper sentiment
analysis using machine translation, In Proceedings of the
20th international conference on computational linguistics
(COLING 2004), August 23 27, pp. 494500, Geneva,
Switzerland, 2004
[13] T. Nasukawa and J. Yi, Sentiment analysis: capturing
favorability using natural language processing, In
Proceedings of the 2nd international conference on
Knowledge capture, pp. 494500, 2003
[14] Chen and C.L. Hwang, Fuzzy multiple attribute decision
making: methods and applications, Springer-Verlag, Berlin,
1992
[15] C.L. Hwang and K. Yoon, Multiple Attribute Decision
Making, Lecture Notes in Economics and Mathematical
System 186, Springer-Verlag, Berlin, 1981
[16] S. Opricovic, Multicriteria optimization of civil
engineering systems, Faculty of Civil Engineering,
Belgrade, 1998
[17] L. Dong, Control problems of grey systems, Systems and
Controls Letters, vol. 5, pp. 288294, 1982
[18] M.S. Kuo, G.L. Liang, and W.C. Huang, Extensions of the
multicriteria analysis with pairwise comparison under a
fuzzy environment, International Journal of Approximate
Reasoning, vol. 43, no. 3, pp. 268285, 2006

249