Information Retrieval Techniques in Sentiment

Analysis - A Review
Graduate Review Paper

Sunanda Bansal
Computer Science Department
Concordia University
Montreal, Canada
sunandabansal92@gmail.com

Abstract—We discuss 5 research papers from the field of
sentiment analysis selected from over a range of about 15 years
with respect to their approaches and use of information retrieval
techniques.
I. I NTRODUCTION
Sentiment analysis is the identification of correct sentiment
associated with a document unit or topic. The sentiment can
be a very a broad term ans thus the major focus of sentiment
analysis is to process document units that exhibit positive or
negative sentiments. The research has been focused majorly,
not exclusively, on the two subproblems
• Subjectivity Classification - subjective/objective opinion
• Polarity Detection - positive/negative
Sentiment Analysis has been approached with various strate-
gies and has been investigated at different levels of the text
- document, sentence, entity and aspect level [3]. Lexicon
based approach and the text classification approach are the two
major approaches taken to extract the sentiment automatically.
The latter is where most of the work in sentiment analysis is
focused on, i.e. supervised learning techniques, which can also
be described as statistical or machine learning approach [6].
Research papers by Pang et al. [1] and Pang et al. [4] mainly
focus on various machine learning algorithms while research
paper by JTaboada et al. [5] focuses on lexicon based approach
in which they use dictionary compilation. We also discuss the
approach used in Maas et al. [7] where they use a mix of
supervised and unsupervised learning to learn word vectors
and propose a model that predicts sentiment annotations on
contexts of words by using the vector representation of the
same. Paltoglou et al. [6], on the other hand examine whether
term weighting functions from the information retrieval rank-
ing approaches could be useful in increasing the accuracy of
classification.
II. A PPROACHES
A. Bag of words model
Out of the various approaches used in [1], it made use
of standard bag-of-features framework, which is essentially
similar to bag-of-words model, to implement the machine
learning algorithms. It drew a comparison between various
settings for implementing features in the algorithms, like
bag-of-unigrams and bigrams. In the end, unigram presence
information was found out to be most effective for employing
features. In the beginning, on a closer inspection of the
problem, a hypothesis suggesting classification on the basis of
bag-of-words was experimented with, which later gave better
results when the same was based on a preliminary examination
of frequency counts and not just the presence in corpus, plus
introspection. It led them to explore corpus-based techniques
to select better sentiment indicators, which further leads to
their machine learning approaches in analyzing sentiments.
B. Machine Learning
One of the main things to consider with supervised learning
has been the representation of a document unit. Some earlier
work [1] used bag-of-words model to represent the document
with the bag-of-unigrams focusing only on the presence or
absence of the words which provides a good baseline classi-
fication accuracy in comparison to other settings like bag-of-
bigrams, adjectives, etc. We will be starting from the research
by Pang et al. in 2002 [1] which focused on classifying
the document on the basis of frequency of appearance of
certain lexical features [1]. Some later approaches focused on
applying text-categorization to only the subjective portions of
the documents [4].
The bag-of-features framework was used in [1] to imple-
ment three standard algorithms: Naive Bayes classification,
maximum entropy classification, and Support Vector Machines
(SVM). Naive Bayes Classification performs optimally for
certain problem classes with highly dependent features. But
maximum Entropy classification makes no assumption about
the conditional independence of features, unlike Naive Bayes
Classification, and thus may potentially outperform Naive
Bayes Classification in some cases.
The Naive Bayes Classification is based on the idea to
assign the document unit to the class that is most likely to be
the right one, given a training set, observed by Bayes’ rule. In
Maximum Entropy approach, it also guesses the most likely
class for a document, but based on a feature/class function
and feature-weight parameters, which are set such that it
maximizes the entropy of the induced distribution. The third
algorithm used is Support Vector Machines (SVM) which is
entirely different from the first two approaches. In the two-
category case, such as this, the basic idea behind the training
approach is to find a hyperplane that separates document
vectors in one class from those in other class with as large
a margin as possible. The classification is simply determined
by which side of the hyperplane they fall on.
Reference [4], on the other hand, focuses on applying
machine learning technique such that it focuses only on the
subjective portions of the document. Unlike [1], [4] focuses
on discarding the sentences that can be labeled as objective
in the document unit. The approach used for extracting these
portions is finding minimum cuts in graphs. In reference [4],
the approach is to integrate sentence level-subjectivity detec-
tion with document-level sentiment polarity. For determining
whether two sentences that exhibit some proximity in terms of
the subjectivity level, the idea used here, is to use graph-based
formulation relying on finding minimum cuts. It works on an
undirected graph and calculates partition cost on the basis of
individual and associative scores for the class for each item.
In [5], Taboada et al. argue that the performance of classi-
fiers so trained are domain specific and the polarity assigned
by these classifiers to the words makes sense mostly within the
context of that domain. In this way the lexicon based approach
seems more suitable for generalized analysis. Tabokada et al.
also argue that the classifiers that learn from Machine learning
approaches do not handle negation and intensification well but
their lexicon based model does in a way that ”generalizes to
all the worlds that have a semantic orientation value” [5].
Another approach is to capture semantic term-document
information an rich semantic content by using a mix of
supervised and unsupervised learning techniques as proposed
in [7]. Using indices in a vocabulary to represent words fail
to capture the relational structure of the lexicons which can
be useful in determining the continuous similarities between
words. The vector-based model, on the other hand does that
much better by encoding these similarities between words as
distance of angle between word vectors. The model proposed
by Maas et al. (2011) attempts to capture both semantic and
sentiment similarities among words. The semantic component
learns via unsupervised model of documents and thus can
determine the words to be semantically close but does not
capture the crucial sentiment information. Thus the model is
extended to use a supervised sentiment component with the
vector representation of words where it predicts the sentiment
annotations on the context of the words.
The model presented in [7] uses a probabilistic model
of documents, based on probabilistic topic model such as
Latent Dirichlet Allocation (LDA) [8], to capture semantic
similarities. The model attempts to directly model the word
probabilities conditioned on the topic mixture, instead of
modeling individual topics. This component doesn’t explicitly
capture sentiments associated, rather just produces representa-
tions of words that occur together with similar representations.
The unsupervised semantic similarity evaluating component of
the model does not require labeled data. But the supervised
sentiment component uses sentiment labels so that the word
vectors should predict the sentiment label.
C. Dictionary Compilation
Dictionary compilation is a lexicon based approach where
most of the research regarding this approach has focused on
using adjectives as indicators of semantic orientation of text
[5]. Reference [5], uses dictionary of words annotated with
the word’s semantic orientation or polarity which is calculated
by aggregation of the sentiment orientation score of all the
adjectives in the document into a single score for the text.
So, the sentiment orientation of entire document is basically
a combined effect of the adjectives or other sentiment-bearing
words found withing the document, based upon a dictionary
containing these words and their corresponding sentiment
orientation scores. This paper criticizes machine learning
based approach for it is highly domain specific and does
not take the negation and intensification into account. This
paper also attempts to address the criticism against lexicon-
based approach that says the dictionaries are unreliable, by
presenting results that attempt to show that their dictionaries
are robust and reliable.
D. Weighting Schemes
Reference [6] hypothesize that the tf-idf weighting schemes
and the fact that these proved quite effective for information
retrieval may indicate that these weighting schemes may ap-
propriately model the distinctive power of terms of documents.
The experiment is not restricted to tf-idf weighting schemes.
The use of weighting schemes is shown to have significantly
improved the accuracy of classification compared to other
approaches.
In an attempt to evaluate their model, [7] attempts document
level polarity classification using bag of words vector v and
features obtained from their model Rv with arbitrary tf-idf
weights. Cosine normalization is not applied to the v instead
it is applied to the final feature vector Rv, though in the
experiment they found that ’bnn’ weighting works best for
v, when document features are generated through the product
Rv, and was then used to get multi-word representations in all
their experiments.
III. C ONCLUSION
The simplistic model with bag-of-unigrams and supervised
learning, proposed by Pang et al. [1] gives the performance that
is not much outperformed by using other complicated feature
settings. Though most research in sentiment analysis focuses
on supervised learning methods, it has its disadvantages for it
is domain specific and does not account for the negation and
intensification of sentiments. For these reasons, lexicon based
model is more appropriate for general and domain-independent
purposes. Lexicon based model can be criticized for the
unreliability of dictionaries. In the lexicon based approach
most research is focused on using adjectives as appropriate
indicators for sentiment orientation. No one approach can be
assumed to be the best approach, in this case. As for, use
of information retrieval techniques, from weighting to bag-
of-words model to vector space model, it seems that these
techniques are widely used in the research field of Sentiment
Analysis.
R EFERENCES
[1] Pang, Bo, Lillian Lee, and Shivakumar Vaithyanathan. ”Thumbs up?:
sentiment classification using machine learning techniques.” Proceedings
of the ACL-02 conference on Empirical methods in natural language
processing-Volume 10. Association for Computational Linguistics, 2002.
[2] Pang, Bo, and Lillian Lee. ”Opinion mining and sentiment analysis.”
Foundations and Trends in Information Retrieval 2.12 (2008): 1-135.
[3] Liu, Bing. ”Sentiment analysis and opinion mining.” Synthesis lectures
on human language technologies 5.1 (2012): 1-167.
[4] Pang, Bo, and Lillian Lee. ”A sentimental education: Sentiment analysis
using subjectivity summarization based on minimum cuts.” Proceedings
of the 42nd annual meeting on Association for Computational Linguis-
tics. Association for Computational Linguistics, 2004.
[5] JTaboada, Maite, et al. ”Lexicon-based methods for sentiment analysis.”
Computational linguistics 37.2 (2011): 267-307.
[6] Paltoglou, Georgios, and Mike Thelwall. ”A study of information
retrieval weighting schemes for sentiment analysis.” Proceedings of the
48th annual meeting of the association for computational linguistics.
Association for Computational Linguistics, 2010.
[7] Maas, Andrew L., et al. ”Learning word vectors for sentiment analy-
sis.” Proceedings of the 49th Annual Meeting of the Association for
Computational Linguistics: Human Language Technologies-Volume 1.
Association for Computational Linguistics, 2011.
[8] D. M. Blei, A. Y. Ng, and M. I. Jordan. 2003. Latent dirichlet allocation.
Journal of Machine Learning Research, 3:9931022,May.