Classification Annotated

Classification:
Analyzing Sentiment
Emily Fox & Carlos Guestrin
Machine Learning Specialization
University of Washington
1 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization
Predicting sentiment by topic:
An intelligent restaurant
review system

Its a big day & I want to book a table at
a nice Japanese restaurant
Seattle has many

sushi restaurants
What are people

saying about
the food?
the ambiance?...
Positive reviews not positive about everything
Experience
Sample review:
Watching the chefs create
incredible edible art made
the experience very unique.
My wife tried their ramen

and it was pretty forgettable.
All the sushi was delicious!

Easily best sushi in Seattle.
From reviews to topic sentiments
All reviews
for restaurant
Experience

Novel intelligent Ramen
restaurant review app
Sushi

Easily best sushi

in Seattle.
Intelligent restaurant review system
All reviews Break all reviews

for restaurant into sentences
The seaweed salad was just OK,
vegetable salad was just ordinary.
I like the interior decoration and

the blackboard menu on the wall.
All the sushi was delicious.
My wife tried their ramen and

it was pretty forgettable.
The sushi was amazing, and

the rice is just outstanding.
The service is somewhat hectic.

Core building block
Sentence Sentiment
Classifier

Intelligent restaurant review system
All reviews BreakSelect

all reviews
sentences
for restaurant into sentences
about sushi
The seaweed salad was just OK, Average
vegetable salad was just ordinary.
predictions
Sentence
I like the interior decoration and
the blackboard menu on the wall.
Sushi
Sentiment
All the sushi was delicious. All the sushi was delicious.
My wife tried their ramen and

it was pretty forgettable.
The sushi was amazing, and The sushi was amazing, and Classifier
the rice is just outstanding. the rice is just outstanding. Most
The service is somewhat hectic. &
Easily best sushi in Seattle. Easily best sushi in Seattle.
Easily best
sushi
in Seattle.
Classifier applications

Classifier
Sentence
Classifier
from
review MODEL
Output: y
Input: x Predicted
class

Example multiclass classifier
Output y has more than 2 categories
Education
Finance
Technology
Input: x Output: y
Webpage
Spam filtering
Not spam
Spam
Input: x Output: y
Text of email,
12
sender, IP, 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization
Image classification
Input: x Output: y
Image pixels Predicted object
Personalized medical diagnosis
Input: x Output: y
Healthy
Disease Cold
Classifier Flu
MODEL Pneumonia


Reading your mind
Hammer
House

Linear classifiers

Representing classifiers
How does it work???
Sentence
Classifier
from
review MODEL
Output: y
Input: x
Predicted class

List of positive List of negative
words words
great, awesome, bad, terrible,
good, amazing, disgusting, sucks,
Simple threshold classifier

Count positive & negative words
in sentence
Sentence
from If number of positive words >
review number of negative words:
=
Input: x Else:
=
List of positive List of negative
words words
great, awesome, bad, terrible,
good, amazing, disgusting, sucks,
Simple threshold classifier

Sushi was
Count positive & negative words
in sentence
great, the 2
food was
awesome, If number of positive words >
but the number of negative words:
service was =
terrible.
Else:
1
=
Problems with threshold classifier
How do we get list of
positive/negative words?
Addressed
Words have dierent by learning
degrees of sentiment: a classifier
- Great > good
- How do we weigh
dierent words?
Addressed
Single words are not enough: by more
- Good Positive elaborate
- Not good Negative features
A (linear) classifier
Will use training data to learn a weight
for each word
Word Weight
good 1.0
great 1.5
awesome 2.7
bad -1.0
terrible -2.1
awful -3.3
restaurant, the, we, where, 0.0

Scoring a sentence
Word Weight Input x:
Sushi was great,
good 1.0
the food was awesome,
great 1.2 but the service was terrible.
awesome 1.7
bad -1.0
terrible -2.1
awful -3.3
restaurant, the, 0.0
we, where,

Called a linear classifier, because output is weighted sum of input.
Word Weight

Simple linear classifier

Score(x) = weighted count of
Sentence words in sentence
from
review If Score (x) > 0:
=
Input: x Else:
=

Decision boundaries

Suppose only two words had non-zero weight
Word Weight
awesome 1.0
Score(x) = 1.0 #awesome 1.5 #awful
awful -1.5
awful
Sushi was awesome,

the food was awesome,
4 but the service was awful.
3
0
0 1 2 3 4
awesome
Decision boundary example
Word Weight
awesome 1.0
Score(x) = 1.0 #awesome 1.5 #awful
awful -1.5
awful
Score(x) < 0

0
Score(x) > 0
0 1 2 3 4
awesome
Decision boundary separates
positive & negative predictions
For linear classifiers:
-When 2 weights are non-zero
line
-When 3 weights are non-zero
plane
-When many weights are non-zero
hyperplane
For more general classifiers
more complicated shapes

Training and evaluating
a classifier

Training a classifier = Learning the weights
Word Weight
good 1.0
awesome 1.7
bad -1.0
awful -3.3

Training Learn
set classifier
Data
(x,y) Test
Evaluate?
(Sentence1, ) set
(Sentence2, )

Classification error
Learned classifier
=
Test example Correct 1
0
Correct!
Mistake!
(Food
Foodwas
(Sushi
Sushi wasgreat,
OK
OK,
great )) Mistakes 0
1
Hide label
Classification error & accuracy
Error measures fraction of mistakes
error = .
-Best possible value is 0.0
Often, measure accuracy

-Fraction of correct predictions
accuracy= .
-Best possible value is 1.0

Whats a good accuracy?

What if you ignore the sentence, and just guess?
For binary classification:
-Half the time, youll get it right! (on average)
accuracy = 0.5
For k classes, accuracy = 1/k

-0.333 for 3 classes, 0.25 for 4 classes,
At the very, very, very least,

you should healthily beat random
Otherwise, its (usually) pointless
Is a classifier with 90% accuracy good? Depends
2010 data shows:
90% emails sent are spam!
Predicting every email is spam

gets you 90% accuracy!!!
Majority class prediction
Amazing performance when

there is class imbalance
(but silly approach)
One class is more common than others
Beats random (if you know the majority class)
So, always be digging in and asking the
hard questions about reported accuracies
Is there class imbalance?
How does it compare to a simple,
baseline approach?
-Random guessing
-Majority class
-
Most importantly:
what accuracy does my application need?
-What is good enough for my users experience?
-What is the impact of the mistakes we make?
False positives, false negatives,
and confusion matrices

Types of mistakes
Predicted label
True False
True label
Positive Negative
Positive
(FN)
(FP)
False True
Positive
Negative Negative
(FP)
(FN)
Cost of dierent types of mistakes can be
dierent (& high) in some applications
Spam Medical
filtering diagnosis
False
Disease
negative Annoying not treated
False
Wasteful
positive Email lost
treatment

Confusion matrix
binary classification
Predicted label
True False
True label
Positive Positive
(FP)
False True
Negative Negative
(FN)

Confusion matrix
multiclass classification
Predicted label
Healthy Cold Flu
Healthy
True label
Cold
Flu

Learning curves:
How much data do I need?

How much data does a model need to learn?
The more the merrier J
- But data quality is most important factor
Theoretical techniques sometimes

can bound how much data is needed
- Typically too loose for practical application
- But provide guidance
In practice:
- More complex models require more data
- Empirical analysis can provide guidance

Learning curves
Test error
Amount of training data

Is there a limit?
Yes, for most models
Test error
Bias of model

More complex models tend to have less bias
Sentiment classifier using
single words can do OK, but
Never classifies correctly:

The sushi was not good.
More complex model:

consider pairs of words (bigrams)
Word Weight
good +1.5 Less bias
not good -2.1 potentially more accurate,
needs more data to learn
Models with less bias tend to
need more data to learn well,
but do better with sucient data
Test error
Classifier based
on single words

Class probabilities

How confident is your prediction?
Thus far, weve outputted a prediction
But, how sure are you about the prediction?

- The sushi & everything P(y=+|x) = 0.99
Definite
else were awesome!
- The sushi was good, P(y=+|x)
the service was OK. Not sure = 0.55
Many classifiers provide a confidence level:
P(y|x)
Output label Input sentence
Extremely useful in practice
Summary of classification

What you can do now
Identify a classification problem and
some common applications
Describe decision boundaries and linear
classifiers
Train a classifier
Measure its error
- Some rules of thumb for good accuracy
Interpret the types of error associated
with classification
Describe the tradeos between model
bias and data set size
Use class probability to express degree
of confidence in prediction

Classification Annotated

Diunggah oleh

Informasi Dokumen

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Classification Annotated

Diunggah oleh

Hak Cipta:

Format Tersedia

Classification:

2 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

What are people

My wife tried their ramen

All the sushi was delicious!

Easily best sushi

All reviews Break all reviews

I like the interior decoration and

All the sushi was delicious.

My wife tried their ramen and

The sushi was amazing, and

The service is somewhat hectic.

Easily best sushi in Seattle.

6 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

Easily best sushi in Seattle.

7 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

All reviews BreakSelect

My wife tried their ramen and

9 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

10 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

14 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

15 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

16 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

17 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

Simple threshold classifier

Simple threshold classifier

Simple linear classifier

23 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

24 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

Sushi was awesome,

27 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

28 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

-Best possible value is 0.0

Often, measure accuracy

-Best possible value is 1.0

32 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

For k classes, accuracy = 1/k

At the very, very, very least,

Predicting every email is spam

Majority class prediction

Amazing performance when

36 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

38 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

39 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

40 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

41 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

Theoretical techniques sometimes

42 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

Amount of training data

43 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

Amount of training data

44 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

Never classifies correctly:

More complex model:

Amount of training data

47 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

But, how sure are you about the prediction?

Many classifiers provide a confidence level:

49 2015 Emily Fox & Carlos Guestrin Machine Learning Specialization

Anda mungkin juga menyukai