Discriminant and Cluster

Analysis

1
 R
 

 



! —tatistical methods that allow the simultaneous

investigation of more than two variables
! Multivariate Methods: Collection of procedures for
analysing association between two or more sets of
measurements that have been made on each object in one
or more samples of objects
! Object: Entities on which measurements are taken
Objects are themselves are only the carriers of
the measurements; they are not measured in
their entirety but only with respect to certain
variables of interest
! Variables: Characteristics or Properties:
Aspects of the objects that are measured
 

R
 

! Measurement : Process by which numbers

(sometimes symbols) are attached to
the characteristics or properties of the
objects according to a pre-determined
procedure
! Measurement —cales :
! Nominal
! Ordinal
! Interval
! Ratio
 

R
 

! Nominal : Data are described categorically

Objects based on nominal scale of measurement
falls into one of a set of mutually exclusive and
collectively exhaustive groups
Not meaningful to perform an arithmetic operation
! Ordinal : Ranked data
Based on ordinally measured values for object ?
and object , one can know only that object ? has
more, less or the same amount of the measured
characteristics as object ?

Not meaningful to perform an arithmetic operation
 

R
 

! Interval :Allow a researcher to say how much more of

the measured characteristic is possessed by
one object or another, but they do not allow to
make absolute comparisons relative to a
meaningful zero point
It is, however, valid to compare the difference
between items on an interval scale
Ex: Temperature measured through centigrade
! Ratio : Ratio —cale data have the same properties as
interval scale but also possess a meaningful
origin.
Ex: Age, Price
 

R
 

Ratio Metric
Interval
Ordinal
Non-metric
Nominal



 


     
 


      


 

R
 

Measure Measurement Level Examples Mathematical

ment properties
scale
^ 
   
      







 
   
 
  

  
  
 
   


  

! 

 
   
   
     "# 

 




$


 

 
     



     


   



%
D
 


  
R


R

All multivariate
methods

Are some of the

variables dependent
on others?

Yes No

Dependence Interdependence
methods methods

Dependence methods explain or Interdependence methods give

predict a dependent variable(s) on meaning to a set of variables or
the basis of two or more seek to group things together
independent variables
 


  
R


R

Method Interdepende Explora Metric Objectives

nce vs. tory vs. vs. Non-
Dependence Confirm metric
atory
Principal Interdependen Explorat Metric Dimension reduction
Component ce ory
Exploratory Interdependen Explorat Metric Understand Patterns of
Factor Analysis ce ory intercorrelations;
uncover latent traits
Confirmatory Interdependen Confirm Metric Verify measurement
Factor Analysis ce atory models
Multidimension Interdependen Mainly Metric or Create spatial
al —caling ce explorat Non- representation from
ory metric object similarities
Cluster Interdependen Explorat Metric or Create groupings from
Analysis ce ory Non- object similarities
metric
 


  
R


R

Method Interde Exploratory vs. Metric vs. Objectives

penden Confirmatory Non-metric
ce vs.
Depend
ence
Canonical Depend Mainly Metric Explain covariation between
Correlation ence exploratory two sets of multiple
variables
—tructural Depend Confirmatory Metric Dependence model with
Equation Models ence measurement error
with Latent
Variables
Analysis of Depend Confirmatory Metric and —pecial case of canonical
Variance ence Non-metric correlation with discrete X
variables
Discriminant Depend Exploratory or Metric and —pecial case of canonical
Analysis ence Confirmatory Non-metric correlation with discrete Y
variables
Logit Choice Depend Exploratory or Metric and Nonlinear probability model
Models ence Confirmatory Non-metric for discrete choice
outcomes



&%

11
 p
   



! Discriminant analysis is a multi-variable technique which

ensures classification of N individuals or units in two or
more groups based on various (p in number) qualities and
which offers the relevant functions.
! In this analysis, a discriminant function is found to allocate
units to groups and this function is determined in a way as
to maximise the difference between function group means

Major Uses

 p


12
Person Evaluation X1 (Amount of Protein X2 (Daily
per standard serving) requirement of
Vitamin D)
1 Dislike 2 4
2 Dislike 3 2
3 Dislike 4 5
4 Dislike 5 4
5 Dislike 6 D
Average 4 4.4
6 like D 6
D like 8 4
8 like 9 D
9 like 10 6
10 like 11 9
Average 9 6.4
Grand Mean 6.5 5.4
— Deviation 3.028 13 2.011
p
    ' (

!




14
 )
p
   



! To identify the linear combinations or discriminant

functions of the predictive (independent) variables which
best distinguish predetermined groups (dependent). This
is done through maximising the rate of intra-group
change to inter-group change.
! To test if there is a difference between groups in terms of
independent or dependent variables (Malhotra, 1996;
Nakip 2006).
! To identify the independent variable which best
determines the intergroup difference.
! To identify which group the outside observation belongs
to via discriminant function (Aaker, Kumar and Day, 1998:
Nakip 2006).
15
 )
p
   



! To test how correct the assumptions are by re-

classifying the available observations according to
the available distribution function (Nakip, 2006); in
other words, to evaluate the classification accuracy
(Yaprak, 200D).
! To identify how much of the variant of the dependent
variable can be explained by independent variables.
! To identify the contributions and degrees of
importance of the discriminant variables (Garson,
2008).

16
 


p
   



! Two or more groups should be found (g • 2 ).

! A minimum of two individuals should be
included in each group (n k • 2 ).
! Variants of p in numbers which are used to
categorise units in groups are called
³discriminant variables´. The number of
these variables should be ( 0 ” p ” N í 2 ).
! Discriminant variables should be measured at
least by an equally intervaled scale.

1D
 


p
   



! A discriminant variable should not be the linear

compound of any other discriminant variable.
! The variance-covariance matrix for each group
should be equal.
! It is assumed that each group is withdrawn from
a mass of multivariable normal distribution
(Çakmak, 1992).
! There should be no multi-linear connection
problem among independent variables (Kalaycı,
2005).

18
 º
 

! Discriminating variables: These are the independent

variables, also called J
? .
! The criterion variable: This is the dependent variable, or
object of classification efforts also called the î J?î

 ?
in —P——.
! Discriminant function: Also called a ? 
Is a latent variable which is created as a linear combination
of discriminating (independent) variables, such that L = b1x1
+ b2x2 + ... + bnxn + c, where
b = Discriminant coefficients, which maximize the
distance between the means of the criterion
(dependent) variable.
x = Discriminating variables
c = Constant. 19
 º
 

! Pairwise group comparisons :

Display the distances between group means (of the dependent
variable) in the multidimensional space formed by the
discriminant functions.
The pairwise group comparisons table gives an F test of
significance (based on Mahalanobis distances) of the distance
of the group means.
The magnitude of the F values enable the researcher to determine
if every group mean is significantly distant from every other
group mean.

20
 º
 

! Number of discriminant functions%

The number of functions in DA is the lesser of (g - 1), where g is
the number of categories in the grouping variable, or p,the
number of discriminating (independent) variables.
The first function maximizes the differences between the values of
the dependent variable and the most powerful differentiating
dimension.
The second function is orthogonal to it (uncorrelated with it) and
maximizes the differences between values of the dependent
variable, controlling for the first factor and represent additional
significant dimensions of differentiation.
And so on.

21
 º
 

! Eigenvalue :
Called the  
??  of each discriminant function.
Reflects the ratio of importance of the dimensions which classify
cases of the dependent variable.
There is one eigenvalue for each discriminant function.
The eigenvalues assess relative importance because they reflect the
percents of variance explained in the dependent variable,
cumulating to 100% for all functions.
The first will be the largest and most important, the second next
most important in explanatory power, and so on.

22
 º
 

ù Canonical correlation, R*:

Measure of the association between the groups formed by the
dependent and the given discriminant function.
R* is zero ĺ No relation between the groups and the function
R* is large ĺ High correlation between the discriminant functions
and the groups.
R* is used to tell how much each function is useful in determining
group differences.
An R* of 1.0 indicates that all of the variability in the discriminant
scores can be accounted for by that dimension

23
 º
 

ù Discriminant score
Value resulting from applying a discriminant function formula to the
data for a given case.
The U 
is the discriminant score for standardized data.

ù Cutoff:
If the discriminant score of the function is less than or equal to the
cutoff, the case is classed as 0, or if above it is classed as 1.
When group sizes are equal, the cutoff is the mean of the two
centroids (for two-group DA).
If the groups are unequal, the cutoff is the weighted mean.

24
 º
 

Ui = b1 X1 + b2 X2 + b3 X3 + ... + bn Xn

Where U = discriminant score

b = discriminant weights
X = predictor (independent) variables

In a particular group, each individual has a discriminant score (zi)

Ȉ zi = centroid (group mean)
Where i = individual
Indicates most typical location of an individual from a particular group

25
 º
 

! Criterion against which each individual¶s discriminant score is

judged to determine into which group the individual should be
classified

For equal group sizes For unequal group sizes

A B

26
 º
 

ù Unstandardized discriminant coefficients

Used in the formula for making the classifications in DA, much as b
coefficients are used in regression in making predictions.
The discriminant function coefficients are partial coefficients,
reflecting the unique contribution of each variable to the
classification of the criterion variable.

ù —tandardized discriminant coefficients

Termed the  ?
 ? ? ? ? ?

??
,
Used to compare the relative importance of the independent
variables, such as beta weights are used in regression.

2D
 º
 

! Functions at group centroids

The mean discriminant scores for each of the dependent variable
categories for each of the discriminant functions in MDA.
Two-group discriminant analysis has two centroids, one for each
group.
The means to be well apart to show the discriminant function is
clearly discriminating.
The closer the means, the more errors of classification there likely
will be.

28
 º
 
*+

   

ù Used to test the significance of the discriminant function as a whole.

In —P——, the "Wilks' Lambda" table will have a column labeled "Test of
Function(s)" and a row labeled "1 through n" (where n is the number
of discriminant functions).
The "—ig." level for this row is the significance level of the discriminant
function as a whole.
The larger the lambda, the more likely it is significant.
A significant lambda means one can reject the null hypothesis that the
two groups have the same mean discriminant function scores and
conclude the model is discriminating.
ù (Variable) Wilks' lambda
Used to test which independents contribute significantly to the
discrimiinant function.
The smaller the variable Wilks' lambda for an independent
variable, the more that variable contributes to the discriminant
function. 29
 º
 
*+

   

ù `
` 
 ? ? ? 


Another overall test of the DA model.
It is an F test, where a "—ig." p value < .05 means the model
differentiates discriminant scores between the groups
significantly better than chance (than a model with just the
constant).

30
 º
 
*, -R



! Classification functions*
! —
 
 
.



   





 -
   
%
! Ê  
 
 
 
R
 

 p
 

  
%
!

  

 



  


  

  
 
 



 %+
/^^

    
 


  
%

31
 º
 
*, -R



! Classification functions*
! Classification table- Classification matrix
!



 p%
+




  



   



  
%
  




 


  
%
+ 

  

 




%
+
 


hit ratio%

32
 º
 
*!  
p
   


! —tructure coefficients - —tructure correlations Discriminant

loadings
Correlations between a given independent variable and the discriminant
scores associated with a given discriminant function.
Used to tell how closely a variable is related to each function in MDA.
The structure coefficients are whole (not partial) coefficients and reflect the
uncontrolled association of the discriminating variables with the
criterion variable
Looking at all the structure coefficients for a function allows the researcher
to assign a label to the dimension it measures, much like factor
loadings in factor analysis.
Canonical structure matrix actor structure matrix
A table of structure coefficients of each variable with each
discriminant function

33
 º
 
*!  
p
   


! —tandardized discriminant function coefficients.

Indicate the semi-partial contribution (the unique, controlled association) of
each variable to the discriminant function(s), controlling the
independent but not the dependent for other independents entered in
the equation
The standardized discriminant function coefficients should be used to
assess the importance of each independent variable's unique
contribution to the discriminant function.

! Wilks's lambda
Tests the significance of each discriminant function in MDA
The smaller the lambda, the greater the differences. Lambda varies from 0 to
1, with 0 meaning group means differ (thus the more the variable
differentiates the groups), and 1 meaning all group means are the same.

34

 ,
 0" 1  03 1
 0"3 033 0"03
 
    

 2 4 p2
1 Dislike 2 4 4 16 8
2 Dislike 3 2 9 4 6
3 Dislike 4 5 16 25 20
4 Dislike 5 4 25 16 20
5 Dislike 6 D 36 49 42
Average 4 4.4 —um 90 110 96
6 like D 6 49 36 42
D like 8 4 64 16 32
8 like 9 D 81 49 63
9 like 10 6 100 36 60
10 like 11 9 121 81 99
Average 9 6.4 —um 415 218 296
Grand Mean 6.5 5.4
— Deviation 3.028 2.011
35
Inferences from Table
ö Two groups are much more widely separated on X1
than they are on X2
ö X1 appears better for group separation
ö However, there is information provided by group
separation on X2
ö Better to take composite both X1 and X2
ö Linear Equation -> Z = K1X1 + K2X2

36
How to find K1 and K2
ö The ratio to be maximised through
appropriate selection of K1 and K2
ö (—um of squared deviations of the group
means around their grand mean) / (—um of
squared deviations of the pooled variability
of the individual cases around their
respective group mean
ö Procedure similar to that of multiple
regression
3D
How to find K1 and K2

ö —olve two normal equations

K1™x12 + K2™x1x2 = 1(L) - 1(DL)
K1™x1x2 + K2™x22 = 2(L) - 2(DL)
ö We have
p



+

™x12 = ™ (X1 - 1)
2 = ™ X12 ± N 1
2 10 10 20
™x22 = ™ (X2 - 2)
2 = ™ X22 ± N 2
2 13.2 13.2 26.4
™x1x2 = ™ (X1 - 1) (X2 - 2) = ™X1X2 - N 1 2 8 8 16

ö^
  
  
    
    
   !  !
38
Discriminant —cores for the Means

ö U

   

ö U

   
 
ö U
 
   


ö !" # !$#" % &' ($( &#"

  ) *() # %   ($ # %

39
 Discriminant —cores for the Means
öu


  


Dislikers Likers
Person Discriminant Person Discriminant
—core —core
1 0.148 6 1.691
2 0.809 D 2.353
3 0.D35 8 2.2D9
4 1.250 9 2.D94
5 1.1D6 10 2.D21
Mean 0.824 Mean 2.368
Grand Mean 1.596
40
ö º * #"+ ,& '

    - 
    

ö. ) #"+ ,& '
ö 
    - / -
 
 

ö 
     - / -
 
  

ö 0#   - 

ö  1 # 1
 2 

41
ö     

  

   !
" # 

ö $  
  % 

   
    
 
   
  
ö $  
  % 

   
    
   
    

42
ö -     
 


ö -   
  
 
ö & 
 ' %
( ! ) 
ö 
   

 

 
  
   

ö '  
 '*++  + , --./
++ , .+ , -..0%  1+ , --.


ö  *
ö '*
ö -)'+12.*2 3
ö 45
6


 6
ö 7   (  
% 
)) 3
43
 


  4


Holdout Method
! Uses part of sample to construct classification rule; other
subsample used for validation
! Uses classification matrix and hit ratio to evaluate groups
classification
! Uses discriminant weights to generate discriminant scores for
cases in subsample

44
 


  4
 1 %2

U - method or Cross Validation

! Uses all available data without serious bias in estimating error

rates
! Estimated classification error rates
P1 = m1/ n1 P2 = m2 / n2

where m1 and m2 = number of sample observations mis-classified

in groups G1 and G2

45
 
 p
   



1. Form groups

2. Estimate discriminant function

3. Determine significance of function and variables

4. Interpret the discriminant function

5. Perform classification and validation

46
 ,5p 


 

1"2 (132 + 1562 -15"2 ,51572 1532 8
159215:2
" : 6 " :; " "### 6%6 7
3 9 : " :7 " "### 7%6 :
9 6 : " 6: " "### 7%# <
: 3 9 " 9" # 9### 7%# 6
6 : 9 " 6# " 3### 7%6 <
7 6 : " 7; " "### 6%6 ;
% % % % % % % % %
% % % % % % % % %
% % % % % % % % %
""6: 9 " :6 " 3### 7%# 7
""76 : " :: " 3### 6%= ""
""<9 : " :7 # "### <%# 9
""=9 : " 6: " "### <%# :
"";: 9 " :; " "### 7%6 <
"3#: 6 " 6: " :### 7%6 <

R 
= , / p 4D
 p
 4


Variable Description Corresponding —cale Values

Name in Output
Willingness to Export (Y1) Will 1(definitely not interested) to 5
(definitely interested)
Level of Interest in —eeking Govt Govt 1(definitely not interested) to 5
Assistance (Y2) (definitely interested)
Employee —ize (X1) —ize Greater than Zero

Firm Revenue (X2) Rev In millions of dollars

Years of Operation in the Years Actual number of years

Domestic Market (X3)
Number of Products Currently Prod Actual number
Produced by the Firm (X4)
Training of Employees (X5) Train 0 (no formal program) or 1
(existence of a formal program)
Management Experience in Exp 0 (no experience) or 1 (presence
International Operation (X6) of experience)

48
,5p  p
   







r St ti ti

 
2
 
 %p 
>
   
% 5 % %
5 %
5 %
5 %
5 %
5 %
% 5 % %
5 %
5 %
5 %
5 %
5 %

5 %
5 %
5 %
5 %
5 %
5 %

 ?1

%
% %
% %
% %
% %
% %
%
% %
% %
% %
% %
% %
% %
% %
% %
% %
% %
% %

  

 
  ,  
 @   

 @



%


%
% % 

% 

  
 
   
 
   
 


% 49
p
   





1 %2




   1
2 A  B
   %
% % %

50
p
   





1 %2

Classification Resultsa

( 
R

 "%## 3%## +

 
  "%## 6= 3 7#
3%## 6 66 7#
"%## ;7%< 9%9 "##%#
3%## =%9 ;"%< "##%#
% ;:%3  
 





%

51