PARAMETER MODELLING/ PARAMETER LEARNING: parameters w that produces class probability estimates.
parameters w that produces class probability estimates. We would like p+(x+) =1 and
1. Data miner specifying the structure of the model with certain numeric parameters
left unspecified. e.g. parameterized mathematical function or equation of a set of
numeric attributes (attributes used in the model chosen based on domain knowledge
regarding which attributes is informative in predicting the target variable)
b. data miner specifies the form of the model and the attributes
2. data mining calculates the best parameter values given a particular set of training
data. This is the goal; tune the parameters so that the model fits the data = PL
Instance-space view: shows the space broken up into regions by decision boundaries;
examples in each space should have similar target variables and homogenous regions
help predict the target variable of a new unseen instance
LINEAR DISCRIMINANT FUNCTIONS
Linear Split = a linear classifier and is a weighted sum of the values for various
attributes
This is a linear discriminant because it discriminates between the classes, and the
function of the decision boundary is a linear combinationa weighted sumof the
attributes.(y=mx+b where y
and x are attributes)
- 2D=line, 3D=Plane, 4D= Hyperplane
Using the values of class(x) or f(x) we are then able to classify a new instance:
The more general form of a linear
discriminant function; a numeric
classification model:
Parameterized model: the weights of the linear function (wi) are the parameters
When the weights change, the answer and classification will also change accordingly
Supervised segmentation by using a mathematical function of multiple attributes
these weights magnitude are loosely interpreted as importance of the features:
OBJECTIVE FUNCTION FOR LINEAR DISCRIMINANT -WHAT PARAMETERS
SHOULD WE USE?
Define an objective function (loss function) that represents our goal, and can be
calculated for a particular set of weights and a particular set of data find the optimal
value for the weights by maximizing or minimizing the objective function.
A loss function will also determine how much penalty should be assigned to an
instance based on the error in the model's predicted
value. Different loss functions will result in different
answers: => seek to minimise errors
***weights are best only if we believe that the objective
function truly represents what we want to achieve faith and experience.
LINEAR DISCRIMINANT FUNCTIONS FOR SCORING & RANKING INSTANCES
need a score that will rank cases by the likelihood of belonging to one class or the
other.
f(x) will be zero when x is sitting on the decision boundary (points are randomly
classified, 50:50); f(x) will be relatively small when x is near the boundary. (points here
are sensitive to the position of the line; highly uncertain); And f(x) will be large (and
positive) when x is far from the boundary in the + direction.
f(x) gives a ranking of the instances by their (estimated) likelihood of belonging to the
class of interest. => but no probability estimate
LINEAR DISCRIMINANT 1: SUPPORT VECTOR MACHINES (LD)
Classify instances based on a linear function of the features
Objective Function: Maximise the margins = fattest bar between the classes
once the widest bar is found, the linear discriminant will be the center line through the
bar
margin-maximizing boundary gives the maximal leeway for classifying new points that
may fall in the margin
If the data are not linearly separable, the best fit is some balance between a fat margin
and a low total error penalty. The penalty for a misclassified point is proportional to the
distance from the decision boundary, so if possible the SVM will make only small
errors.
Objective fn = Max(Margin - |distance (pt, MB)|
LINEAR REGRESSION
Standard linear regression
procedures minimize the sum or mean of the squares of these errorswhich gives the
procedure its common name least squares regression.
2 types of objective functions: 1. Sum of absolute errors (more robust) 2. Sum of
squared errors (more convenient)
SHORTCOMINGS WITH THE LEAST SQ REG: For least squares regression a
serious drawback is that it is very sensitive to the data: erroneous or otherwise outlying
data points can severely skew the resultant linear function; Penalizes large errors
*systems that are building and applying models automatically, modelling needs to be
more robust **choose the objective function to optimizeand we should do so
with the ultimate business application in mind
LINEAR DISCRIMINANT #2: LOGISTICS REGRESSION
Logit gives accurate estimates of class probability (estimate the probability that
a new instance belongs to the class of interest).
Logit = model for the probability of class membership; It is a class probability
estimation model and not a regression model; usually used for binary problems = (Y/
N)
***the same linear function f(x) (since the magnitudes of f(x) were used to proxy the
probability) but given that probability ranges from 0 to 1, we seek to convert the f(x)
distance to a probability => by using f(x) as a measure of the log-odds of the event
of interest => f(x) is an estimation of the log-odds that x belongs to the positive class
LIKELIHOOD: The odds of an event is p(occurring):p(not occurring). So, for example,
if the event has an 80% probability of occurrence, the odds are 80:20 or 4:1. 90% is
90:10 or 9 ln=2.19. 0.001 is 1:999 or 0.001001 then ln=-6.9 ln(p(occurring)/p(not
occurring))
p+(x) represents the model's estimate of the probability of the class membership of a
data item by feature vector x
1) For probability estimation,
Logit uses the same linear model
as linear discriminants for
classification and linear
regression for estimating numeric
target values. 2) The output of the
Logit model is interpreted as the
log-odds of class membership. 3)
These log-odds can be translated
directly into the probability of class membership. 4) fit ln(odds) with the linear function
of the attributes through optimising the parameters
WHAT'S THE OBJECTIVE FUNCTION?
Consider the following function
computing the likelihood that a
particular labeled example belongs
to the correct class, given a set of
After determining that the classification tree is a better method (choose higher mean
p+(x.) to be =0. The g function gives the models estimated probability of seeing xs
actual class given xs features.
The model (set of weights (w)) that gives the highest product is the model that gives
the highest likelihood to the datathe maximum likelihood model. The
maximum likelihood model on average gives the highest probabilities to the positive
examples and the lowest probabilities to the negative examples.
Target variable is not a representation of the probability of class membership.
The training data comprise a set of statistical draws from the underlying probabilities,
rather than representing the underlying probabilities themselves. The Logit tries to
estimate the probabilities with a linear-log-odds model based on the observed data.
Logit =/= Regression: The way to distinguish classification/ regression is via the
target variable - is it categorical or numerical **in log regression, the model produces a
numeric estimate but the target values are still categorical
THE DIFFERENCE BETWEEN LOG REG & TREE INDUCTION:
A classification tree uses decision boundaries that are perpendicular to the instance-
space axes; linear classifier can use decision boundaries of any direction/ orientation
classification trees select a single attribute at a time whereas linear classifiers use
a weighted combination of all attributes.
A classification tree is a piecewise classifier that segments the instance space
recursively using a divide-and-conquer approach cut up arbitrarily finely into very
small regions. A linear classifier places a single decision surface through the entire
space; great freedom in the orientation of the surface, but limited to a single division
into 2 segments (linear eqn) that uses all of variables,must fit the entire data space.
Accuracy & Comprehensibility will be an important factor in which method to use:
Logit model = with a strong background in statistics
A decision tree, if it is not too large =without a strong statistics or mathematics
background.
WHAT IS OVERFITTING?
Overfitting: Finding chance occurrences in data that look like interesting patterns, but
which do not generalize, is called overfitting the data. Overfitting is the tendency of
data mining procedures to tailor models to the training data, at the expense of
generalization to previously unseen data points. => want models to apply to the
general population
Generalization is the property of a model or modeling process, whereby the model
applies to data that were not used to build the model.
100% accurate = table model => customer not part of the historical dataset, lookup will
fail since there will be no exact match (no generalisation = memorising the training
data)
HOW TO RESOLVE OVERFITTING?
All data mining procedures have the tendency to overfit to some extentsome more
than others. There is no single choice/ procedure that will eliminate overfitting. The
best strategy is to recognize overfitting and to manage complexity in a
principled way.
Trade-off between model complexity (better capture the real complexities & be more
accurate) and the possibility of overfitting
HOW TO IDENTIFY
OVERFITTING?
HOLD OUT DATA: Need to hold
out some data (or test set/ lab test)
where we know target variable, but
not be used to build the model
model will then be used to predict
the values of the holdout data and
this will help estimate the generalization performance by comparing the predicted
values with the hidden true values. FITTING GRAPH: fitting graph shows the accuracy
of a model as a function of complexity. More overfitting when the complexity increases.
model is not allowed to be complex enough -> not very accurate.
models get too complex -> look very accurate on the training data, but are overfitting
-> the training accuracy diverges from the holdout (generalization) accuracy.
Base error rate = % of churn cases in the population. (A corresponding baseline for a
regression model is a simple model that always predicts the mean or median value of
the target variable.)
Complexity: log reg = # of attributes; classification = # of nodes (not leaves)
OVERFITTING IN TREE INDUCTION
Tree induction: find impt predictive individual attributes recursively (divide parent node
to child node to get purer subset) splitting on attributes -> left with a single instance at
each leaf node, which is pure => lookup table => give a nontrivial classification (better
than lookup table) Generally: a procedure that grows trees until the leave are pure
tends to overfit = allowed to grow without bound, the decision trees can fit any data to
arbitrary precision
The sweet spot represents the best
trade-off between the extremes of
(i) not splitting the data at all (using
the average target value) and (ii)
building a complete tree out until the
leaves are pure.
OVERFITTING IN
MATHEMATICAL FUNCTIONS
#1: More complexity:
- adding more
variables (more attributes);
Adding non-linear terms (will increase flexibility) #2: As you increase the
dimensionality via more attributes, you can perfectly fit larger and larger sets of
arbitrary points. = To avoid overfitting: manual/automatic attribute selection
Comparing SVM to Logit: Logit appears to be overfitting; SVM tends to be less
sensitive to individual examples; procedure incorporates complexity control
CROSS VALIDATION
Cross-validation is a
more sophisticated
holdout training and
testing procedure => 1)
statistics such as mean
and variance of the accuracy,
to understand how the
performance is expected to
vary across datasets 2)
makes better use of a limited
dataset (computes its
estimates over all the data)
The purpose of CV is to use the original labeled data efficiently to estimate the
performance of a modeling procedure.
How does it work?
1 Cross-validation begins by splitting a labeled dataset into k partitions called folds.
2 Typically, k will be five or ten. These folds are created randomly.
3 Cross-validation then iterates training and testing k times, in a particular way.
4 In this iteration, the other k1 folds are combined to form the training data.
So, in each iteration we have (k1)/k of the data used for training and 1/k used for
testing. Each iteration produces one model, and thereby one estimate of
generalization performance, for example, one estimate of accuracy.
accuracy, St. dev lower), you will then use the entire data set to build the model
Generalisation Performance:
1 Different modeling procedures = different performance of the same data
2 Diff training sets = diff. generalisation performance
3 Diff test sets may = diff. estimates of the generation performance
4 Training set size change = diff gene performance for the resultant model
LEARNING CURVES
Generalization performance generally improves (since more accurate also) as more
training data become available, up to a point (learning curve steep initially; marginal
advantage of having more data decreases). A plot of the generalization performance
against the amount of training data is called a learning curve
Difference between learning curve and fitting curve: A learning curve shows the
generalization performance (performance only on testing data), plotted against the
amount of training data used. A fitting graph shows the generalization performance &
the performance on the training data, but plotted against model complexity (fixed
amount of training data).
for smaller training-set sizes, Logit
yields better generalization accuracy
than tree induction as the training sets
get larger, tree induction soon is more
accurate.
Tree Induction: Overfit more, Better for
larger datasets, Tree can represent
nonlinear relationships between the
features and the target; more flexible,
leading it to overfit more with small data, but to model more complex regularities with
larger training sets Logit: Perform better for smaller datasets; Since log regress is only
a line, cannot capture everything; has less flexibility, which allows it to overfit less with
small data, but keeps it from modeling the full complexity of the data
AVOIDING OVERFITTING
Tree Induction
Stop growing a tree before it gets too complex:
1) The simplest method to limit tree size is to specify a minimum number of instances
that must be present in a leaf. (this minimum-instance stopping criterion )
using the data at the leaf to make a statistical estimate of the value of the target
variable on a very small subset of data= inaccurate; amt is det. based on experience.
2) conduct a hypothesis test at every leaf to determine whether the observed
difference in information gain could have been due to chance. If the hypothesis test
concludes that it was likely not due to chance, then the split is accepted
Prune back a tree that is too large
Pruning means to cut off leaves and branches, replacing them with leaves.
Estimate whether replacing a set of leaves or a branch with a leaf would reduce
accuracy. If not, then go ahead and prune.
Build trees of varying sizes and then compare using CV which of the trees are
performing better
2. Linear Models
Regularisation: optimise fit and simplicity
Complexity control via regularization works by adding to this objective function a
penalty for complexity (depending whether the objective function is max or min, have
to change accordingly):
After maximum likelihood/ accuracy overfitting. The term is simply a weight that
determines how much importance the optimization procedure should place on the
penalty, compared to the data fit.
We can choose using cross-validation Then this would be used to learn a
regularized model on all the training data So when you increase the number of
parameters, it will be harmful for the objective function
There are different types of penalties that can be applied:
The most commonly used penalty is the sum of the squares of the weights, sometimes
called the L2-norm of w L2-norm+standard least squares linear reg=ridge regression
functions can fit data better if they are allowed to have very large positive and negative
weights; sum of the squares of the weights gives a large penalty when weights
have large absolute values.
Sum of squares given a large penalty when weights have large values
sum of the absolute values (rather than the squares), known as the L1-norm+standard
least squares linear reg=lasso
ends up zeroing out many coefficients= Since these coefficients are the multiplicative
weights on the features,
L1-regularization effectively performs an automatic form of feature selection.
Similarity and Distance:
- similarity between objects = the distance between objects: Data is represented as
feature vectors hence, the closer two objects = the more similar they are.
NEAREST NEIGHBOUR REASONING:
M
ost
similar instances are
called nearest neighbours
Heuristic for Nearest Neighbour: given a new example whose target variable we
want to predict, we scan through all the training examples and choose several that are
the most similar to the new example (this is still unsupervised) Then we predict
the new examples target value, based on the nearest neighbors (known) target
values. (this makes it supervised)
Combining Functions --> Prediction: Classification: highest count/ majority vote;
Regression: mean/ median of neighbours; Note that in retrieving neighbors we do not
use the target variable because were trying to predict it -> it will not enter into the
distance calculation
CLASSIFICATION: From the clusters, you calculate the distances and find the closest
neighbours. Look at the target variable of the k neighbours are retrieved and consult
their known target variables (classes), decide on the new entrys predicted target
variable via a majority vote
PROBABILITY ESTIMATION: Usually important not just to classify a new example but
to estimate its probabilityto assign a score to it; score gives more information than
 just a Yes/No decision. Probability = (Out of k who said "Yes")/(Out of k who said "No"+ Because the y axis represents the distance between clusters, the dendrogram can
Out of k who said "Yes") might want to use more than just three nearest neighbors
to compute the probability estimates
REGRESSION: Once we can retrieve nearest neighbors We could use the average
(about 42) or the median (40)
HOW TO DECIDE WHICH K TO USE?
Main parameter here is k ; using different Ks, you can compare the performance
(using CV) to decide which K you want to use
Odd numbers are convenient for breaking ties for majority vote classification.
Too small a k= overfitting; shading each pixel of the graph
the greater k =more the estimates are smoothed out among neighborsincrease k to Input: number of clusters k and a database containing n objects
the maximum possible (so that k = n) the entire dataset would be used for every Output: set of k clusters that minimizes the squared-error criterion
prediction: 1) For classification= predict the majority class in the entire dataset;2) for The k-means algorithm:
regression, the average of all the target values; 3) for class probability estimation, the
base rate probability
SOME ARE NEARER THAN OTHERS?
nearest-neighbor methods often use weighted voting or similarity moderated voting
such that each neighbors contribution is scaled by its similarity** can use all neighbors
but scaling each by its similarity (scaling weight = 1/square of the distance; the
contributions are proportional to the weights, adding up to one) => weighted probability
Benefits of using weighted scoring:
reduces the importance of deciding how many neighbors to use; => avoid committing
to a k by retrieving a very large number of instances (e.g., all instances, k = n) and
depend upon distance weighting to moderate the influences.
COMPLEXITY CONTROL
irregular concept boundaries; do not impose any particular geometric form on the
classifier; form boundaries in instance space tailored to the specific data training.
=1-NN overfits strongly => When classifying each training data point; retrieval of that
training point itself as its own nearest neighbor; own value for the target variable would
be used to predict itself --> perfect classification; n-NN model: do not allow much
complexity at all in our model
ISSUES WITH NEAREST NEIGHBOUR METHODS AS PREDICTIVE MODELS:
Intelligibility 2 aspects to it: the justification of a specific decision and the
intelligibility of an entire model.(difficult is to explain more deeply what knowledge
has been mined from the data) = KNN methods lack of specific decision models
Dimensionality and Domain Knowledge 1)numeric attributes may have vastly different
ranges, 2) Problem with having too many attributes, or many that are irrelevant to the
similarity judgment. curse of dimensionality = incidental information => instance
similarity can be confused and misled by the presence of many irrelevant attributes.
To fix this problem: feature selection, the judicious determination of features that
should be included in the data mining model => done manually by the data miner,
using background knowledge as what attributes are relevant.
An also inject domain knowledge into similarity calculations is to tune the
similarity/distance function manually => assigning diff weights to the different attributes
Computational Efficiency
training is very fast because it usually involves only storing the instances; main
computational cost of a nearest neighbor method is borne by the
prediction/classification step, when the database must be queried to find nearest
neighbors of a new instance ==> expensive and might take a long while
CLUSTERING
This idea of finding natural groupings in the data may be called unsupervised
segmentation, or more simply clustering. => Unsupervised modeling does not focus on quality of clustering is improved (e.g. the actually square-error E is reduced)*** same
a target variable. => objects within groups are similar but the objects in different
groups are not similar
2 main purposes: 1. Discovery of all overall distribution patterns and interesting
correlations among data attributes 2. Data reduction: cluster to get 1 group for
applications. Requirements of Clustering: small low dimensional data sets and Processing is
numerical attributes => cluster algo works well; large data sets: need to deal with 1)
scalability 2) different types of attributes (categorical, numerical) not all clusters are
shaped like a circle. Domain knowledge required to determine input parameters and
evaluate results, deal with noisy data (outliers have large impact on clustering,
insensitivity to order of input records), high dimensionality, interpretability and usability
Choice of clustering techniques: the type of data available, the particular purpose
and application (choice depends on domain knowledge)
Common method for focusing on the clusters themselves is to represent each cluster
by its cluster center, or centroid = not necessarily one of the instances; it is the
geometric center of a group of instances.
1 Hierarchical Clustering:
a Using Euclidean distance renders points more similar to each
other if they are closer to each other in the plane.
Circles Chart:
Circles labeled 1-5 are placed over the points to indicate
clusters
It is a clustering because it groups the points by their similarity.
Notice only overlap between clusters is when one cluster contains other clusters.
circles actually represent a hierarchy of clusterings
most general (highest-level) clustering is just the single cluster that contains everything
cluster 5 in the example
lowest-level clustering is when we remove all the circles, and the points themselves
are six (trivial) clusters
Dendrogram:
shows explicitly the
hierarchy of the
clusters
The numbers at the
joins of the
dendrograms
correspond to the
numbered circles
It creates a collection
of ways to group the
points --> clipping
the dendrogram
=>allows the data
analyst to see the groupings before deciding on the number of clusters
PROBLEMS WITH UNEQUAL
give an idea of where natural clusters may occur/ outlier, COSTS AND BENEFITS:
Hierarchical clusterings generally are formed by simple classification accuracy as a
starting with each node as its own cluster -> metric is that it makes no
merged iteratively until only a single cluster distinction between false positive
remains. and false negative errors = it
Distance function between clusters= called the makes the tacit assumption that both errors are equally important
linkage function: the Euclidean distance between the closest points in each of Should estimate the cost or benefit of each decision a classifier can make. Once
the clusters, aggregated, these will
CLUSTERING: NEAREST NEIGHBOURS (K-MEANS): produce an expected
profit (or expected
benefit or expected
cost) estimate for the
The algorithm starts by creating k initial cluster centers, usually randomly, but classifier.
USING APPROPRIATE METRICES: mean-
sometimes by choosing k of the actual data points, or by being given specific initial
squared-error (measure accuracy of a regression
starting points by the user, or via a pre-processing of the data to determine a good
model)= the value of the target variable, which is
set of starting centers.
the number of stars that a user would give as a rating for the movie
The clusters corresponding to these cluster centers are formed, by determining
EXPECTED VALUE: decomposes data-analytic thinking into (i) the structure of the
which is the closest center to each point.
problem, (ii) the elements of the analysis that can be extracted from the data, and (iii)
Next, for each of these clusters, its center is recalculated by finding the actual the elements of the analysis that need to be acquired from other sources
centroid of the points in the cluster. Method: expected value calculation
The process simply iterates: since the cluster centers have shifted, we need to the possible outcomes of a situation are enumerated and expected value is then the
recalculate which points belong to each cluster weighted average where the weight given to each value is its probability of occurrence
Once these are reassigned, we might have to shift the cluster centers again.
The k-means procedure keeps iterating until there is no change in the clusters (or
possibly until some other stopping criterion is met). **Each oi is a possible decision outcome; p(oi) is its probability and v(oi) is its value.
Evaluation of k-means: The probabilities often can be estimated from the data (ii), but the business values
No guarantee that a single run of the k-means algorithm will result in a good often need to be acquired from other sources (iii)
clustering: result of a single clustering run will often terminate local optimum but Specifically, lets calculate the expected benefit (or cost) of targeting consumer x:
depends on the initial centroid locations (-> random initial partitioning)
k-means algorithm is scalable and efficient in large data sets: generally
***What we care about is, in aggregate, how well does each model do: what is its
relatively fast, because it only computes the distances between each data point and
the cluster centers on each iteration; Hierarchical clustering is generally slower, as it
needs to know the distances between all pairs of clusters on each iteration, which at
the start is all pairs of data points.
Works well when clusters are compact clouds and well separated
Mean must be defined (e.g. not suitable for categorical attributes), Sensitive to noisy
and outlier data (-> mean calculation)
How to determine a good value for k?
One answer is simply to experiment with different k values and see which
ones generate good results. (NO CV) = value for k can be decreased if some clusters
are too small and overly specific, and increased if some clusters are too broad and
diffuse.
k-means is often used for exploratory data mining
K-MEDOIDS: Medoid: most centrally located datapoint in a cluster; Algorithm:
Find k cluster in n objects by first arbitrarily determining a representative
object (medoid) for each cluster
Each remaining object is clustered with the medoid which is the most similar
The medoids are iteratively replace by one of the non-medoids as long as the
distance measure as k-means
Compared to K-means: expected value.
ERROR RATES these probabilities (of errors and correct decisions) can be
It is more robust to noise and outliers as compared to k-means
estimated from the confusion matrix.: Each o i corresponds to one of the possible
more
costly than K-meanS
K-MEDIAN: Use Manhattan distance and
use median instead
Plain Accuracy (1-Error Rate): T=110, P=61,
Classification accuracy is popular (very easy to measure), but usually too
simplistic
Decompose and count the different types of correct and incorrect decisions
made by classifier
Accuracy is a common evaluation metric that is often used in data mining
studies because it reduces classifier performance to a single number and it is very
easy to measure
need a way to decompose and count the different types of correct and
incorrect decisions made by a classifier ==> CONFUSION MATRIX
Confusion Matrix: combinations of the class we predict/the actual class
A type of contingency table COST & BENEFITS
A confusion matrix for a problem involving n classes is an n n matrix with the While the probabilities can be estimated from data, the costs and benefits often
cannot.
Using this equation, we can now compute and compare the expected profits for
various models and other targeting strategies.
A cost-benefit matrix specifies for each (predicted, actual) pair the cost or benefit
making such a decision
columns labeled = actual classes; rows =predicted classes.
A confusion matrix separates out the decisions made by the classifier, making
explicit how one class is being confused for another. => different sorts of errors
may be dealt with separately = false positives and false negatives
a bad outcome = positive example [alarm!]
a good outcome = negative example [uninteresting]
Problems with Unbalanced Classes:
consider a classification problem where one class is rare ==> classifiers often The class priors, p(p) and p(n), specify the likelihood of seeing positive and negative
are used to sift through a large population of normal or uninteresting entities instances, respectively = factoring out allows us to separate the influence of class
because the unusual or interesting class is rare among the general population, the imbalance from the predictive power of the model
class distribution is unbalanced or skewed
becomes more skewed, evaluation based on accuracy breaks down
THEREFORE: we need more information about the data