Statistics Review

Lec03
ECE 452/552

Learning Outcomes
Students will be able to
Derive a statistic from a given sample set of observations.

Evaluate its estimator based on properties like bias, consistency and accuracy.
Assess if the statistic is sufficient, given the statistical model.

Use different types of distributions to model observed data.

Exponential Family (Gaussian, Exponential and Chi-Squared)

Model problems as Hypothesis or Significance testing.

Statistic
What is a statistic?

Given a random sample

, from a distribution
parameterized by in some parameter space
Any real valued function on the random sample is called a statistic

Examples:

mean, variance, median, mode, min, max, any constant

In general, when we use an estimator on an observation, we estimate

some statistic
Statistical Parameter numerical characteristic of the statistical model
A statistic is often an estimator for a statistical parameter, e.g., sample mean

Sufficient Statistic
A statistic
is sufficient if and only if the joint density
of samples can be factored as follows, according to the NeymanFisher factorization theorem:

where u,v are nonnegative functions

The function u can depend on the full random sample
but not on unknown .
The function v can depend on , but on the random sample only
through the statistic

Sufficient Statistic Example: Uniform

Suppose {X1,X2,,Xn} are uniformly distributed on [0,], with
unknown , then what is a sufficient statistic? T=max(X1,X2,,Xn)

Uniform pdf

Joint pdf

Sufficient Statistic Example: Uniform

If any xi < 0, then fn = 0 for all . So only consider the factorization for
xi0, i=1,,n
Define an indicator function

xi for i=1,,n if and only if max(x1,,xn) . Therefore, for xi0,

i=1,,n we can write

Sufficient Statistic Example: Normal

Let {X1,X2,,Xn} be a random sample drawn from a normal
distribution for which the mean is unknown but the variance 2 is
known. Find a sufficient statistic for

Sufficient Statistic Example: Normal

Let {X1,X2,,Xn} be a random sample drawn from a normal
distribution for which the mean and variance 2 unknown. Find a
sufficient statistic

Examples
Sample mean is an unbiased and consistent estimator of E[X]
Unbiased but inconsistent

Let {X1,X2,,Xn} be an iid sample and the statistic T(X)=X1 is the estimator of E[X]
Is this consistent?
Is this unbiased?

Biased but consistent

Let {X1,X2,,Xn} be an iid sample and T(X) =

Is this consistent?
Is this unbiased?
Is this asymptotically unbiased?

as the estimator of E[X]

Example: The Sample Mean

Example: The Sample Mean

sufficient

Example Significance Test

Suppose X1,,Xn are iid samples of an exponential () random variable
X with unknown parameter . Using the observations X1,,Xn, the
significance test can answer a question regarding the unknown .
H0: =3.5

Question: Should we accept or reject the hypothesis that =3.5?

Answer: Yes/No

Example Hypothesis Test

Suppose X1,,Xn are iid samples of an exponential () random variable
X with unknown parameter . Using the observations X1,,Xn, the
significance test can answer a question regarding the unknown .
H1: =2.5; H2: =3.5; H3: =4.5

Question: Which of the hypotheses is true H1, H2 or H3?

Does equal to 2.5, 3.5 or 4.5?

(a)

(b)

Types of Errors
Type-I error

Significance Testing: Reject H0 when H0 was true

Binary Hypothesis Testing: Reject H0 (Accept H1) when H0 was true (False alarm)

Type-II error

Significance Testing: Accept H0 when H0 was false

Binary Hypothesis Testing: Accept H0 (Reject H1) when H0 was false (False reject)

Jargon: Statisticians vs. Electrical Engineers

Statisticians

Electrical Engineers

Statistic T(x) and threshold

Null Hypothesis (H0)
Alternative Hypothesis (H1)
Type-I error (H1, when H0 true)
Type-II error (H0, when H1 true)
Significance level

Detector
Noise only hypothesis
Signal + Noise hypothesis
False Alarm
Miss or False Reject
Probability of False alarm

Error Metrics

Sensitivity = Recall = Hit/Detection rate = True Positive rate

TP/(TP + FN)
Specificity = True Negative rate = 1-False Positive Rate
TN/(TN + FP)
Precision
Recall
TP/(TP + FP)
TP/(TP + FN)
F-measure
F = 2TP/(2TP + FP + FN)
Accuracy
Acc = (TP + TN)/(TP + FN + TN + FP)

TP True Positive
TN True Negative
FP False Positive
FN False Negative

Detection rate

False Negative rate

Receiver Operating Characteristic (ROC) Curve

False Alarm

False Alarm

Next Lecture
Reading from KD-Ch-3
Bayesian Hypothesis testing
Neyman Pearson based Hypothesis testing
Likelihood Ratio Test (LRT)
Matched Filters (Maybe)