Assessing the

Validity and
Reliability of
Diagnostic and
Screening Tests

Adora F. Mendoza – Abat,

M.D., CFP
 Screening
• The application of a test to people
who are as yet asymptomatic for the
purpose of classifying them with
respect to their likelihood of having a
particular disease
• Not diagnostic tests but preliminary
tests seeking to identify those who
are most likely to have the disease
Criteria of Disease Suitable
for Screening Program
• 1. The disease should be serious
• 2. Early treatment must be
beneficial in terms of reducing
mortality and morbidity comparable
to late treatment
• 3. The prevalence of preclinical
disease must be high in the
population screened
Biologic Variation of Human
Populations (Bimodal
Curve)
25

20

15
Induration
# of Subj.
10

0
0 3 6 9 12 15 18 21 24 27
Biologic Variation of Human
Populations (Unimodal
Curve)
140

120

100

80 S. BP
60 # of Men

40

20

0
<110 120 140 160 >180
 Validity of Screening
Tests
• Validity – the ability of the test to
distinguish between who has a
disease and who does not
• 2 Components:
• 1. Sensitivity – the ability of the test
to identify correctly those who have
the disease
• - The proportion of diseased people
who were correctly identified as such
by the test
 Factors that Increase
Sensitivity
• 1. Lowering the cut-off and making
criteria for disease definition less
stringent
• 2. Parallel Testing – the
administration of 2 screening tests at
the same time and persons with a
positive result on any of these tests
are considered positive
When Should Sensitivity Be
Increased
• 1. When the disease is serious and a
definitive treatment exists
• 2. When the disease can be spread
• 3. When subsequent diagnostic
evaluations of positive screening
tests are associated with minimal
costs and risks
 Validity of Screening
Tests
• 2. Specificity – the ability of the test
to identify correctly those who do not
have the disease
• - proportion of non-diseased people
who are correctly called negative by
the test
• Note : In order to quantitatively
assess the sensitivity and specificity
of a test, we must have another
source of truth with which to
compare the test results
 Factors that Increase
Specificity
• 1. Making the criteria for disease
definition more stringent or
increasing the cut-off level
• 2. Series Testing – an initial
screening test is administered and
then, only persons who are positive
on this preliminary test will undergo
a second additional screening
procedure
When Should Specificity Be
Increased
• 1. When the costs or risks
associated with further diagnostic
techniques are substantial
• 2. When the stigma associated with
having the disease is severe
 Concept of the Sensitivity
and Specificity of Screening
Exams
Test Dse. No Tota • Sensitivity = 80 x
Dse. l 100
• 100
+ 80 100 180
• = 80%

- 20 800 820
• Specificity = 800 x
100
Tota 100 900 1 • 900
l 000 • = 89%
 Comparison of the Results of a
Dichotomous Test with Actual
Disease Status
Test W/ Dse. W/O Sensitivity = TP
Results Dse.
x 100
TP + FN
+ Have No dse.
dse. & & have
have (+) (+) test Specificity = TN
test = = false x 100
true (+) negative
- or (TP)
Have (FP)
No dse. TN + FP
dse. but and
have (-) have (-)
test = test =
false (-) true (-)
or (FN) (TN)
Issues of False Positives
• All people who screened positive are
brought back to a more sophisticated
and more expensive tests
• The anxiety and worry induced in
persons who have been told that
they have tested positive
 Importance of Problems in
False Negatives
• Depends on the nature and severity
of the disease being screened for,
the effectiveness of available
intervention measures, and whether
the effectiveness is greater if the
intervention is administered early in
the natural history of the disease
Tests of Continuous
Variables
Diabetics Non-Diabetics

High

Blood
Sugar

Low
 Test
Test
 Screening Using a
Dichotomous Test
Test Disease Total

Positive Negative

(+) a (True b (False a + b (all people with positive

positives) Positives) tests)

(-) c (False d ( True c + d ( all people with negative

negatives) negatives) tests)
 Two-Stage Screening
• First Stage – less expensive, less
invasive, or less uncomfortable test
• - screened positive are recalled for
further testing
• Second Stage – more expensive,
more invasive which may have
greater sensitivity and specificity
• - reduces the problem of false
positives
 Two Stage Screening
Program
• Test 1 (Blood Sugar)
• Assume: Disease Prevalence = 5%
• Population = 10, 000
• Test 1 (Blood Sugar) : Sensitivity = 70%
• Specificity = 80%

Test Disease Total

Results (+) (-)
(+) 350 1,900 2,250
(-) 150 7,600 7,750
Total 500 9,500 10,000
 Two Stage Screening
• Test 2 (OGTT)
Test Dse. -Dse Total
• Given:
+ 350 1 2250
Sensitivity = 90%
- 150 900
7 7750 Specificity = 90%
Tota 500 600 9 10000 • Solution :
l 500 Net Sensitivity
Test Dse. -Dse Tota = (315/500)(100)= 63%
+ 315 190 l505 Net Specificity
- 35 1 1 = [ (7600+1710) /
Tota 350 710
1 745
2 9500] [100]
l 900 250 = 98%
 Predictive Value of a
Test
• The proportion of people correctly
identified by the test to have a
disease
• Positive Predictive Value – the
proportion of patients who test
positive and who have actually the
disease in question
• - divide the number of true positives
by the total number who tested
positive (true positives + False
• Negative Predictive Value – the
probability that those who tested
negative do not have the disease
• - divide the number of true negatives
by all those who tested negative
(true negatives + false negatives)
 Predictive Value of a
Test
Test Disease No Dse. Total
Results
Positive 80 100 180
Negative 20 800 820
Total 100 900 1,000

• Positive Predictive Value = (80/180) x 100 =

44%
• Negative Predictive Value = (800/820) x
100= 98%
 Relation of Disease
Prevalence to Predictive
Value
Ex.; Sensitivity = 99% Specificity = 95%
Dse. Test Sick Not Totals
Predictive
Prevalence
1% Results
+ 99 Sick
495 594
Value - 1 9405 9406
Totals 100 9900 10000
5% + 495 475 970
- 5 9025 9303
Totals 500 9500 10000
 Relation of Disease
Prevalence to Predictive
Value
Ex.; Sensitivity = 99% Specificity = 95%
Dse. Test Sick Not Totals Predictive
Prevalence Results Sick Value
1% + 99 495 594 (99/594) x
100 = 17%
- 1 9405 9406
(9405/9406)
5% x 100=100%
+ 495 475 970 (495/970)
x 100 = 51%Totals 100 9900 10000
- 5 9025 9303
(9025/9303) x 100 = 97%
Relation of Predictive Value
to the Specificity of the
Test
Test + -Tota • Prevalence = 50%
+ 250 250 l500 • Sensitivity = 50%
- 250 250 500 • Specificity = 50%
Tota 500 500 1 • Predictive Value?
l 000
Test + - Tota
• Prevalence = 20%
+ 100 400 l500
• Sensitivity = 50%
- 100 400 500 • Specificity = 50%
Tota 200 800 1 • Predictive Value?
l 000
Relation of Predictive Value
to the Specificity of the
Test + - Test
Tota
l • Prevalence = 20%
+ 180 400 580 • Sensitivity = 90%
• Specificity = 50%
- 20 400 420 • Predictive Value?

Test + - Tota
• Prevalence = 20%
+ 80 l180
• Sensitivity = 50%
100
• Specificity = 90%
- 720 870
• Predictive Value?
100
Relation of Predictive Value
to the Specificity of the
Test
 Interpretation
• Sensitivity = 90%
-90% of those with the disease will be
labeled as positive by the test
• Specificity = 90%
- 90% of those without the disease will be
labeled as negative by the test
Positive Predictive Value = 75%
75% of those who test positive will really
have the disease
Negative Predictive Value = 75%
75% of those who test negative will really
Reliability (Repeatability of
Tests)
• Regardless of the specificity and
sensitivity, if the test results cannot
be reproduced, the value and
usefulness of the test are minimal.
 Factors that Contribute to
Variation in Test Results
• 1. Intrasubject Variation – variation
between individual subjects
• - In evaluating any test result, it is
important to take into consideration
the conditions under which the test
was carried out, including the time of
day.
• 2. Interobserver Variation – variation
between those reading the test
results
Relation Between Validity
and Reliability
 Exercise 1
• Calculate for the sensitivity, specificity,
positive predictive value, and negative
predictive value
• Interpret your answers

THROAT CULTURE

Test With W/O Dse. Total

Results
+ Disease
27 35 62
- 10 77 87
Total 37 112 149
 Exercise 2
• Complete the table with needed data
• Solve for the positive predictive
value and interpret your answer
• State the relationship of predictive
value and specificity using the
positive predictive value
• Solve for the negative predictive
value and interpret your answer
 Relationship of Specificity
to Predictive Value
Ex. Prevalence = 10% ; Sensitivity =
100%
Specificity Test Sick Not Totals
Predictive
70% +
Results Sick
-
Value
Totals 10000
95% +
-
Totals 10000