2270
ORIGINAL ARTICLE
The Validity of Prospective and Retrospective Global Change
Criterion Measures
John Schmitt, PhD, PT, Richard P. Di Fabio, PhD, PT
ABSTRACT. Schmitt J, Di Fabio RP. The validity of pro-
spective and retrospective global change criterion measures.
Arch Phys Med Rehabil 2005;86:2270-6.
Objective: To assess the validity of retrospective versus
prospective criterions of change.
Design: Single cohort pretest-posttest design.
Setting: Physical or occupational therapy outpatient clinics.
Participants: Volunteer sample of 211 patients with upper-
extremity musculoskeletal problems.
Interventions: Not applicable.
Main Outcome Measures: Disabilities of the Arm, Shoul-
der, and Hand questionnaire, the Shoulder Pain and Disability
Index, the Patient-Rated Wrist Evaluation, the Medical Out-
comes Study 12-Item Short-Form Health Survey; global dis-
ability rating (GDR), retrospective global rating of change
(GRC), and patient satisfaction.
Results: Correlations were calculated among the baseline,
3-month follow-up, and change scores for each outcome mea-
sure with the change criterion instruments. Retrospective GRC
and patient satisfaction ratings showed moderate correlations
with the 3-month follow-up scores, but nonsignificant correla-
tions with baseline scores. By contrast, the prospective GDR
criterion showed significant correlations with both baseline and
3-month follow-up scores ranging between 0.3 and 0.4 (abso-
lute value).
Conclusions: Retrospective self-report measures of change
do not accurately reflect true change over time. The retrospec-
tive GRC and patient satisfaction were heavily influenced by
current (posttreatment) status whereas the prospective global
change measure reflected both baseline and posttreatment sta-
tus equally and thus appeared to be a more valid measure of
change over time. This study demonstrates the need for an
alternative criterion for establishing true individual change.
Key Words: Outcome assessment (health care); Recall;
Rehabilitation; Treatment effectiveness; Validity of results.
© 2005 by the American Congress of Rehabilitation Medi-
cine and American Academy of Physical Medicine and
Rehabilitation
From the College of St. Catherine, Minneapolis, MN (Schmitt); and Department of
Physical Medicine & Rehabilitation, School of Medicine, University of Minnesota,
Minneapolis, MN (Di Fabio).
Presented as a poster to the American Physical Therapy Association Combined
Sections Meeting, February 2004, Nashville, TN.
Supported in part by the American Physical Therapy Association Orthopaedic
Section.
No commercial party having a direct financial interest in the results of the research
supporting this article has or will confer a benefit upon the author(s) or upon any
organization with which the author(s) is/are associated.
Reprint requests to Richard P. Di Fabio, PhD, PT, Program in Physical Therapy,
Mayo Mail Code 388, University of Minnesota, Minneapolis, MN 55455, e-mail:
difab001@umn.edu.
0003-9993/05/8612-9785$30.00/0
doi:10.1016/j.apmr.2005.07.290
Arch Phys Med Rehabil Vol 86, December 2005
ESPONSIVENESS TO CLINICALLY important change
R over time is an essential characteristic of an instrument
1-4
designed to assess patients’ response to treatment. There is
no agreement in the literature on a method to determine re-
sponsiveness. A critical challenge is to identify a group that has
truly changed. A randomized controlled trial using a treatment
of known efficacy is the strongest design to accomplish this
purpose because it provides for the differentiation of “changed”
from stable patients. However, drawbacks of this approach
include the requirement for treatments of proven efficacy for a
given condition, and the ethical considerations of withholding
such efficacious treatment from a placebo group simply to
determine the responsiveness of an outcome measure.5 Instead,
an observational design is often employed, using an external
criterion to determine change status.
There is no accepted criterion to ascertain true change sta-
tus.5-8 One approach is to utilize a retrospective global rating of
change instrument, consisting of a single question about degree
of change since baseline.5,9-11 The retrospective global rating
serves as an external criterion to differentiate between changed
and unchanged patients at the end of a longitudinal study. It has
been described as a “credible alternative” in the absence of a
criterion standard for change in functional status.12 Retrospec-
tive ratings are also known as “transitional” instruments that
require the respondent to compare current status with baseline
status and judge whether change has taken place (yes/no) and,
if so, how much change.
Norman et al13 have challenged the use of such global
ratings of change (GRC) to categorize improvement. These
authors point out that retrospective judgments require compar-
ison of a current state to that experienced weeks or months
previously. This is a difficult recall task for patients. Ross14 has
discussed the potential for recall bias related to recent history
using numerous examples from the social science literature.
Rather than an accurate representation of true change, retro-
spective ratings are more likely to correlate with the subject’s
current status. These arguments bring into question the validity
of responsiveness indices based on retrospective ratings of
change. Nevertheless, authors continue to use these global
instruments as a criterion to determine responsiveness8,15-19
and to define treatment success.20,21
We have recently reported22 on a comparison of various
statistics for reporting responsiveness. As an alternative to
retrospective global ratings, we chose to use the change score
for a patient’s global rating of current status, assessed at
baseline and again 3 months later. Our rationale was to avoid
the necessity for patient recall inherent in retrospective global
ratings. However, few studies have compared the validity of
this approach with other methods for determining a patient’s
change status.
According to Norman,13 a retrospective estimate of change
is invalid to the extent that it correlates more highly with the
current status of the patient (at follow-up) than with the base-
line state. A true measure of change would correlate equally in
magnitude (but with opposite sign) with the initial and final
patient condition.13 In other words, true change would be
equally influenced by baseline and follow-up status. In a 2002
 VALIDITY OF CLINICAL CHANGE MEASURES, Schmitt
article, Guyatt et al23 offer a mathematical proof of this equal
and opposite correlation with a valid transition rating. How-
ever, few studies have used clinical data to test the validity of
change criterions. In the analysis described in Norman et al,13
the investigator’s retrospective judgment of change was used as
a criterion standard for comparison with the global ratings. No
evidence was offered to support the validity of the investiga-
tor’s ratings. Further research is needed to explore the validity
of other possible criterion measures for change. The purpose of
this study was to compare the validity of retrospective mea-
sures of change to alternative criterions for change, including a
prospectively administered global function measure.
METHODS
Participants
Patients with a physician’s diagnosis of a musculoskeletal
upper-extremity problem were recruited at the initial visit to
local physical and occupational (hand therapy) outpatient clin-
ics. All patients 18 years and older who could read and under-
stand English were eligible for the study. Subjects were ex-
cluded if they had primary or coexisting systemic conditions
including multiple sclerosis, cancer, rheumatoid arthritis,
stroke, or mental or cognitive impairment. The recruitment
goal was at least 200 subjects to minimize sampling error and
allow sufficient power for subgroup analysis. This study was
approved by local institutional review boards and informed
consent was obtained from each research subject.
Study Design
A single cohort of patients with upper-extremity diagnoses
was followed-up from the initial clinical visit for a total of 3
months. The duration and method of therapy provided was left
to the individual therapist. The 3-month follow-up was com-
pleted by mail. The interval between baseline and 3-month
follow-up was used as a construct for change.
Outcome Measures
Disabilities of the Arm, Shoulder, and Hand Question-
naire. The Disabilities of the Arm, Shoulder, and Hand ques-
tionnaire (DASH)24 has 30 items and is intended to evaluative
change over time.25 The DASH is region-specific and so allows
comparisons across diagnoses of the upper extremity. Scores
may range from 0 to 100, with 0 reflecting no disability. The
DASH has shown high internal consistency25 using the Cron-
bach ␣ and intraclass correlation coefficients (ICCs) for test-
retest reliability using have been reported as .92 and above.26-28
The construct validity and responsiveness of the DASH have
been supported in a number of patient populations.26-30
Shoulder Pain and Disability Index. The Shoulder Pain
and Disability Index (SPADI)31,32 is a 13-item questionnaire
related to pain and function of the shoulder. The SPADI
includes a 5-item pain scale and an 8-item disability scale; each
scale is scored as a percentage of the maximum possible scale
score. The total SPADI is calculated as the mean of the pain
and disability scales. Scores range from 0 to 100 with higher
scores indicating greater disability. The SPADI has shown
reliability,33 validity,31,32,34 and responsiveness33,35 among pa-
tients with various shoulder diagnoses. The SPADI was given
to the subgroup of patients with diagnoses affecting the prox-
imal upper extremity.
Patient-Rated Wrist Evaluation. The Patient-Rated Wrist
Evaluation (PRWE)30,36 is a joint-specific instrument that con-
tains 5 items regarding pain and 10 items pertaining to function
of the wrist and hand.30 Item totals are summed and trans-
2271
formed to a 0 to 50 scale for both pain and function. The total
score is the sum of the 2 subscales and may range from 0 to
100, with higher scores indicating greater pain and disability.
Reliability, validity, and responsiveness to change have been
reported in patients recovering from wrist fractures.30,36 The
PRWE was given to the subgroup of patients with conditions
affecting the distal upper extremity.
Medical Outcomes Study 12-Item Short-Form Health Sur-
vey. The Medical Outcomes Study 12-Item Short-Form
Health Survey (SF-12) is a generic health-related quality of life
measure with items derived from the 36-Item Short-Form
Health Survey (SF-36).37,38 It has been shown to reproduce the
physical (PCS) and mental component summary (MCS) scales
of the SF-36 with less respondent burden.38
Global disability rating. A prospectively administered global
disability rating (GDR) scale was used to provide the patient’s
current global functional assessment without requiring exten-
sive recall.22 Patients were instructed to answer the question,
“How much has this problem affected your normal daily ac-
tivities using the arm(s) during the past week?” Disability
levels ranged from 1 (no disability) to 7 (maximum disability).
Written descriptors defined each level of disability. Patients
completed this single global question at each outcome assess-
ment. The examining therapists also rated the patient’s baseline
disability using this scale.
The clinician interrater reliability was examined using
records of therapy evaluations obtained for 10 patients with a
variety of disorders affecting the shoulder, elbow, wrist, and/or
hand. Each participating therapist was asked to independently
rate the upper-extremity disability on each case. A subset of
therapists was asked to repeat this process approximately 5
months later to examine intrarater reliability. Therapists were
blind to their earlier ratings of these identical cases.
The test-retest reliability of the GDR in this population has
been previously reported with ICCs of .75 for patient ratings
and .85 for therapist ratings.22 Significant correlations between
the patient and therapist ratings supported the construct validity
of this instrument.22 Patient ratings showed Spearman correla-
tions ranging from .56 to .71 with the 3 upper-extremity out-
come measures (DASH, SPADI, PRWE) at baseline and 3
months.22
Retrospective GRC. This global rating format has been
used previously to provide an external standard for change in
functional status.5 The patient judges whether change has oc-
curred and rates both the extent and the importance of this
change on 7-point Likert scales. Degree and importance ratings
of change are summed for a range of 1 to 14 for improvement,
0 for no change, or ⫺1 to ⫺14 for deterioration.
Global satisfaction. At the 3-month follow-up, patients
were asked, “How satisfied were you with your physical/
occupational therapy treatment(s)?” The 5-item responses
ranged from “very dissatisfied” to “very satisfied” with “neu-
tral” as the middle descriptor.
Protocol
At the initial therapy visit, each patient completed a demo-
graphic summary form, the DASH, SF-12, SPADI, and GDR.
In addition, patients with injuries affecting the distal extremity
(elbow, wrist, and/or hand) completed the PRWE. The exam-
ining therapist completed a rating of disability based on his/her
evaluation of the patient’s injury.
Three months after the initial visit, the complete packet of
questionnaires was mailed to each subject with the addition of
the retrospective GRC. If there was no response, follow-up
postcards and a second set of questionnaires was mailed in
subsequent weeks.
Arch Phys Med Rehabil Vol 86, December 2005
2272 VALIDITY OF CLINICAL CHANGE MEASURES, Schmitt

Table 1: Demographic Summary (model 2,1) for interrater reliability. Test-retest reliability data
3-Month were entered into a 1-way ANOVA with subjects as the inde-
Baseline Follow-up pendent variable. We used mean square terms to calculate an
Characteristics (N⫽211) (n⫽155) ICC1,1 for intrarater reliability.
Mean age ⫾ SD (y) 47.5⫾14.1 49.6⫾14.3 To examine the validity of the GDR, the relation between
Age range (y) 18–88 18–88 baseline therapist and patient ratings of upper-extremity func-
Sex, n (%) tion were analyzed using the Spearman rank-order correlation
Male 105 (49.8) 70 (45.2) coefficient. We used Spearman rank-order coefficients and
Female 106 (50.2) 85 (54.8) 95% confidence intervals (CIs)39 to correlate the GDR, retro-
Symptom location, n (%) spective GRC, and patient satisfaction with the 3 upper-ex-
Shoulder 133 (63.0) 98 (63.2) tremity outcome measures (DASH, SPADI, PRWE) for base-
Elbow, wrist, or hand 69 (32.7) 52 (33.5) line, follow-up, and change scores.
Both 9 (4.3) 5 (3.2)
Symptom duration (mo) RESULTS
Mean ⫾ SD 11.1⫾23 12.7⫾26 A summary of the subjects’ demographic information is
Median 4 4 listed in table 1. The patients had a variety of diagnoses
Workers’ compensation, n (%) 38 (18) 19 (12) involving the upper extremity as is seen in many outpatient
Surgery, n (%) 33 (15.6) 24 (15.5) orthopedic practices. The most common diagnoses were shoul-
Occupation, n (%) der pain (n⫽45 patients, 21% of the sample), shoulder tendon-
Manual labor 67 (31.8) 35 (22.6) itis (n⫽33, 16%), shoulder impingement (n⫽16, 8%), and
Office/professional 101 (47.9) 80 (51.6) elbow tendonitis (n⫽24, 11%).
Retired 29 (13.7) 27 (17.4) Six percent of the SF-12 questionnaires had missing items
Other 14 (6.8) 13 (8.4) (typically 1 item was omitted). The correlation of the imputed
scores with scores using median response replacement was .98
NOTE. Descriptive statistics for eligible subjects enrolled at baseline for both the SF-12 PCS and MCS scores.
and the subset of subjects completing the 3-month follow-up.
Abbreviation: SD, standard deviation.
Reliability and Validity of GDRs
Thirty-nine physical and occupational therapists participated
A total of 326 subjects were invited to participate in this in subject recruitment and therapist GDRs. The ICC2,1 for
study. Of these, 115 subjects declined to participate (50% interrater reliability for case study ratings by the 39 participat-
women; mean age, 49.1y), with lack of time being the primary ing therapists was .67. Five months later, 9 of the participating
reason for refusal. A total of 211 eligible subjects were enrolled therapists rated the same set of 10 case studies with an in-
and 155 (73.5%) returned the 3-month follow-up packet of trarater ICC1,1 of .96. At baseline, the correlations between
questionnaires (table 1). This subset of patients completing patient and therapist GDRs were .64 for patients with proximal
follow-up was similar to the baseline sample in age, sex, diagnoses and .53 for patients with distal diagnoses.
location of symptoms, occupation, education, and race.
Correlations Among Change Measures
Analysis The correlations of both retrospective and prospective crite-
All statistics were run using NCSS 2001.a Instruments were rion measures of change with change scores of the generic
consistently scored by 1 investigator according to the original SF-12 PCS, the region-specific DASH, and the joint-specific
authors’ instructions. When items were missing from the SPADI and PRWE total scores are listed in table 2. Both the
DASH, SPADI, or PRWE, percentage scores were calculated prospective and retrospective criterion measures showed mod-
based on the reduced total number of items. When responses erate correlations with change scores on the upper-extremity
were missing from the SF-12, the PCS and MCS scores were outcome measures (range, .54 –.60), while correlations with
imputed using a multiple linear regression formula with com- patient satisfaction were somewhat lower (range, .27–.46).
pleted items as predictors of the final scale scores. To test the A breakdown of the correlation of the retrospective and
effect of this procedure, Pearson correlation coefficients were prospective change criterion measures with baseline and
calculated to compare imputed scores with scores resulting 3-month follow-up status is shown in table 3. As predicted, the
from substitution of sample median item responses for the retrospective GRC correlated much more highly with the cur-
missing items. rent status of the patient (Spearman ␳ range, ⫺.42 to ⫺.59)
The clinician GDRs of the case records were analyzed with than at baseline (␳ range, .05–.17). In fact, correlations between
a 2-way repeated-measures analysis of variance (ANOVA) the retrospective ratings and baseline scores were nonsignifi-
with case as a repeated factor and rater as an independent cant for each questionnaire. Similarly, the correlations with
variable. We used mean square terms to calculate the ICC patient satisfaction ranged from ⫺.26 to ⫺.32 at follow-up, but

Table 2: Correlations of External Criterion Measures With Questionnaire Change Scores

DASH Change SF-12 PCS Change SPADI Change PRWE Change
Outcome Measures (95% CI) (n⫽139) (95% CI) (n⫽139) (95% CI) (n⫽91) (95% CI) (n⫽40)

Prospective change* .67 (.57–.75) .54 (.41–.65) .63 (.49–.74) .61 (.37–.77)
Retrospective change .66 (.55–.74) .57 (.45–.67) .62 (.48–.73) .62 (.38–.78)
Patient satisfaction .43 (.28–.56) .27 (.11–.42) .32 (.12–.49) .46 (.17–.68)

*Based on change scores on the GDR.

Arch Phys Med Rehabil Vol 86, December 2005

 VALIDITY OF CLINICAL CHANGE MEASURES, Schmitt 2273

were nonsignificant at baseline. The correlation with prospec-

⫺.34 (⫺.03, ⫺.59)

⫺.52 (⫺.25, ⫺.72)
tive GDR change scores, however, were opposite in sign and

⫺.31 (.00,⫺.57)
nearly equal in magnitude, as predicted for an unbiased mea-

3-Month
sure of change. Baseline correlations ranged from .26 to .42,

PRWE Scores (n⫽40)

and follow-up correlations were from ⫺.20 to ⫺.40. Despite

Table 3: Spearman Correlations of Global Change Criterion With Baseline, 3-Month Follow-Up, and Change Scores of Upper-Extremity Outcome Measures
this finding, table 2 shows that the retrospective and prospec-
tive global change criterion measures had similar correlations
with the change scores of the DASH, SPADI, and PRWE.

.26 (⫺.06, .53)

.10 (⫺.22, .40)
.04 (⫺.28, .35)
Initial
DISCUSSION
Retrospective global change instruments do not accurately
measure change over time. The retrospective measures used in
this study were heavily influenced by current (posttreatment)
status. As predicted, baseline (pretreatment) status had little or
no impact on the retrospective ratings. Van Stel et al40 reported
⫺.40 (⫺.21, ⫺.56)
⫺.59 (⫺.44, ⫺.71)
⫺.26 (⫺.06, ⫺.44)

similar results in a validation study for the Quality of Life for

3-Month

Respiratory Illness Questionnaire. These findings have impli-

cations for researchers who want to categorize patients accord-
SPADI Scores (n⫽91)

ing to change status. Perhaps more important, it should concern

clinicians who base treatment decisions on a patient’s retro-
spective judgment.41 Transitional questions such as, “How
much have you changed since your first visit” yield responses
.05 (⫺.16, .25)
.09 (⫺.12, .29)

that are poor reflections of true change across time. Accurate

.30 (.10, .48)

assessment of change in health status would require consider-

Initial

ation of baseline health relative to current health, and both

assessments should contribute equally to a valid rating of
change.
Advocates of evidence-based practice have placed more
⫺.20 (⫺.04, ⫺.36)
⫺.42 (⫺.27, ⫺.55)
⫺.26 (⫺.10, ⫺.41)

Sign of correlation coefficients were reversed for consistency with other outcome measures in this table.

emphasis on dichotomous measures of treatment effect such as

experimental and control event rates and number needed to
3-Month
SF-12 PCS Scores (n⫽139)†

treat. Frequently, retrospective global measures have been used

to categorize patients as improved or not.20,21 Despite the
questions raised by Norman et al,13 researchers continue to
assume validity in the use of such outcome measures. These
assumptions do not appear to be warranted.
Guyatt et al23 cite data from 3 different clinical trials to argue
.01 (⫺.16, .18)
.37 (.22, .50)
.17 (.00, .33)

for the use of transition measures as valid retrospective crite-

Initial

rions for change, useful guides to score interpretation, and even

as primary outcome measures in both research and clinical
practice. This argument was based on a regression analysis
where the baseline scores were significant predictors of the
retrospective global change scores. However, these authors
⫺.30 (⫺.14, ⫺.44)
⫺.54 (⫺.41, ⫺.65)
⫺.32 (⫺.12, ⫺.49)

acknowledge that rarely did the transition measures perform

as predicted for a true change criterion, with follow-up
3-Month

scores accounting for a much greater variance in the transi-

DASH Scores (n⫽139)

tion scores in approximately two thirds of the measures cited.

This study involved a 4-week recall period. For longer interim
periods, transition ratings would be even more suspect.
Retrospective ratings are prone to bias. Factors such as
.16 (⫺.01, .32)
.10 (⫺.07, .26)

social desirability or transient variations in health produce

.42 (.27, .55)

undue influence on a rating that is intended to reflect actual

Initial

change across weeks or months of time. The patient making a

*Based on change scores on the GDR.

retrospective assessment of global change appears to be greatly

affected by perceived health status on the day the instrument is
completed. Ross14 asserts that memory may be selective, and
Retrospective global change

events consistent with our current state may be recalled pref-

Prospective global change*

erentially. Patients may not be able to accurately recall baseline

status, or they are unable to make judgments regarding differ-
Change Criterion

ences in disability or health between 2 points in time. Yet

Patient satisfaction

retrospective GRC have frequently been used as the criterion

standard for validating change in health-assessment question-
naires. Retrospective ratings of change appear to be inadequate
for this purpose.
Previous research has highlighted the difficulty of retrospec-
tive recall of health status. Mancuso and Charlson42 have
shown that patients do not accurately recall baseline state, and
†

Arch Phys Med Rehabil Vol 86, December 2005

2274 VALIDITY OF CLINICAL CHANGE MEASURES, Schmitt
that the direction of error (ie, under- or overestimation) is often not
predictable. Overall, however, patients tended to overestimate
treatment effects following hip replacement. Aseltine et al41 con-
firmed this tendency among patients with prostate and gyne-
cologic conditions, especially following surgical interventions.
Ross14 also reviewed several studies that revealed a tendency
for subjects to recall greater change than actually occurred.
Various investigators have reported that retrospective GRC
were more responsive than a number of functional outcome
measures,43-45 but this may simply reflect overestimation of
change due to recall or experimenter bias.
Herrmann46 has summarized arguments against relying on
the patient’s recall ability with retrospective global measures,
including the lack of specificity of the question, the common
occurrence of “guessing” when recall is deficient, and recall
bias due to intervening treatment. Inability to recall baseline
status calls into question the validity of retrospective judgments
of change.
Other researchers have reported different results from
prospective versus retrospective global change instru-
ments.26,41,44,47,48 Different estimations of change yield differ-
ent samples for calculation of responsiveness indices. It would
be useful to identify a preferred criterion measure of global
change. As an alternative to GRC, previous investigators have
used patient satisfaction,7,49-52 retrospective rating of change in
overall pain or function,6 and estimates of the degree of treat-
ment effectiveness43 or recovery from surgery.53 In each case,
recall to an earlier state is required to make retrospective
judgments about change over time.
In contrast to retrospective ratings, change scores from a
prospectively administered global function item correlated
equally and in opposite direction to baseline and follow-up
functional status measured by the DASH, SPADI, and PRWE.
That is, both baseline and follow-up status contributed equally
to the prospective change criterion, as would be expected for a
true measure of change. Clearly, prospectively measured global
change would be less prone to bias by recall or therapist
influence. The single-item GDR used in this study was shown
to be reliable and showed validity concurrent with clinician
global ratings. However, both prospective and retrospective
global change measures demonstrated similar modest correla-
tions (␳ range, .59 –.67) with change scores on these same
upper-extremity questionnaires. Individually, the coefficient of
determination calculated from correlations of either of the
change criterions with the DASH, SPADI, and PRWE ranged
from 9% to 45%. If the global change instrument is chosen as
the criterion standard, then the DASH and other accepted
functional questionnaires leave a large proportion of variance
unexplained.
Overall, these results highlight the inherent difficulty of
using a single-item global rating as a criterion for change. As
Norman et al comment, “Indeed, if the single global rating
could be shown to have superior measurement characteristics,
there is no reason to not simply use this as a measure of
health-related quality of life.”13(p871-2) The assumption in using
the global change rating as a criterion is that the patient’s
judgment will account for the varying symptom levels and
disabilities for varying activities over an extended period of
time and create an accurate “overall” rating of health and
disability. By contrast, multiple-item health status question-
naires are designed to assess functional ability over a range of
daily tasks and activities. Errors in the estimation of difficulty
with any given activity would not unduly influence the overall
score. Because transitional questions are avoided, error caused
by inaccurate recall would be minimized.
Arch Phys Med Rehabil Vol 86, December 2005
There is agreement in the literature that there is no criterion
standard for measuring function or functional change. Given
the challenges inherent in determining a valid global criterion
for change, it seems preferable to avoid the use of one alto-
gether. A priori prognostic ratings have been used to identify
the subgroup of patients most likely to change in response to
intervention.54,55 Although a priori ratings may not produce the
bias inherent in retrospective measures, they are still likely to
result in a high degree of misclassification. Patient satisfaction
has been used as a criterion for change, but this approach has
drawbacks similar to the use of retrospective global ratings.
Satisfaction levels reflect many aspects of care in addition to
functional change. In this study, satisfaction also reflected the
current state of the patient rather than true change over time.
Previous research as shown a high correlation between satis-
faction and retrospective global change ratings, but a low
correlation between satisfaction and a serial (prospectively
administered) assessment of global change, suggesting that
satisfaction is also biased toward posttreatment status rather
than true change over time.44 There is a need for an acceptable
alternative to the use of global change criterion measures.
We have recently described an approach to determine
change status based on pre-post treatment scores on the stan-
dardized multi-item functional outcome measures used in this
study.22 Each subject’s change score is compared with an
established threshold to determine whether true or meaningful
change has occurred. The threshold for change may be defined
according to statistically important (minimum detectable
change) or clinically important change (minimal clinically im-
portant difference). These approaches do not require the use of
a global change criterion and may serve as a valid alternative
for determining true change status.
CONCLUSIONS
Criteria for global change are used frequently in research and
clinical practice. Retrospective global change and satisfaction
questions appear to reflect follow-up status rather than true
change over time. Change scores from a prospectively mea-
sured global functional status question reflected both baseline
and follow-up status fairly equally, but still accounted for a
small to moderate amount of the variance in commonly ac-
cepted functional outcome measures for the upper extremity.
This study suggests that global change measures are not valid
indicators of true change. We recommend the development of
alternatives to global change instruments in studies of respon-
siveness.
Acknowledgments: We thank the physical and occupational ther-
apists at Park Nicollet Clinic and the Institute for Athletic Medicine for
their support in recruiting patients.
References
1. Beurskens AJ, de Vet HC, Koke AJ, van der Heijden GJ,
Knipschild PG. Measuring the functional status of patients with
low back pain. Assessment of the quality of four disease-specific
questionnaires. Spine 1995;20:1017-28.
2. Guyatt G, Walter S, Norman G. Measuring change over time:
assessing the usefulness of evaluative instruments. J Chronic Dis
1987;40:171-8.
3. Deyo RA, Diehr P, Patrick DL. Reproducibility and responsive-
ness of health status measures. Statistics and strategies for eval-
uation. Control Clin Trials 1991;12:142S-58S.
4. Kirshner B, Guyatt G. A methodological framework for assessing
health indices. J Chronic Dis 1985;38:27-36.
5. Stratford PW, Binkley FM, Riddle DL. Health status measures:
strategies and analytic methods for assessing change scores. Phys
Ther 1996;76:1109-23.
 VALIDITY OF CLINICAL CHANGE MEASURES, Schmitt
6. Wright JG, Young NL. A comparison of different indices of
responsiveness. J Clin Epidemiol 1997;50:239-46.
7. Stucki G, Liang MH, Fossel AH, Katz JN. Relative responsive-
ness of condition-specific and generic health status measures in
degenerative lumbar spinal stenosis. J Clin Epidemiol 1995;48:
1369-78.
8. Middel B, Stewart R, Bouma J, van Sonderen E, van den Heuvel
WJ. How to validate clinically important change in health-related
functional status. Is the magnitude of the effect size consistently
related to magnitude of change as indicated by a global question
rating? J Eval Clin Pract 2001;7:399-410.
9. Taylor SJ, Taylor AE, Foy MA, Fogg AJ. Responsiveness of
common outcome measures for patients with low back pain. Spine
1999;24:1805-12.
10. Santanello NC, Zhang J, Seidenberg B, Reiss TF, Barber BL.
What are minimal important changes for asthma measures in a
clinical trial? Eur Respir J 1999;14:23-7.
11. Stratford PW, Binkley JM, Riddle DL, Guyatt GH. Sensitivity to
change of the Roland-Morris Back Pain Questionnaire: part 1.
Phys Ther 1998;78:1186-96.
12. Jaeschke R, Singer J, Guyatt GH. Measurement of health status.
Ascertaining the minimal clinically important difference. Control
Clin Trials 1989;10:407-15.
13. Norman GR, Stratford P, Regehr G. Methodological problems in
the retrospective computation of responsiveness to change: the
lesson of Cronbach. J Clin Epidemiol 1997;50:869-79.
14. Ross M. Relation of implicit theories to the construction of per-
sonal histories. Psychol Rev 1989;96:341-57.
15. van der Lee JH, Beckerman H, Knol DL, de Vet HC, Bouter LM.
Clinimetric properties of the motor activity log for the assessment
of arm use in hemiparetic patients. Stroke 2004;35:1410-4.
16. Williams NH, Wilkinson C, Russell IT. Extending the Aberdeen
Back Pain Scale to include the whole spine: a set of outcome
measures for the neck, upper and lower back. Pain 2001;94:
261-74.
17. Dunkl PR, Taylor AG, McConnell GG, Alfano AP, Conaway MR.
Responsiveness of fibromyalgia clinical trial outcome measures.
J Rheumatol 2000;27:2683-91.
18. Chang E, Abrahamowicz M, Ferland D, Fortin PR. Comparison of
the responsiveness of lupus disease activity measures to changes
in systemic lupus erythematosus activity relevant to patients and
physicians. J Clin Epidemiol 2002;55:488-97.
19. Locker D, Jokovic A, Clarke M. Assessing the responsiveness of
measures of oral health-related quality of life. Community Dent
Oral Epidemiol 2004;32:10-8.
20. Hoving JL, Koes BW, de Vet HC, et al. Manual therapy, physical
therapy, or continued care by a general practitioner for patients
with neck pain. A randomized, controlled trial. Ann Intern Med
2002;136:713-22.
21. Smidt N, van der Windt DA, Assendelft WJ, Deville WL,
Korthals-de Bos IB, Bouter LM. Corticosteroid injections, phys-
iotherapy, or a wait-and-see policy for lateral epicondylitis: a
randomised controlled trial. Lancet 2002;359:657-62.
22. Schmitt JS, Di Fabio RP. Reliable change and minimum important
difference (MID) proportions facilitated group responsiveness
comparisons using individual threshold criteria. J Clin Epidemiol
2004;57:1008-18.
23. Guyatt GH, Norman GR, Juniper EF, Griffith LE. A critical look
at transition ratings. J Clin Epidemiol 2002;55:900-8.
24. Hudak PL, Amadio PC, Bombardier C. Development of an upper
extremity outcome measure: the DASH (disabilities of the arm,
shoulder and hand) [corrected]. The Upper Extremity Collabora-
tive Group (UECG). Am J Ind Med 1996;29:602-8.
25. Marx RG, Bombardier C, Hogg-Johnson S, Wright JG. Clinimet-
ric and psychometric strategies for development of a health mea-
surement scale. J Clin Epidemiol 1999;52:105-11.
2275
26. Beaton DE, Katz JN, Fossel AH, Wright JG, Tarasuk V,
Bombardier C. Measuring the whole or the parts? Validity, reli-
ability, and responsiveness of the Disabilities of the Arm, Shoul-
der and Hand outcome measure in different regions of the upper
extremity. J Hand Ther 2001;14:128-46.
27. MacDermid JC. Outcome evaluation in patients with elbow pa-
thology: issues in instrument development and evaluation. J Hand
Ther 2001;14:105-14.
28. Turchin DC, Beaton DE, Richards RR. Validity of observer-based
aggregate scoring systems as descriptors of elbow pain, function,
and disability. J Bone Joint Surg Am 1998;80:154-62.
29. Kirkley A, Griffin S, McLintock H, Ng L. The development and
evaluation of a disease-specific quality of life measurement tool
for shoulder instability. The Western Ontario Shoulder Instability
Index (WOSI). Am J Sports Med 1998;26:764-72.
30. MacDermid JC, Richards RS, Donner A, Bellamy N, Roth JH.
Responsiveness of the short form-36, disability of the arm, shoul-
der, and hand questionnaire, patient-rated wrist evaluation, and
physical impairment measurements in evaluating recovery after a
distal radius fracture. J Hand Surg [Am] 2000;25:330-40.
31. Roach KE, Budiman-Mak E, Songsiridej N, Lertratanakul Y.
Development of a shoulder pain and disability index. Arthritis
Care Res 1991;4:143-9.
32. Williams JW Jr, Holleman DR Jr, Simel DL. Measuring shoulder
function with the Shoulder Pain and Disability Index. J Rheumatol
1995;22:727-32.
33. Beaton D, Richards RR. Assessing the reliability and responsive-
ness of 5 shoulder questionnaires. J Shoulder Elbow Surg 1998;
7:565-72.
34. Beaton DE, Richards RR. Measuring function of the shoulder. A
cross-sectional comparison of five questionnaires. J Bone Joint
Surg Am 1996;78:882-90.
35. Heald SL, Riddle DL, Lamb RL. The shoulder pain and disability
index: the construct validity and responsiveness of a region-
specific disability measure. Phys Ther 1997;77:1079-89.
36. MacDermid JC, Turgeon T, Richards RS, Beadle M, Roth JH.
Patient rating of wrist pain and disability: a reliable and valid
measurement tool. J Orthop Trauma 1998;12:577-86.
37. Jenkinson C, Layte R, Jenkinson D, et al. A shorter form health
survey: can the SF-12 replicate results from the SF-36 in longi-
tudinal studies? J Public Health Med 1997;19:179-86.
38. Ware J Jr, Kosinski M, Keller SD. A 12-Item Short-Form Health
Survey: construction of scales and preliminary tests of reliability
and validity. Med Care 1996;34:220-33.
39. Luus HG, Muller FO, Meyer BH. Statistical significance versus
clinical relevance. Part III. Methods for calculating confidence
intervals. S Afr Med J 1989;76:681-5.
40. van Stel HF, Maille AR, Colland VT, Everaerd W. Interpretation
of change and longitudinal validity of the quality of life for
respiratory illness questionnaire (QoLRIQ) in inpatient pulmonary
rehabilitation. Qual Life Res 2003;12:133-45.
41. Aseltine RH Jr, Carlson KJ, Fowler FJ Jr, Barry MJ. Comparing
prospective and retrospective measures of treatment outcomes.
Med Care 1995;33:AS67-76.
42. Mancuso CA, Charlson ME. Does recollection error threaten the
validity of cross-sectional studies of effectiveness? Med Care
1995;33:AS77-88.
43. Gotzsche PC. Sensitivity of effect variables in rheumatoid arthri-
tis: a meta-analysis of 130 placebo controlled NSAID trials. J Clin
Epidemiol 1990;43:1313-8.
44. Fischer D, Stewart AL, Bloch DA, Lorig K, Laurent D, Holman
H. Capturing the patient’s view of change as a clinical outcome
measure. JAMA 1999;282:1157-62.
45. Ziebland S, Fitzpatrick R, Jenkinson C, Mowat A. Comparison of
two approaches to measuring change in health status in rheuma-
Arch Phys Med Rehabil Vol 86, December 2005
2276 VALIDITY OF CLINICAL CHANGE MEASURES, Schmitt
toid arthritis: the Health Assessment Questionnaire (HAQ) and
modified HAQ. Ann Rheum Dis 1992;51:1202-5.
46. Herrmann D. Reporting current, past, and changed health status.
What we know about distortion. Med Care 1995;33:AS89-94.
47. Barber BL, Santanello NC, Epstein RS. Impact of the global on
patient perceivable change in an asthma specific QOL question-
naire. Qual Life Res 1996;5:117-22.
48. Timmerman AA, Anteunis LJ, Meesters CM. Response-shift bias
and parent-reported quality of life in children with otitis media.
Arch Otolaryngol Head Neck Surg 2003;129:987-91.
49. L’Insalata JC, Warren RF, Cohen SB, Altchek DW, Peterson MG. A
self-administered questionnaire for assessment of symptoms and
function of the shoulder. J Bone Joint Surg Am 1997;79:738-48.
50. BenDebba M, Heller J, Ducker TB, Eisinger JM. Cervical spine
outcomes questionnaire: its development and psychometric prop-
erties. Spine 2002;27:2116-24.
51. Katz JN, Gelberman RH, Wright EA, Lew RA, Liang MH. Re-
sponsiveness of self-reported and objective measures of disease
severity in carpal tunnel syndrome. Med Care 1994;32:1127-33.
Arch Phys Med Rehabil Vol 86, December 2005
52. Bessette L, Sangha O, Kuntz KM, et al. Comparative responsive-
ness of generic versus disease-specific and weighted versus un-
weighted health status measures in carpal tunnel syndrome. Med
Care 1998;36:491-502.
53. van der Windt DA, van der Heijden GJ, de Winter AF, Koes BW,
Deville W, Bouter LM. The responsiveness of the Shoulder Dis-
ability Questionnaire. Ann Rheum Dis 1998;57:82-7.
54. Binkley JM, Stratford PW, Lott SA, Riddle DL. The Lower Extrem-
ity Functional Scale (LEFS): scale development, measurement prop-
erties, and clinical application. North American Orthopaedic Reha-
bilitation Research Network. Phys Ther 1999;79:371-83.
55. Stratford PW, Binkley JM, Riddle DL. Development and initial
validation of the back pain functional scale. Spine 2000;25:2095-
102.
Supplier
a. Number Cruncher Statistical Systems, 329 N 1000 E, Kaysville, UT
84037.