Article

Annals of Internal Medicine

The Effect of Including C-Reactive Protein in Cardiovascular Risk

Prediction Models for Women
Nancy R. Cook, ScD; Julie E. Buring, ScD; and Paul M. Ridker, MD

Background: While high-sensitivity C-reactive protein (hsCRP) is an

independent predictor of cardiovascular risk, global risk prediction
models incorporating hsCRP have not been developed for clinical
use.
Objective: To develop and compare global cardiovascular risk prediction models with and without hsCRP.
Design: Observational cohort study.
Setting: U.S. female health professionals.
Participants: Initially healthy nondiabetic women age 45 years and
older participating in the Womens Health Study and followed an
average of 10 years.
Measurements: Incident cardiovascular events (myocardial infarction, stroke, coronary revascularization, and cardiovascular death).
Results: High-sensitivity CRP made a relative contribution to global
risk at least as large as that provided by total, high-density lipoprotein (HDL), and low-density lipoprotein (LDL) cholesterol individually, but less than that provided by age, smoking, and blood pressure. All global measures of fit improved when hsCRP was
included, with likelihood-based measures demonstrating strong

he Framingham risk model (1) is used extensively for

detecting risk for coronary heart disease and has been
adapted by the Adult Treatment Panel III (ATP III) of the
National Cholesterol Education Program (2). The traditional risk factors included are strong predictors of cardiovascular risk, and the model has been validated in several
populations (3). However, despite the models success, up
to 20% of all coronary events occur in the absence of any
major risk factor (4, 5). In addition, most individuals who
do not develop coronary heart disease have at least 1 clinically elevated Framingham risk factor (6).
Given these modest levels of sensitivity and specificity,
research over the past decade has focused on novel bloodbased atherosclerotic risk factors that, like cholesterol, can
be inexpensively obtained and interpreted in the primary
care setting. One of the most promising of these is highsensitivity C-reactive protein (hsCRP), a biomarker of inflammation that has consistently been shown to predict
incident myocardial infarction, stroke, and cardiovascular
death among apparently healthy men and women after
adjustment for all components of the Framingham risk
score (716). Blood levels of hsCRP also correlate with
hypofibrinolysis and abnormal glucose metabolism, and
thus reflect pathophysiologic processes that are related to
vascular occlusion but are not easily measured with traditional risk factors (1720). On that basis, in 2003 the
Centers for Disease Control and Prevention and the Amer-

preference for models that include hsCRP. With use of 10-year risk
categories of 0% to less than 5%, 5% to less than 10%, 10% to
less than 20%, and 20% or greater, risk prediction was more
accurate in models that included hsCRP, particularly for risk between 5% and 20%. Among women initially classified with risks of
5% to less than 10% and 10% to less than 20% according to the
Adult Treatment Panel III covariables, 21% and 19%, respectively,
were reclassified into more accurate risk categories. Although addition of hsCRP had minimal effect on the c-statistic (a measure of
model discrimination) once age, smoking, and blood pressure were
accounted for, the effect was nonetheless greater than that of total,
LDL, or HDL cholesterol, suggesting that the c-statistic may be
insensitive in evaluating risk prediction models.
Limitations: Data were available only for women.
Conclusions: A global risk prediction model that includes hsCRP
improves cardiovascular risk classification in women, particularly
among those with a 10-year risk of 5% to 20%. In models that
include age, blood pressure, and smoking status, hsCRP improves
prediction at least as much as do lipid measures.
www.annals.org

Ann Intern Med. 2006;145:21-29.

For author affiliations, see end of text.

ican Heart Association published the first set of guidelines

to endorse the use of hsCRP as a potential adjunct to
traditional risk factor screening (21).
Despite these data, no simple clinical algorithm that
includes Framingham covariables and hsCRP has been developed, and thus it has not been possible to determine
whether individuals might be more accurately classified if
hsCRP were added to global risk prediction models for
major cardiovascular disease (CVD), including myocardial
infarction, coronary revascularization, stroke, and cardiovascular death (22).

See also:
Print
Editors Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Editorial comment. . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Related article. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Summary for Patients. . . . . . . . . . . . . . . . . . . . . . . I-19
Web-Only
Appendix
Appendix Table
Conversion of figures and tables into slides
2006 American College of Physicians 21

Article

Risk Prediction Models with and without hsCRP

Development of Risk Prediction Models

Context
The value of adding high-sensitivity C-reactive protein
(hsCRP) to a global risk assessment model is unknown.

Contribution
The authors used the Womens Health Study, a nationwide cohort of 15 048 initially healthy women, to develop
a cardiovascular disease (CVD) risk prediction model using
hsCRP and Framingham risk model predictors. While
hsCRP improved overall model fit, the clinical utility of
hsCRP in terms of reclassification was most substantial for
those with a 5% or greater 10-year risk based on traditional risk factors.

Cautions
The study does not address the clinical value of lowering
hsCRP level.

Implications
In this largely low-risk population, adding hsCRP to the
Framingham model reclassified patients into groups that
better reflected their actual CVD risk. This effect was most
clinically relevant for those at intermediate risk.
The Editors

METHODS
We compared the clinical utility of global cardiovascular risk prediction models based on Framingham covariables with and without hsCRP among participants in the
Womens Health Study (WHS) (2325), a large-scale, nationwide cohort of U.S. women age 45 years and older
who were free of CVD and cancer at study entry. Women
were followed annually for the development of CVD, with
an average follow-up of 10 years. All reported CVD outcomes, including myocardial infarction, ischemic stroke,
coronary revascularization procedures, and deaths from
cardiovascular causes, were adjudicated by an end points
committee after medical record review. All study participants provided written informed consent, and the study
protocol was approved by the institutional review board of
Brigham and Womens Hospital in Boston, Massachusetts.
Baseline blood samples were assayed for C-reactive
protein with a validated, high-sensitivity assay (Denka
Seiken, Tokyo, Japan) and for total, high-density lipoprotein (HDL), and low-density lipoprotein (LDL) cholesterol
with direct-measurement assays (Roche Diagnostics, Basel,
Switzerland). Women who were diabetic at baseline were
excluded from predictive modeling because ATP III labeling considers diabetes to be a risk equivalent for coronary
heart disease (2). In parallel with guidelines established for
lipid evaluation (26), models were initially fitted in a derivation cohort limited to women not taking hormone replacement therapy at baseline (n ! 15 048 with data on all
variables) and were then applied to all nondiabetic women
(n ! 26 927) for clinical risk prediction.
22 4 July 2006 Annals of Internal Medicine Volume 145 Number 1

Models were fitted by using Cox proportional hazards

models (27), restricting predictors to components of the
Framingham risk score (including either total or LDL cholesterol as well as HDL cholesterol), and adding hsCRP.
To determine the functional form used for each predictor,
we examined spline plots and fitted power functions to
determine the best fit for each variable. When fit was similar, we chose the simplest form, usually a linear term or
log transformation, especially when supported by previous
data from existing ATP III algorithms.
To model blood pressure more fully, we included a
nonlinear term for systolic blood pressure. As in the Framingham score, we included antihypertensive medication
use and considered its interaction with blood pressure. At
baseline in the WHS, use of cholesterol-lowering medications was rare, was not composed of statins, and was not
statistically significant in these models. As also in the Framingham score, current but not past smoking was included. We considered interactions of all predictors with
smoking and age, particularly on the basis of inclusion in
the ATP III risk score. To enhance model simplicity, these
were not included when they were only of marginal statistical significance. Of interest, when body mass index was
added to the final model, it was not a statistically significant predictor of CVD in these data, an observation consistent with its absence from the Framingham risk score.
In addition to comparing the performance of the final
model derived from the WHS data with hsCRP to the
WHS model without hsCRP, we also compared the performance of the formal ATP III model (2), with and without hsCRP (Appendix, available at www.annals.org). The
ATP III model includes terms for the natural logarithms of
age, of total and HDL cholesterol, and of systolic blood
pressure and was developed for the prediction of hard
coronary heart disease events, including myocardial infarction and death from coronary heart disease. Total CVD,
including ischemic stroke as well as revascularization procedures, was used as the end point in these analyses. To be
conservative and allow the best possible fit for all traditional covariables, we recalculated !-coefficients for the
ATP III model in the WHS data before evaluating any
additive effects of hsCRP.
Measures of Model Fit

The primary means of comparing predictive models

based on Framingham covariables with and without hsCRP
was the Bayes information criterion, a likelihood-based
measure that adds a penalty for model complexity (28, 29).
Lower values indicate better fit. Because of its common use
in the medical literature, we also computed the c-index
(28), or concordance probability, which is a generalization
of the c-statistic, or the area under the receiver-operating
characteristic curve (30), that allows for censored data. For
these analyses, the c-index was computed and adjusted for
optimism due to overfitting with bootstrap sampling (31)
www.annals.org

Risk Prediction Models with and without hsCRP

Article

Table 1. Best-Fitting Global Cardiovascular Risk Prediction Model among the Model Derivation Cohort of 15 048 Women from
the Womens Health Study*
Variable

WHS Model

!-Coefficient
Age
SBP # 125
(SBP # 125)2
Antihypertensive use
Current smoking
Ln(HDL)
Ln(Total cholesterol)
Ln(hsCRP)
Measures of fit
BIC
C-index
Calibration P value (risk percentage)

SE

0.074
0.032
#0.0003
0.264
0.965
#1.244
1.569
0.196

0.006
0.006
0.0002
0.123
0.117
0.205
0.269
0.050

With hsCRP
6960.26
0.815
0.23

Without hsCRP
6969.60
0.813
0.039

Hazard Ratio

P Value

1.08
1.03
1.00
1.30
2.62
0.29
4.80
1.22

"0.001
"0.001
0.058
0.032
"0.001
"0.001
"0.001
"0.001

* BIC ! Bayes information criterion; HDL ! high-density lipoprotein; hsCRP ! high-sensitivity C-reactive protein; Ln ! natural logarithm; SBP ! systolic blood pressure;
WHS ! Womens Health Study.
SBP is measured in mm Hg, HDL and total cholesterol are measured in mg/dL, and hsCRP is measured in mg/L.
Lower values of BIC and higher values of the c-index and calibration P value reflect better fit.

using the Design library in S-PLUS software (Insightful

Corp., Seattle, Washington) (28).
For comparison, we also computed several other measures of global model fit (provided in the Appendix, available at www.annals.org), including other likelihood-based
measures such as model weights for the Bayes information
criterion, which provide an estimate of the posterior probability of each model given the set of candidate models
considered (29, 32); the Akaike information criterion and
its corresponding model weights (32); and Nagelkerkes
generalized model R2 (33, 34). We computed the D-statistic of Royston and Sauerbrei (35), based on the separation
of survival curves by predictor variables, and again adjusted
for optimism. Differences in statistics between nested models were tested with a 1-sided test using bootstrap sampling
(31). We also calculated the Brier score (28), which directly compares the observed outcomes with the fitted
probabilities.
To assess model calibration, or how closely the predicted probabilities reflect actual risk, observed risk was
calculated on the basis of 8 years of follow-up (available for
all participants) and was extrapolated to 10 years for display purposes. We computed the HosmerLemeshow calibration statistic (36) comparing observed and predicted
risk using 10 categories based on 2percentage point increments in predicted risk, ranging from less than 2% to 18%
or greater. We also computed this statistic using decile
categories of predicted probabilities.
To address clinical utility, we directly compared predicted risk estimates that are based on models using Framingham covariables with and without hsCRP with actual
risk that was observed during study follow-up among all
26 927 women for whom data were available. We used
weighted " statistics (37) to compare the predicted probabilities with and without hsCRP. To approximate clinical
www.annals.org

criteria commonly used in current treatment guidelines (2,

38), we grouped the predicted probabilities into 10-year
risk categories of 0% to less than 5%, 5% to less than 10%,
10% to less than 20%, and 20% or greater.
Finally, to address the generalizability of the final
WHS risk prediction model with hsCRP, we calibrated the
predicted probabilities to observed risk in the Framingham
Heart Study, using a limited-access data set available from
the National Heart, Lung, and Blood Institute at www
.nhlbi.nih.gov/resources/deca/default.htm. Mean values
Figure 1. Relative risk (RR) of future cardiovascular events
according to baseline high-sensitivity C-reactive protein
(hsCRP) levels in the model derivation cohort (n " 15 048),
adjusted for Framingham covariables.

Risk estimates are provided on a natural log scale and were derived from
a Cox regression model using a flexible spline curve. Dotted lines represent 95% CIs.
4 July 2006 Annals of Internal Medicine Volume 145 Number 1 23

Article

Risk Prediction Models with and without hsCRP

Figure 2. Cardiovascular point scoring system for women based on Framingham covariables and high-sensitivity C-reactive protein
(hsCRP).

This scoring system is intended as an illustration only. CVD ! cardiovascular disease; HDL ! high-density lipoprotein; RR ! relative risk; SBP !
systolic blood pressure. To convert cholesterol values to mmol/L, multiply by 0.02586.

among women age 47 years and older at the 10th Framingham examination were evaluated. Mean HDL cholesterol
level at the 10th examination was estimated on the basis of
age and HDL cholesterol level at the 15th examination.
The population mean hsCRP value was estimated from a
24 4 July 2006 Annals of Internal Medicine Volume 145 Number 1

model derived from the WHS, including the other risk

factors plus body mass index. The average 10-year risk for
major CVD in the Framingham data, including myocardial infarction, stroke, and cardiovascular death, was estimated by using a product-limit estimator. The projected
www.annals.org

Risk Prediction Models with and without hsCRP

10-year risk from the WHS models was calibrated to the

10-year rate of cardiovascular outcomes among women in the
Framingham data (Appendix, available at www.annals.org).

Article

Table 2. Relative Contribution of Individual Framingham

Covariables and High-Sensitivity C-Reactive Protein to
Global Cardiovascular Risk*

Role of the Funding Sources

This work was supported by grants from the Donald

W. Reynolds Foundation, the Leducq Foundation, the
Doris Duke Charitable Foundation, and the National Institutes of Health. The funding agencies had no role in the
design, conduct, or reporting of the study or in the decision to submit the manuscript for publication.

RESULTS
Among the 15 048 women used for model development, the mean age was 54 years (SD, 8); 1841 women
(12%) were current smokers. A total of 2227 women
(15%) had blood pressure of 140/90 mm Hg or higher,
and 1802 women (12%) were taking antihypertensive
medication at baseline. Median lipid values were as follows:
total cholesterol level, 5.3 mmol/L (interquartile range, 4.7
to 6.1 mmol/L) [206 mg/dL (interquartile range, 181 to
234 mg/dL)]; LDL cholesterol level, 3.2 mmol/L (interquartile range, 2.6 to 3.8 mmol/L) [124 mg/dL (interquartile range, 102 to 147 mg/dL)]; and HDL cholesterol level,
1.3 mmol/L (interquartile range, 1.1 to 1.5 mmol/L) [49
mg/dL (interquartile range, 42 to 59 mg/dL)]. The median
hsCRP level was 1.5 mg/L (interquartile range, 0.6 to 3.4
mg/L). Over the mean 10-year follow-up, 390 women developed CVD, including 116 myocardial infarctions, 100
ischemic strokes, 217 coronary revascularization procedures, and 65 deaths due to cardiovascular cause. Some
women experienced more than 1 of these events.
The best-fitting global cardiovascular risk prediction
model in these data included all Framingham covariables
and hsCRP (Table 1). Figure 1 shows that after adjustment for all Framingham variables, there was a log-linear
relationship between hsCRP and risk for CVD. While risk
estimates based on the model coefficients given in Table 1
provide the best estimates, for clinical utility we also calculated a point scoring system (39) for cardiovascular risk
analogous to that used by the National Cholesterol Education Program (Figure 2). The absolute risk estimates in
Figure 2 are calibrated to the overall incidence in the Framingham Heart Study sample to enhance generalizability.
Relative Contributions to Global Risk of Age, Blood
Pressure, Smoking, Lipids, and hsCRP

Table 2 presents a comparison of the relative contributions to global risk made by each individual Framingham covariate and by hsCRP. Age is the strongest predictor of risk in these data, leading to a high likelihood ratio
and a c-index of 0.73. After adjustment for age, likelihood
ratio statistics demonstrated the strongest improvement in
fit for systolic blood pressure, followed in descending order
by hsCRP; current smoking; and HDL, total, and LDL
cholesterol. In models that included age, systolic blood
www.annals.org

Variable

Variable LR
Chi-Square

Overall Model Fit

Model LR
Chi-Square

C-Index

Age only
$ SBP
$ Ln(hsCRP)
$ Current smoking
$ Ln(HDL cholesterol)
$ Ln(total cholesterol)
$ LDL cholesterol

100.60
86.72
75.04
70.19
36.72
31.13

267.05
367.66
353.77
342.10
337.24
303.78
298.18

0.731
0.768
0.763
0.757
0.765
0.747
0.746

Age, SBP, and smoking

$ Ln(hsCRP)
$ Ln(HDL cholesterol)
$ Ln(total cholesterol)
$ LDL cholesterol

44.05
41.89
26.28
22.94

444.59
488.64
486.48
470.87
467.53

0.791
0.800
0.801
0.796
0.796

* Estimated from Cox proportional hazards models. Variable chi-square is the

1-degree-of-freedom likelihood ratio chi-square statistic for inclusion of each variable separately, with larger values indicating greater improvement in fit. Model
chi-square is the multi degree-of-freedom test for the fit of the entire model. The
$ sign indicates the addition of each variable separately to the model with age only
or with age, SBP, and smoking only. HDL ! high-density lipoprotein; hsCRP !
high-sensitivity C-reactive protein; LDL ! low-density lipoprotein; Ln ! natural
logarithm; LR ! likelihood ratio; SBP ! systolic blood pressure.

pressure, and current smoking (the 3 strongest predictors

in aggregate), likelihood ratio statistics similarly demonstrated the strongest fit for hsCRP, followed in order by
HDL, total, and LDL cholesterol.
Table 2 also demonstrates the relative insensitivity of
the c-index as a method of determining model fit. For
example, while the likelihood ratio statistic was able to
rank the relative contributions of systolic blood pressure,
hsCRP, smoking, and HDL cholesterol in the age-adjusted
analyses, the c-index had little ability to distinguish among
any of these (all values between 0.75 and 0.77). Similarly,
in the analyses adjusted for age, systolic blood pressure, and
smoking, the likelihood ratio statistic indicated that either
hsCRP or HDL cholesterol improved fit more than did
total or LDL cholesterol, yet the c-index was again unable
to distinguish between these measures (all values 0.80).
Discrimination and Calibration in Models with and
without hsCRP

For both the ATP III model and the final WHS
model, the likelihood ratio test for the inclusion of hsCRP
was highly significant (P " 0.001). Of note, the Bayes
information criterion indicated a strong preference for the
inclusion of hsCRP (Table 1) after adjustment for adding
a variable. This suggests that the model including hsCRP
provided better fit, even after adjustment for the increase in
number of predictors. Similarly, models that included
hsCRP demonstrated better calibration (higher P value for
calibration), while models without hsCRP had larger deviations between the observed and predicted probabilities in
the higher-risk categories (Figure 3). By contrast, the c4 July 2006 Annals of Internal Medicine Volume 145 Number 1 25

Article

Risk Prediction Models with and without hsCRP

Figure 3. Calibration curves for risk prediction models without (top) and with (bottom) high-sensitivity C-reactive protein (hsCRP)
in the model.

The model that includes hsCRP shows closer agreement between observed and model-based predicted risk. WHS ! Womens Health Study.

index again showed minimal ability to detect differences in

model fit. As shown in the Appendix Table (available at
www.annals.org), all global measures of fit showed improvement when hsCRP was added to prediction models
based on Framingham covariables alone.
Clinical Risk Classification and Accuracy

To better compare model performance within clinical

categories, we classified all nondiabetic women (n !
26 927) into 4 risk groups defined by the ATP III categories of 10-year risk for CVD of 0% to less than 5%, 5% to
less than 10%, 10% to less than 20%, and 20% or greater.
We then compared the WHS models with and without
hsCRP by cross-classifying expected risks and comparing
these to the observed proportions of events in each group.
While there was general agreement between these classifications (weighted " ! 0.86), the predicted risk categories
changed substantially with the addition of hsCRP for
women with at least a 5% 10-year risk according to only
26 4 July 2006 Annals of Internal Medicine Volume 145 Number 1

the Framingham risk variables (Table 3). Specifically, more

than 20% of all participants with intermediate risk were
reclassified with the addition of hsCRP; among those originally classified as having 5% to less than 10% risk, 12%
moved down a category in risk and 10% moved up.
Among those originally classified as having 10% to less
than 20% risk, 19% were reclassified: 14% to a lower and
5% to a higher category. Among those at high risk (#20%
risk), 14% were reclassified into a lower-risk category. By
contrast, among those with less than 5% risk according to
Framingham covariables, only 2% were reclassified. Thus,
overall in this low-risk cohort, with 88% of women in the
lowest risk group, 4% were reclassified. However, this
overall percentage depends heavily on the underlying risk
and would be substantially greater in an older or higherrisk population. For comparison, according to ATP III risk
categories based on National Cholesterol Education Program !-coefficients rather than those derived from the
www.annals.org

Risk Prediction Models with and without hsCRP

WHS, among women originally classified as having less

than 5%, 5% to less than 10%, 10% to less than 20%, and
20% or greater 10-year risk, 4%, 38%, 42%, and 20%,
respectively, were reclassified in the WHS model that included hsCRP.
Of note, for women reclassified, the models that included hsCRP also estimated the actual risk more accurately. For example, among those classified in the 5% to
less than 10% category according to the model without
hsCRP, the 12% reclassified to the lower-risk category actually experienced only a 2% risk, and the 10% reclassified
to a higher-risk category actually experienced a higher 15%
risk (Table 3). The addition of hsCRP also led to better
calibration for women initially classified into the other 3
groups.

Article

hsCRP, and the Bayes information criterion weights, corresponding to the posterior model probabilities, strongly
supported the addition of hsCRP. As also shown in these
data, the relative contribution to global risk made by
hsCRP was at least as large as that made by total, HDL, or
LDL cholesterol. The added predictive value of including
hsCRP was most evident among those at 5% or greater
10-year risk.
While all the likelihood-based measures of fit improved when hsCRP was added, the c-index, or generalized
c-statistic, changed little with the addition of any bloodbased risk factor (including total, HDL, and LDL cholesterol) once we accounted for age, smoking, and blood pressure. In fact, the increase in the c-index associated with
hsCRP was greater than that for any of the lipids. This
observation underscores the limitations of the c-index, or
area under the receiver-operating characteristic curve, as a
method for determining model fit, despite its continued
popular use in the medical literature. The c-index is particularly suited to retrospective case control studies, in
which the actual outcome probabilities cannot be estimated (40). Since it is based exclusively on ranks, however,
it measures only how well the predicted values can rankorder the responses. It may not be as sensitive as the likelihood function in choosing between models, especially
when the models are strong (28). This may be particularly
true in settings where many individuals fall into low-risk
groups, as is true of cardiovascular risk detection in the
general population as well as in the WHS. In the current

DISCUSSION
The Framingham risk score provides a useful measure
of risk stratification for coronary heart disease and has been
valuable in clinical practice. Whether it can be improved
by including other simple and inexpensive measures has
not previously been determined. We fitted predictive models for major CVD, including myocardial infarction, coronary revascularization, stroke, and cardiovascular death, using traditional Framingham predictors, with and without
the addition of hsCRP. As shown, all global measures of
predictive accuracy were improved in prediction models
that included hsCRP. In particular, likelihood-based measures showed a strong preference for the models with

Table 3. Observed and Expected Risks among all 26 927 Nondiabetic Women in the Womens Health Study Using the Final
Global Risk Prediction Model with and without High-Sensitivity C-Reactive Protein*
10-Year Risk in WHS Model without hsCRP

10-Year Risk in WHS Model with hsCRP

0% to <5%

0% to <5%
Total participants, n
Participants classified in each risk stratum
by the WHS model with hsCRP, %
10-y risk, %
5% to <10%
Total participants, n
Participants classified in each risk stratum
by the WHS model with hsCRP, %
10-y risk, %
10% to <20%
Total participants, n
Participants classified in each risk stratum
by the WHS model with hsCRP, %
10-y risk, %
>20%
Total participants, n
Participants classified in each risk stratum
by the WHS model with hsCRP, %
10-y risk, %

5% to <10%

10% to <20%

Total Reclassified
>20%

23 174
97.9

488
2.1

1.6

5.8

267
11.9

1773
78.7

213
9.5

0
0.0

2.4

7.8

15.2

0
0.0

110
13.7

653
81.3

40
5.0

6.8

11.5

19.9

0
0.0

0
0.0

30
14.4

179
85.7

18.8

27.1

0
0.0

0
0.0

2.1

21.3

18.7

14.4

* All estimated and observed risk estimates have been extrapolated to 10-y risk. hsCRP ! high-sensitivity C-reactive protein; WHS ! Womens Health Study.
www.annals.org

4 July 2006 Annals of Internal Medicine Volume 145 Number 1 27

Article

Risk Prediction Models with and without hsCRP

data, the c-index could not distinguish between blood pressure, smoking, or any of the measured blood predictors,
even though the likelihood ratio statistics clearly ordered
variable contributions, with systolic blood pressure being
the strongest predictor after age. In fact, hsCRP was a
stronger individual predictor than any of the lipid measures in these data, including HDL cholesterol, its closest
competitor. Thus, reliance solely on the c-statistic for
model development could erroneously lead to the exclusion of lipids as well as hsCRP from risk prediction models.
Accuracy, or the predictive ability of a model, has 2
major components, discrimination and calibration (28).
The c-statistic is a measure of discrimination, or the ability
to separate 2 groups, such as case-patients and controls.
Particularly in a prospective study, calibration, or how well
the predicted probabilities reflect actual risk, is another
aspect of accuracy not captured by the c-statistic. A model
could discriminate well but lack even internal calibration if
the fitted scores do not reflect the true probability of an
event. The predicted probability given the risk factors, or
the post-test probability, can be more useful clinically in
assessing future cardiovascular risk than sensitivity or specificity, on which the c-statistic is based. Put another way, as
Moons and Harrell have stated (41), sensitivity and specificity have no direct diagnostic meaning; for the patient,
the issue is not the risk for having a positive test result, but
the risk for developing the disease. In our data, the model
that included hsCRP was better calibrated and was a better
predictor of the probability of disease.
Clinically, the inclusion of hsCRP in global prediction
models more accurately predicted true cardiovascular risk
in these data. While it did not strongly affect estimated risk
among women originally at very low risk, hsCRP had a
substantial effect among women at 5% or higher 10-year
risk. Among those classified as having 5% to less than 20%
risk, the addition of hsCRP reclassified about 20% of
women into more accurate risk strata. This would suggest
that an effective clinical strategy to improve risk prediction
might be to evaluate hsCRP among women with at least
5% predicted risk based on traditional risk factors.
One of the primary goals of any risk prediction algorithm for CVD is to identify individuals at increased risk
who will benefit from aggressive lifestyle changes, including dietary moderation, exercise, and smoking cessation, all
of which reduce hsCRP levels in addition to decreasing
cardiovascular risk. Thus, knowledge of hsCRP level and
more precise global risk estimation may help motivate patients to improve adherence to therapeutic lifestyle
changes, as currently advocated in the ATP III guidelines
(2). Ten-year risks are also commonly estimated to make
therapeutic decisions with regard to lipid-lowering therapy,
particularly with statin agents. Previous research demonstrates that statin agents decrease hsCRP levels in a manner
largely independent of LDL cholesterol, and the efficacy of
statin therapy is linked in part to underlying hsCRP levels
(42 46). As shown in our data, approximately 20% of
28 4 July 2006 Annals of Internal Medicine Volume 145 Number 1

individuals at intermediate risk had their risk estimates

substantively increased or decreased with improved accuracy when hsCRP was added to the Framingham covariables. Thus, the use of an hsCRP-modified Framingham
risk score also has the potential to help more patients and
physicians to better direct the use of preventive statin therapy to appropriate risk groups.
Limitations of our analysis merit consideration. First,
our analysis is limited to women, and thus care must be
taken before these data are generalized to men. However,
data from several other large cohorts of men have found
hsCRP to predict risk independently of the traditional Framingham covariables (7, 9 12, 14, 16, 47). Second, our
data are based on a single determination of hsCRP instead
of 2 measures, as is currently recommended (21). This
limitation would tend to increase variability in our measures of hsCRP and thus lead, if anything, to an underestimation of true effects. Third, in contrast to LDL cholesterol, there remains no evidence to date that reducing
hsCRP level itself will reduce cardiovascular risk. As such,
although these data show that in women the addition of
hsCRP can improve global risk prediction models, they
should not be construed as implying a direct benefit of
hsCRP reduction, an issue now being evaluated in several
clinical trials. Finally, we evaluated only hsCRP in this
study and recognize the possibility that other biomarkers of
inflammation, hemostasis, and thrombosis may become
available in the future. In this regard, we believe that the
methods developed here to address incremental value for
risk screening will be of use as other novel biomarkers emerge.
From Brigham and Womens Hospital, Harvard Medical School, and
Harvard School of Public Health, Boston, Massachusetts.
Grant Support: By grants from the Donald W. Reynolds Foundation
(Las Vegas, Nevada), the Leducq Foundation (Paris, France), and the
Doris Duke Charitable Foundation (New York). The overall Womens
Health Study cohort is supported by grants HL-43851 and CA-47988
from the National Heart, Lung, and Blood Institute and the National
Cancer Institute (both in Bethesda, Maryland).
Potential Financial Conflicts of Interest: Honoraria: P.M. Ridker

(Dade Behring); Grants received: P.M. Ridker (Reynolds Foundation,

Leducq Foundation, Doris Duke Foundation, National Heart, Lung,
and Blood Institute, National Cancer Institute, American Heart Association,
Dade Behring, AstraZeneca, Novartis, Sanofi-Aventis). Dr. Ridker is listed as
a co-inventor on patents held by the Brigham and Womens Hospital that
relate to the use of inflammatory biomarkers in cardiovascular disease.
Requests for Single Reprints: Nancy R. Cook, ScD, Division of Pre-

ventive Medicine, Brigham and Womens Hospital, 900 Commonwealth

Avenue East, Boston, MA 02215; e-mail, ncook@rics.bwh.harvard.edu.
Current author addresses and author contributions are available at www
.annals.org.

References
1. Wilson PW, DAgostino RB, Levy D, Belanger AM, Silbershatz H, Kannel
WB. Prediction of coronary heart disease using risk factor categories. Circulation.
1998;97:1837-47. [PMID: 9603539]
www.annals.org

Risk Prediction Models with and without hsCRP

2. Executive Summary of The Third Report of The National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, And Treatment
of High Blood Cholesterol In Adults (Adult Treatment Panel III). JAMA. 2001;
285:2486-97. [PMID: 11368702]
3. DAgostino RB Sr, Grundy S, Sullivan LM, Wilson P. Validation of the
Framingham coronary heart disease prediction scores: results of a multiple ethnic
groups investigation. JAMA. 2001;286:180-7. [PMID: 11448281]
4. Ridker PM, Brown NJ, Vaughan DE, Harrison DG, Mehta JL. Established
and emerging plasma biomarkers in the prediction of first atherothrombotic
events. Circulation. 2004;109:IV6-19. [PMID: 15226246]
5. Khot UN, Khot MB, Bajzer CT, Sapp SK, Ohman EM, Brener SJ, et al.
Prevalence of conventional risk factors in patients with coronary heart disease.
JAMA. 2003;290:898-904. [PMID: 12928466]
6. Greenland P, Knoll MD, Stamler J, Neaton JD, Dyer AR, Garside DB, et al.
Major risk factors as antecedents of fatal and nonfatal coronary heart disease
events. JAMA. 2003;290:891-7. [PMID: 12928465]
7. Ridker PM, Cushman M, Stampfer MJ, Tracy RP, Hennekens CH. Inflammation, aspirin, and the risk of cardiovascular disease in apparently healthy men.
N Engl J Med. 1997;336:973-9. [PMID: 9077376]
8. Ridker PM, Rifai N, Rose L, Buring JE, Cook NR. Comparison of C-reactive
protein and low-density lipoprotein cholesterol levels in the prediction of first
cardiovascular events. N Engl J Med. 2002;347:1557-65. [PMID: 12432042]
9. Koenig W, Lowel H, Baumert J, Meisinger C. C-reactive protein modulates
risk prediction based on the Framingham Score: implications for future risk assessment: results from a large cohort study in southern Germany. Circulation.
2004;109:1349-53. [PMID: 15023871]
10. Ballantyne CM, Hoogeveen RC, Bang H, Coresh J, Folsom AR, Heiss G,
et al. Lipoprotein-associated phospholipase A2, high-sensitivity C-reactive protein, and risk for incident coronary heart disease in middle-aged men and women
in the Atherosclerosis Risk in Communities (ARIC) study. Circulation. 2004;
109:837-42. [PMID: 14757686]
11. Danesh J, Wheeler JG, Hirschfield GM, Eda S, Eiriksdottir G, Rumley A,
et al. C-reactive protein and other circulating markers of inflammation in the
prediction of coronary heart disease. N Engl J Med. 2004;350:1387-97. [PMID:
15070788]
12. Pai JK, Pischon T, Ma J, Manson JE, Hankinson SE, Joshipura K, et al.
Inflammatory markers and the risk of coronary heart disease in men and women.
N Engl J Med. 2004;351:2599-610. [PMID: 15602020]
13. Ridker PM, Wilson PW, Grundy SM. Should C-reactive protein be added
to metabolic syndrome and to assessment of global cardiovascular risk? Circulation. 2004;109:2818-25. [PMID: 15197153]
14. Cushman M, Arnold AM, Psaty BM, Manolio TA, Kuller LH, Burke GL,
et al. C-reactive protein and the 10-year incidence of coronary heart disease in
older men and women: the cardiovascular health study. Circulation. 2005;112:
25-31. [PMID: 15983251]
15. Ridker PM, Rifai N, Cook NR, Bradwin G, Buring JE. Non-HDL cholesterol, apolipoproteins A-I and B100, standard lipid measures, lipid ratios, and
CRP as risk factors for cardiovascular disease in women. JAMA. 2005;294:32633. [PMID: 16030277]
16. Laaksonen DE, Niskanen L, Nisskanen K, Punnonen K, Tuomainen TP,
Salonen JT. C-reactive protein in the prediction of cardiovascular and overall
mortality in middle-aged men: a population-based cohort study. Eur Heart J.
2005;26:1783-9. [PMID: 15821003]
17. Festa A, DAgostino R Jr, Howard G, Mykkanen L, Tracy RP, Haffner
SM. Chronic subclinical inflammation as part of the insulin resistance syndrome:
the Insulin Resistance Atherosclerosis Study (IRAS). Circulation. 2000;102:42-7.
[PMID: 10880413]
18. Festa A, DAgostino R Jr, Tracy RP, Haffner SM. Elevated levels of acutephase proteins and plasminogen activator inhibitor-1 predict the development of
type 2 diabetes: the insulin resistance atherosclerosis study. Diabetes. 2002;51:
1131-7. [PMID: 11916936]
19. Pradhan AD, Manson JE, Rifai N, Buring JE, Ridker PM. C-reactive
protein, interleukin 6, and risk of developing type 2 diabetes mellitus. JAMA.
2001;286:327-34. [PMID: 11466099]
20. Ridker PM, Buring JE, Cook NR, Rifai N. C-reactive protein, the metabolic
syndrome, and risk of incident cardiovascular events: an 8-year follow-up of 14
719 initially healthy American women. Circulation. 2003;107:391-7. [PMID:
12551861]
21. Pearson TA, Mensah GA, Alexander RW, Anderson JL, Cannon RO 3rd,
Criqui M, et al. Markers of inflammation and cardiovascular disease: application
www.annals.org

Article

to clinical and public health practice: a statement for healthcare professionals from
the Centers for Disease Control and Prevention and the American Heart Association. Circulation. 2003;107:499-511. [PMID: 12551878]
22. Grundy SM. The changing face of cardiovascular risk [Editorial]. J Am Coll
Cardiol. 2005;46:173-5. [PMID: 15992653]
23. Buring JE, Hennekens CH. The Womens Health Study: summary of the
study design. Journal of Myocardial Ischemia. 1992;4:27-9.
24. Rexrode KM, Lee IM, Cook NR, Hennekens CH, Buring JE. Baseline
characteristics of participants in the Womens Health Study. J Womens Health
Gend Based Med. 2000;9:19-27. [PMID: 10718501]
25. Ridker PM, Cook NR, Lee IM, Gordon D, Gaziano JM, Manson JE, et al.
A randomized trial of low-dose aspirin in the primary prevention of cardiovascular disease in women. N Engl J Med. 2005;352:1293-304. [PMID: 15753114]
26. Hainline A, Karon J, Lippel K, eds. Manual of Laboratory Operations, Lipid
Research Clinics Program, and Lipid and Lipoprotein Analysis. 2nd ed. Bethesda,
MD: Department of Health and Human Services; 1982.
27. Cox DR. Regression models and life tables (with discussion). Journal of the
Royal Statistical Society B. 1972;34:187-220.
28. Harrell FE Jr. Regression Modeling Strategies. New York: Springer; 2001.
29. Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning:
Data Mining, Inference and Prediction. New York: Springer-Verlag; 2001.
30. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver
operating characteristic (ROC) curve. Radiology. 1982;143:29-36. [PMID:
7063747]
31. Efron B, Tibshirani R. An Introduction to the Bootstrap. New York: Chapman & Hall; 1993.
32. Burnham KP, Anderson DR. Model Selection and Inference: A Practical
Information-Theoretic Approach. New York: Springer-Verlag; 1998.
33. Nagelkerke NJ. A note on a general definition of the coefficient of determination. Biomka. 1991;78:691-2.
34. Ash A, Shwartz M. R2: a useful measure of model performance when predicting a dichotomous outcome. Stat Med. 1999;18:375-84. [PMID: 10070680]
35. Royston P, Sauerbrei W. A new measure of prognostic separation in survival
data. Stat Med. 2004;23:723-48. [PMID: 14981672]
36. Hosmer DW, Hosmer T, Le Cessie S, Lemeshow S. A comparison of
goodness-of-fit tests for the logistic regression model. Stat Med. 1997;16:965-80.
[PMID: 9160492]
37. Fleiss JL. Statistical Methods for Rates and Proportions. 2nd ed. New York:
Wiley; 1981.
38. Greenland P, Smith SC Jr, Grundy SM. Improving coronary heart disease
risk assessment in asymptomatic people: role of traditional risk factors and noninvasive cardiovascular tests. Circulation. 2001;104:1863-7. [PMID: 11591627]
39. Sullivan LM, Massaro JM, DAgostino RB Sr. Presentation of multivariate
data for clinical use: The Framingham Study risk score functions. Stat Med.
2004;23:1631-60. [PMID: 15122742]
40. Gail MH, Pfeiffer RM. On criteria for evaluating models of absolute risk.
Biostatistics. 2005;6:227-39. [PMID: 15772102]
41. Moons KG, Harrell FE. Sensitivity and specificity should be de-emphasized
in diagnostic accuracy studies. Acad Radiol. 2003;10:670-2. [PMID: 12809422]
42. Ridker PM, Rifai N, Pfeffer MA, Sacks FM, Moye LA, Goldman S, et al.
Inflammation, pravastatin, and the risk of coronary events after myocardial infarction in patients with average cholesterol levels. Cholesterol and Recurrent
Events (CARE) Investigators. Circulation. 1998;98:839-44. [PMID: 9738637]
43. Ridker PM, Rifai N, Clearfield M, Downs JR, Weis SE, Miles JS, et al.
Measurement of C-reactive protein for the targeting of statin therapy in the
primary prevention of acute coronary events. N Engl J Med. 2001;344:1959-65.
[PMID: 11430324]
44. Kent SM, Taylor AJ. Usefulness of lowering low-density lipoprotein cholesterol to "70 mg/dl and usefulness of C-reactive protein in patient selection. Am
J Cardiol. 2003;92:1224-7. [PMID: 14609606]
45. Ridker PM, Cannon CP, Morrow D, Rifai N, Rose LM, McCabe CH, et
al. C-reactive protein levels and outcomes after statin therapy. N Engl J Med.
2005;352:20-8. [PMID: 15635109]
46. Nissen SE, Tuzcu EM, Schoenhagen P, Crowe T, Sasiela WJ, Tsai J, et al.
Statin therapy, LDL cholesterol, C-reactive protein, and coronary artery disease.
N Engl J Med. 2005;352:29-38. [PMID: 15635110]
47. Boekholdt SM, Hack CE, Sandhu MS, Luben R, Bingham SA, Wareham
NJ, et al. C-reactive protein levels and coronary artery disease incidence and
mortality in apparently healthy men and women: The EPIC-Norfolk prospective
population study 1993-2003. Atherosclerosis. 2005. [PMID: 16257408]
4 July 2006 Annals of Internal Medicine Volume 145 Number 1 29

Annals of Internal Medicine

Current Author Addresses: Drs. Cook, Buring, and Ridker: Division of

Preventive Medicine, Brigham and Womens Hospital, 900 Commonwealth Avenue East, Boston, MA 02215.
Author Contributions: Conception and design: N.R. Cook, P.M. Ridker.

Analysis and interpretation of the data: N.R. Cook, P.M. Ridker.

Drafting of the article: N.R. Cook, P.M. Ridker.
Critical revision of the article for important intellectual content: N.R.
Cook, J.E. Buring, P.M. Ridker.
Final approval of the article: N.R. Cook, P.M. Ridker.
Statistical expertise: N.R. Cook.
Obtaining of funding: P.M. Ridker.
Collection and assembly of data: J.E. Buring.
48. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in
developing models, evaluating assumptions and adequacy, and measuring and
reducing errors. Stat Med. 1996;15:361-87. [PMID: 8668867]

APPENDIX
Computation of 10-Year Risk
The 10-year risk for cardiovascular disease calibrated to the
Framingham population may be estimated for each individual
woman using the !-coefficients in Table 1. First, multiply each
womans risk factor x by the appropriate coefficient in Table 1
and sum these (! "! # x). The risk may then be computed
from the following equation:
Risk ! 1 $ (0.903)exp("! # x

$8.795)

Additional Measures of Model Fit

The Appendix Table presents additional measures comparing the best-fitting models in the WHS data with and without
hsCRP, as well as the ATP III model with coefficients refitted to
the WHS data, with and without hsCRP. As indicated, all global
measures of fit showed a preference for the models that included
hsCRP. The measures are as follows:
1. The likelihood ratio chi-square provides a global test of
model fit. It is a function of the degrees of freedom, or number of
terms in the model. The difference between chi-square values
provides a test of the model improvement with hsCRP (P %
0.0001 for both the WHS and ATP III models).
2. The Bayes information criterion is a function of the log
likelihood but adds a penalty for added variables based on the
sample size (28). It is not influenced by the number of predictors,
so models can thus be compared directly. Lower values reflect
better fit, suggesting improvement with the addition of hsCRP.
3. The Bayes information criterion weight provides an estimate of the posterior probability of each model given the set of
candidate models considered (29, 32). The weights suggest a

W-12 4 July 2006 Annals of Internal Medicine Volume 145 Number 1

much higher probability that the WHS model that includes

hsCRP is correct.
4. The Akaike information criterion is a function of the log
likelihood that adds a penalty of 2 for each added variable (32),
less extreme than the penalty used in the Bayes information criterion. Lower values are better, again suggesting improvement
with hsCRP.
5. The Akaike information criterion weights reflect the relative likelihood of a model given the data and the set of models
(32). These weights display a clear preference for the models with
hsCRP.
6. Nagelkerkes generalized model R2 (33, 34) is a measure
of the fraction of the $2 log likelihood explained by the predictors, analogous to the percentage of variance explained in a linear
model. It is adjusted to a range of 0 to 1 and is higher for models
with hsCRP, both in the original data and after adjustment for
optimism using the bootstrap (31, 48).
7. The D-statistic of Royston and Sauerbrei (35) measures
the separation of survival curves across levels of the predictor
variables, analogous to distance between KaplanMeier curves.
This is higher for models that included hsCRP, even after adjustment for optimism, suggesting better prediction for these models.
8. The Brier score (28) computes the sum of squared differences between the observed outcome and the fitted probability. It
is lower for models that included hsCRP, indicating that the
predicted probabilities are closer to the observed outcomes.
9. The c-index represents the area under the receiver-operating characteristic curve (30), allowing for censored data. This is
a measure of discrimination based on ranks and is similar but
slightly higher for models that included hsCRP, even after adjustment for optimism. The c-statistic is the probability that, for
a randomly selected pair of subjects, one diseased and the other
nondiseased, the person with disease will have the higher estimated disease probability according to the model.
10. The HosmerLemeshow calibration statistic (36) classifies predicted probabilities into categories and compares the
mean predicted probability with the observed risk within each
category. A P value representing a significant difference indicates
a lack of fit. When decile categories are used, the predicted probability is less than 5% for the first 9 of 10 categories. Calibration
is adequate for all models that use this measure and is somewhat
better for models without hsCRP. The calibration statistic based
on risk percentage compares observed and predicted risk by using
10 categories based on 2percentage point increments in predicted risk, from 0% to 2% risk to 18% or greater risk. This
statistic indicates significant deviation of observed and predicted
values in models without hsCRP, suggesting a lack of fit in higher-risk categories.

www.annals.org

Appendix Table. Comparison of Discrimination and Calibration for Global Risk Prediction Models with and without
High-Sensitivity C-Reactive Protein*
Variable

WHS Model

ATP III Model

With hsCRP

Without hsCRP

With hsCRP

Without hsCRP

556.75 (8)
6960.26
0.874
6928.53
0.506
9.28
9.05
1.948
1.914
0.01960

541.44 (7)
6969.60
0.0082
6941.84
0.0007
9.03
8.92
1.922
1.893
0.01965

558.69 (9)
6964.28
0.117
6928.58
0.492
9.31
8.97
1.951
1.913
0.01959

542.54 (8)
6974.46
0.0007
6942.74
0.0004
9.05
8.84
1.918
1.884
0.01964

Discrimination
C-index
Adjusted c-index

0.815
0.813

0.813
0.811

0.814
0.810

0.812
0.809

Calibration
HosmerLemeshow statistic
P value, deciles
P value, risk percentage

0.19
0.23

0.59
0.039

0.71
0.25

0.79
0.008

Global measures
LR chi-square (df)
BIC
BIC weight
AIC
AIC weight
R2 , %
Adjusted R2, %
D-statistic
Adjusted D-statistic
Brier score

* Lower values of the BIC, AIC, and Brier score and higher values of all other statistics, including the calibration P values, indicate better fit. See text for descriptions. AIC !
Akaike information criterion; ATP ! Adult Treatment Panel; BIC ! Bayes information criterion; df ! degrees of freedom; hsCRP ! high-sensitivity C-reactive protein;
LR chi-square ! multi degree-of-freedom likelihood ratio chi-square statistic for fit of the entire model; WHS ! Womens Health Study.
Superior fit in comparison of models with and without hsCRP.
Adjusted for optimism using bootstrap resampling. Tests for improvement in the D-statistic with the addition of hsCRP to the ATP III model reached statistical
significance (P ! 0.03).
Based on 10 categories defined by 2percentage point increments in predicted risk.

www.annals.org

4 July 2006 Annals of Internal Medicine Volume 145 Number 1 W-13