Associations between Internet-based patient ratings and conventional quality measures in the English NHS

November 6th 2012

Felix Greaves Department of Primary Care and Public Health Imperial College London felix.greaves08@imperial.ac.uk @felixgreaves

We love rating things on the internet

We now do this in health too

USA

The number of ratings is increasing

Gao et al., JMIR, 2012

Cumulative number of hospital ratings

UK

10000

15000

25000

20000

5000 0

Greaves et al., JMIR, 2012

Aug-08 Oct-08 Dec-08 Feb-09 Apr-09 Jun-09 Aug-09 Oct-09 Dec-09 Feb-10 Apr-10 Jun-10 Aug-10 Oct-10 Dec-10 Feb-11 Apr-11 Jun-11 Aug-11 Oct-11 Dec-11

What some clinicians think

What people say about online rating

Why its bad 1. Potential straining of doctor-patient relationships 2. Reviews may be malicious or fake 3. Selection bias by those leaving reviews 4. Lack of meaningful data on technical quality of health care Why its good 1.Doctors can often be poor judges of their patients satisfaction and experience 2.Feedback changes doctors performance 3.People will inevitably use the Internet to voice opinions, so why not capture this information in a useful form

Where we are

Physician and hospital ratings websites are becoming more popular They remain controversial The relationship between ratings and hard measures of quality is not known Can we compare ratings to other healthcare quality measures?

The opportunity for a natural experiment

Getting the data

We thought we could compare ratings with some traditional measures of quality

Vs.

Average scores
Score
The environment where I was treated was I was treated with dignity and respect by the hospital staff I was involved with decisions about my care The hospital staff worked well together

Mean rating (out of 5)

Range

3.6 4.0 3.8 4.1

2.6-5.0 2.7-5.0 2.4-5.0 2.9-5.0

Greaves et al. BMJ Quality and Safety, 2012

Average scores
Score
The environment where I was treated was

Mean rating (out of 5)

Range

3.6

2.6-5.0

I was treated with dignity 4.0 2.7-5.0 and respectby themajority of ratings are positive The hospital staff 67% would recommend to a friend I was involved with 3.8 2.4-5.0 decisions about my care The hospital staff worked 4.1 2.9-5.0 well together
Greaves et al. BMJ Quality and Safety, 2012

Ratings compared to patient experience surveys

NHS Choices Measure
Proportion of patients recommending Rating of being treated with dignity and respect Survey question: Overall, how would you rate the quality of care you received Overall, did you feel you were treated with dignity and respect while in hospital?

Spearman Rho
0.40

p value <0.001

0.33

<0.001

Rating of staff working together

How well would rate how well the doctors and nurse worked together?

0.32

<0.001

Rating of cleanliness How clean was the hospital ward or 0.48 room you were in?

<0.001

Greaves et al. BMJ Quality and Safety, 2012

Ratings compared to patient experience surveys

NHS Choices Measure
Proportion of patients recommending Survey question: Overall, how would you rate the quality of care you received

Spearman Rho
0.40

p value <0.001

There is a moderate, highly Rating of being Overall, did you feel you were 0.33 significant association between treated with dignity treated with dignity and respect ratings online and large and respect while in hospital? surveys of patient experience
Rating of staff working together How well would rate how well the doctors and nurse worked together?

<0.001

0.32

<0.001

Rating of cleanliness How clean was the hospital ward or 0.48 room you were in?

<0.001

Greaves et al. BMJ Quality and Safety, 2012

Ratings compared to outcomes

NHS Choices Measure Other variable Spearman Rho p value

Proportion of patients recommending Proportion of patients recommending Proportion of patients recommending Proportion of patients recommending Proportion of patients recommending

Hospital Standardised Mortality Ratio Standardised morality rate for high risk conditions Standardised morality rate among surgical inpatients with serious treatable complications Standardised mortality rate from low mortality conditions Emergency readmission rate within 28 days

-0.20

0.01

-0.22

0.01

-0.00

0.99

0.03

0.70

-0.31

<0.001

Ratings compared to outcomes

NHS Choices Measure Other variable Spearman Rho p value

Proportion of patients recommending Proportion of patients recommending Proportion of patients recommending Proportion of patients recommending Proportion of patients recommending

Hospital Standardised Mortality Ratio Standardised morality rate for high risk conditions

-0.20

0.01

-0.22

0.01

There is a mild, significant Standardised morality rate among -0.00 association between ratings surgical inpatients with serious online complications treatable and mortality
Standardised mortality rate from low mortality conditions Emergency readmission rate within 28 days 0.03

0.99

0.70

-0.31

<0.001

Comparing cleanliness ratings with infection rates

Spearman Rho

NHS Choices Measure

Other variable

p value

Patient perception of cleanliness

Rate of MRSA bacteraemia -0.30 (per 1,000 bed days)

<0.001

Greaves et al, 2012 Arch Int Med

Comparing cleanliness ratings with infection rates

Spearman Rho

NHS Choices Measure

Other variable

p value

There is a moderate, Patient Rate of MRSA bacteraemia -0.30 significant association perception of (per 1,000 bed days) cleanliness between ratings of cleanliness online and infection rates

<0.001

Greaves et al, 2012 Arch Int Med

A wisdom in the crowd of patients?

Comparison of the NHS Inpatient Survey and ratings on the NHS Choices website NHS Inpatient Survey NHS Choices ratings
Mechanism Number of responses Selection Paper-based survey 69,000 per year Random; patients receive a survey requesting completion after leaving hospital 79% rated their overall care as excellent or very good Likely more expensive Ratings left on a website 5,000 per year Self-selecting; patients are not solicited

Proportion positive Cost

67% would recommend to a friend Likely less expensive

Comparison of the NHS Inpatient Survey and ratings on the NHS Choices website NHS Inpatient Survey NHS Choices ratings
Mechanism Number of responses Selection Paper-based survey Ratings left on a website

Proportion positive Cost

There were 10,000 hospital ratings inper year 69,000 per year 5,000 the UK over 2 years Random; patients receive a Self-selecting; patients are Over the same time not were survey requesting period there solicited 29,118,009 hospital completion after leaving admissions hospital 79% rated hospital admissions are rated 0.04% of their overall care 67% would recommend to a as excellent or very good friend Likely more expensive Likely less expensive

The stereotypical reviewer?

Where are people rating their care?

Associations between whether a practice is rated with population and practice characteristics
Independent variable Z statistic p value

15.38 <0.001 Practice population size -7.82 <0.001 IMD score of Practices serving younger people patients <0.001 Population more6.72 likely to be rated density

are

Singlehander -4.50 <0.001 -3.88 Proportion are more likely to be rated <0.001 of population aged Practices in urban areas are more over 65 years 0.11 Proportion likely -1.58be rated of to population who are white Type of contract -0.71 0.48 Training practice 0.35 0.73

Practices serving less deprived people

The cloud of patient experience

Thanks to my co-authors:

Christopher Millett Ara Darzi Dominic King Henry Lee Utz Pape Azeem Majeed Robert Wachter Daniel Cano-Ramirez

Thank you
felix.greaves08@imperial.ac.uk