Appraising implementation when evaluating the health effects of organisational interventions.

M Egan
MRC/CSO Social and Public Health Sciences Unit
24th May 2012

MRC/CSO Social and Public Health Sciences Unit

Why do evaluators care about implementation?

So we can learn how to improve implementation To help us assess the extent to which findings are generalisable and transferable (relationship between context, implementation and outcomes). So we can distinguish between theory failure and implementation failure. A consequence of our growing interest in complex interventions, natural experiments and theory led evaluation.

MRC|CSO Social and Public Health Sciences Unit

Theory based approaches

MRC|CSO Social and Public Health Sciences Unit

The Cochrane logo

MRC|CSO Social and Public Health Sciences Unit

Clinical trial phases (MRC guidance, 2000)

MRC|CSO Social and Public Health Sciences Unit

Key elements of the development and evaluation process (MRC guidance, 2008)
Feasibility/piloting 1 Testing procedures 2 Estimating recruitment /retention 3 Determining sample size

Development 1 identifying evidence base 2 identifying /developing theory 3 Modelling process and outcome

Evaluation 1 Assessing effectiveness 2 Understanding change process 3 Assessing cost effectiveness

MRC|CSO Social and Public Health Sciences Unit

Implementation 1 dissemination 2 surveillance 3 long term follow up

Appropriate study designs

Research question Effectiveness Does this work? Effectiveness of service delivery: How does it work ? Salience Does it matter ? Acceptability Will children/parents want to use it? Appropriateness Is this the right service for these children? ++ + Qualitative Surveys Case control Research studies Cohort RCTs Studies + ++

++

++

++

++

++

(Muir Gray, 1997)

MRC|CSO Social and Public Health Sciences Unit

Evaluating social interventions: a false alternative

Black box evaluations of effectiveness Experimental designs Research hierarchy No interest in context, process, theory of change, or differential effects Focused on attribution of intervention to effect

vs

Theory based evaluation Suspicious of experimental designs (espec. RCTs) Breaks a complex intervention into a series of hypothesised causal links. What works, for whom and in what context?

MRC|CSO Social and Public Health Sciences Unit

Evaluating social interventions: a false alternative

Black box evaluations of effectiveness Experimental designs Research hierarchy No interest in context, process, theory of change, or differential effects Focused on attribution of intervention to effect

Theory based evaluation Suspicious of experimental designs (espec. RCTs) Breaks a complex intervention into a series of hypothesised causal links. What works, for whom and in what context?

Why cant you do both?

MRC|CSO Social and Public Health Sciences Unit

We should do both (sometimes)

Evaluators are often interested in 1. What works? (i.e. effects and effectiveness) 2. How does it work?/Why does it now work? 3. For whom? 4. In what context? A mixed methods approach needed to answer all 4 questions Quasi-experimental designs address What Works? (sub-(group analysis can sometimes help address the third and fourth question too). Often, the how, for whom and what context questions can only be properly interpreted in the light of whether or not a particular intervention works.
MRC|CSO Social and Public Health Sciences Unit

Gatehouse study

MRC|CSO Social and Public Health Sciences Unit

Note: I did not work on the Gatehouse Study (M Egan).

Aim & rationale of the Gatehouse Project School-based universal intervention Focus on improving the school environment (rather than simply teaching health promotion to children) Aim to promote emotional well-being & reduce health risk behaviours related to this, by increasing connectedness in young people Each school given resources and expertise to develop their own unique approach.

MRC|CSO Social and Public Health Sciences Unit

Process: Increasing connectedness

MRC|CSO Social and Public Health Sciences Unit

Implementation factors
Understanding of intervention

Measures/observations
Key informant interviews

Broader political & social environment Education region characteristics

Facilitator journals

School engagement questionnaires Minutes of meetings

School characteristics

School background audit

MRC|CSO Social and Public Health Sciences Unit

Cluster randomised controlled trial

Year 8 students

Intervention
16 districts 32 schools

12 schools*

1,620 82% response

Control

14 schools*

1,874 71% response

* 4 schools closed or threatened with closure, 2 did not agree to participate

MRC|CSO Social and Public Health Sciences Unit

Implementation appears to be broadly successful in terms of schools response to the Project

Shift in opinion from burden & irrelevant to ownership & utility strong sense of partnership in research
respected, valued, worked with schools

provided framework for change

opportunity to take risks a lens to review activities & practices

empowered, built capacity

actively seek new research opportunities to continue work

MRC|CSO Social and Public Health Sciences Unit

The Gatehouse Project: changes in health risk behaviour in year 8 students after 2 years

120 100
% of group

80 60 40 20 0
Smoking Regular smoking Binge drinking Cannabis Weekly Cannabis Comparison schools Intervention

All analyses adjusted for previous level of substance use in the school

MRC|CSO Social and Public Health Sciences Unit

But...
No change in explanatory mechanisms - school connectedness did not change Implementation

Explanatory mechanism

Outcome

MRC|CSO Social and Public Health Sciences Unit

MRC|CSO Social and Public Health Sciences Unit

MRC|CSO Social and Public Health Sciences Unit

Why am I here?

MRC|CSO Social and Public Health Sciences Unit

Based on the following systematic reviews

Egan M, Bambra C, Thomas S, Petticrew M, Whitehead M, Thomson H. The psychosocial and health effects of workplace reorganisation. 1. A systematic review of organisational-level interventions that aim to increase employee control. Journal of Epidemiology and Community Health (JECH) 2007; 61:945-54. Bambra et al. The psychosocial and health effects of workplace reorganisation. 2. A systematic review of task restructuring interventions. JECH 2007; 61:1028-37 Bambra, C. et al. Shifting schedules - The health effects of reorganizing shift work. American Journal of Preventive Medicine, 2009 Bambra, C et al. "A hard day's night?" The effects of Compressed Working Week interventions on the health and work-life balance of shift workers: a systematic review. JECH, 2008; 62(9):764-77
MRC|CSO Social and Public Health Sciences Unit

Interventions

MRC|CSO Social and Public Health Sciences Unit

Workplace reorganisation reviews: notable features

Organisation change rather than individual-level
organisational-level interventions intended to increase employees opportunities to make decisions or participate in decision making.

Broad review question and inclusion criteria

A range of health outcomes linked to stress (physical, psychological, sick days) Many study designs (experimental, prospective, retrospective studies: best available evidence)

Mediating mechanism
Psychosocial outcome (e.g. demand, control, support)

Implementation appraisal Health inequalities Narrative synthesis not meta-analysis

The Cochrane logo

MRC|CSO Social and Public Health Sciences Unit

So what where we looking for?

Psychosocial improvement Health improvement

Differential effects Differential effects

Intervention
Psychosocial deterioration Health deterioration

Psychosocial improvement

Health improvement Health deterioration

Control

Psychosocial deterioration

Reliable/plausible data on context, implementation to help future interventions and help us distinguish between implementation failure and theory failure.
MRC|CSO Social and Public Health Sciences Unit

What did we find?

Psychosocial and health outcomes

Intervention
and sometimes...

(ill)matched control

Psychosocial and health outcomes

Hardly anything on differential effects / health inequalities

Hardly anything on context or implementation

MRC|CSO Social and Public Health Sciences Unit

Nonetheless
Our findings did generally fit the demand, control support model in that Evidence of improving psychosocial environment tended to co-occur with health improvements. Evidence of deteriorating psychosocial environments tended to cooccur with worsening health. This applies particularly to changes in employee control.

MRC|CSO Social and Public Health Sciences Unit

Review methods: search and selection for psychosocial reviews

65282 search hits

733 retrieved

18 Selected for participation

21 Selected for task restructuring

Note: similar process for shiftwork reviews

MRC|CSO Social and Public Health Sciences Unit

Review methods: critical appraisal checklist

1. prospective 2. representative sample 3. appropriate comparison group 4. baseline response 60% 5. follow-up 80% in cohort, 60% in cross-section 6. adjustment for non-response and drop-out; 7. conclusions substantiated by data 8. adjustment for confounders 9. no contamination 10. appropriate statistical tests. Also emphasised controlled and prospective studies.

MRC|CSO Social and Public Health Sciences Unit

Problems with critical appraisal

Involves criteria that are either arbitrary or vague Implies that sources of bias are of equal important and that higher scores mean less biased studies. What are we appraising? Study methods or reporting of study methods? How useful is evidence hierarchy?
See Cochrane Handbook
MRC|CSO Social and Public Health Sciences Unit

Review methods: implementation checklist

MRC|CSO Social and Public Health Sciences Unit

Review methods: implementation checklist

MRC|CSO Social and Public Health Sciences Unit

Problems with implementation checklist

Involves criteria that are either arbitrary or vague Implies that sources of bias are of equal important and that higher scores mean less biased studies. What are we appraising? Study methods or reporting of study methods? What distinguishes good and bad implementation no established hierarchy

In other words, similar problems to those that relate to methodological checklists.

MRC|CSO Social and Public Health Sciences Unit

Specific problems with our checklist

Terminology: we use implementation to cover context, planning, delivery, programme theory and differential effects. Ignorance: we didnt know what a well implemented workplace reorganisation intervention looked like. Uninformative: the checklist rewards studies that mention certain implementation issues, but does not distinguish between (i) good and bad implementation; nor (ii) robust or non-robust evaluation of implementation There wasnt much to appraise.
MRC|CSO Social and Public Health Sciences Unit

There wasnt much to appraise.

More than half studies reported on Motivation for intervention Employee support for intervention and thats it.
MRC|CSO Social and Public Health Sciences Unit

Problem not just about quantity of reporting also quality

General reliance on anecdotal evidence rather than a planned study of implementation. Sometimes appeared to privilege one view point e.g. either managers or employees. Few studies discussed how implementation influenced outcomes. Quality of implementation reporting did not vary much by type or date of publication, or methodological quality, or health outcome.

MRC|CSO Social and Public Health Sciences Unit

Whats the point of all that?

Good reporting of implementation is a pre-condition to being able to appraise whether the implementation was good or bad. We have confirmed that this pre-condition is generally not being met which means we cannot know whether the implementation was good or bad.

MRC|CSO Social and Public Health Sciences Unit

Whats the point of all that?

Good reporting of implementation is a pre-condition to being able to appraise whether the implementation was good or bad. We have confirmed that this pre-condition is generally not being met which means we cannot know whether the implementation was good or bad.

Is that right?

MRC|CSO Social and Public Health Sciences Unit

Qualitative version of the checklist (no score)

MRC|CSO Social and Public Health Sciences Unit

This more qualitative approach produced more useful findings

Context: Participation interventions dont work in downsizing organisations Mechanism(?): Task structure interventions motivated by business reasons (efficiency, profit, productivity) were more likely to have negative health impacts than interventions motivated by stress reduction/health goals. Differential effects: Some evidence that the positive and negative health effects of organisational changes to the psychosocial work environment are felt more by men and lower socio-economic groups.

MRC|CSO Social and Public Health Sciences Unit

Workplace psychosocial reviews (demand, control, support model)

Increase participation in the workplace 18 studies Most robust = prospective with non-random control Some participation interventions may benefit employee physical and (especially) psychological health as predicted by the demand-control-support model, but may not protect employees from generally poor working conditions. Task restructuring interventions 19 studies Most robust = prospective with non-random control Task-restructuring interventions that increase demand or decrease control adversely affect the health of employees. Adverse effects are less likely if health goals are a planned into the intervention
MRC|CSO Social and Public Health Sciences Unit

Changing shifts reviews

Changing shifts 26 studies Most robust = prospective with non-random control Beneficial effects on health and worklife balance: (1) switching from slow to fast rotation, (2) changing from backward to forward rotation, and (3) self-scheduling of shifts Compressed working week 40 studies Most robust = prospective with non-random control Little effect on self-reported health. Work-life balance was generally improved

MRC|CSO Social and Public Health Sciences Unit

Key messages from review

Workplaces can be made more or less healthy. Findings often supportive of psychosocial models explaining workplace health especially control. Negative outcomes more likely in workplaces undergoing downsizing and/or some forms of economically motivated task restructuring. Little evidence on differential effects although some suggestion that organisational workplace change can potentially widen or narrow health inequalities. Barriers to robustly evaluating these kinds of workplace interventions

These conclusions are, we believe, useful (e.g. see Bambra, 2010 and Marmot Review, 2010).
MRC|CSO Social and Public Health Sciences Unit

Summing up
I have suggested a mixed methods approach to evaluating the health impacts of organisational change including quasi-experimental, qualitative and programme theory methods. The Gatehouse study shows such an approach is possible and replicable, even for complex interventions, although the approach is very challenging. Systematic reviews can assess outcomes, mechanisms and implementation although it takes more than just a checklist score. Using a best available evidence approach we can derive useful findings from flawed studies and reviews.

MRC|CSO Social and Public Health Sciences Unit

But
This is definitely a field where there is room for improvement

MRC|CSO Social and Public Health Sciences Unit

But
This is definitely a field where there is room for improvement.

Im sure the rest of this conference will demonstrate the progress that has already been made, and how further improvements can be made.

MRC|CSO Social and Public Health Sciences Unit

Key References
Publications (Co-)Authored by Presenter Bambra et al. The psychosocial and health effects of workplace reorganisation. 2. A systematic review of task restructuring interventions. JECH 2007; 61:1028-37 Egan M, Bambra C, Petticrew M, Whitehead M. Reviewing evidence on complex social interventions: appraising implementation in systematic reviews of the health effects of organisationallevel workplace interventions. Journal of Epidemiology & Community Health 2009; 63:4-11. Egan M, Bambra C, Thomas S, Petticrew M, Whitehead M, Thomson H. The psychosocial and health effects of workplace reorganisation. 1. A systematic review of organisational-level interventions that aim to increase employee control. JECH 2007; 61:945-54. Other Publications Bambra, C. et al. Shifting schedules - The health effects of reorganizing shift work. American Journal of Preventive Medicine, 2009 Bambra, C et al. "A hard day's night?" The effects of Compressed Working Week interventions on the health and work-life balance of shift workers: a systematic review. JECH, 2008; 62(9):764-77 Bambra C, Gibson M, Sowden A, Wright K, Whitehead M, Petticrew M. Tackling the wider social determinants of health and health inequalities: evidence from systematic reviews. Journal of Epidemiology & Community Health 2010; 64:284-91. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions. London: Medical Research Council, 2008. MRC. A framework for the development and evaluation of RCTs for complex interventions to improve health. London: Medical Research Council, 2000:18. Muir Gray JA. (1997) Evidence-based healthcare: how to make health policy and management decisions. London: Churchill Livingstone. Patton GC, Bond L, Carlin J, Thomas L, Butler H, Glover S, et al. Promoting social inclusion in schools: a cluster randomised trial. American Journal of Public Health. 2006; 96(9): 1582-7.

Thanks co-authors and funders. Thanks to Lyndal Bond for permission to use Gatehouse slides.