Experiments on Interactive Groups, and Network Effects

Examples from Legislative Studies

Sayali Phadke1

Bruce A. Desmarais2

1 PhD student
Department of Statistics
2 Associate Professor
Department of Political Science

Joint Statistical Meeting 2016

Supported by NSF Grants: SES-1558661, SES-1619644,
CISE-1320219, and DGE-1144860

(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

1 / 20

Motivation: Bergan and Cole (Political Behavior, 2015)

Field experiment on 148 Michigan legislators

Treatment: Calls from constituents
Outcome: Vote on anti-bullying bill
Could there have been spillover effects?

(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

2 / 20

Overview

Motivation

Methodology

Applications

Final remarks

(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

3 / 20

Motivation

Motivation: Identification of causal effect

Homophily and Contagion Are Generically Confounded in Observational
Social Network Studies (Shalizi and Thomas, Sociological methods and
research 2011)

(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

4 / 20

Motivation

Motivation: Network plot

Stable Unit Treatment Value Assumption (SUTVA)

Treatment status of a unit does not affect the outcome of another.
(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

5 / 20

Motivation

Research objectives

Model spillover of treatment effect via network

structures.
Examine sensitivity to specification of the
model of treatment effects
Evaluate models using data from field
experiments on US State legislatures.

(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

6 / 20

Methodology

Methodology: Intuition

Researcher specifies network model of effects

How do we compare outcomes across groups?
How do we measure effect of experiment?

(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

7 / 20

Methodology

Causal model (Bowers et al., Political Analysis, 2012)

Sharp null of no effects assumed

Spillover depends on # treated neighbors
Spillover modeled as a nonlinear growth curve
Separate direct and indirect effect parameters

(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

8 / 20

Methodology

Testing (Bowers et al., Political Analysis, 2012)

Kolmogorov-Smirnov (KS) test statistic used

to compare treatment-control outcomes
Compared under large number of permutations
p-value is proportion of permutation statistics
greater than observed statistics

(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

9 / 20

Methodology

Salient dimensions

Network selection: ties through which units

interact
Neighborhood specification: extent of
spreading of treatment
Diffusion model: form taken by the spreading
of treatment

(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

10 / 20

Applications

Application: Butler, Nickerson et al. (2012) experiment

Field experiment on 70 New Mexico legislators

Letters indicating constituent opinion about a
spending bill
Analysis concluded significant treatment effect
Coppock (2014) extended analysis to model
spillovers

(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

11 / 20

Applications

Application: Replication of Coppock (2014)

Plot of pvalues

0.4

1.0

yi,z = yi,0 + 1 zi + 2 g (z )

0.4

0.2
0.4

Higher p-value indicates

evidence for effect of
experiment

0.0

2 on Y axis

0.6

0.2

1 on X axis

Indirect effects

0.2

0.8

0.0

0.4

0.2

0.0

0.2

0.4

Direct effects

(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

12 / 20

Extension: K-nearest ideological neighbors

0.4

Plot of pvalues: k=12

0.4

Plot of pvalues: k=3

0.8
0.2

0.2

0.8

0.6
0.0

0.0

Indirect effects

0.6

0.4
0.2

0.2

0.4

0.2

0.2
0.4

0.4

Indirect effects

Applications

0.0

0.4

0.2

0.0

0.2

0.4

0.0

0.4

Direct effects

(Pennsylvania State University)

0.2

0.0

0.2

0.4

Direct effects

Examples from Legislative Studies

JSM 2016

13 / 20

Extension: Committee network

Plot of pvalues: Committees>0

Plot of pvalues: Committees>1

0.4

0.4

1.0

0.8
0.2

0.2

0.8

0.6
0.0

0.0

Indirect effects

0.6

0.4
0.2

0.2

0.4

0.2

0.2
0.4

0.4

Indirect effects

Applications

0.0

0.4

0.2

0.0

0.2

0.4

0.0

0.4

Direct effects

(Pennsylvania State University)

0.2

0.0

0.2

0.4

Direct effects

Examples from Legislative Studies

JSM 2016

14 / 20

Application: Bergan and Cole (2015)

Plot of pvalues

Plot of pvalues: k=5

0.4

0.4

1.0

0.8
0.2

Indirect effects

0.2

0.8

0.6
0.0

0.0

0.6

0.4
0.2

0.2

0.4

0.2
0.4

0.2
0.4

Indirect effects

Applications

0.0

0.4

0.2

0.0

0.2

0.4

0.0

0.4

Direct effects

(Pennsylvania State University)

0.2

0.0

0.2

0.4

Direct effects

Examples from Legislative Studies

JSM 2016

15 / 20

Applications

Replication overview

Treatment spreads among New Mexico

legislators via ideological network
Spending is a fairly partisan issue
Spillover not observed among Michigan
legislators

(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

16 / 20

Final remarks

Summary

Interference exists in experiments on

interactive social groups
Several dimensions important to models for
interference
Simulation and power analysis can assist in
optimal design

(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

17 / 20

Final remarks

Next steps

Replicate Broockman (2013) that studies

legislators from multiple states
Consider other networks
Model a mixture of networks
Model proportion of treated neighbors

(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

18 / 20

Final remarks

Thank you
Questions?

(Pennsylvania State University)

Examples from Legislative Studies

JSM 2016

19 / 20

Final remarks

Network plots

Figure: Ideological network: New Mexico

(Pennsylvania State University)

Figure: Committee network: New Mexico

Examples from Legislative Studies

JSM 2016

20 / 20