Supplementary Information for: Partisanship, Political

Knowledge, and the Dunning-Kruger Effect

Feb. 19, 2018

Study 1: A Political Dunning-Kruger Effect

Question Wording
Demographics
• Which best describes your gender?
– Male (0)
– Female (1)
• What is your current age?
– [Pull-down]
• Which of the following best describes your annual income before taxes?
– Less than $25,000 (1)
– $25,000-$49,999 (2)
– $50,000-$74,999 (3)
– $75,000-$99,999 (4)
– More than $100,000 (5)
• Generally speaking, do you usually think of yourself as a Republican, a Democrat, an
Independent, or what?
– Republican (1)
– Democrat (2)
– Independent (3)
– Something else (4)
• (If 1 is selected): Would you call yourself a strong Republican or a not very strong
Republican?
– Strong Republican (1)
– Not very strong Republican (2)
• (If 2 is selected): Would you call yourself a strong Democrat or a not very strong
Democrat?
– Strong Democrat (1)
– Not very strong Democrat (2)
• (If 3 is selected:) Do you think of yourself as closer to the Republican Party or to the
Democratic Party?
– Closer to the Republican (1)
 – Closer to the Democratic (2)
– Neither (3)
• What is the highest level of education you have completed?
– Less than high school (1)
– High school graduate/ GED (2)
– Some college (3)
– 2 year degree (4)
– 4 year degree (5)
– Masters degree (6)
– Doctorate (7)
– Professional Degree (JD, MD) (8)
• Are you currently married, widowed, divorced, separated, or have you never been
married?
– Married (1)
– Widowed (2)
– Divorced (3)
– Separated (4)
– Never married (5)
• How many children under 18 years old, if any, live in your household?
– None (1)
– 1 (2)
– 2 (3)
– 3 (4)
– 4 or more (5)

Political Quiz
• Now we would like you to respond to a series of questions with right and wrong
answers. Please try to answer this Political Quiz to the best of your ability. There is NO
penalty for incorrect answers. Please do not cheat in any way on this quiz.
• For how many years is a United States Senator elected--that is, how many years are
there in one full term of office for a U.S. Senator? [randomize]
– 6 years (1)
– 4 years (0)
– 2 years (0)
– 8 years (0)
• On which of the following does the U.S. federal government currently spend the
LEAST? [randomize]
– Foreign Aid (1)
– Medicare (0)
– National Defense (0)
– Social Security (0)
• Do you happen to know which party currently has the FEWEST members in the U.S.
House of Representatives? [randomize]
– Democrats (1)
– Republicans (0)
• Which political party is more conservative when it comes to healthcare policy?
[randomize]
– Democratic Party (0)
– Republican Party (1)
– They are about the same (0)
• Who is the current U.S. Secretary of Energy? [randomize]
– Rex Tillerson (0)
– Rick Perry (1)
– Elaine Chao (0)
– Ben Carson (0)
– Steven Mnuchin (0)

Treatments
• (Control) Please rank the following objects from largest (1) to smallest (4), by clicking
and dragging the choices. [randomize]
– A pineapple
– A tree
– A mouse
– A pea
• (Treatment 1) Now we would like you to think about Americans who identify as
members of the two major political parties. If you think a word in the list below
applies to Republicans and/or Democrats, check the corresponding box or boxes. If a
word applies to neither group, leave the boxes unchecked.
• (Treatment 1 Cont'd) [Republicans/Democrats] Are...
– Intelligent
– Compassionate
– Hardworking
– Irresponsible
– Misguided

Self-Assessment
• (Mastery) How would you rate your performance on the Political Quiz you took a few
moments ago?
– Excellent (5)
– Very Good (4)
– Good (3)
– Fair (2)
– Poor (1)
• (Self-Placement) Next we would like you to rate your own performance relative to
everyone else who has taken the quiz. Please use the sliding scale to evaluate your
performance.

Table A1: Descriptive Statistics Summary, Study 1

Statistic N Mean St. Dev. Min Max

Nonwhite 1047 0.308 0.462 0 1

Democrat 1047 0.420 0.494 0 1
Republican 1047 0.283 0.451 0 1
Income 1047 3.101 1.470 1 5
Education 1045 4.203 1.569 1 8
Married 1047 0.568 0.496 0 1
Homeowner 1047 0.611 0.488 0 1
Employed Full-Time 1047 0.527 0.499 0 1
Children Under 18 1047 1.758 1.072 1 5
Age 1047 44.203 15.964 18 90
 Table A2: Linear Regression Model Predicting Perceived Performance by Quiz
Score, Study 1 (Corresponds to Fig. 2 in Main Text)

Zero Correct 4.000***

(0.185)
One Correct 3.968***
(0.076)
Two Correct 3.865***
(0.064)
Three Correct 3.929***
(0.062)
Four Correct 4.342***
(0.085)
Five Correct 4.598***
(0.101)
N 1028
R-squared 0.938
Adj. R-squared 0.938
Residual Std. Error 1.047 (df = 1022)
F Statistic 2577.958*** (df = 6; 1022)

p < .01; p < .05; p < .1

Note: To show all group estimates, this is a zero-intercept model.
It presents each group’s estimated score and tests the null
hypothesis that a given estimate is zero.
 Table A3: Linear Regression Models Predicting Perceived Mastery, Study 1
(Corresponds to Fig. 3 in Main Text)

Model 1 Model 2

Treated 0.141* 0.108

(0.074) (0.158)
Quartile 2 -0.215
(0.341)
Quartile 3 -0.224
(0.333)
Quartile 4 0.408
(0.340)
Treated: Q2 0.074
(0.215)
Treated: Q3 0.087
(0.212)
Treated: Q4 -0.017
(0.216)
Constant 3.379*** 3.379***
(0.117) (0.250)
N 1047 1034
R-squared 0.003 0.032
Adj. R-squared 0.003 0.025
Residual Std. Error 1.196 (df = 1045) 1.184 (df = 1026)
F Statistic 3.658* (df = 1; 1045) 4.841*** (df = 7; 1026)

p < .01; p < .05; p < .1

 Table A4: Linear Regression Models Predicting Perceived Performance Quintile,
Study 1 (Corresponds to Figs. 3 and 4 in Main Text)

Model 1 Model 2

Treated 0.138** 0.036

(0.066) (0.140)
Quartile 2 -0.578*
(0.302)
Quartile 3 -0.298
(0.295)
Quartile 4 0.616**
(0.302)
Treated: Q2 0.308
(0.190)
Treated: Q3 0.172
(0.188)
Treated: Q4 -0.094
(0.191)
Constant 3.852*** 3.919***
(0.105) (0.222)
N 1040 1028
R-squared 0.004 0.057
Adj. R-squared 0.003 0.050
Residual Std. Error 1.068 (df = 1038) 1.044 (df = 1020)
F Statistic 4.344** (df = 1; 1038) 8.744*** (df = 7; 1020)

p < .01; p < .05; p < .1

Evaluation of Political Knowledge Question Battery

In order to assess the properties of the political knowledge question battery, I engage in an
exploratory factor analysis which demonstrates that the question battery, despite
acceptable reliability, possesses two factors which likely relate to current events awareness
and political sophistication, respectively. Before performing this examination it is helpful to
observe the distribution of quiz question accuracy, to assess the reasons for variation in the
number of correct responses. Table A5, below, shows this pattern.

Table A5: Distribution of Quiz Response Accuracy, Study 1

Statistic N Mean St. Dev. Min Max

Term Length 1045 0.436 0.496 0 1

Energy Secretary 1046 0.293 0.455 0 1
Healthcare Ideology 1044 0.617 0.486 0 1
Spending 1039 0.460 0.499 0 1
House Minority 1044 0.826 0.380 0 1

We see from Table A5 that there is evidence of overall question accuracy that mirrors
accuracy in other studies. For example, the term length question was correctly answered
by roughly 43.6% of qBus respondents, a question which was accurately reported by
35.4% of the ANES 2012 unweighted pre-election sample. However, these questions are
not directly comparable because the ANES measure was open-ended, meaning that it is
more likely for our respondents to have correctly guessed. This is also the case for the
question about the energy secretary's names, in which roughly 29.3% of respondents
answered correctly out of four options. It is likely that this surprisingly high proportion of
correct answers is partially thanks to guessing.
 Table A6: Regression Models Predicting Perceived Performance Quintile, Study 1

Full Battery Term Length Spending Healthcare Energy Sec. House Min.
OLS logistic logistic logistic logistic logistic
Model 1 Model 2 Model 3 Model 4 Model 5 Model 6

Education 0.135*** 0.137*** 0.218*** 0.129** 0.060 0.145**

(0.028) (0.050) (0.053) (0.051) (0.049) (0.065)
Age 0.023*** 0.026*** 0.006 0.022*** 0.030*** 0.033***
(0.003) (0.005) (0.005) (0.005) (0.005) (0.007)
Income 0.077** 0.113* 0.052 0.138** 0.031 0.027
(0.033) (0.058) (0.062) (0.059) (0.058) (0.075)
Homeownership 0.287*** 0.212 0.265 0.483*** 0.051 0.567***
(0.087) (0.158) (0.169) (0.159) (0.156) (0.201)
Employed 0.043 0.035 0.122* 0.063 -0.068 0.143*
(0.033) (0.060) (0.063) (0.061) (0.060) (0.080)
Female 0.398*** 0.443*** 0.387** 0.279* 0.517*** 0.244
(0.082) (0.144) (0.156) (0.147) (0.143) (0.187)
Nonwhite -0.190** -0.293* -0.242 -0.564*** 0.187 0.061
(0.084) (0.151) (0.164) (0.147) (0.148) (0.189)
Constant -0.674 -3.480*** -3.773*** -3.650*** -2.229*** -3.380***
(0.472) (0.864) (0.924) (0.867) (0.845) (1.086)
N 1016 1026 1027 1025 1021 1025
R-squared 0.196
Adj. R-squared 0.191
Log Likelihood -650.786 -598.922 -637.873 -661.591 -444.022
Residual Std. Error 1.178 (df = 1008)
F Statistic 35.209*** (df = 7; 1008)
AIC 1317.571 1213.845 1291.746 1339.182 904.045

p < .01; p < .05; p < .1

Cronbach's Alpha
Next we can assess the Cronbach's Alpha of the five items. I derive this measure from a
tetrachoric correlation matrix, which is appropriate due the binary scales used for each of
the knowledge measures. The Cronbach's Alpha is 0.61 (the standardized Alpha is also
0.61). The G6 value is 0.57.

Principal Axis Factoring

I next present the results of an exploratory factor analysis using principal axis factoring.1
This analysis, which relied upon tetrachoric correlations and varimax rotation, favored the
existence of two principal axes. On the results of this two-axis model we fail to reject the
null hypothesis of two factors (𝜒 2 = 0.35, p = 0.55). We see in Fig. 1 below the loadings of
each item on these axes (with item numbers corresponding to the row position in Table 5).
The results show that the identifaction of the House minority party, the question most
frequently correctly answered by the sample (𝜇 = 0.826), loads strongly onto a second
axis, while the other items are closely associated to a first.

Factor Plot, Question Battery for Study 1

Next, in Fig. 2, we see a factor analysis diagram confirming the notion that the house
minority party question appears to be distinct from the other questions. It is unclear what
this second principal axis might indicate, though it is possible that the increased rate of

1Factor analysis was performed using the 𝑝𝑠𝑦𝑐ℎ library in R. The 𝑝𝑎 command was used to
generate the results presented in the discussion and graphics below.
correct responses indicates that this question taps baseline current events knowledge.
Because the question was known to so many respondents, it is possible that this rationale
finds support. Nevertheless, we are encouraged by the strong similarity of the other four
items, as well as the acceptable Cronbach's Alpha measure.

Factor Analysis Diagram, Knowledge Questions, Study 1

Lastly, a parallel analysis corroborates the preceding discussion by demonstrating the
existence of a fairly evident "elbow" in the Scree plot at 2 principal axes.
Parallel Analysis Scree Plot, Knowledge Questions, Study 1
As a further point of analysis, we might additionally investigate the correlates of
performance on the five knowledge items. Table 6 demonstrates above all that education is
positively correlated with success on the overall knowledge battery, as well as the five
individual questions. This dearth of variation in the demographic predictors of success on
each question lends further support to the validity of the measure.

Power Analysis
A power analysis (seen below in Fig. 2) demonstrates that simple two-sample t-test
comparisons are adequately powered if we assume the expected effect size of a particular
treatment is between 0.5 and 0.3; only expected effect sizes of 0.2 or lower (which are
trivial in nature) would be inappropriately underpowered. The dashed horizontal line
shows an Alpha level of 0.95, meaning that 95% confidence intervals are expected to
produce significant estimates of group differences when such differences exist. As the lines
for effect size estimates of 0.3 to 0.5 intersect this line, we see that sample sizes between
105 to 290 respondents per group are preferred. In no test in the study are any control or
treatment groups smaller than 105, even when engaging in comparisons across subsets of
respondents.

Power over N across Effect Size Estimates, Two-Sample t-Test Comparisons

Study 2: Peer Evaluations

Question Wording
• Please indicate your AGE in years by clicking on the pull-down tab below.
– (Pull-down)
• Which best describes your gender?
– Male (0)
– Female (1)
• What is the highest level of education you have completed?
– Less than High School (1)
– High School Graduate/GED (2)
– Some College (3)
– 2-year College Degree (4)
– 4-year College Degree (5)
– Post-Graduate Degree (6)
• What racial or ethnic group best describes you?
– White (0)
 – Black (1)
– Hispanic (1)
– Asian (1)
– Native American (1)
– Middle Eastern (1)
– Mixed (1)
– Another racial or ethnic group (1)
• Thinking about politics these days, how would you describe your own political
viewpoint?
– Very Liberal (1)
– Liberal (2)
– Moderate (3)
– Conservative (4)
– Very Conservative (5)
– Not Sure (.)
• Some people seem to follow what's going on in government and public affairs most of
the time, whether there's an election going on or not. Others aren't that interested.
Would you say you follow what's going on in government and public affairs...?
– Most of the time (1)
– Some of the time (2)
– Only now and then (3)
– Hardly at all (4)
• How many days in the past week did you discuss politics with family or friends?
– Zero days (0)
– One day (1)
– Two days (2)
– Three days (3)
– Four days (4)
– Five days (5)
– Six days (6)
– Seven days (7)
• Thinking back over the last year, what was your household's annual income?
– Less than $10,000 (1)
– $10,000 - $19,999 (2)
– $20,000 - $29,999 (3)
– $30,000 - $39,999 (4)
– $40,000 - $49,999 (5)
– $50,000 - $59,999 (6)
– $60,000 - $69,999 (7)
– $70,000 - $79,999 (8)
– $80,000 - $99,999 (9)
 – $100,000 - $119,999 (10)
– $120,000 - $149,999 (11)
– More than $150,000 (12)
• Generally speaking, do you usually think of yourself as a Republican, a Democrat, an
Independent, or what?
– Republican (1)
– Democrat (2)
– Independent (3)
– Something else (4)
• (If 1 is selected): Would you call yourself a strong Republican or a not very strong
Republican?
– Strong Republican (1)
– Not very strong Republican (2)
• (If 2 is selected): Would you call yourself a strong Democrat or a not very strong
Democrat?
– Strong Democrat (1)
– Not very strong Democrat (2)
• (If 3 is selected:) Do you think of yourself as closer to the Republican Party or to the
Democratic Party?
– Closer to the Republican (1)
– Closer to the Democratic (2)
– Neither (3)

Political Quiz
• Now we would like you to respond to a series of questions with right and wrong
answers. Please try to answer this Political Quiz to the best of your ability. There is NO
penalty for incorrect answers. Please do not cheat in any way on this quiz.
• For how many years is a United States Senator elected--that is, how many years are
there in one full term of office for a U.S. Senator? [randomize]
– 6 years (1)
– 4 years (0)
– 2 years (0)
– 8 years (0)
• On which of the following does the U.S. federal government currently spend the
LEAST? [randomize]
– Foreign Aid (1)
– Medicare (0)
– National Defense (0)
– Social Security (0)
• Do you happen to know which party currently has the FEWEST members in the U.S.
House of Representatives? [randomize]
– Democrats (1)
 – Republicans (0)
• Which political party is more conservative when it comes to healthcare policy?
[randomize]
– Democratic Party (0)
– Republican Party (1)
– They are about the same (0)
• Who is the current U.S. Secretary of Energy? [randomize]
– Rex Tillerson (0)
– Rick Perry (1)
– Elaine Chao (0)
– Ben Carson (0)
– Steven Mnuchin (0)
• Next we would like you to rate your own performance relative to everyone who has
taken the test. Please use the sliding scale to evaluate your performance.

Peer Evaluation
• Now we'd like you to take a look at another respondent's answers. In a moment, we'll
ask you to grade that respondent's performance.
• Here is some information about the respondent you will be grading. Please look over
their answers carefully. (Treatment 1)
– Participant #0001272

– Age: 36
– Gender: Male
– Political Party: Democrat
• Here is some information about the respondent you will be grading. Please look over
their answers carefully. (Treatment 2)
– Participant #0001272

– Age: 36
– Gender: Male
– Political Party: Republican
• What is the political party identification of Participant #0001272?
– Republican (1)
– Democrat (2)
– Independent (3)
• Test Answers for Participant #0001272Q:
– For how many years is a United States Senator elected--that is, how many years
are there in one full term of office for a U.S. Senator? Respondent's Answer: 6
years
– Q: On which of the following does the U.S. federal government currently spend
the LEAST? Respondent's Answer: Foreign Aid
 – Q: Which party currently has the FEWEST members in the U.S. House of
Representatives? Respondent's Answer: Democrats
– Q: Which political party is more conservative when it comes to healthcare
policy? Respondent's Answer: Republicans
– Q: Who is the current U.S. Secretary of Energy? Respondent's Answer: Rick
Perry
• How would you rate Participant #0001272's performance on the test?
– Excellent (5)
– Above Average (4)
– Average (3)
– Below Average (2)
– Poor (1)

Table A7: Descriptive Statistics Summary, Study 2

Statistic N Mean St. Dev. Min Max

Age 1559 35.401 11.393 18 78

Democrat 1559 0.453 0.498 0 1
Republican 1559 0.198 0.398 0 1
Nonwhite 1559 0.239 0.427 0 1
Education 1559 4.162 1.295 1 6
Ideology (5-pt.) 1559 2.667 1.115 1 5
Pol. Interest 1559 3.212 0.785 1 4
Pol. Discussion 1556 2.716 2.005 0 7
Income 1559 5.777 2.938 1 12
 Table A8: Linear Regression Models Predicting Perceived Performance by Quiz
Score, Study 2 (Corresponds to Fig. 5 in Main Text)

Zero Correct 2.136***

(0.210)
One Correct 2.683***
(0.109)
Two Correct 2.885***
(0.060)
Three Correct 3.079***
(0.046)
Four Correct 3.462***
(0.048)
Five Correct 4.200***
(0.056)
N 1557
R-squared 0.922
Adj. R-squared 0.922
Residual Std. Error 0.984 (df = 1551)
3050.352*** (df =
F Statistic
6; 1551)

p < .01; p < .05; p < .1

Note: To show all group estimates, this is a zero-intercept model.
It presents each group’s estimated score and tests the null
hypothesis that a given estimate is zero.
Table A9: Linear Regression Models Predicting Perceived Peer Performance, Study
2 (Corresponds to Fig. 6 in Main Text)

Model 1 Model 2 Model 3

Rep. Peer -0.007 0.008 -0.107

(0.046) (0.166) (0.271)
Republican -0.035 -0.019
(0.062) (0.277)
Democrat 0.026 -0.076
(0.049) (0.238)
Quartile 2 0.186 0.112
(0.122) (0.202)
Quartile 3 0.320*** 0.294
(0.114) (0.192)
Quartile 4 0.846*** 0.783***
(0.110) (0.185)
Rep. Peer: Quartile 2 0.046 0.202
(0.194) (0.318)
Rep. Peer: Quartile 3 -0.032 0.071
(0.183) (0.301)
Rep. Peer: Quartile 4 0.042 0.311
(0.177) (0.292)
Rep. Peer: Republican 0.126
(0.413)
Rep. Peer: Democrat 0.233
(0.400)
Quartile 2: Republican -0.158
(0.331)
Quartile 3: Republican -0.033
(0.306)
Quartile 4: Republican 0.054
(0.296)
Quartile 2: Democrat 0.237
 (0.276)
Quartile 3: Democrat 0.078
(0.261)
Quartile 4: Democrat 0.119
(0.251)
Rep. Peer: Q2
0.383
Republican
(0.502)
Rep. Peer: Q3
-0.049
Republican
(0.467)
Rep. Peer: Q4
-0.423
Republican
(0.452)
Rep. Peer: Q2 Democrat -0.549
(0.460)
Rep. Peer: Q3 Democrat -0.244
(0.437)
Rep. Peer: Q4 Democrat -0.441
(0.425)
Constant 3.746*** 3.214*** 3.250***
(0.041) (0.104) (0.174)
N 1473 1471 1471
R-squared 0.001 0.145 0.153
Adj. R-squared -0.001 0.141 0.139
Residual Std. Error 0.838 (df = 1469) 0.777 (df = 1463) 0.778 (df = 1447)
0.380 (df = 3; 35.369*** (df = 7; 11.329*** (df = 23;
F Statistic
1469) 1463) 1447)

p < .01; p < .05; p < .1

Evaluation of Political Knowledge Battery

Below I include a replication of the political knowledge evaluation seen above. We see
overall that the results of this second test of the political knowledge battery are similar to
the earlier examination, in that the findings suggest two underlying principal axes for the
knowledge battery. However, these axes differ somewhat from the earlier battery, a finding
which merits some discussion.
Table A10: Distribution of Quiz Response Accuracy, Study 2

Statistic N Mean St. Dev. Min Max

Term Length 1559 0.597 0.491 0 1

Spending 1559 0.486 0.500 0 1
Healthcare Ideology 1559 0.921 0.270 0 1
House Minority 1559 0.890 0.313 0 1
Energy Secretary 1557 0.451 0.498 0 1

Cronbach's Alpha
First we can assess the Cronbach's Alpha for Study 2, calculated on the basis of tetrachoric
correlations. This value is is 0.70 (the standardized Alpha is also 0.70). The G6 value is 0.68.
These measures provide increased confidence that the items in the knowledge battery are
related.
Factor Plot, Question Battery, Study 2
Fig. 5, however, demonstrates the results of a varimax-rotated principal axis factor analysis
calculated using a tetrachoric correlation matrix. This figure demonstrates again the
existence of two principal axes. In this case, the least-correctly-answered item, the name of
the current Secretary of Energy, appears to load strongly on this second factor. Two other
items, Senate term length and the spending question, occupy a middle space in the graphic,
with moderately high loadings on both axes.
As might be expected, we correspondingly fail to reject a statistical test of the null
hypothesis of two principal axes (𝜒 2 = 0.09, p = 0.74).
Factor Analysis Diagram, Question Battery, Study 2
Fig. 6 shows a factor analysis diagram identifying these two underlying dimensions. It
appears that again, these principal axes have a lot to do with the overall difficulty of the
questions. The two most difficult questions in the battery, the spending measure and the
identification of the energy secretary, are related to principal axis 2, while the rest of the
items are related to PA1.
Parallel Analysis Scree Plot, Study 2
Finally, a parallel analysis once again shows support for the findings presented above.