1

Stanovich and Wests (2007) Actively Open-Minded Thinking Scale: An Examination of Factor Structure Kimberly R. Marsh Dena A. Pastor James Madison University

Paper presented at the annual meeting of the Association for Psychological Science, Washington, D.C., May 2011

2 Abstract Research examining the divergence of human thinking from what is considered to be the ideal standard has indicated that human thinking is often prone to error and everyday thinking processes, judgments, and decision making are negatively affected by our own biased viewpoints. These cognitive shortcomings have inspired researchers to examine characteristics of what they deem to be good thinking such as actively open-minded thinking. In order to examine the relationship between characteristics of good thinking and performance on critical thinking tasks, Stanovich and West (2007) developed their own Actively Open-Minded Thinking (AOT) scale. The current study examined the factor structure of Stanovich and Wests (2007) AOT scale by examining multiple factor solutions via exploratory factor analysis. Results indicated that AOT items did not load on the six subscales originally developed by Stanovich and West (2007) nor were they unidimensional; instead, a 4-factor solution emerged. Theoretical and practical implications of these findings are discussed.

3 Stanovich and Wests (2007) Actively Open-Minded Thinking Scale: An Examination of Factor Structure There is an extensive body of research examining the divergence of human thinking from what is considered to be the ideal standard. Perhaps unsurprisingly, this research has indicated that human thinking is often prone to error and everyday thinking processes, judgments, and decision making are negatively affected by our own biased viewpoints. Knowledge of these common cognitive shortcomings has inspired research that seeks to identify aspects of good thinking practices. Ideally, in identifying aspects of good thinking, researchers in this domain may identify some individual skills or characteristics that could be cultivated and lend themselves to the improvement of reasoning. According to Baron (1991), good thinking is a complex process that incorporates, optimal search for possibilities, evidence, and goals, and fairness in the search for evidence and in inference (p. 172). Conversely, Baron characterizes aspects of poor thinking as inadequate search for evidence and the general tendency of people to allow their own biases to affect their reasoning (Baron, 1991, p. 172). Baron further delineates good and poor thinking by identifying what he believes to be the defining aspect of good thinking; that it is actively open-minded. This actively open-minded thinking component of good thinking incorporates a search for evidence, the extent to which is determined by the importance of the question, and an objectivity in ones consideration and review of the evidence (Baron, 1991). There has been some research conducted in the area of cognitive psychology that suggests the distinct nature of cognitive ability (i.e., general intelligence) and individual thinking dispositions, such as actively open-minded thinking (Baron, 1985, 1988, 1991; Ennis, 1987; Moshman, 1994; Norris, 1992; Perkins, Faraday, & Bushey, 1991). More specifically, there is

4 some evidence to suggest that each of these variables is able to account for unique variance in performance on critical thinking tasks, suggesting that cognitive ability and individual thinking dispositions may address different aspects of what is considered to be good thinking (Stanovich & West, 1997, 1998, 2007). Stanovich and West have devoted much of their research to examining the nature of the relationship between cognitive ability, individual thinking dispositions, such as actively open-minded thinking, and whether these constructs can uniquely predict performance on a number of cognitive tasks. For instance, Stanovich and West (1997) evaluated individual performance on an instrument they refer to as the Argument Evaluation Task. Participants were instructed to objectively evaluate the quality of a number of arguments crafted by a fictitious individual after previously indicating their own opinions and thoughts on the subject matter of these arguments (e.g., social and political issues, such as, crime, gun control). Stanovich and West (1997) found that after controlling for cognitive ability (e.g., selfreported SAT scores), individual thinking disposition variables, such as actively open-minded thinking accounted for a significant amount of variance in performance on the Argument Evaluation Task; ultimately providing more support for the argument that cognitive ability and individual thinking disposition variables may both contribute to characteristics of critical thinking. Stanovich and West (1997, 1998, 2007) have been instrumental in the development and utilization of multiple versions of their own Actively Open-minded Thinking scale (AOT). Stanovich and West (1997) conceptualized actively open-minded thinking as the tendency of people to engage in flexible or open-minded thinking and avoid dogmatic or categorical thinking. Stemming from this definition of actively open-minded thinking, six subscales were conceptualized for the current Stanovich and West (2007) 41 item version of the AOT scale:

5 Flexible thinking (Stanovich & West, 1997), Openness-Values (Costa & McCrae, 1992), Dogmatism (Paulhus & Reid, 1991; Robinson, Shaver, & Wrightsman, 1991; Troldahl & Powell, 1965), Categorical thinking (Epstein & Meier, 1989), Belief identification (S et al., 1999), and Counterfactual thinking (Stanovich & West, 1997). Although there have been multiple versions of the AOT scale (S, West, & Stanovich, 1999; Stanovich & West, 1997) consisting of different combinations of items and subscales leading up to the most current, 41 item version of Stanovich and Wests (2007) AOT scale, the rationale behind scale modifications and revisions is unclear. Purpose of the Current Study To date, no published factor analysis studies have been conducted on the most current, 41 item version of Stanovich and Wests (2007) AOT scale. Stanovich and West (2007) currently suggest summing participant item responses for the six subscales, indicating that they believe the actively open-minded construct to be unidimensional. The purpose of the current study is to perform exploratory factor analysis (EFA) on the most current version of the Stanovich and West (2007) AOT scale in order to examine the emergent factor structure. An EFA will inform the validity of the AOT scores by indicating the extent to which the relationships among items conform to Stanovich and Wests conceptualization of the construct, aid in identifying items not functioning as intended, and inform understanding of the actively open-minded thinking construct itself. Methods Participants and Procedures Undergraduate students from a mid-sized university participated in the current study. Participants registered to participate through the university subject pool and were awarded class credit for an introductory psychology course in exchange for their participation in the study.

6 Undergraduate research assistants conducted study sessions in an on-campus computer lab, running four participants at a time. Participants completed a series of measures in addition to the Stanovich and West (2007) AOT scale via Qualtrics Survey Software. A total of 452 students initially completed the survey, however, seven participants were listwise deleted due to missing data. The remaining sample (N = 445) was screened for both univariate and multivariate normality. Upon data screening, variable distributions were deemed univariate normal (skewness and kurtosis values falling within the acceptable range of |2| and |7|, respectively; Kline, 2005). Using DeCarlos (1997) macro to screen for multivariate outliers, 15 cases were flagged as outliers at the .05 significance level. These cases were removed from the data set, resulting in the final sample used for the subsequent EFA (N = 430). Of the 430 undergraduate students who were included within the analyses for the current study, the majority were first-years (70.2%), female (70.2%), and Caucasian (85.1%). Mean participant age was 19.00 years (SD = 1.43). Actively Open-minded Thinking (AOT) Scale. Participants completed Stanovich and Wests (2007) 41 item version of the AOT. This version of the AOT consists of six subscales including: Flexible thinking (Stanovich & West, 1997), Openness-Values (Costa & McCrae, 1992), Dogmatism (Paulhus & Reid, 1991; Robinson et al., 1991; Troldahl & Powell, 1965), Categorical thinking (Epstein & Meier, 1989), Belief identification (S et al., 1999), and Counterfactual thinking (Stanovich & West, 1997). Participants indicated their agreement with each of the 41 statements on a Likert scale consisting of 6 response options: Disagree Strongly (1), Disagree Moderately (2), Disagree Slightly (3), Agree Slightly (4), Agree Moderately (5), and Agree Strongly (6). Items were reflected in such a way that higher AOT scores indicated a

7 higher level of agreement with items that endorsed actively open-minded thinking. Cronbachs alpha for the Stanovich and West (2007) AOT scale was .83. Results Data Screening Descriptive statistics for each of the Stanovich and West (2007) AOT scale items are presented in Table 1. Referring to the estimates provided in Table 1, we can see that, overall, the distribution of the variables are univariate normal with skewness and kurtosis values falling within the acceptable range of |2| and |7|, respectively (Kline, 2005). It should be noted that these descriptive statistics are based upon the reduced sample (N = 430) utilized within the analyses after the data were screened and cleaned. AOT inter-item Pearson correlation coefficients ranged from -.21 to .48, indicating that multicollinearity was not an issue within the current sample. It should be noted that these interitem correlations are rather low; suggesting that AOT items are not substantially related to one another and thus factor analysis may be inappropriate (Nunnally & Bernstein, 1994). However, Bartletts test of sphericity, the Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO) index, and the Measure of Sampling Adequacy (MSA) indices were still examined in order to determine whether it was appropriate move forward with factor analysis. Bartletts test of sphericity ( 2(820) = 4088.23, p < .001) indicated that we should reject the null hypothesis that the AOT variables are completely uncorrelated, thus suggesting that the variables are inter-correlated and it is appropriate to run a parallel analysis procedure and conduct factor analysis (Gorsuch, 1983). The KMO index (KMO = .85) approached one, indicating that the correlation matrix was amenable to factoring. This KMO index value is considered to be meritorious by Kaisers (1974) standards, indicating that the AOT variables

8 are likely to share common factors. Based upon the values in the anti-image correlation matrix, we can conclude that the amenability of the variables to factoring is acceptable. Overall, the Measure of Sampling Adequacy (MSA) values along the diagonal in the anti-image correlation matrix were fairly high (approaching a value of one), indicating that each of the AOT variables share an adequate amount of variance with the remaining AOT variables. Furthermore, the absolute values on the off-diagonal of the anti-image correlation matrix were rather small (close to 0). That is, the negative partial correlations amongst the AOT variables were quite small, indicating that any combination of two AOT variables do not relate to one another after controlling for how these two variables relate to the remaining AOT variables. Exploratory Factor Analysis Both a scree plot of eigenvalues based on the reduced correlation matrix and the results of a parallel analysis (Horn, 1965) were examined in order to determine the number of factor solutions for review (see Figure 1). All possible AOT factor eigenvalues were plotted on the scree plot. Upon visual examination of the scree plot, it appears that plotted eigenvalues begin to level off after the first seven plotted eigenvalues, suggesting that these subsequent factors account for insignificant amounts of variance in AOT scores above and beyond that accounted for by the first seven factors. The results of the parallel analysis procedure mirror those of the visual examination of the scree plot suggesting the extraction of seven factors based upon the fact that the first seven eigenvalues extracted from the AOT data set are greater than those eigenvalues calculated based upon randomly generated data sets that have similar characteristics. Because of the exploratory nature of the analyses, all seven factor solutions (i.e., 1-factor, 2factor, 3-factor, etc.) were subsequently examined. However, only the results for the 4-factor

9 solution are presented and discussed in detail due to the fact that this emergent factor solution displayed what most closely resembles simple structure and was most interpretable. Table 2 displays the resulting pattern coefficients for the 4-factor EFA solution using Direct Oblimin rotation with delta set to zero. Pattern coefficients provide the most interpretable factor loading estimates as they reflect the correlation between each item and the factor after controlling for the inter-correlations amongst other items. Item loading criteria employed within the current study was |.40|, thus items were deemed to load on a factor if their pattern coefficient was greater than or equal to |.40|. Results indicated that AOT item functioning was problematic. There were nine items that failed to load on any of the factors across the seven factor solutions examined. Failure of these items to load on any factor may indicate that these items are confusing for respondents or they fail to measure similar aspects of the actively open-minded thinking construct. There were nine additional items that failed to load on any of the four factors in the 4-factor solution. This may also suggest that these items are problematic for respondents. There were 13 items that loaded solely on the first factor of the 4-factor solution and accounted for 15.84% of the total variance in AOT scores. Nine of the 13 items that loaded on the first factor in the 4-factor solution consistently loaded on the first factor across multiple solutions. Overall, these items appeared to represent participant proclivity to engage in a kind of rigid, black and white thinking style that is characterized by dogmatic and categorical thinking. This suggests that these previously hypothesized aspects of actively open-minded thinking merged together to form one factor. There were five items that loaded solely on the second factor of the 4-factor solution and accounted for 4.33% of the total variance in AOT scores. Overall, items that loaded on this

10 second emergent factor appeared to reflect aspects of flexible, open-minded thinking. Similar to those items that loaded on factor 1, the most salient item loadings on factor 2 were items that were originally identified as two aspects of the actively open-minded thinking construct, flexible thinking and openness- values; thus, it appears as though these two previously hypothesized aspects of actively open-minded thinking have merged to form the second emergent factor in the 4-factor solution. Items that loaded on factors 3 and 4 were relatively minor, or secondary, in respect to the first two emergent factors of the 4-factor solution. There were three items that loaded on factor 3 and accounted for 3.36% of the total variance in AOT scores. Unlike the first two emergent factors in the 4-factor solution, these items did not appear to be the result of previously specified aspects of the actively open-minded thinking construct merging to form one factor. Moreover, these items appear to reflect a perseverance in your beliefs factor not originally hypothesized. Finally, there were only two items that loaded on factor 4 and accounted for 2.63% of the total variance in AOT scores. These two items were originally specified as the counterfactual thinking subscale by Stanovich and West (1997, 2007). Although these items load on their originally hypothesized factor, they do not account for much variance in AOT scores suggesting that counterfactual thinking may not be a central component of actively open-minded thinking. Inter-factor correlations for the 4-factor solution were rather low, ranging from .05 to .23; providing further evidence to suggest the distinct nature of AOT items statements that load on each of the 4 emergent factors. AOT item communalities reported in Table 2 range from .06 to .48. Item communality estimates indicate the amount of common variance in the item explained by the factor solution. These item communality estimates are quite low. We would hope that these item communality estimates would be higher, indicating that the set of four emergent

11 factors explains a substantial amount of common variance within a specific item; however, this is not the case. The total variance accounted for in AOT score by the four emergent factors combined was 26.15%. It should be noted that this estimate of total variance reflects overlap amongst the factors as the rotation method allowed them to inter-correlate, thus it reflects a slight overestimate of the variance accounted for by the factors. The total variance estimate is rather low, leaving approximately 74% of the variance in AOT scale scores unaccounted for by the four emergent factors. The large amount of variance left unaccounted for in AOT scores could have several implications for the current version of the AOT scale. First, item wording could be such that it confuses participants, causing them to respond in a contradictory manner to items meant to address similar aspects of the construct. Secondly, item statements could be measuring distinct concepts that do not covary, or relate in such a way, to adequately represent Stanovich and Wests (1997, 2007) original conceptualization of actively open-minded thinking. Discussion The purpose of the current study was to examine the emergent factor structure of Stanovich and Wests (2007) AOT scale in an effort to determine whether scores on the AOT accurately represent actively open-minded thinking as conceptualized by Stanovich and West (1997, 2007). Stanovich and West (2007) suggest calculating an AOT scale total score by summing participant responses across item statements originally developed to address six facets of the actively open-minded thinking construct, including, dogmatic thinking, categorical thinking, flexible thinking, openness to values, belief identification, and counterfactual thinking. The suggested calculation of an AOT total score implies that Stanovich and West believe the scale to be unidimensional. However, this does not appear to be the case based upon the findings

12 of the current study. AOT items did not load on the six subscales originally developed by Stanovich and West (2007) nor were they unidimensional; instead, a 4-factor solution emerged. EFA results not only indicated that calculation of an AOT total score is inappropriate, but that the current version of the Stanovich and West (2007) AOT scale needs to be revised and conceptualization of the actively open-minded thinking construct revisited. Examination of the AOT inter-item correlation matrix revealed that inter-item correlations were quite low, indicating that the 41 items currently on the AOT scale may be measuring distinct, or seemingly unrelated concepts. Given the low inter-item correlations, it is not surprising that although the EFA resulted in an interpretable 4-factor solution, there were a series of problematic items identified in need of substantial revision or removal (e.g., items that failed to load across factor solutions, items that failed to load on any of the emergent four factors). In addition, the emergent factors failed to adequately represent theorized aspects of actively open-minded thinking (e.g., factor 3 which did not appear to represent a theorized aspect of actively open-minded thinking, factor 4 which only consisted of two items). Moreover, estimates regarding the percentage of variance accounted for in AOT scores by each of the four factors, item communalities, and total variance accounted for in AOT scores by all four factors further support the distinct nature of AOT item statements. Confusing AOT scale item content and/or wording may contribute to the distinctness of the item statements and, ultimately, the failure of items to form factors, particularly factors that are able to explain a large percentage of variance in scores. For example, one of the nine items that failed to load on any of the factors across multiple solutions reads, Even though freedom of speech for all groups is a worthwhile goal, it is unfortunately necessary to restrict the freedom of certain political groups. The wording for this particular item is confusing and it may be difficult

13 for respondents to decipher the meaning of the statement. Another item statement that failed to load on any factor solution reads, My blood boils over whenever a person stubbornly refuses to admit hes wrong. This particular item does not seem to relate to the other items on the scale in any logical or intuitive way. There appear to be other aspects of AOT items that pose additional threats to comprehension, such as wording that encourages respondents to misconstrue subject matter. For example, there are several items that inquire about ones beliefs or opinions about lifestyles (e.g., It is important to persevere in your beliefs even when evidence is brought to bear against them, I consider myself broad-minded and tolerant of peoples lifestyles). Wording such as this may cause the respondent to think of religious beliefs or about ones lifestyle in regards to sexual orientation, thus causing participants to respond to the items in differential manners based upon their own interpretation of the item content. There is evidence to suggest that respondents are interpreting items in different ways. Zilberberg and Pastor (2009) conducted a study in which they asked respondents to think aloud as there were responding to a subset of items statements from the AOT scale. The think aloud exercise revealed that item wording was ambiguous and participants had differential conceptualizations of item subject matter (Zilberberg & Pastor, 2009). The emergent 4-factor solution offered insight regarding how AOT scores reflect Stanovich and Wests (2007) conceptualization of actively open-minded thinking. As mentioned previously, Stanovich and West (1997, 2007) originally drafted items to reflect six facets of actively open-minded thinking, but two of the four factors within the emergent 4-factor solution appeared to be the result of a merging of originally hypothesized subscales. More specifically, factor 1 seemed to represent a merging of Stanovich and Wests (2007) originally specified dogmatic and categorical thinking subscales, with items from each subscale loading on the

14 factor. These items reflected a similar, concrete or dichotomizing thinking style. Similar to factor 1, factor 2 appeared to be the result of a merging of previously specified subscales. Items loading on factor 2 appeared to represent the merging of the flexible thinking and openness to values subscales originally specified by Stanovich and West (2007), with items from both subscales loading on the factor to represent an emergent factor that reflected a flexible/open-minded thinking style. One could make the argument that none of the items loading on the four factors in the emergent 4-factor solution address central components of actively open-minded thinking previously detailed by researchers who have defined the construct. However, items loading on factors 3 and 4 in the 4-factor solution seem to be addressing thinking characteristics particularly tangential to the actively open-minded thinking construct. More specifically, items loading on factor 3 appeared to represent a previously unspecified aspect of Stanovich and Wests (2007) concept of actively open-minded thinking construct, with these items reflecting perseverance in ones beliefs. Factor 4 was defined by only two items with salient factor loadings that reflected the propensity of individuals to engage in counterfactual thinking. Although these two items that loaded on factor 4 were the two items originally developed to represent Stanovich and Wests (2007) counterfactual thinking subscale, it is unclear as to whether they should represent core aspects of actively open-minded thinking. The presence of these two factors (factors 3 and 4) in particular raises the question of what are the essential components of actively open-minded thinking. Stanovich and West (1997) provide one way of thinking about actively open-minded thinking when they described it as the tendency of people to engage in flexible or open-minded thinking and avoid dogmatic or categorical thinking. It was this conceptualization of actively

15 open-minded thinking that inspired and informed the development of their AOT scale. Other researchers, such as Baron, conceptualize actively open-minded thinking differently. Baron (1991) characterizes actively open-minded thinking as a search for evidence, the extent to which is determined by the importance of the question, and an objectivity in ones consideration and review of the evidence. An instrument measuring actively open-minded thinking as described by Baron would look quite different than the Stanovich and West (2007) AOT scale. As mentioned previously, it could be argued that the components of Stanovich and Wests (2007) instrument might be related to actively open-minded thinking, but are not necessarily essential features of the construct. The Stanovich and West (2007) AOT scale has been used in many research endeavors, however, development and purpose of the scale is relatively unclear. Did Stanovich and West set out to develop a psychometrically sound (i.e., valid and reliable) measure of respondents propensity to engage in actively open-minded thinking? Or was their intent to develop a measure that evaluates tangentially related aspects of actively open-minded thinking that relates well to other measures of similar characteristics (e.g., cognitive ability, need for cognition, argument evaluation)? One way of viewing the current Stanovich and West (2007) AOT instrument, consistent with the latter question, is that it is comprised of features related to actively openminded thinking, but not necessarily essential to the definition of the construct. The development of the instrument may have been guided by the goal of measuring things related to good thinking, not necessarily by the intention to reliably and validity measure actively open-minded thinking itself. It is therefore not surprising that scores on the AOT scale relates to other constructs in expected ways; however these relationships are between aspects related to actively open-mined thinking and other constructs, not between the construct and actively open-minded

16 thinking itself. If the AOT scale was developed with the first purpose in mind, we recommend that researchers review the literature and revisit the conceptualization of the actively openminded thinking construct in order to develop a more valid and reliable scale that reflects core, theoretical components of actively open-minded thinking. This process might begin with revisiting Barons conceptualization and initiating the development of a measure of actively open-minded thinking that directly reflects this characteristic of good thinking.

17
Table 1 Item Descriptive Statistics
Item 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 M 4.07 3.44 4.67 5.40 4.54 5.26 3.86 4.82 3.71 4.77 4.30 3.17 3.16 4.97 3.27 4.15 4.67 4.91 3.08 3.94 3.80 4.66 4.17 4.87 3.93 4.43 4.32 3.78 4.52 4.56 2.60 3.64 4.65 3.96 3.76 3.36 3.96 4.96 4.89 3.22 4.51 SD 1.37 1.19 1.19 0.80 1.28 0.98 1.50 1.17 1.41 0.94 1.41 1.08 1.37 1.20 1.27 1.19 1.26 0.98 1.28 1.19 1.15 1.01 1.23 1.25 1.46 1.01 1.02 1.41 0.95 1.01 1.13 0.95 1.06 1.24 1.19 1.01 1.06 1.15 1.06 1.05 0.93 Skewness -0.20 0.03 -0.80 -1.65 -0.76 -1.56 -0.07 -1.04 0.10 -0.82 -0.44 0.09 0.09 -1.08 0.10 -0.15 -0.79 -1.22 0.18 -0.18 0.11 -0.51 -0.15 -1.24 -0.34 -0.30 -0.49 -0.29 -0.16 -0.40 0.55 -0.03 -0.68 -0.34 0.08 0.30 -0.08 -1.00 -0.86 0.49 -0.37 Kurtosis -0.93 -0.43 0.04 3.79 0.04 2.45 -1.10 0.75 -0.93 1.14 -0.82 -0.13 -0.86 0.38 -0.61 -0.72 -0.01 2.21 -0.51 -0.41 -0.50 0.03 -0.58 1.18 -0.84 -0.25 0.40 -0.76 -0.43 0.16 0.12 0.82 0.50 -0.33 -0.54 0.04 0.12 0.33 0.37 0.34 0.60

Note. N = 430.

18
Table 2 4-Factor Solution Pattern Coefficients, Item Communalities, and Variance Accounted for by Each Factor
Factor Item Item text 1 Even though freedom of speech for all groups is a worthwhile goal, it is unfortunately necessary to restrict the freedom of certain political groups. (R) 2 What beliefs you hold have more to do with your own personal character than the experiences that may have given rise to them. (R) 3 I tend to classify people as either for me or against me. (R) 4 5 6 7 8 9 A person should always consider new possibilities. There are two kinds of people in this world: those who are for the truth and those who are against the truth. (R) Changing your mind is a sign of weakness. (R) I believe we should look to our religious authorities for decisions on moral issues. (R) I think there are many wrong ways, but only one right way, to almost anything. (R) It makes me happy and proud when someone famous holds the same beliefs that I do. (R) 1 0.22 0.20 0.53 0.36 0.55 0.57 0.16 0.50 0.35 0.07 0.53 -0.32 0.11 0.45 0.04 0.44 0.52 0.24 0.08 0.49 0.36 0.59 0.42 2 0.05 -0.03 3 -0.04 0.13 4 0.05 0.18 Item communalities 0.06 0.11 0.27 0.27 0.36 0.33 0.25 0.37 0.20 0.15 0.29 0.21 0.19 0.27 0.38 0.29 0.33 0.23 0.36 0.29 0.26 0.36 0.48 0.37 0.47 0.09 0.28 0.30 0.13 0.30 0.13 0.15 0.37 0.20 0.20 0.09 0.27 0.30 0.41 0.12 0.27

-0.02 -0.03 -0.05 0.27 0.02 0.07 0.28 0.11 -0.01 0.32 -0.26 -0.02 0.12 0.06

-0.20 -0.01 0.31 0.19 0.21 -0.19 0.00 0.04 -0.06 0.03

10 Difficulties can usually be overcome by thinking about the problem, rather than through waiting for good fortune. 11 There are a number of people I have come to hate because of the things they stand for. (R) 12 Abandoning a previous belief is a sign of strong character. 13 No one can talk me out of something I know is right. (R) 14 Basically, I know everything I need to know about the important things in life. (R) 15 It is important to persevere in your beliefs even when evidence is brought to bear against them. (R) 16 Considering too many different opinions often leads to bad decisions. (R) 17 There are basically two kinds of people in this world, good and bad. (R) 18 I consider myself broad-minded and tolerant of other people's lifestyles. 19 Certain beliefs are just too important to abandon no matter how good a case can be made against them. (R) 20 Most people just don't know what's good for them. (R) 21 It is a noble thing when someone holds the same beliefs as their parents. (R) 22 Coming to decisions quickly is a sign of wisdom. (R) 23 I believe that loyalty to one's ideals and principles is more important than "open-mindedness." (R)

-0.12 -0.08 -0.23 0.37 0.01 0.21 0.07 0.07 0.34 0.12 -0.06 0.07 0.02 0.35 0.28 0.22 0.12 0.54 0.17 0.14 -0.07 0.00 0.15 0.05 0.08 -0.08 -0.06 0.40

-0.15 -0.11 0.56 0.17 0.28 -0.02 0.25 0.22 -0.07 -0.02 -0.02 0.08 0.02 0.10

24 Of all the different philosophies which exist in the world there is probably only one which is 0.36 correct. (R) 25 My beliefs would not have been very different if I had been raised by a different set of parents. (R) 0.03 26 If I think longer about a problem I will be more likely to solve it. 27 I believe that the different ideas of right and wrong that people in other societies have may be valid for them. 28 Even if my environment (family, neighborhood, schools) had been different, I probably would have the same religious views. (R) 29 There is nothing wrong with being undecided about many issues. 30 I believe that laws and social policies should change to reflect the needs of a changing world. 31 My blood boils over whenever a person stubbornly refuses to admit he's wrong. (R) 32 I believe that the "new morality" of permissiveness is no morality at all. (R) 33 One should disregard evidence that conflicts with your established beliefs. (R) 34 Someone who attacks my beliefs is not insulting me personally. 35 A group which tolerates too much difference of opinion among its members cannot exist for long. (R) 36 Often, when people criticize me, they don't have their facts straight. (R) 37 Beliefs should always be revised in response to new information or evidence. 38 I think that if people don't know what they believe in by the time they're 25, there's something wrong with them. (R) 39 I believe letting students hear controversial speakers can only confuse and mislead them. (R) 40 Intuition is the best guide in making decisions. (R) 41 People should always take into consideration evidence that goes against their beliefs. Percent of Variance Accounted for by each Factor 0.03 0.09 0.06 0.19 0.01 0.15 0.29 0.35 0.04 0.37 0.25

-0.15 -0.05 0.68 0.28 0.49 -0.06 0.26 0.53 -0.17 0.05 0.26 0.42 0.13 -0.12 -0.08 -0.06 0.04 0.02 0.07 0.54

-0.12 -0.06 0.05 0.14 0.16 0.15 0.05 0.09 0.12 0.15 -0.15 -0.04 0.25 0.08 3.36 0.05 -0.26 0.07 0.23 -0.09 -0.01 -0.05 0.05 0.04 0.25 0.15 0.05 2.63

-0.15 0.51 0.56 0.50 0.08 0.06 15.84 0.00 0.19 -0.11 0.48 4.33

Note. N = 430. (R) indicates that item is reflected.

19
Figure 1. AOT Scale EFA Parallel Analysis Results 7.00

6.00

5.00

4.00 Raw Data Eigenvalues 3.00 Generated Data Eigenvalues

2.00

1.00

0.00 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Figure 1. AOT scale EFA parallel analysis results. N = 430.

20 References Baron, J. (1985). Rationality and intelligence. New York: Cambridge University Press. Baron, J. (1988). Thinking and deciding. New York: Cambridge University Press. Baron, J. (1991). Beliefs about thinking. In J. F. Voss, D. N. Perkins, & J. W. Segal (Eds.), Informal reasoning and education, pp. 169-186. Hillsdale, N. J. : Erlbaum. Costa, P. T., & McCrae, R. R. (1992). Revised NEO personality inventory. Odessa, FL: Psychological Assessment Resources. DeCarlo, L. T. (1997). On the meaning and use of kurtosis. Psychological Methods, 3(2), 292307. Ennis, R. H. (1987). A taxonomy of critical thinking dispositions and abilities. In J. Baron & R. Sternberg (Eds.), Teaching thinking skills: Theory and practice (pp. 9-26) New York: Freeman. Epstein, S., & Meier, P. (1989). Constructive thinking: A broad coping variable with specific components. Journal of Personality and Social Psychology, 57, 332 350. Gorsuch, R.L. (1983). Factor Analysis, 2nd Edition, Hillsdale, N.J.: Erlbaum. Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185. Kaiser, H.F., 1974. Index of factorial simplicity. Psychometrika 39, 3136. Kline, R. B. (2005). Principles and Practice of Structural Equation Modeling (2nd Edition). New York: Guilford.

21 Moshman, D. (1994). Reasoning, metareasoning, and the promotion of rationality. In A. Demetriou & A. Efklides (Eds.), Intelligence, mind, and reasoning: Structure and development (pp. 135-150), Amsterdam: Elsevier. Norris, S. P. (1992). Testing for the disposition to think critically. Informal Logic, 14, 157-164. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGrawHill. Paulhus, D. L., & Reid, D. B. (1991). Enhancement and denial in socially desirable responding. Journal of Personality and Social Psychology, 60, 307 317. Perkins, D. N., Faraday, M., & Bushey, B. (1991). Everyday reasoning and the roots of intelligence. In J. Voss, D. Perkins, & J. Segal (Eds.), Informal reasoning and education (pp. 83-105). Hillsdale, NJ: Erlbaum. Robinson, J. P., Shaver, P. R., & Wrightsman, L. S. (1991). Measures of Personality and Social Psychological Attitudes (Vol. 1, pp. 560 564). San Diego, CA: Academic Press. S, W. C., West, R. F., & Stanovich K. E. (1999). The domain specificity and generality of belief bias: Searching for generalizable critical thinking skill. Journal of Educational Psychology, 91, 497-510. Stanovich, K. E., & West, R. F. (1997). Reasoning independently of prior belief and individual differences in actively open-minded thinking. Journal of Educational Psychology, 89(2), 342-357. Stanovich, K. E., & West, R. F. (1998). Individual differences in rational thought. Journal of Experimental Psychology, 127 (2), 161-188. Stanovich, K. E. & West, R. F. (2007). Natural myside bias is independent of cognitive ability. Thinking & Reasoning, 13, 225-247.

22 Troldahl, V., & Powell, F. (1965). A short-form dogmatism scale for use in field studies. Social Forces, 44, 211 215. Zilberberg, A. & Pastor D. A. (2009). Willingness to consider contradictory evidence scale: A mixed methods approach. Paper presented at the Northeastern Educational Research Association, Rocky Hill, CT.