EngineeringEducation
Virginia Tech
January 2009
Journalof EngineeringEducation 53
Journalof EngineeringEducation
Variables
Analyses
Predictive
Analyses
Categorical
Contingency Tables
Logistic regression
ANOVA, MANOVA,
t-tests Pearson's
Correlation
Discriminant
Analysis
Linear Regression
Multiple Linear
Regression
Interpretivist
(constructivism,
social
constructionism,
hermeneutics,
phenomenology)
Theoretical
perspective
Postpositivist
View on
reality
Single
falsifiable
reality
Multiple subjective
realities
Multiple
subjective
and political
realities
Multiple fragmented
realities
Purpose
To find
relationships
among
variables, to
define
cause-and
effect
To describe a
situation, experience,
or phenomenon
To produce
a sociopolitical
critique
To deconstruct existing
'grand narratives'
Methods
Methods
and
variables
defined in
advance,
hypothesis
driven
Methods and
approaches emerge
and are to be
adjusted during study
Methods
and
approaches
designed to
capture
inequities
Methods and
approaches generated
during the study
The role
of
researcher
Researcher
is detached
Researcher and
participants are
partners
Researcher
and
participants
are activists
Researchers and
participants have
various changing roles
Outcome
or
research
product
Context-free
generalizations
Situated descriptions
Critical
essays,
policy
changes
Reconceptualized
descriptions of the
phenomenon
Critical/
emancipatory
Postmodern/poststructural
been conducted from a post-positivist perspective, often in combination with the collection of quantitative data. These mixed methods studies are described further in Section IV. More commonly,
however, qualitative research uses alternate perspectives, which are
summarized in Table 2, taken from Koro-Ljungberg and Douglas
(2008). For a full discussion of these perspectives see Crotty (2003),
and for descriptions in the context of engineering see KoroLjungberg and Douglas (2008) and Chism et al. (2008). Examples
of how these perspectives are used exist within the engineering education literature. Klukken, Parsons, and Columbus (1997) used a
phenomenological interpretive perspective to describe the experience of being a creative engineer; Donath et al. (2005) used the
interpretive perspective of social constructionism to examine how
knowledge is created within an active learning group; and Tonso
(1996) utilized a critical feminist perspective to examine how the
discourse in a design class defines gender roles. As with all aspects of
the research design, the theoretical perspective one chooses,
whether positivist, interpretivist, or critical, is ultimately driven by,
and must be consistent with, the research questions of the study.
B. The Role ofTheory
The role of theory (a description or explanation of a phenomenon) is very different in qualitative and quantitative research. As
stated earlier, in quantitative studies, theory is utilized early in the
research design to identify hypotheses and to select appropriate
measurement instruments. In contrast, the use of theory in qualitative research comes much later, if at all, as a lens through which the
findings can be interpreted. In general, qualitative research takes an
inductive approach to data analysis. The data are examined without
preconceptions as to existing theory or pre-determined categories,
allowing themes or categories to emerge from the data. In order to
accomplish this, potential biases of the researcher are examined and
made known (in phenomenology, the process of identifying biases
and then setting them aside is called "bracketing"). This process allows new insight that would not be possible if an existing theory or
concept were imposed on the data. Klukken et al. (1997) used a
phenomenological approach to identify the characteristics of creative engineers, the first step towards building a theory of creativity
in engineering. Even when theory development is not the goal of a
study, one of the strengths of qualitative research is that its open,
emerging approach to the conduct of the research allows new phenomena to be identified which would not have been expected (and
thus not identified if a priorihypotheses or specific measurement
instruments drove the research). As one example, McLoughlin
(2005) identified a new type of gender bias in her study of women in
engineering. Because this type of bias had not been previously identified, a quantitative study of gender bias would not have revealed it.
Only by examining the words of the participants and interpreting
those words with qualitative analysis techniques was McLoughlin
able to identify a new phenomenon. Her results now provide
opportunities for additional qualitative and quantitative studies to
examine this phenomenon further.
C. Sampling
In comparison to quantitative studies, with their emphasis on
large, representative samples, qualitative research focuses on smaller
groups in order to examine a particular context in great detail. The
goal is not to provide a broad, generalizable description that is representative of most situations, but rather to describe a particular
January 2009
Design Type
Timing of quan
and qual
phases
Relative
weighting of
quan and qual
components
Mixing - when
quan and qual
phases are
integrated
Triangulation
Concurrent
Equal
During
interpretation
or analysis
QUAN + QUAL
Embedded
Concurrent or
Sequential
Unequal
One is
embedded within
the other
QUAN(qual) or
QUAL(quan)
Explanatory
Sequential,
quan then qual
Usually quan is
given priority
Phase 1 informs
phase 2
Exploratory
Sequential,
qual then quan
Usually qual is
given priority
Phase 1 informs
phase 2
Notation
58 Journalof EngineeringEducation
Another recent example actually used mixed methods terminology to describe the overall project design:
The [Academic Pathways Study] uses a concurrent
triangulation mixed-methods design, in which both
qualitative and quantitative methods are employed to collect
and analyze data. The integration of results occurs during
the interpretation phase (Creswell et al., 2003), enabling
researchers to address a broad range of research questions
toward discerning complex phenomena like student learning
and development (Johnson and Onwuegbuzie, 2004)
(Kilgore et al., 2007, p. 323).
The article itself describes analysis of an open-ended design task,
with a thick description of the categories and their relationships, as
well as quantifying frequencies of various behaviors and strategies
employed by their participants (Kilgore et al., 2007).
B. Embedded Designs
Embedded designs are not distinguished by the concurrent or
sequential nature of data collection (either is allowed). Rather, one
type of data takes a supplemental role to the other. Creswell and Plano
Clark offer the criterion that a study is embedded if the secondary
data are not useflul or meaningful without the primary study (2007).
Following this criterion, we can identify several engineering education articles with very limited secondary components as embedded designs. As might be expected, engineering education studies
exhibited a strong preference for assigning priority to quantitative
data and analysis. For example, it was quite common to develop an
extensive control group or pre-test/post-test quantitative design to
assess learning and augment it with one or a few open-ended questions to students about their attitude toward the intervention
(Campbell et al., 2002; Lee, Castella, and Middleton, 1997; Morell
et al., 2001; Raju and Sankar, 1999; Weisner and Lan, 2004). The
extent to which authors presented the qualitative results ranges
from referencing another publication (Brown, Morning, and
January 2009
Watkins, 2005) to nearly a full page (Morell et al., 2001; Raju and
Sankar, 1999). Most included one or two paragraphs discussing
student attitudes and intervention logistics (Campbell et al., 2002;
Lee et al., 1997; Weisner and Lan, 2004).
C. Explanatory Designs
Explanatory mixed methods designs are characterized by an initial and extensive quantitative phase built upon by a subsequent
qualitative phase. Usually, the qualitative results serve to explain the
quantitative results. Integration occurs between phases, as the quantitative results often inform the questions or sampling in the second
phase (Creswell and Plano Clark, 2007; Creswell et al., 2003).
A few JEE articles provide hints that authors pursued a mixed
methods approach (likely explanatory) to provide complementary data:
As a complement to the statistical FCQ_analysis, two midsemester classroom interviews were conducted (Gall et al.,
2003, p. 340).
Finally, open-ended comments were analyzed as a
qualitative component to shed light on numerical results
(Hackett and Martin, 1998, p. 87).
Since we found no statistically significant ACT/SAT or
GPA data that could explain the dramatic increases in the
graduation rates of Connections students, we administered a
follow-up survey in the fall of 2000 asking participants in
the program to reflect about their experiences (Olds and
Miller, 2004, p. 30).
In a few cases, authors were explicit in how the first phase informed the second:
This study employed both quantitative and qualitative
analysis methods. Quantitative data came from statistical
analyses of hundreds of megabytes of access log files stored
on the web server.... Qualitative data came from user
interviews. ... Empirical findings from the quantitative
analysis helped to form some of the initial questions used
during the interviews (Liang, Bell, and Leifer, 2001, p. 520).
Here, quantitative results provided ideas for qualitative interview
questions. In another example, the quantitative phase identified interview participants (who had the highest number of entries in the
database), as well as providing findings for participants to react to in
interviews:
Readers will see that the publication analysis alone raises
many questions with respect to the motivation of the
publication authors. To provide additional context for
understanding the impact of the coalitions, selected
coalition personnel reflected on the quantitative analysis
results and described their perceptions of the work at the
time and in light of recent calls for rigor. For the interview
component of the study, the preliminary [quantitative]
results of the publication analysis were shared with coalition
leaders and first authors of the highest number of
publications (Borrego, 2007b, p. 9).
January 2009
Transferability: applicability of
research findings to other settings,
achieved through thick description
Journalof EngineeringEducation
REFERENCES
cation, eds. A.E. Kelly and R.A. Lesh, 429-456. Hillsdale, NJ:
Lawrence Erlbaum.
Foor, C.E., S.E. Walden, and D.A. Trytten. 2007. "I wish
that I belonged more in this whole engineering group:" Achieving
individual diversity. Journal of Engineering Education 96 (2):
103-15.
French, B.F.,J.C. Immekus, and W.C. Oakes. 2005. An examination of indicators of engineering students' success and persistence.
JournalofEngineeringEducation
94 (4): 419-25.
87 (1): 87-95.
Haines, V.A.,J.E. Wallace, and M.E. Cannon. 2001. Exploring
the gender gap in engineering: A re-specification and test of the
hypothesis of cumulative advantages and disadvantages. Journalof
EngineeringEducation 90 (4): 677-84.
Hawks, B.K., and J.Z. Spade. 1998. Women and men engineering students: Anticipation of family and work roles. Journalof
EngineeringEducation87 (3): 249-56.
1-7.
Heckel, R.W. 1995. Disciplinary patterns in degrees, faculty and
research funding.JournalofEngineeringEducation84 (1): 1-10.
Heckel, R.W. 1996. Engineering freshman enrollments: Critical and non-critical factors. Journal ofEngineeringEducation85 (1):
15-21.
Heywood, J. 2005. Engineering education.: Research and development in curriculum andinstruction. Hoboken, NJ: IEEE Press.
Howe, K.R. 1988. Against the quantitative-qualitative incompatibility thesis or dogmas die hard. EducationalResearcher 17 (8):
10-16.
Hsi, S., M.C. Linn, and J.E. Bell. 1997. The role of spatial reasoning in engineering and the design of spatial instruction. Journal
ofEngineeringEducation86 (2): 151-58.
321-32.
Jesiek, B.K., L. Newswander, and M. Borrego. 2009. Engineering education research: Field, community, or discipline? Journalof
EngineeringEducation 98 (1): 39-52.
January 2009
Moeller-Wong, C., and A. Eide. 1997. An engineering student retention study. Journalof Engineering Education 86 (1):
7-15.
Morell, L., R. Buxeda, M. Orengo, and A. Sanchez. 2001. After
so much effort: Is faculty using cooperative learning in the class90 (3): 357-62.
room?JournalofEngineeringEducation
Morse, J.M. 1991. Approaches to qualitative-quantitative
methodological triangulation. NursingResearch 40: 120-23.
Morse, J.M. 2003. Principles of mixed methods and multimethod research design. In Handbook of mixed methods in socialand
behavioralresearch, eds. A. Tashakkori and C. Teddlie, 189-208.
Thousand Oaks: Sage Publications.
Moskal, B.M., J.A. Leydens, and MJ. Pavelich. 2002. Validity,
reliability and the assessment of engineering education. Journalof
EngineeringEducation91 (3): 351-54.
Napp, J.B. 2004. Survey of library services at engineering news
record's top 500 design firms: Implications for engineering education.JournalofEngineeringEducation (3): 247-52.
Ogot, M., G. Elliott, G., and N. Glumac. 2003. An assessment
of in-person and remotely operated laboratories. Journalof Engineering Education 92 (1): 57-64.
Ohland, M.W., and G. Zhang. 2002. A study of the impact of
minority engineering programs at the FAMU-FSU college of engi91 (4): 435-40.
neering.JournalofEngineeringEducation
Olds, B.M., and R.L. Miller. 2004. The effect of a first-year integrated engineering curriculum on graduation rates and student
satisfaction: A longitudinal study. JournalofEngineeringEducation
93 (1): 23-35.
Olds, B.M., B.M. Moskal, and R.L. Miller. 2005. Assessment
in engineering education: Evolution, approaches and future collab94 (1): 13-25.
orations.JournalofEngineeringEducation
Parsons, C.K., E. Caylor, and H.S. Simmons. 2005. Cooperative education work assignments: The role of organizational and individual factors in enhancing ABET competencies and co-op
workplace well-being. Journal of Engineering Education 94 (3):
309-18.
Patton, M.Q. 2002. Qualitative research & evaluation methods.
Thousand Oaks, CA: Sage Publications.
Pavelich, MJ., and W.S. Moore. 1996. Measuring the effect of
experiential education using the Perry Model. JournalofEngineeringEducation85 (4): 287-92.
Peters, M., P. Chisholm, and B. Laeng. 1994. Spatial ability,
student gender, and academic performance. Journalof Engineering
Education 83 (1): 1-5.
Raju, P.K., and C.S. Sankar. 1999. Teaching real-world issues
through case studies. Journalof Engineering Education 88 (4):
501-08.
Rayne, K., T. Martin, S. Brophy, NJ. Kemp, J.D. Hart, and
K.R. Diller. 2006. The development of adaptive expertise in biomedical engineering ethics. JournalofEngineeringEducation 95 (2):
165-73.
Robinson, D.A.G., and B.A. Reilly. 1993. Women engineers:
A study of educational preparation and professional success.Journal
ofEngineeringEducation 82 (2): 78-82.
Rutz, E., R. Eckhart, J.E. Wade, C. Maltbie, C. Rafter, and V.
Elkins. 2003. Student performance and acceptance of instructional
technology: Comparing technology-enhanced and traditional
instruction for a course in statics. JournalofEngineeringEducation
92 (2): 133-40.
Journalof Engineering Education 65
Vanderburg, W.H., and N. Khan 1994. How well is engineering education incorporating societal issues? Journalof Engineering
685-93.
Sandelowski, M. 2003. Tables or tableux? The challenges
of
writing and reading mixed methods studies. In Handbook of mixed
neering.JournalofEngineeringEducation
92 (2): 167-79.
modes for introductory computing. JournalofEngineeringEducation 2002. The scholarship of teaching and learning in engineering. In
94 (4): 355-62.
Disciplinarystyles in the scholarshipofteachingand learning:Exploring
tionalResearcber3l(8): 25-27.
Sage Publications.
Tebbs, J.M., K.M. Bower. 2003. Some comments
on the ro-
AUTHORS' BIOGRAPHIES
92 (1): 91-94.
Teddlie, C., and A. Tashakkori. 2003. Major issues and controversies in the use of mixed methods in the social and behavioral sci-
Maura Borrego is an assistant professor of Engineering Education at Virginia Tech. Dr. Borrego holds an M.S. and Ph.D. in
Materials Science and Engineering from Stanford University. Her
ences. In Handbook ofmixed methods in socialand behavioralresearch, current research interests center around interdisciplinary collaboeds. A. Tashakkori and C. Teddlie, 3-50. Thousand Oaks, CA: ration in engineering and engineering education. Funded by her
Sage Publications.
CAREER grant, she also studies interdisciplinarity in engineering
Terenzini, P.T., A.F. Cabrera, C.L. Colbeck,J.M. Parente, and graduate programs nationwide.
S.A. Bjorklund. 2001. Collaborative learning vs. lecture/discussion:
Address: Engineering Education (0218), Blacksburg, Virginia
Students' reported learning gains. JournalofEngineeringEducation 24061; telephone: (+ 1) 540.231.9536; e-mail: mborrego@vt.edu.
90 (1): 123-30.
, Thorne, M., and M. Giesen. 2002. Statisticsforthe behavioralsciElliot P. Douglas is an associate professor in the Department
ences. New York: McGraw-Hill.
of Materials Science and Engineering. He received his doctorate
Tinto, V. 1993. Leaving college: Rethinking the causes and cures of in Polymer Science and Engineering from the University of
student attrition.Illinois: University Of Chicago Press.
Massachusetts-Amherst. His research interests in engineering
Todd, R.H., S.P. Magleby, C.D. Sorensen, B.R. Swan, and education are in the areas of active learning, critical thinking, and
D.K. Anthony. 1995. A survey of capstone engineering courses the use of qualitative methods.
in North America. Journalof EngineeringEducation 84 (2):
Address: 323 Materials Engineering Building, Box 116400, Uni165-74.
versity of Florida, Gainesville, FL 32611; telephone: (+1) 352.
Tonso, K. 1996. The impact of cultural norms on women.Jour- 846.2836; fax: (+ 1) 352.392.3771; e-mail: edoug@mse.ufl.edu.
nalofEngineeringEducation85 (3): 217-25.
66 Journalof EngineeringEducation
January 2009
COPYRIGHT INFORMATION