The Validity and Reliability of

Qualitative Research

Training Overview
Section 1: Section 2: Section 3: Section 4:
Section 5: Philosophical Orientation: Getting in

the Right Mindset

Enhancing the Validity of Qualitative Research Enhancing the Reliability of Qualitative Research Applications of Qualitative Research Design Group Work to Apply Learning

Section 1

Philosophical Orientation
The philosophical underpinnings of qualitative research (as contrasted against quantitative research) Implications for sampling

Implications for addressing issues of validity and reliability

Quantitative and Qualitative Paradigms Contrasted

Quantitative Research is
Fundamentally an inferential enterprise that seeks to uncover universal principles Philosophically and methodologically built or designed around the ability to infer from a sample to a larger population

Quantitative and Qualitative Paradigms Contrasted

Qualitative Research is
Fundamentally an interpretive enterprise that is context-dependent

Philosophically and methodologically built or designed around the ability to interpret (comprehend/understand) a phenomenon from an emic (insider), as well as an etic (outsider) perspective
This inter-subjective (i.e., shared) understanding serves as a proxy for objectivity

Implications for Research Questions

A client wants to know about how well a program is working for youth:
Quantitative Research Questions (descriptive, explanatory ->inferential)

Qualitative Research Questions (descriptive, explanatory ->interpretive)

Implications for Sampling

Sampling Strategies used in Quantitative Research:
Obtaining a random or representative sample (based on probabilities) Permits the researcher to infer from a segment of the population (from which it is more feasible to collect data) to a larger population

Implications for Sampling

Sampling Strategies used in Qualitative Research:
Purposive sampling (to ensure that the researcher has adequately understood the variation in the phenomena of interest) Theoretical sampling (to test developing ideas about a setting by selecting phenomena that are crucial to the validity of those ideas)
Example: Case study selection in ACY (see, also, handout on sampling techniques)

Implications for Handling Threats to Validity and Reliability

In quantitative research, threats to validity are addressed by prior design features (such as randomization and controls)

Implications for Handling Threats to Validity and Reliability

In qualitative research, such prior elimination of threats to validity is less possible because:
qualitative research is more inductive, and it focuses primarily on understanding particulars rather than generalizing to universals.

Qualitative researchers view threats as an opportunity for learning

- e.g. researcher effects and bias are part of the story that is told; they are not controlled for

Section 2

Implications: Enhancing the Validity of Qualitative Research

Defining validity within the qualitative paradigm Major types of validity within the qualitative paradigm Design considerations

Validity
Validity is not a commodity that can be

purchased with techniques Rather,

validity is, like integrity, character and

quality, to be assessed relative to

purposes and circumstances.
Brinberg and McGrath 1985:13

Validity
In general, validity concerns the degree to which an account is accurate or truthful
In qualitative research, validity concerns the degree to which a finding is judged to have been interpreted in a correct way

Assessing the Validity of Qualitative Research

Can another research read your field (and other types of) notes (i.e., the explication of your logic) and come to the same understandings of a given phenomenon?
Concern about validity (as well as reliability) is the primary reason thick description is an essential component of the qualitative research enterprise Resources:
Handout: Different Types of Notes

Example: ACY Site Visit Toolkit

Major Types of Validity in Qualitative Research

Descriptive Validity Interpretive Validity Theoretical Validity External Validity (i.e., generalizability)

Descriptive Validity
Concerned with the factual accuracy of an account (that is, making sure one is not making up or distorting the things one hears and sees)

All subsequent types of validity are dependent on the existence of this fundamental aspect of validity

Descriptive Validity
Behavior must be attended to, and with

some exactness, because it is through

the flow of behavior or, more

precisely, social action that cultural

forms find articulation.
Geertz 1973:17

Interpretive Validity
Interpretive accounts are grounded in the language of the people studied and rely, as much as possible, on their own words and concepts

At issue, then, is the accuracy of the concepts as applied to the perspective of the individuals included in the account

Interpretive Validity: Design Consideration

While the relevant consensus about the terms used in description rests in the research community, the relevant consensus for the terms used in interpretation rests, to a substantial extent, in the community studied
An important design element, for increasing interpretive validity, therefore, is to employee, at some level/to some degree, a participatory research approach (e.g., through member checks, peer to peer research model, etc.)

Theoretical Validity
Theoretical understanding goes beyond concrete description and interpretation; its value is derived based on its ability to explain succinctly the most amount of data A theory articulates/formulates a model of relationships as they are postulated to exist between salient variables or concepts Theoretical validity is thus concerned, not only with the validity of the concepts, but also their postulated relationships to one another, and thus its goodness of fit as an explanation

Major Threats to Validity

Type I error: believing a principle to be true when it is not (i.e., mistakenly rejecting the null hypothesis) Type II error: rejecting a principle when, in fact, it is true Type III error: asking the wrong question

Triangulation: An Important Theoretical Validity Check

Case example: Parable of the blind men and the elephant

Triangulation: An Important Theoretical Validity Check

The most fertile search for validity comes from a combined series of different measures, each with its

own idiosyncratic weaknesses, each pointed to a

single hypothesis. When a hypothesis can survive the confrontation of a series of complementary

methods of testing, it contains a degree of validity

unattainable by one tested within the more constricted framework of a single method.
Webb et al. 1966:174

External Validity in Qualitative Research

There is broad agreement that generalizability (in the sense of producing laws that apply universally) is not a useful standard or goal for qualitative research This is not to say, however, that studies conducted to examine a particular phenomenon in a unique setting cannot contribute to the development of a body of knowledge accumulating about that particular phenomenon of interest Consensus appears to be emerging that for qualitative researchers generalizability is best thought of as a matter of the fit between the situation studied and others to which one might be interested in applying the concepts and conclusions of that study.

Enhancing External Validity

Thick descriptions are crucial. Such descriptions of both the site in which the studies are conducted and of the site to which one wishes to generalize (or apply ones findings) are critical in allowing one to search for the similarities and differences between the situations. Analysis of these similarities and differences makes it possible to make a reasoned judgment about the extent to which we can use the findings from one study as a working hypothesis about what might occur in another situation.

Multi-site Studies: Another Way to Enhance Generalizability

A finding emerging repeatedly in the study of numerous sites would appear to be more likely to be a good working hypothesis about some as yet unstudied site than a finding emerging from just one or two sites. A finding emerging from the study of several very heterogeneous sites would be more robust and, thus, more likely to be useful in understanding various other sites than one emerging from the study of several very similar sites.

Heterogeneity may be obtained by creating a sampling frame that maximizes the variation inherent in the sample, specifically in terms of potentially theoretically important dimensions

Section 3

Implications: Enhancing the Reliability of Qualitative Research

Defining reliability Key strategies for enhancing the reliability of qualitative research

Reliability
Reliability concerns the ability of different researchers to make the same observations of a given phenomenon if and when the observation is conducted using the same method(s) and procedure(s)

Enhancing the Reliability of Qualitative Research

Researchers can enhance the reliability of their qualitative research by:
Standardizing data collection techniques and protocols

Again, documenting, documenting, documenting (e.g., time day and place observations made)
Inter-rater reliability (a consideration during the analysis phase of the research process)

Section 4

Applications of Qualitative Research Design

Core Qualitative Methods Guiding Principles

Qualitative Research Techniques

Core Qualitative Methods

Semi- or Un-structured, Open-Ended:

In-depth Interviews

(in the field, face-to-face) (field/site visits)

Participant Observation Archival Research

(document review and analysis)

Guiding Principles
Qualitative research designs consider ways to foster:
Reflexivity (an ongoing process of reflecting on
the researchers subjective experience, ways to broaden and enhance this source of knowing, & examining how it informs research)

Iteration (a spiraling process: sequential and

repetitive steps in examining preliminary findings for the purposes of guiding additional data collection and analysis) shared/ objective agreement about how to assign meaning to a social experience - with insiders and outsiders)

Intersubjectivity (a process of reaching a

The Iterative Process of Qualitative Research: A Model

Analysis

Data Collection

Reflection

Qualitative Research Techniques

Instrumentation:
Key Informants (question development and piloting of instrument)
Unstructured to Semi-structured Probing Data Processing and Analytic Tools

Qualitative Research Techniques/ Considerations

Sampling
Single v. Multiple Cases (not an individual)
Expert and Key Informants (identification and
recruitment of sample) recruitment of sample)

Roles of the Researcher (identification and

Qualitative Research Techniques

Data Collection
Participants as Data Collectors
Field Notes (personal reflections, observations,
emerging concepts/theories)

Debriefing (a participant, a participating

researcher, a non-participating researcher)

Qualitative Research Techniques

Analysis
Key Informant Feedback

Codebooks (specifies definitions and relationships of concepts and terms)

Memos (emerging patterns, concepts; documentation of analytic pathways) Case Analysis Meeting (a meeting of a research team for the purposes of reflecting on analytic process, tools, and findings) Matrices or Diagrams (to identify and examine time sequencing, the structure of relationships, conditions of cross case events)

Section 5

Group Work to Apply Learning

 What is a given program achieving with homeless and runaway youth?

Key Methodological Issues

(instrumentation, sampling, data collection, analysis)

What More Do You Need to Know? Initial Methodological Approach and Justification

In-House Qualitative Resources

Handouts, OMNI Reports and Proposals Qualitative Research Design, An Interactive Approach
(Maxwell, 1996)
Sage, Applied Social Research Methods Series

Qualitative Data Analysis (Miles & Huberman, 1994) The Quality of Qualitative Research (Seale, 1999) Focus Groups, Theory and Practice (Stewart &
Shamdasani, 1990)
Sage, Applied Social Research Methods Series