Improving and Assessing Instructional Effectiveness

Theodore Frick
Department of Instructional Systems Technology School of Education Indiana University Bloomington

Invited Lecture to IUB Medical Sciences Faculty October 14, 2002

Overview
Confounding of instructional effectiveness by other factors Types of knowledge about education Type 2, 3 and 4 examples Recommendations

October 14, 2002

Improving and Assessing Instructional Effectiveness -- Ted Frick

The Logic of Effectiveness

Before instruction
1. Student nonmastery

After instruction
Student mastery

Conclusion
Instruction appears to be effective enough, but Instruction apparently was not effective enough Can't tell if instruction is effective, since student had achieved the goal before instruction began ??? Something wrong with the content - factual errors, incorrect procedures, bad models ??? Bad luck ???

2. Student nonmastery

Student nonmastery

3. Student mastery

Student mastery

4. Student mastery

Student nonmastery

October 14, 2002

Improving and Assessing Instructional Effectiveness -- Ted Frick

Confounding Factors
1. 2. 3. 4.

Learning occurs without intentional instruction. Learning occurs despite poor instruction. Were not the only teachers a student has. Results of instruction may occur LONG afterwards.
Improving and Assessing Instructional Effectiveness -- Ted Frick

October 14, 2002

Conflation of Research Methods and Outcomes

In addition to the confounding problem in determining effectiveness of instruction, there has been a lot of debate in education about inquiry methods e.g., quantitative vs. qualitative. Continuing problem of bridging research and practice in education

October 14, 2002

Improving and Assessing Instructional Effectiveness -- Ted Frick

Consider this Debate

Imagine for a moment three carpenters arguing about which tool is best.

Quanta: "Hammers and nails are clearly superior." Qualia: "I disagree. Screwdrivers and screws are much more effective." Performa: "You're both wrong. Saws are best for cutting wood." Quanta: "Who said anything about cutting wood? I thought we were talking about fastening wood together." Qualia: "Right. Who needs saws?" Performa: "I do. I need to cut this board in half."
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

Premise
The kind of knowledge about education that we create through disciplined inquiry determines what research methods are appropriate and useful.

October 14, 2002

Improving and Assessing Instructional Effectiveness -- Ted Frick

Types of Educology: Knowledge about Education

Methodology of Theory Building (Elizabeth Steiner, 1988) (link)

Non-axiological knowledge what is Axiological knowledge pertaining to values:

Instrumental value: good for what is effective Intrinsic value: good in itself what is worthwhile

The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles (Lee Cronbach, et al., 1972)
Improving and Assessing Instructional Effectiveness -- Ted Frick

October 14, 2002

Six Types of Knowledge (Frick)

Scope of Knowledge

Purpose of Inquiry
What is?

Unique

Generalizable

What is effective?
What is worthwhile?

3
5

4
6

See Frick: R690 Syllabus for further details

October 14, 2002

Improving and Assessing Instructional Effectiveness -- Ted Frick

Six Types of Knowledge (Frick)

Scope of Knowledge

Purpose of Inquiry
What is?

Unique

Generalizable

What is effective?
What is worthwhile?
Example: Type 1 Knowledge Claim

3
5

4
6

Elizabeth Steiner is an educational philosopher.

Improving and Assessing Instructional Effectiveness -- Ted Frick

October 14, 2002

10

Six Types of Knowledge (Frick)

Scope of Knowledge

Purpose of Inquiry
What is?

Unique

Generalizable

What is effective?
What is worthwhile?

3
5

4
6

Example: Type 2 Knowledge Claim Students who spend more time engaged successfully in arithmetic and reading tasks score more highly on achievement tests in those content areas. (Fisher, et al., 1976)
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

11

Six Types of Knowledge (Frick)

Scope of Knowledge

Purpose of Inquiry
What is?

Unique

Generalizable

What is effective?
What is worthwhile?

3
5

4
6

Example: Type 3 Knowledge Claim The DISTAR program for teaching arithmetic and reading to elementary students works.
Improving and Assessing Instructional Effectiveness -- Ted Frick

October 14, 2002

12

Six Types of Knowledge (Frick)

Scope of Knowledge

Purpose of Inquiry
What is?

Unique

Generalizable

What is effective?
What is worthwhile?

3
5

4
6

Example: Type 4 Knowledge Claim Instruction is most effective when it is problem-based, activates prior
learner knowledge, demonstrates what is to be learned, provides opportunities for guided practice, and encourages integration with everyday life. (Merrill, 2001)
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

13

Six Types of Knowledge (Frick)

Scope of Knowledge

Purpose of Inquiry
What is?

Unique

Generalizable

What is effective?
What is worthwhile?
Example: Type 5 Knowledge Claim

3
5

4
6

The corporal punishment policy in Houston schools is a bad policy.

October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

14

Six Types of Knowledge (Frick)

Scope of Knowledge

Purpose of Inquiry
What is?

Unique

Generalizable

What is effective?
What is worthwhile?
Example: Type 6 Knowledge Claim

3
5

4
6

Teachers and students should respect each other.

Improving and Assessing Instructional Effectiveness -- Ted Frick

October 14, 2002

15

Do not confuse ends and means

The kind of knowledge about education that we create through disciplined inquiry determines what research methods are appropriate and useful.

Outcome of research is knowledge. Inquiry methods are a means to that end. Criteria for evaluating adequacy of research methodology are NOT the same for each knowledge type e.g.,

criteria for statistical inference to a broad population do not apply to Types 1, 3, and 5; effectiveness not of concern in Types 1, 2, 5, and 6, but main concern in 3 and 4.
Improving and Assessing Instructional Effectiveness -- Ted Frick

October 14, 2002

16

For today
Focus on a few examples of Types 2, 3 and 4 knowledge of education
Type 2: Academic learning time research Type 3 outcomes: Program/product evaluation Kirkpatricks 4 levels (1959) Type 4 process: Effective Web Instruction Frick & Boling (2002) Type 4: Principles of 5-Star Instruction David Merrill (2001)

October 14, 2002

Improving and Assessing Instructional Effectiveness -- Ted Frick

17

Six Types of Knowledge (Frick)

Scope of Knowledge

Purpose of Inquiry
What is?

Unique

Generalizable

What is effective?
What is worthwhile?

3
5

4
6

Type 2 Example: Relationship of academic learning time and student achievement

Improving and Assessing Instructional Effectiveness -- Ted Frick

October 14, 2002

18

Type 2: Academic Learning Time

Task Success Engaged Time

Allocated Time

ALT is positively correlated with academic achievement in the same content areas (Fisher, et al., 1976; Rieth & Frick, 1983; Berliner; 1985).
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

19

Six Types of Knowledge (Frick)

Scope of Knowledge

Purpose of Inquiry
What is?

Unique

Generalizable

What is effective?
What is worthwhile?

3
5

4
6

Type 3: Evaluation of a particular instructional program or product

Improving and Assessing Instructional Effectiveness -- Ted Frick

October 14, 2002

20

Type 3: Program/Product Evaluation Kirkpatricks Levels

Techniques for Evaluating Training Programs -- Donald Kirkpatrick (1959)
1. 2. 3. 4.

Reaction (satisfaction) Learning (achievement) Behavior (transfer to real context) Results (impact on organization/context)

More on Kirkpatricks levels from Encyclopedia of Educational Technology

Improving and Assessing Instructional Effectiveness -- Ted Frick

October 14, 2002

21

Six Types of Knowledge (Frick)

Scope of Knowledge

Purpose of Inquiry
What is?

Unique

Generalizable

What is effective?
What is worthwhile?

3
5

4
6

Type 4 Example: how to create effective instructional products design theory

Improving and Assessing Instructional Effectiveness -- Ted Frick

October 14, 2002

22

Type 4: Effective Web Instruction: An Inquiry-Based Process Frick & Boling

More
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

23

Type 4: First Principles of Instruction David Merrill

Integration Activation

Problem

Application

Demonstration

October 14, 2002

Improving and Assessing Instructional Effectiveness -- Ted Frick

24

Type 4: First Principles of Instruction (contd)

Learning is facilitated when:
Learners are engaged in solving real-world problems. 2. Existing knowledge is activated as a foundation for new knowledge. 3. New knowledge is demonstrated to the learner. 4. New knowledge is applied by the learner. 5. New knowledge is integrated into the learners world. (Merrill, 2001, p. 2) Does your instruction rate 5 stars? A rating scale
1.
October 14, 2002
Improving and Assessing Instructional Effectiveness -- Ted Frick

25

Six Types of Knowledge (Frick)

Scope of Knowledge

Purpose of Inquiry
What is?

Unique

Generalizable

What is effective?
What is worthwhile?

3
5

4
6

Type 3: Indiana University physician education program

Improving and Assessing Instructional Effectiveness -- Ted Frick

October 14, 2002

26

Recommendations
In physician education, focus on Type 3 knowledge:
Apply Merrills 5 principles as criteria for design. See video (requires RealPlayer). Use Kirkpatricks 4 levels of evaluation for your programs at IU.

October 14, 2002

Improving and Assessing Instructional Effectiveness -- Ted Frick

27

Questions?

This presentation is available at:

http://www.indiana.edu/~tedfrick/improveinstruction.ppt

Contact: frick@indiana.edu

October 14, 2002

Improving and Assessing Instructional Effectiveness -- Ted Frick

28