Effective Training: Systems, Strategies, and Practices, 4th Edition

Chapter Nine Evaluation of Training

P. Nick Blanchard and James W. Thacker

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-1

Evaluation Phase
Input

Process
Evaluation Strategy and Design

Output
Process Measures

Evaluation Objectives

Organizational Constraints Design Issues

Outcome Measures - Reaction - Learning - Behavior - Results

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-2

Potential Questions to Be Addressed in a Process Analysis (Before Training) Part 1 of 2

Were needs diagnosed correctly? What data sources were used? Was a knowledge/skill deficiency identified? Were trainees assessed to determine their prerequisite KSAs? Were needs translated into training objectives? Were all objectives identified? Were the objectives written in a clear, appropriate manner?
Was an evaluation system designed to measure objectives?
Was the training program designed to meet all the training objectives? Was previous learning that supports or inhibits training identified? Were individual differences assessed/factored into training design? Was trainee motivation to learn assessed?
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-3

Potential Questions to Be Addressed in a Process Analysis (Before Training) Part 2 of 2

What steps were taken to address trainee motivation to learn? Were processes built into training to facilitate recall and transfer? Were steps included to call attention to key learning events? What steps are included in the training to aid trainees in symbolic coding and cognitive organization? What opportunities are included in the training to provide symbolic and behavioral practice? What actions are included in the training to ensure transfer of learning to the job? Are the training methods appropriate for the learning objectives?

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-4

Potential Questions to Be Addressed in a Process Analysis (During Training)

Were the trainer, training techniques, and training/learning objectives well matched? Were lecture portions of the training effective? Was involvement encouraged/solicited? Were questions used effectively? Did the trainer conduct the various training methodologies (case study, roleplay, etc.) appropriately? Was enough time allotted? Did the trainer use the allotted time for activities? Was enough time allotted? Did trainees follow instructions? Was there effective debriefing following exercises? Did the trainer follow the training design and lesson plans? Was enough time given for each of the requirements? Was time allowed for questions?
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-5

Who Is Interested in the Process Data

Training Department
Trainer: Yes, to determine what works well and what does not. Other trainers: Yes, to the extent that process is generalizable. Training manager: Only if training is not successful or a problem is present with a particular trainer. Customers Trainees: Trainees supervisor: Upper management:

No No No

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-6

Reaction Questionnaire for the Trainer Part 1 of 2

Please circle the number that reflects the degree to which you agree or disagree with the following statements. 1= Strongly disagree, 2= Disagree, 3= Neither agree nor disagree 4= Agree 5 =Strongly agree
1. The trainer did a good job of stating the objectives at the beginning of training .....1 2 3 4 5
2. The trainer made good use of visual aids (easel, white board) when making the presentations .....1 2 3 4 5

3. The trainer was good at keeping everyone interested in the topics ... 1 2 3 4 5
4. The trainer encouraged questions and participation from trainees .....1 2 3 4 5
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-7

Reaction Questionnaire for the Trainer Part 2 of 2

5. The trainer made sure everyone understood the concepts

before moving on to the next topic..1 2 3 4 5 6. The trainer summarized important concepts before moving to the next module1 2 3 4 5 7. Overall, how would you rate this trainer? (Check one.) ____1. Poor; I would not recommend this trainer to others. ____2. Adequate; I would recommend this trainer only if no others were available. ____3. Average ____4. Good; I would recommend this trainer above most others. ____5. Excellent; this trainer is among the very best Ive ever worked with. 8. Additional comments:
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-8

Steps to Consider in Developing a Reaction Questionnaire Part 1 of 2

1. 2. Determine what you want to find out (consider training objectives). Develop a written set of questions to obtain the information.

3.
4. 5.

Develop a scale to quantify respondents data.

Make forms anonymous so participants will feel free to respond honestly. Ask for information useful in determining differences in reactions by subgroups (e.g., young vs. old; minority vs. non-minority). This could be valuable in determining effectiveness of training for different cultures, for example, which may be lost in an overall assessment.

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-9

Steps to Consider in Developing a Reaction Questionnaire Part 2 of 2

Note: Care must be taken when asking for this information. If you ask too many questions about race, gender, age, tenure, and so on, participants will begin to feel that they can be identified without their name on the questionnaire. 6. Allow space for Additional Comments in order to allow participants the opportunity to mention things you might not have considered. 7. Decide the best time to give the questionnaire to get the information you want. a. If right after training, ask someone other than the instructor to administer and pick up the information. b. If some time later, develop a mechanism to obtain a high response rate (e.g., encourage the supervisor to allow trainees to complete the questionnaire on company time).

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-10

Procedures for Developing a Multiple-Choice Test

1.
2. 3.

4.
5. 6. 7.

Examine objectives to gain a clear understanding of the content area you wish to test. Write the questions in a clear manner. Shorter is better. Choose alternatives to the correct response from typical errors made during training. Make alternatives realistic. Do not consistently make the correct response always longer than incorrect responses. Provide four options. More than four takes longer to read, and it is difficult enough to write three reasonable alternatives along with the correct answer. Pretest items by giving the test to those expected to know the material. Ask for feedback on clarity. Note any questions that many of them get wrong. Give revised items to a group of fully trained (experienced) employees and a group of untrained (inexperienced) employees. The former should score well and the latter should do poorly.

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-11

Test of Knowledge Organization for Civil Engineers 1 of 2

The following list of concepts is related to road construction. Use them to fill in the appropriate blank boxes in the map. Try to fill in the boxes so that related terms or concepts are clustered together. Concepts can be related because they occur at the same time, one is necessary for the other, or one leads to the other. Each of the listed concepts is used only once. Note that some of the concepts are already mapped for you. Concepts: Asphalt placement Compaction/rolling Cut/fill Dump truck Hot materials

Prime/tack coat Rollers Signage Site access

Striper Striping Survey Traffic

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-12

Test of Knowledge Organization for Civil Engineers 2 of 2

Water Trucks Compaction

earthwork

Safety
Heavy equipment

paver

Spray truck curing finishing safety Road access

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-13

Safety

Example of an Attitudinal Measure

Attitudes Toward Empowerment Please indicate the degree to which you agree or disagree with the following statements. 1 = Strongly disagree 2 = Disagree 3 = Neither agree nor disagree 4 = Agree 5 = Strongly agree
1. Empowering employees is just another way to get more work done with fewer people. (reverse scored)..1 2 3 4 5 2. Empowering of employees allows everyone to contribute their ideas to the betterment of the company.1 2 3 4 5 3. The empowerment program has improved my relationship with my supervisor1 2 3 4 5

4. Empowerment has brought more meaning to my life at this company..1 2 3 4 5

5. Empowerment interventions should be introduced in other plants in this company..1 2 3 4 5 6. The empowerment process has been a positive influence in labor-management relations.1 2 3 4 5
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-14

Guidelines for Writing Effective Questionnaires Part 1 of 4

1. Write simply and clearly, and make the meaning obvious. Bad: To what extent do supervisors provide information regarding the quality of performance of people at your level? Good: How often does your boss give you feedback on your job? 2. Ask one question at a time. Bad: Both the organizations goals and my role within the organization are clear. Good: The organizations goals are clear. My role within the organization is clear.

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-15

Guidelines for Writing Effective Questionnaires Part 2 of 4

3. Provide discrete response options.
Bad: During the past three months how often did you receive feedback on your work?

1 rarely
Good:

3 occasionally

5 frequently

During the past three months how often did you receive feedback on your work? 2 13 times a month 3 about once a week 4 2-4 times a week 5 once a day or more
9-16

1 not once

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

Guidelines for Writing Effective Questionnaires Part 3 of 4

4. Limit the number of response options. Bad: What percent of the time are you sure of what your compensation will be?
1 2 3 4 5 6 7 8 9 10 010% 1120% 2130% 3140% 4150% 5160% 6170% 7180% 8190% 91100%

Good: What percent of the time are you sure of what your compensation will be?
1 020% 2 2140% 3 4160% 4 6180% 5 81100%

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-17

Guidelines for Writing Effective Questionnaires Part 4 of 4

5. Match the response mode to the question. To what extent are you satisfied with your job? 2 disagree 3 4 agree 5 strongly agree

Bad:

1 strongly disagree Good: 1 not at all

To what extent are you satisfied with your job? 2 a little bit 3 some 4 quite a lot 5 very much

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-18

Scripted Situation Item for Evaluation of a School Superintendent

The following is a scenario about a school superintendent. Read the scenario and place an X next to the behavior you believe your superintendent would follow.
The administrator receives a letter from a parent objecting to the content of the science section on reproduction. The parent strongly objects to his daughter having exposure to such materials and demands something be done. The effective administrator would most likely: (check one) ____ ____ ____ ____ Ask the teacher to provide handouts, materials, and curriculum content for review. Check the science curriculum for the board-approved approach to reproduction, and compare board guidelines to course content. Ask the head of the science department for his or her opinion about the teachers lesson plan. Check to see if the parent has made similar complaints in the past.
9-19

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

Types of Outcomes and Examples of Factors Affecting those Outcomes

Organizational Results
External environment of the organization: Economy, regulations, suppliers, etc. Internal environment of the organization: Policies, procedures, systems Employee performance, KSAs, and needs Transfer of training Motivational forces in the job setting Opportunity to applying training on the job KSAs Trainee readiness for the course Trainee motivation to learn Design, materials, and content Trainer(s) behaviors Perceived match between trainee Exceptions and what training provided
9-20

Job Behavior

Learning

Reactions

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

Cost Savings for Grievance Reduction Training

Pretraining
Management time (for those going to 3rd step) 10 hours per grievance Union Reps time (paid by management) 7.5 hrs per grievance Total Cost Management time Union reps time Total 10 hrs. X 63 grievances = 630 hrs.

Post training
10 hrs. X 8 grievances = 80 hrs.

7.5 X 63 grievances = 472 hrs

7.5 X 8 grievances = 60 hrs.

630 hrs X $50 per hr.

80 hrs. X $50 per hr. = $4,000.00 472 hrs X $25 per hr.= $11,812.50 60 hrs. X $25 per hr. = $1,500.00 $43,312.50 $5,500.00
= $31,500.00

Reduction in cost of grievances going to the third step 43,312.50 5,500.00 = $37,812.50 Cost of training -32,430.00 Cost saving for the first year $ 5,382.50
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-21

Training Investment Analysis Work SheetPart 1 of 5

Objective:

Audience:

Returns measured over:

Other:

One year:

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-22

Training Investment Analysis Work SheetPart 2 of 5

Part 1: Calculating the Revenue Produced by Training Option A-Itemized Analysis Increased sales: Additional sales per employee X Revenues (or margin) per sale X Number of employees = Revenue Produced by Training Higher Productivity: Percent increase in productivity X Cost per employee (salary plus benefits plus overhead) X Number of employees = Revenue Produced by Training
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-23

Training Investment Analysis Work SheetPart 3 of 5

Reduced errors: X X = Client retention: X = Average cost per error Number of errors avoided per employee Number of employees Revenue Produced by Training Average revenue per client Number of clients retained Revenue Produced by Training

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-24

Training Investment Analysis Work SheetPart 4 of 5

Employee retention: Average cost of a new employee ( training plus lost productivity) Number of employees retained Revenue Produced by Training

Other:

X =

Total Revenue Produced by Training Option B-Summary Analysis $ = Revenue Revenue After Training Without Training
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

Revenue Produced by Training

9-25

Training Investment Analysis Work SheetPart 5 of 5

Part 2: Calculating the Return Revenue Cost of Produced Training By Training

=
Total Return on Training Investment

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-26

Calculation of the Utility of the Grievance Training Part 1 of 2

Formula: U = (N)(T)(DT)(SDY) - C
N = 30 T = 1 year (This is probably an overly conservative estimate) DT = .2 DT = Xt Xu SD (r yy) Xt = average job performance of the trained supervisors Xu = average job performance of the untrained supervisors SD =standard deviation of job performance for the untrained supervisors r yy = reliability of job performance measure

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-27

Calculation of the Utility of the Grievance Training Part 2 of 2

DT is a measure of the improvement (in standard deviation units) in performance that trained supervisors will exhibit. SDY = $14,000 This is based on: .40 X $35,000 = $14,000

The above assumes average salary of 35,000 dollars. The .40 comes from the 40% rule, which is a calculation based on 40% of the average salary of trainees.

So based on the above information, the utility of the training based on this formula is:
30 X 1 X .2 X 14,000 - 32,020 = $51,980
Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-28

Who Is Interested in the Outcome Data

Outcome Data
Reaction Training Department Learning Behavior Results

Trainer
Other Trainers Training Manager Customers Trainees Trainees Supervisor Upper Management

Yes
Perhaps Yes Yes Not really No

Yes
Perhaps Yes Yes Only if no transfer No

Yes
Perhaps Yes Yes Yes Perhaps

No
No Yes Perhaps Yes Yes

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

9-29

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher. Printed in the United States of America.

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall

Copyright 2010 Pearson Education, Inc. publishing as Prentice Hall 9-30