Evaluation of University Extension HTML/XHTML Training for Website Development

Project IV: Evaluation of University Extension HTML/XHTML Training for Website Development James Becker Purdue University Strategic Assessment and Evaluation 577 Evangelia van Barneveld October 12, 2012
Identification of evaluation goals and scope (5) Description of process (includes Gantt chart) (5) Data collection (instruments description) (20) Analysis process (20) Report (10) Requirements (5) TOTAL Comments: Jim a good start. Still lots of opportunity to evolve the plan. 5 5 15 15 10 5 55
Comment [A1]: Ah yea I know her. Like her a lot!

Evaluation of University Extension HTML/XHTML Training for Website Development

Table of Contents
Executive Summary ...................................................................................................................................... 2 Key Findings and Recommendations.....3 Introduction .................................................................................................................................................. 4 Data Collection and Analysis ........................................................................................................................ 6
Level 1 Reaction to Regional Trainings...6 Level 2 Learning.7 Level 3 Behavior Change..7 Level 4 Results8 Conclusion..9 References10 Appendices...11

Evaluation of University Extension HTML/XHTML Training for Website Development To: The Extension Administrative Team From: IDEA Consulting Date: 10/9/2012

EXECUTIVE SUMMARY

University Extension as an organization provides each county Extension office with a basic website. It is up to each county to develop the content of the website and populate it with local activity and program information. Many University Extension county websites have done an inadequate job of website development. The purpose of this training program is to improve the quality of University Extension County websites. With 92 counties in the state of Indiana, University Extension is faced with a large task to bring up the quality of so many websites. IDEA Consulting began a needs analysis process by (see Appendix B) meeting with University Extensions administrative team in June. Regional and County Directors said Extension Educators do not presently possess the knowledge and skills for creating quality web pages on their county websites. Improved county websites would help meet the main objective to give the public a better web interface with University Extension. Several Extension Educators were interviewed and confirmed the issue of lack of training and heavy workloads, as factors preventing them from doing a better job with the websites. Extension technology trainers will utilize MindLeaders, Web Publishing and Design with HTML 4.01 and XHTML: Basics of HTML and XHTML, online curriculum. They will train Extension Educators statewide, to create web pages on their websites during the formal training and provide support beyond the training. The objective of the training process will be to create capable Extension Educators to act as resources and web page developers in county offices, statewide, to improve the overall quality
Comment [A2]: Gotcha

Evaluation of University Extension HTML/XHTML Training for Website Development of county Extension web pages. Approximately 120 Extension Educators will attend one of six regional trainings, 3 Days each. They will be held around the state from September through

October. An additional 5 months will allow for collecting additional outcomes and holding focus groups with end users of the many Extension websites around the state. Using Kirkpatricks four levels for evaluating training programs we will collect and analyze data from different parts of the training. Data will include information about reactions to the training, knowledge and skills received and behavior change and results.
Comment [A3]: Sounds good.

Key Findings The results from different parts of the evaluation may include: Satisfaction or dissatisfaction with the training facilities Satisfaction or dissatisfaction with trainers Satisfaction or dissatisfaction with online training modules Increased employee knowledge of Web design and HTML/XHTML Increased employee knowledge of interactive multi-media web features Increased employee knowledge of urls, cascading style sheets and links Employee creation of new county web pages Types of new county web pages created How new knowledge and skills are being utilized Additional training that is needed Clientele reaction to new web pages and information on web pages Clientele expectations and desire for future web pages

Recommendations Possible recommendations may include: Follow-up with Extension Educators regarding additional training they feel is needed, once they have had time to use knowledge and skills from the training. County Directors and Regional Directors should meet with Extension Educators regarding the overall development of web pages to review, progress, quality and to address problems. Extension Educators should continue to be supported by University Trainers and by online training resources.

Evaluation of University Extension HTML/XHTML Training for Website Development University Extension as a whole could consider adopting a Balanced Scorecard program to encourage performance improvement as it relates to training of its employees. This would help the organization monitor how it is doing with professional development.

Introduction University Extension has a long tradition of providing research based information to people who would not have access to the University otherwise. The importance of county websites has increased tremendously in recent years. Still the potential for using websites for more than mere information is not being reached. Many competitors have better websites with increased abilities to provide e-learning, educational videos, and interaction with clientele. Evaluating training and website development will aid University Extensions goal to create high tech avenues that continue to do as their slogan suggests, Bringing Knowledge to the People. The training includes the following courses:
Web Publishing and Design with HTML 4.01 and XHTML: Getting Organized Web Publishing and Design with HTML 4.01 and XHTML: Basics of HTML and XHTML Web Publishing and Design with HTML 4.01 and XHTML: HTML, XHTML, and Cascading Style Sheets (CSS) Web Publishing and Design with HTML 4.01 and XHTML: Adding Interactivity and Multimedia with HTML and XHTML

The four courses are broken into smaller modules; each course will take 2-3 hours to complete. With breaks, lunch and time for discussions, evaluation surveys, pre and post tests, the regional trainings will fill 3 days each. Each of the 6 trainings will have approximately 20 Extension Educators. Combining the on-line courses with live instructors, allows Extension Educators to have live support. Later when the training is over they will continue to have access to the courses online and can return to them to refresh information or check on something they need help with. Instructors at the regional trainings will also work with Extension Educators to set-up a peer organized Facebook group, which will serve as their own learning community to support each

Evaluation of University Extension HTML/XHTML Training for Website Development other. The Extension employed trainers will also continue to be available by phone from their campus location. Specific reasons for evaluating this training and subsequent results, is to clearly identify how well the problem of upgrading county websites can be addressed with this kind of training. The objective of this evaluation process will be to measure and quantify how well Extension Educators learn the information in the training. Then it will look at how well they use it in their work to create better web pages, transferring the skills to their jobs (Kirkpatrick & Kirkpatrick, 2006).

Comment [A4]: Good

Target Performers Extension Educators are on the ground where clientele live, it is easy to see they have very busy schedules. Doing one more thing is a major concern to them. All Extension Educators have Masters degrees, yet much of the new cloud technologies have come along since many of them finished their degrees. Many have never participated in an online course before, so using online courses with live instructors will be a new experience. In order to use the web and other online technologies to communicate better with clientele, they must be willing to learn and commit time and effort. Success will depend upon the trainings ability to help Extension Educators see what advantages their new knowledge and skills give them.

Evaluation of University Extension HTML/XHTML Training for Website Development Data Collection and Analysis

As mentioned earlier in the Executive Summary, this evaluation will utilize Kirkpatricks evaluation framework, which is divided into 4 levels. There will be 6 regional trainings held in different parts of the state. This will keep the size of each training group relatively small, close to 20 people in each. Level 1 and Level 2 data will be collected at all 6 regional trainings. Level 1 Reaction to Regional Trainings An online survey will be administered as part of the training to 100% of the attendees. The survey will be a mixture of multiple choice and Likert scale questions (see Appendix F). There will be some open ended questions to invite specific comments. This will result in a mixture of quantitative and qualitative data to analyze. The questions pertain to the appropriateness of the training content, effectiveness of the trainers, the training location, facilities, the online training quality, ease of understanding and ability to get questions answered. The survey results will be tabulated and the averages will be used to apply ranges as guidelines. For example if a Likert scale such as Very Good, Good, Average Not Good and Very Bad were scored 5 = Very Good, 4 = Good, 3 = Average, 2 = Not Good and 1= Very Bad, you could tabulate all the answers to that question. Lets say 30 people take the survey, 10 people answer Very Good, 10 answer Good, 7 answer Average, 2 Not Good and 1 Very Bad. You would add 50+40+21+4+1=116/30=3.8 average for that question (see Appendix H). The scores then can be used to rate the satisfaction of the different groups taking the training at different locations around the state. The can see how well they are doing in regard to the different questions. Answers to open ended questions are considered qualitative. These answers will be sorted into categories and quantified to the extent they are clearly saying the similar things. For example if the question asked you what you would change about the training
Comment [A5]: Appendix H does not provide enlightenment on this topic.

Evaluation of University Extension HTML/XHTML Training for Website Development and five people wrote they would prefer more face to face instruction, these answers would be grouped together.

Comment [A6]: Youd probably also need to code those open-ended answers in order to see what the potential groupings might be.

Level 2 - Learning A Pre-Test/Post-Test (see Appendix G) will be administered at the beginning and ending of the regional trainings. Extension Educators will take a Pre-Test at the beginning of the training about HTML/XHTML and other related web design topics that will come up in the training. An identical Post-Test will be given at the end of the training. The scores for both tests can then be compared for each Extension Educator. Areas that need improvement can be identified for the group as a whole and for individuals. Additional training options or repeating certain online training segments will be identified for individuals.
Comment [A8]: Thats not your role as the evaluator, right? You need to figure out WHY the participants were successful or not in their learning. Comment [A7]: Good

Level 3 Behavior Change Extension Educators will have a number of things to encourage their usage of their new knowledge and skills. They will have each other, via a Facebook group dedicated to HTML/XHTML and other web design topics they choose to bring to the learning community. In this way they can become their own support group in addition to on-campus trainers who will be available via phone. Approximately 60 days or more after their training, Extension Educators will be sent an online survey to be completed. The survey (see Appendix C) will ask for examples of how they have used their training to work on their websites. They will rate their confidence to utilize what they have learned to work on their websites, as well as tell how often they are working on their websites. They will provide specific examples of what they have done and how they have done it.
Comment [A10]: And how will you analyze this? Comment [A9]: Again, this would be the role of the instructional designer, not the evaluator who needs to focus on the application of the knowledge and skills learned in training on the job.

Evaluation of University Extension HTML/XHTML Training for Website Development

Level 4 Results University Extensions websites are where many of their clientele and public get information about programs. Timely, reliable information helps them to keep up on news for their farms, homes, children, and family members. This goal to improve service to clientele via websites cannot be proven without feedback from the people who use them. Six regional focus groups will be held. Pre-training focus groups from six different areas of the state will be held in July, 2012 and post-training focus groups will be held mid-February through March, 2013. County Extension offices will provide clientele mailing lists and assist in sending out requests for participation in the focus groups. Focus groups in July, 2012 will set a benchmark of where people think the websites are before the training of Extension Educators. Each focus group member will be asked to fill out a brief website evaluation (see Appendix D), for 2 University Extension County Websites, before participating in the focus groups via Adobe Connect. They will be asked to turn in these forms after the focus groups are held. Though the focus groups will be held by Adobe Connect, local offices may offer to host so that people may attend and utilize a speaker phone system, if they prefer that to calling in from home. In mid-February through March, a second round of focus groups/website evaluation groups will be assembled from 6 different regions of the state. Adobe Connects will be utilized for the focus groups with, volunteers, 4-H parents, 4-H members, Master Gardeners, teachers, farmers, etc. The focus group members will be 6-10 people each from each of 6 regions. On the Adobe Connect sessions they will then be asked questions to help define their concerns about specific websites (see Appendix E).
Comment [A12]: What people? How will you select them? Comment [A11]: What does that mean - where these web sites are at? You know there are a TON of metrics you could gather from log files that have nothing to do with peoples opinions. For example, has traffic increased to the web site? How long are people staying on the web site? What sort of content is downloaded (if any)? What sort of questions are fielded by the CONTACT US people? Basically, you want to know what the impact is on the organization of having these improved web pages (and the organizational goal is NOT nicer web pages). What does the organization envision happening once the web pages are improved? THAT is your level 4.

Evaluation of University Extension HTML/XHTML Training for Website Development

Conclusion Technology training is never an easy task to take on. There are many things to consider, because people must be ready to utilize the newly learned knowledge. In this case Extension Educators are to receive significant training in web page design, but this not an easy subject. There could be a certain amount of push-back from the educators receiving the training. This could impact Level 3 evaluation results for Educators who do not follow through to use their new knowledge and skills. However, if there is success at improving County Websites, Extension Educator will benefit from increased web traffic on their sites and better communication with clientele. The training program itself is relatively inexpensive (see Appendix A). The online curriculum being used is a trial course offered free by MindLeader . The main expenses for the training are travel, food and facilities. The trainers are from University Extensions own IT training staff, already under salary. The evaluation report will hopefully be able to show successful training results at level 2 and usage of the Extension Educators new skills at level 3. The best result of course would be many exciting new web pages on County Extension websites, enjoyed by clientele.
Comment [A13]: Here is your L4 stuff.

Evaluation of University Extension HTML/XHTML Training for Website Development

10

References
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs (3 ed.). San Francisco CA: Berrett-Koehler Publishers, Inc. Lemay, L. (2006). Web publishing and design with html 4.01 and xhtml: basics of html and xhtml. Retrieved from http://central.mindleaders.com/lms/tile.do?def=topFrame

Evaluation of University Extension HTML/XHTML Training for Website Development

11

APPENDICES

Evaluation of University Extension HTML/XHTML Training for Website Development

12

Appendices

Appendix A

Program Chart

Appendix B Project Gantt Charts Appendix C Level 3 Educator Survey Appendix D Website User Evaluation Appendix E Website Focus Group Questions Appendix F. Level 1 Survey Appendix G......................... Pre-Test/Post-Test Appendix H................................ .Sample Tabulation

Evaluation of University Extension HTML/XHTML Training for Website Development

13

Appendix A

Evaluation Program Chart

Project Extension HTML/XHTML Training for Website Development Scope Evaluation of the course and Level 1 through Level 4 data collected. Version 25.6N1 Identification Creator Jim Becker

Purpose

Levels

Objective to Measure

To evaluate the success of technology training for Extension Educators to improve county websites. Step 1. 2. 3. 4. 5. 6. Needs Assessment 3 Day Regional Training Collect Level 1 & 2 Data Pre-Test and Post Test Employee Level 3 Survey Clientele Focus Groups/Collect Level 4 Data 7. Final Data Analysis Complete Data to be Collected Level 4 Clientele reaction to websites Level 3 Application of Knowledge Pre-Test & Post Test Level 1 Survey Level 2 Survey

Level 1Reaction Level 2 Learning Level 3Behavior Change Level 4 - Results Who Extension Training Staff Extension Training Staff Extension Training Staff Extension Training Staff Extension Training Staff Extension Training Staff Extension Training Staff

Measure the increased abilities of Extension Educators to work on county websites and the results based on clientele reaction to websites the Educators have worked on.

Date June 4 July 5 Sept. 4Oct. 18 Sept .4Oct. 18 Sept. 4- Oct. 18 Jan. 24Feb. 8 July 9July 30 & Feb 13 Mar 28

Budget Travel$1200 Food - $2500 Printing $ 950 Trainers $8000 Total $12650.00

Comment [A14]: Sounds like the WHO would also include clientele as per your bullet #6.

When Before & after Extension Educator training At least 60 days after training 3 Day Reg. Ext. Educ.Training 3 Day Reg. Ext. Educ.Training 3 Day Reg. Ext. Educ.Training 3 Day Reg. Ext. Educ.Training 3 Day Reg. Ext. Educ.Training

How Collected Focus Groups & Written website evaluation Summer/Spring Online Survey Online Surveys Online Survey Online Survey Online Survey Online Survey

From Whom Clientele Ext. Educators Ext. Educators Ext. Educators Ext. Educators Ext. Educators Ext. Educators

By Whom Ext. Training Staff Ext Training Staff Ext Training Staff Ext Training Staff Ext Training Staff Ext Training Staff Ext Training Staff Comment [A15]: Im not sure L4 pre-training data needs to be specifically collected by the evaluator per se it seems to me that the client ISSUES that have been brought up about the web pages would serve as good post-training evaluation metrics.

Target Measures

Methods of Analysis

Analysis

Reaction to 3 Day Regional training by Extension Educators

Online survey completed by Extension Educators.

Knowledge of web design, HTML and XHTML, etc.

Increase in knowledge based upon comparison of Pre-Test and Post Test Scores

Analysis will be conducted by University Extension Trainers. Some of the questions on the different surveys will be appropriate for quantitative data, but other open ended questions will be suited only for qualitative data. The different results will be used to create a report that will analyze the training process and its objectives.

Evaluation of University Extension HTML/XHTML Training for Website Development

14

Application of knowledge by Survey completed by Extension using it on County Extension Educators at least 60 days after 3 Websites Day Regional Training.

Clientele reaction to county Extension websites prior to training of Extension Educators. Clientele will each use a paper evaluation to evaluate 3 websites, then participate in a Focus group via Adobe Connect

The paper evaluation of websites will be collected from the University Extension Clientele. Comments and answers to questions during the focus groups will be recorded and transcribed. Similar answers to specific questions will be coded and categorized.

Evaluation of University Extension HTML/XHTML Training for Website Development Appendix B -GANTT Charts Planning and Needs Analysis

15

Regional Extension Educator Trainings

Pre-Test/Post-Test and Level 1 Data Analysis

Evaluation of University Extension HTML/XHTML Training for Website Development Appendix B Contd Level 3 and Level 4 Data Collection

16

Evaluation of University Extension HTML/XHTML Training for Website Development Appendix C Level 3 Extension Educator Survey

17

Evaluation of University Extension HTML/XHTML Training for Website Development Appendix C Contd

18

Evaluation of University Extension HTML/XHTML Training for Website Development Appendix D Website User Evaluation Sheet
Poor
Ease of Navigation Home page link always visible Website looks good in any browser. Text and images load quickly. Information and topics are easy to find.

19
Comment [A16]: Insert instructions for implementation.

Below Average

Average

Above Average

Excellent

Pages are well organized and easy to read. Comment [A17]: You should avoid compound questions (AND)

Information for different Extension Programs, ie. 4H, Agriculture, Nutrition, etc, are current.

Pages for different Extension programs are uniform and attractive.

Page content is not disorganized. Links are not broken and go to correct, up to date information.
Comment [A18]: This is a double negative not disorganized.

Evaluation of University Extension HTML/XHTML Training for Website Development Appendix E Focus Group Questions for Website Evaluation Navigation Is the website easy to navigate or get around to information you want? Are you able to find the information that you came to the site for? Are the pages well organized to make finding information easier? If not, what are they like? Is there a structure to the pages that make the information seem logical? Are the links all connected to the information like they say they are? Content Do the different site areas, ie, 4-H, Agronomy, Health and Human Sciences, have the kind of information you think they should have? If not why not? What could they do better? Do you find the information you need when you come to the site? If not is the issue content or organization? Is the content up to date? Is there old information that needs to be removed? Does the site make use of video and/or sound? Aesthetics Is the site pleasant to look at and visit? Do the colors and images add or detract from the navigation of the site? Do the pages use a common color theme? Does the layout of the site vary a lot from page to page?

20
Comment [A19]: What are the instructions for implementation? How long does this take? Do the participants need access to the web sites to test is individually? Comment [A20]: Are you having them do usability testing of the web sites?

Comment [A21]: Shouldnt the answers to these questions be known before hand? Isnt this part of the analysis that the IDs do? Comment [A22]: How can a user tell WHY the content is not present?

Evaluation of University Extension HTML/XHTML Training for Website Development Appendix F Level 1 Survey

21

Evaluation of University Extension HTML/XHTML Training for Website Development Appendix F Continued Level 1 Survey Continued:

22

Evaluation of University Extension HTML/XHTML Training for Website Development Appendix G - Pre-Test/Post-Test

23
Comment [A23]: Instructions.

Evaluation of University Extension HTML/XHTML Training for Website Development Appendix G Contd

24

Evaluation of University Extension HTML/XHTML Training for Website Development

25

Appendix H Sample Tabulation Ratings for a 10 Point Scale, Excellent to Poor (Kirkpatrick & Kirkpatrick, p. 62)
Comment [A24]: This looks like a 5-pt scale 5 answer options.

_10_Excellent 5_Very Good _3_Good 1_Fair 1 Poor (10x5=50)+(5x4=20)+(3x3=9)+(1x2=2)+(1x1=1)=82 The rating is 82/20 or 4.1