Evaluation of NFHS Creating a Safe and Respectful Environment Training Course for the Wyoming Department of Education

Submitted by Jennifer Anderson of JLA Associates Boise State University July 27, 2012

Table of Contents I. II. Summary Program Description a. Program Objectives b. Program Components III. Evaluation Method a. Participants b. Procedures c. Data Sources IV. V. VI. VII. Results Discussion Project Cost Appendices a. Survey Questions b. Checklist c. Invoice d. Training Course License e. Website Links 5-6 6-7 7-8 9-15 4-5 3 4

2|Page

Summary The following report describes an evaluation of an online training course called, Creating a Safe and Respectful Environment offered by the National Federation of State High School Associations (NFHS). The evaluation began by a request from the Wyoming Department of Education (WDE) to evaluate the training course in order to make a decision regarding the future use of the course. The course is currently offered to state advisors for the Career Technical Student Organizations in the state. The evaluation began with a discussion among stakeholders regarding the goals of the evaluation and the program objectives. The Wyoming Department of Education wanted to know if the training course met its own course objectives and if it was relevant and comprehensive enough for the population they were using it for. After the discussion with the WDE the evaluation specialist began creating data sources to evaluate the training course. The data sources included a participant survey and an evaluator checklist. The participant survey provided results in favor of the training course. Each of the participants agreed or strongly agreed that they could define the necessary information, provide examples, and prevent and control inappropriate behaviors, hazing, bullying, and Cyberbullying. This information may be misleading because it is unknown if the participants were considering their prior knowledge or only the information received from the training course. The narrative feedback from the survey was somewhat underwhelming and lacked convincing information to continue using the course or to search for an alternative. The participants identified the information as relevant and pertinent to their positions. The evaluation checklist measured the completion of the eight course objectives. The evaluation specialist critically observed and completed the training course while using the checklist to assess which objectives were completed. Results showed that all eight course objectives were met by the training course. Using the information, the evaluation specialist has answered the Wyoming Department of Educations question regarding if the course meets its own course objectives. The next step is for the WDE to consider if the information in the training module is relevant enough for their purposes and population. The results show that the participants can define, provide examples, and feel ready to prevent and control inappropriate behaviors, hazing, bullying, and Cyberbullying. However, the narrative feedback identifies areas that the participants would like more information, such as specific state advisor duties. The evaluation specialist recommends that the Wyoming Department of Education continues to use this training module until they can identify an alternative that more closely aligns with the state advisor duties. This training module may not be available currently, and may need to be created in-house by the WDE.

3|Page

Program Description The Wyoming Department of Education (WDE) requested an evaluation of an online training course that is required of the state advisors for each of the five Wyoming Career Technical Student Organizations (CTSOs), including DECA (Distributive Education Clubs of America), FBLA (Future Business Leaders of America), FCCLA (Family, Career, and Community Leaders of America), FFA (Future Farmers of America), and SkillsUSA. The WDE hires a state advisor to oversee each of the CTSOs and organize student leadership conferences over the course of two years. These advisors are in contact with hundreds to thousands of middle school, high school, and college age students. Therefore, the WDE feels it is necessary for the state advisors to complete training on ethical behaviors when working with children. The WDE has identified an online training course called, Creating a Safe and Respectful Environment. This training is geared towards coaches and teachers and is developed by the National Federation of State High School Associations (NFHS). The WDE identified the need to evaluate if this course meets its objectives and if it provides relevant information for state advisors. The training course is self paced and on average takes 45 minutes to complete. The course objectives are described in the introduction of the course and include the following: 1. Identify the roles and responsibilities of teachers and coaches when working with students in an interscholastic activity. 2. Prepare teachers and coaches for their role in controlling and preventing inappropriate behaviors, hazing, bullying, and Cyberbullying. 3. Prepare teachers and coaches for their role in reporting inappropriate behaviors, hazing, bullying, and Cyberbullying. 4. Define inappropriate behaviors, hazing, bullying, and Cyberbullying. 5. Provide examples of inappropriate behaviors, hazing, bullying, and Cyberbullying. 6. Describe four types of abuse. 7. Identify signs of hazing. 8. Identify and describe the elements of a good reporting system. In order to complete the online training course participants must complete a free registration with the NFHS Learning Center online. Once registered, participants must either purchase the online course for $20 or redeem a license allowing them to take the course for free. The course itself has five sections 1) Introduction; 2) Inappropriate, Abusive, or Harassing Behavior; 3) Hazing; 4) Bullying and Cyberbullying; and 5) Conclusion. The three main topics including inappropriate, abusive, and harassing behavior, hazing, and bullying and Cyberbullying present videos, legal cases, definitions, transcripts, and resources on each of the topics. At the end of these three sections there are quizzes to assess what was learned. Upon completion of the entire training participants receive a certificate of completion for the course. Evaluation Method The decision making model was used for this evaluation in order to assess the effectiveness of the Creating a Safe and Respectful Environment online training course. The evaluation consisted of data collection from a population including five Career Technical Student Organization (CTSO) state advisors and the evaluation specialist. The five state advisors completed the NFHS online training and evaluation survey. The participants were all in the
4|Page

education field and have had moderate levels of experience working with students. The evaluation specialist completed the training as well, but did not participate in the survey. The evaluation began with the development of an Evaluation Program Description (EPD). The EPD allowed the evaluation specialist to identify the need for the evaluation and address specific questions the WDE wanted answered. The evaluation specialist sent notifications to the state advisors for them to complete the online training course. The evaluation specialist also completed an observational analysis of the training course by using a checklist to assess if the objectives were met. After the training was completed by all six participants, the evaluation specialist created a survey to give to the remaining five participants in order to collect data on the module itself and the relevancy to the state advisors. Participant performance was measured by completion of the course. All participants were issued a certificate of completion upon passing 3 quizzes within the training course. The data sources included a checklist and survey. The checklist (appendix a) used to assess the training course was developed by the evaluation specialist and used during observational analysis of the course. The checklist identified which objectives were met. The surveys were given to the participants after the survey was completed. The survey (appendix b) used Likert scale items to assess the attitudes of the advisors, relevancy of the training course, and objectives met. The five level Likert items included the following choices: strongly agree, agree, neutral, disagree, and strongly disagree. The survey also provided an area for participant narratives and feedback regarding each module. Results Participant Survey The results for this evaluation were found using a participant survey and an evaluation specialist checklist. Five participants completed the survey after taking the online training course. All five participants were females with 20 or more years of experience working with students. Table 1 shows that 100% or five of the five advisors agreed or strongly agreed they could define inappropriate behaviors, hazing, bullying, and Cyberbullying after completing the training. All five advisors also agreed or strongly agreed they understood the elements of a good reporting system. Five out of five agreed or strongly agreed they could identify signs of hazing and bullying after the training. Table 2 shows that five of the five advisors could provide examples of inappropriate behaviors, hazing, bullying, and Cyberbullying after the training. All five advisors agreed or strongly agreed they could describe four types of abuse and inappropriate behaviors. Table 3 shows that five of the five advisors agreed or strongly agreed they understood their role in preventing and controlling inappropriate behaviors, hazing, bullying, and Cyberbullying. Table 4 shows that five of the five advisors agreed or strongly agreed they felt prepared to report any inappropriate behaviors, hazing, bullying, or Cyberbullying. Two of the five advisors agreed that the training was excellent while three of the advisors rated it as neutral. All five advisors agreed that the training was relevant to their positions. When asked to describe their overall thoughts on the training the responses were: None; The training was really not any new information but a great reminder on how to handle and prevent situations; good; I appreciated the format of the course. It was convenient and effective. It was updated and pertinent information; and it was ok. When asked if they had any comments regarding the categories one participant responded, liked the statement what you permit you promote. The
5|Page

other participants answered the additional narrative questions by saying no, none, or NA. When asked what important information the training was missing one participant stated, specific information regarding state advisors.

% Who Strongly Agree or Agree they can define the category

% Who Strongly Agree or Agree they can identify examples of the category

Chart 1
100 75 50 25 0

Chart 2
100 75 50 25 0

% Who Strongly Agree or Agree they they understand their role in preventing and

100 75 50 25 0

% Who Strongly Agree or Agree they feel prepared to report the category

Chart 3

Chart 4
100 75 50 25 0

Evaluator Checklist The evaluation specialist generated a checklist (appendix B) to assess if the online training course met its 8 identified course objectives. The checklist shows that all 8 of the objectives were accomplished through the instruction provided in the online training course. Discussion The purpose of this evaluation was to provide the WDE with information in order to make a decision on using the NFHS online training course in the future. The WDE wanted to know if the training module was applicable, feasible and comprehensive enough to use in order to prepare state advisors for their positions, but the overall focus of this evaluation was to identify if this course met its own objectives. The results proved that the course did complete all 8 objectives identified within its description and introduction. The participant survey was beneficial in that it provided information from the actual population that the training course will be used for in the future. The results from the survey show that the participants feel prepared to define, report, and control inappropriate behaviors,
6|Page

hazing, bullying, and Cyberbullying. In response to the question asking for the participants to describe their overall thoughts on the training course one of the participants stated, The training was really not any new information but was a great reminder on how to handle and prevent situations. It appears the participants understood the information and are able to discuss examples of the behaviors. The survey indicates that the training course was successful in educating them and preparing them for inappropriate behaviors, hazing, bullying, and Cyberbullying. It did appear that the participants may have wanted more information regarding their positions as state advisors, this it may be necessary to continue looking for alternative training courses or possibly develop one in house. One question that arises from the survey is if the survey results are accurate based upon the training provided in the course or if they come from prior knowledge? The questions were to be answered after the training and based upon the information received from the training. However, it is not known if the participants answered them based on the training information or their own prior knowledge, as they commented that the information was not very new. The participants also identified that they all have 20 or more years experience working with students. There may also be a want to please or prove their knowledge to the Wyoming Department of Education by answering positively to the survey questions. The advisors would not want to be portrayed as incapable based upon their answers to the survey questions. The checklist provided data on the course objectives and showed that all eight of the objectives were met. Objective number one was met with many roles and responsibilities described in each section. The course discussed the reporting requirements and major areas for communication when handling these situations. The training did not provide specific steps for reporting but provided nice examples of what not to do. Each of the categories was defined and provided relevant examples for the participants. All eight of the course objectives were met by using the evaluation checklist. The course met its objectives but the Wyoming Department of Education will need to take into consideration some of the comments made by participants and their prior knowledge. There appeared to be agreement between the participants that the training was relevant to their positions. However, comments suggested that the information was not new to them but rather a review. The Wyoming Department of Education may need to consider if this training really meets their expectations for what they want the state advisors to know. The WDE must also consider the possibility of inaccurate survey responses, based upon the participants prior knowledge and want to appear knowledgeable to the department. It is recommended that the Wyoming Department of Education continue to use the NFHS training course until they can either identify a training course that aligns with their expectations for a state advisor, or until they can develop and in house training that directly meets their needs and expectations. Overall, the evaluation proved that the online training course accomplished its eight course objectives and is a valuable resource for those eight items. Project Cost The evaluation took 20 days to complete, allowing participants enough time to complete the training module and survey. The evaluation specialist required time to meet with the WDE team to create the EPD, time to develop the data sources and contact the participants and allow them time to complete the training. Finally, the specialist required time to complete the online
7|Page

training and complete the data analysis and reporting requirement. There were no travel costs incurred, as the evaluation was completed in Cheyenne, Wyoming and through virtual tools. The cost of the training was paid for by the Wyoming Department of Education. The total cost of the evaluation was $10,000. See the attached invoice (appendix C) for itemized description of costs.

8|Page

Appendix A Survey 5 point Likert Scale Items: 1. I can define inappropriate behaviors 2. I can describe four types of abuses. 3. I can list examples of inappropriate behaviors. 4. I understand my role in controlling and preventing inappropriate behaviors. 5. I feel prepared to identify inappropriate behaviors. 6. I feel prepared to report inappropriate behaviors. 7. I understand the key elements of a good reporting system. 8. I can define hazing 9. I can identify signs of hazing. 10. I can describe types of hazing. 11. I feel prepared to identify hazing activities. 12. I feel prepared to report hazing activities. 13. I understand my role in controlling and preventing hazing. 14. I can define bullying. 15. I can define Cyberbullying. 16. I can describe examples of bullying. 17. I can describe examples of Cyberbullying. 18. I understand my role in controlling and preventing bullying. 19. I understand my role in controlling and preventing Cyberbullying. 20. I feel prepared to prevent bullying. 21. I feel prepared to prevent Cyberbullying. 22. I will use the three rules to preventing bullying and Cyberbullying. 23. Overall, I would rate the course as excellent. 24. The course content was relevant to my position as state advisor. 25. The course was understandable. 26. The online registration was easy. 27. The course navigation was easy. 28. The transcript was beneficial to the module. 29. The additional resources and links were helpful. Narrative Questions below: 30. Please describe your overall thoughts on this training. 31. Please list any information or topics you think were missing or should be included in a future training. 32. Do you have suggestions related to the inappropriate behaviors unit? 33. Do you have any comments related to the inappropriate behaviors unit? 34. Do you have any suggestions related to the hazing unit? 35. Do you have any comments related to the hazing unit? 36. Do you have any suggestions related to the bullying and Cyberbullying unit? 37. Do you have any comments related to the bullying and Cyberbullying unit? 38. Gender? 39. Years Experience working with students?
9|Page

Appendix B Checklist COURSE OBJECTIVE Objective 1: Identify the roles and responsibilities of teachers and coaches when working with students in an interscholastic activity. MET X

Comments: Responsibilities included modeling appropriate behavior, reporting activities to proper authorities, instructing students and staff in procedures and policies, acting in loco parentis or in place of parent. Educators, judges, and uniformed people are held at a higher standard because they are whom parents entrust their children to mold into responsible citizens. Roles of Coaches: Model appropriate behavior, responsible for effectively communicating abuse and harassment policies, stay current with policies, educate your students and staff about procedures and policies. Follow the elements of a good reporting system. Role in Preventing Hazing: Assure students they are safe and protected from retribution Ask questions Know your school district policies on hazing Know proper reporting requirements and those to report to Role in Preventing Bullying: Understand bullying may be misconstrued of leadership Bullying may be physical or verbalunderstand this is not ok You must be vigilant and responsible to prevent the behavior and communicate that it is not acceptable. Suggestions to prevent bullying and Cyberbullying: Know and follow school or district policies and procedures If policies dont exist, be proactive in helping to develop Communicate a desire for positive team leadership Communicate that no form will be tolerated Social media is not acceptable when used negatively Open door policy Enlist parents help in reporting or preventing bullying Limit use of electronic communication to necessities Know the technology and social media language (netlingo.com) Remind students and staff that electronic communications can be recovered, stolen, traced, and used against you. Dont write it, Dont forward it 1) Dont respond 2) Save evidence
10 | P a g e

3) Tell a trusted adult. Objective 2: Prepare teachers and coaches for controlling and preventing inappropriate behaviors, hazing, bullying, and Cyberbullying. X

Comments: Legal cases provided examples of inappropriate behaviors and hazing activities. Must educate personnel about state reporting requirements, immediately report any discovered cases. Failure to Report is a serious offense. Lacks specific steps to control or prevent, but addresses the issues through video examples. See comments for Objective 1. Objective 3: Prepare teachers and coaches for reporting inappropriate behaviors, X hazing, bullying, and Cyberbullying. Comments: Identified examples of non-reporting legal cases for inappropriate behaviors, hazing, and bullying through social media. Provide resources on other legal cases. See comments from objective 1. No specific steps for reporting identified, but legal case examples provided good information for problems that arise from not reporting. Objective 4: Define inappropriate behaviors, hazing, bullying, and Cyberbullying X

Comments: Provided definitions, examples, videos, recommendations, and legal case examples to explain each. First show examples of the types of behaviors and define each. Inappropriate behaviors included 1) physical abuse such as hitting, shoving, choking, biting, withholding meals, caressing, fondling, etc.; 2) Emotional abuse such as brainwashing, namecalling, accusing, ordering, threatening, countering, minimizing, etc.; 3) Sexual Abuse/Harassment such as requests for favors, verbal conduct, non verbal conduct, physical conduct of a sexual nature, Quid Pro Quo etc.; and 4) Neglect or the unavailability of reasonable services. Hazing: Any humiliating or dangerous activity expected of a student to belong to a group, regardless of their willingness to participate. Bullying: Intentionally hurting or scaring someone unable to adequately defend themselves. Types of bullying: physical pain, spreading rumors, keeping others out of a group, mean teasing, organizing people to gang up on others. Cyberbullying: Bullying through use of electronic and social media to harass. Types: sending mean texts, email, or other messages online. Posting inappropriate pictures or messages, using others usernames inappropriately, to spread rumors, creating false online profiles to embarrass or hurt someone.

11 | P a g e

Objective 5: Provide examples of inappropriate behaviors, hazing, bullying, and X Cyberbullying. Comments: See comments for objective 4. Statistics were provided as well. 48% of students experience some form of sexual harassment. 9% of students harassed report it. 50% of students did nothing in response. Types of hazing include physical punishment, branding, sleep deprivation, kidnapping, tattooing, piercing, head-shaving, inappropriate sexual behavior, unreasonable/unacceptable foods or beverages, forced consumption, etc. Prevalence of hazing1.5 million students were subjected to hazing in American High Schools. 76% were hazed between ages of 10 and 15. Objective 6: Describe four types of abuse. X Comments: Four types of abuse include physical abuse, emotional abuse, sexual abuse, and neglect. See comments for objective 4. Objective 7: Identify signs of hazing. X

Comments: Signs of hazing include excessive fatigue, disheveled appearance, inappropriate clothing, isolation from friends or family, miss class due to lack of sleep, and showing signs of depression. Objective 8: Identify and describe the elements of a good reporting system. X

Comments: Elements of a Good Reporting System 1. Communicate to your students, coaches/activity personnel, officials and parents that you have a policy against abuse, sexual misconduct and other forms of harassment. 2. Make the process of reporting allegations simple and direct. 3. Make the principal, as the highest administrative official in the building, responsible for directing investigations of any allegation of abuse, misconduct or harassment. 4. Document in writing any allegation of abuse, misconduct or harassment and the results of the investigation. 5. Promptly take remedial action as is necessary and appropriate. When in doubt, dont hesitate to contact the proper authorities according to your school district policies.

12 | P a g e

Appendix C

JLA Associates
SUBMITED TO:

INVOICE
Attn: CTE Section Wyoming Department of Education 2300 Capitol Avenue Hathaway Building, 2nd Floor Cheyenne, WY 82002-0050

Invoice #300 Date: July 25, 2012

DESCRIPTION

Personnel Evaluation Specialist 20 days X $500 per day

AMOUNT
$10,000.00

TOTAL Invoices due upon receipt*. Please make checks payable to:

$10,000.00

JLA Associates Inc. 4151 Jander Drive Laramie, Wyoming 82072

*All payments not received within thirty business days will be charged 18% interest on outstanding amounts.

13 | P a g e

Appendix D
Jennifer Anderson <jennifer.anderson@wyo.gov>

No-Reply :: NFHS - Creating a Safe and Respectful Environment

1 message

accounts@nfhslearn.com <accounts@nfhslearn.com> To: jennifer.anderson@wyo.gov Cc: jennifer.anderson@wyo.gov Dear Jennifer Anderson,

Wed, Jun 20, 2012 at 11:08 AM

We are giving you access to the NFHS Creating a Safe and Respectful Environment course. We hope you can take the time to complete the course in its entirety. License #: SE0103839490 Step 1: To redeem your course with your license # you must first be registered and signed in to the site. To register you can go to http://www.nfhslearn.com/registration.aspx. If you are already registered, please go to http://www.nfhslearn.com/ to sign in. Step 2: Once you have signed in, visit your homepage by clicking on the "My Homepage / Courses" link. Step 3: Click on the "Redeem License" button on your homepage, found in the box just below the "My Available Courses" box. Step 4: Enter the License # above into the box and click "Submit". Once you have successfully redeemed your license you may now launch the course from your homepage. NOTE: License numbers are a blend of letters and numbers. If your license is not accepted on the first try, please check to make sure you have entered it correctly (the letter "O" is often mistaken for the number "zero"). Thank you, NFHS

14 | P a g e

Appendix E Website: www.nfhs.learn.com

Survey Link: https://docs.google.com/spreadsheet/viewform?pli=1&formkey=dE9CWm5kY0JOZ2tXMmRuTVVOSG lOX2c6MQ#gid=0

15 | P a g e