1

Evaluation of Smarthinking Tutoring Program

Submitted to: Dr. John Graham May 2013

Submitted by: Christina Jorgensen

Synopsis
Idaho Virtual Learning (IVL) introduced a new tutoring program, Smarthinking, as an additional resource for teachers and students. The program allows students 18 hours a day of available tutors who offer live one-on-one tutoring. Smarthinking also offers English students a new tool for essays and papers. Students can submit one paragraph or more online for a tutor to review and edit. The students receive feedback within 24 hours and there is no limit to the number of times they can get help. The program was introduced in October 2011 and will run through the end of the 2011-2012 school year. Tutors were provided a 2 hour training of the new program, which included limited hands on experience. Students, teachers, and tutors were given surveys to gather feedback on the program. The data was then collected and analyzed to produce evidence of whether the three main objectives of the program were met. Comments were also collected from the students, teachers, and tutors to provide feedback on any changes that could or should be made if the program were to continue in the future. A summative evaluation was done on the three main objectives for the Smarthinking tutoring program. The objectives are listed below. The program is easy to use and navigate through. The program will help improve grades of those that need the additional help. The program is being used by at least 60% of the math and English students enrolled at IDLA.

Upon completion of the evaluation it was determined that the program is not as easy to use and navigate as first thought. It has been suggested that the tutors have additional training and/or more hands on experience before working with the students. It was also suggested that the students receive some sort of orientation before using the program and that the usernames and passwords be given to the teachers to distribute instead of the main office. Many of the students and teachers reported that there was a definite or somewhat of an improvement to grades after using Smarthinking. English teachers noticed more of an improvement then math teachers. Also, many teachers felt that it was somewhat or was an additional resource to use within their classes. There are 1400 math and English students that had the opportunity to participate in the program. 140 students were randomly selected and out of those 103 responded to the survey, meaning that they used Smarthinking at least once. Further feedback could be collected by a bigger sample size to ensure that at least 60% of the math and English students were using the program. Overall the program is successful. Considerations for improvement focus on training and orientation for and on the program.

Introduction
Smarthinking is an online tutoring program that offers students almost 24/7 assistance in math and English. Smarthinking is part of Pearson, and is the leader in on-demand online student support. The program has proven to boost performance and retention. Idaho Virtual Learning Academy (IVL) is an online school that provides assistance to all school districts in the state of Idaho. School districts work with IVL to offer classes not available to students in their schools due to school size, resources, or money. IVL also provides recovery classes for various subjects and allows school districts to use their classes to help students recover classes and get caught up when time or money does not allow for it inside the traditional classroom setting. IVL has partnered with Smarthinking to provide tutoring services to their students in math and English. Math students enjoy live, one-on-one tutoring 18 hours a day, 7 days a week. Students can drop in for instant help, or can set up a live tutoring session with their teachers or any other available tutor. English students can submit essays or papers for review by a tutor and get feedback within 24 hours. English students also have the option of drop in services for instant help, or they can also set up live tutoring sessions with their teacher or any available tutor. All tutors are highly qualified, according the standards set by the state of Idaho, and are certified in the subjects that they are tutoring. All tutoring sessions are archived for future use by the student and tutor for further discussions and/or test preparation. This report provides feedback from teachers, tutors, and students on how well the Smarthinking tutoring program has been implemented into IVL. It is also intended to help administrators understand what changes can or should be made to help both the student and tutor use the program to its full potential. The report will also be used as a resource when presenting the continual use of the program in the future to the Board of Trustees to see if the program is worth the time and money to continue to use it. Surveys were sent to teachers, tutors, and students to get feedback on the tutoring program. Training was given to many of the tutors and surveys were used to compare those that had training to those that did not, and what benefits there were from that training, if any. Documentation of grades was also used to see if there was an improvement from the students that used the tutoring. This evaluation with provide administrators with feedback from students, teachers, and tutors concerning Smarthinking as a resource worth the time and money. The evaluation will discuss the benefits or pit falls of the program and any changes that can help the program succeed as a worthwhile tool in education online. The evaluation will also provide recommendations for use of Smarthinking in the future as a tutoring tool for IVL. For this evaluation, the terms teachers and tutors were used interchangeably and are referring to the same group of people.

Description of program
The Smarthinking tutoring program is intended for students enrolled at IVL as a resource for additional help in math and English classes. The program is limited to math students in classes

4 Math Foundations (7th grade) to Algebra 2, including Statistics and all English classes, regardless of level or class. It allows students additional help outside of teacher office hours or prescheduled tutoring sessions, set up with the teacher. If the program is successful, further discussions will include other subjects as well. Smarthinking was implemented at the beginning of the 2011-2012 school year and will run for the duration of the school year. The program currently only includes math and English students.

Program Objectives
Smarthinking was implemented into IVL as another resource for teachers and students as a live one-on-one tutoring program. There are three main objectives for the program. The program is easy to use and navigate through. The program will help improve grades of those that need the additional help. The program is being used by at least 60% of the math and English students enrolled at IDLA.

IDLA is looking for new technological tools to help facilitate learning and additional resources for students and teachers to enjoy a successful online learning experience. Administration believes that if the above objectives are met, students and teachers will have success and both will have a better overall learning environment and experience.

Program Components
Computers Students participate in the Smarthinking program from a computer at their given school, at home, or at a library. No computers were given to the students. Students had to provide their own computer if they participated away from their school. Computers are necessary with the program and the online classes. Internet connection Internet connections are required to access the Smarthinking website. High speed is highly recommended but lower speeds are acceptable, but require much more patience. Username/password Students were given individualized usernames and passwords to log into Smarthinking. Tutors are also given individualized usernames and passwords. Usernames and passwords allow students access to their own archived sessions and help tutors keep track of various students. Tutors log into the Smarthinking program by going to the Smarthinking website and logging in using his or her designated username and password. They then set up their scheduled tutoring times. Tutors also have the option to create an impromptu tutoring session. They will see a list of

5 students waiting to be tutored and then click on the student. A live whiteboard session is then created and tutors and students then communicate but writing on the whiteboard. Tutors also have the option of reviewing essays and papers submitted for feedback, and then provide the necessary feedback and then return the essay/paper to the student. Students log into the Smarthinking program by going to the Smarthinking website and then logging in using his or her designated username and password. Students then have the option of submitting essays/papers to the English tutors or they can request a live tutoring session. If they request a live tutoring session they are added to a list and wait for a tutor to start a session with them. Once a tutor has selected them, a new whiteboard opens and students and tutors communicate by writing on the whiteboard.

Evaluation Method
Participants
Students Students that participated in the Smarthinking tutoring program were students enrolled in math classes, Math Foundations (7th grade) to Algebra 2, including Statistics, and all English classes at IVL. They are from grades 7th to 12th, and are from all over the state of Idaho. All students in the above classes were sent surveys asking them to respond if they had used the Smarthinking program. It was available to any student in the classes designated. Tutors Tutors were teachers from IVL that were highly qualified, as determined by the State Department of Education, and certified by the state of Idaho to teach either math or English at the secondary level. All teachers teaching math classes, Math Foundations (7th grade) to Algebra 2, including Statistics, and all English classes were sent surveys. Other important participants were school administration that set up the program, and the lead teachers in math and English, Mrs. Smith and Mrs. Jones, respectively. One other important participant, Mrs. Matthews, who was responsible for putting schedules together and helping get the program off the ground.

Procedures
Before the program was implemented into math and English classes, tutors were offered training for the new program. The training consisted of live online training that allowed tutors hands on experience in the program, practicing with other tutors before tutoring students. Training was for one day and lasted roughly 2 hours.

6 Throughout the year, students have been using the tutoring program to get additional help in math and/or English. Tutors have been attending scheduled tutoring hours all year long and have added pre-scheduled tutoring sessions when requested by students.

Data Sources
Surveys were sent to students to collect data on the overall effectiveness of the program and to provide feedback from the students. Surveys were also sent to tutors to collect data on how active the tutors were, effectiveness in the classroom, and provide feedback from the tutors about the program. Surveys were given to tutors to collect data on the success of the training and to document who attended training and who did not. All surveys were created using surveymonkey.com. Student surveys can be found at http://www.surveymonkey.com/s/PH8RL2N. Tutor surveys can be found at http://www.surveymonkey.com/s/PZKQYBT. Copies of questions asked in the student survey and teacher survey can be found in the appendix, respectively.

Results
The following are figures that represent the data collected and analyzed during the evaluation. 70 tutors/teachers were involved and 47 responded to the survey. There were 1400 students involved and 140 were randomly selected to participate in the survey, with 103 responding.

Teachers on Improvment
20 15 10 5 0 Improvements, English Improvements, Math

Smarthinking as a resource

18% 20%

22%

Yes Somewhat Not Really

40%

No

Figure1. The graph shows if teachers saw improvement with students grades after using Smarthinking.

Figure 2. This graph shows the percentage of teachers that felt that Smarthinking helped their classes as an additional resource.

Userbility of Smarthinking
Difficult Somewhat difficult Somewhat easy Really easy 0 20 40 60 Students Tutors

Figure 3. Shows how easy tutors and students thought Smarthinking was to use and navigate. Students' Perspective
No Somewhat Yes 0 20 40 60 80 Grades improve Tutor Helpful

Figure 4. Shows the students perspective of the tutors and the improvement of their grades.

Discussion
The purpose of this evaluation was to see if (1) the program was easy to use and navigate through, (2) the program helped improve grades of those that need the additional help, and (3) the program is being used by at least 60% of the math and English students enrolled at IDLA. According to the data analysis, 23 out of 50 of the tutors and 67 out of 103 students thought that the program was easy or somewhat easy to use and navigate through. 64% of teachers felt as though the program helped improve their students grades and 76% of the students felt as though the program helped or somewhat helped improve their grades. 103 out of 140 students submitted the survey, meaning that they had participated at least once in the Smarthinking tutor program. That is 73.5% of the students randomly selected to participate. Figure 1 showed if the teachers saw improvement with students grades after using Smarthinking. Since English had different tools available to them besides just the live tutoring, English and math were spilt into their own groups. More improvements were seen with the English grades overall, but only one math teacher noted that he or she saw absolutely no improvement at all.

8 Figure 2 showed the percentages of teachers that felt that Smarthinking helped their classes as an additional resource. 62% felt that it did help a lot or somewhat. Only 18% felt that it did not add anything to their classes. Figure 3 shows how easy the students and tutors felt it was to use and navigate through the Smarthinking tutoring program. 15 students and 7 tutors felt that it was difficult to navigate through. Out of 7 tutors, 5 did not attend the training meeting. 30 students and 10 tutors felt that it was somewhat difficult, but 70% of them also noted in the comments box, at the end of the survey, that the more they used the program the more they felt comfortable with it. Figure 4 showed the perspective of the tutors according to the students and also if the students, themselves, felt that the program helped to improve their grades. 68% of the students felt as though the tutors were helpful. 32% felt that the program definitely helped improve their grades, while 44% felt that it had somewhat of an effect, though many noted that they tried harder this semester then in years past because the classes were needed to graduate. After completing the evaluation and reviewing the data, one of the biggest factor that stood out was the ease, or in this case difficulty, to which the students and tutors had in using and navigating the program. The students and tutors had a harder time using the program then planned. Many tutors commented that they would like more hands on practice using the program before having students. Training was given to the tutors, and some hands on practice were given during the two hour training. None of the students received any training or instruction before trying to use the program, and many commented that it was difficult to get their usernames and passwords. Other teachers commented that it would be easier if the teachers had the students usernames and passwords for Smarthinking, instead of having to be the middle man and contact Katie Laubengayer. Upon completion of the data collection and then the analysis, the evaluation of the Smarthinking tutoring program is complete. The three main objectives for this program were to (1) see if the program was easy to use and navigate through, (2) see if the program helped improve grades of those students that needed the additional help, and (3) see if program was being used by at least 60% of the math and English students enrolled at IVL. It was determined that the program was not as easy to use and navigate through. It has been suggested to provide more training opportunities to the tutors prior to students using the program and provide some sort of orientation for the students, at which time the students would receive their usernames and passwords. Many tutors and students did not have a problem with it and this can help provide encouragement to those in management. The program was supposed to help improve the grades of students that needed the additional help. 76% of the students felt that it definitely or somewhat helped their grades. 64% of teachers felt that the program definitely helped or somewhat helped their students improve their grades. About half of the teachers commented that the more the program is used in the school the more they believe that the grades will improve. Teachers are starting to get behind the program and are more willing to announce the program to their classes.

9 The third objective for the program was to have at least 60% of the students use Smarthinking. Out of the 1400 students in the math and English classes that have tutoring available to them, 140 were randomly selected to be surveyed. It was asked that only those that had used Smarthinking complete the survey; out of the 140 randomly selected 103 replied. 74% of those randomly selected have been using Smarthinking. More students can be selected and surveyed to ensure that the numbers are accurate. According to this evaluation, more than 60% of the students are using Smarthinking. Overall the program was very successful and is recommended to continue. Changes that could occur to better the overall production of the program would be to add training, and/or more hand on experience for tutors and provide some sort of orientation for the students before they need to use it. The success of the first year also encouraged those teachers that participated to be more influential in using it next year and encouraging their students to be participants.

10

Project Cost
Staff Mrs. Christina Jorgensen Staff Total Travel Mrs. Christina Jorgensen Travel Total Supplies/Materials Printing Costs Office Supplies Supplies/Materials Total Program Costs Total Rate $500.00 Days 11 Total $5,500.00 $5,500.00 Total $253.55 $253.55 Total $15.00 $20.00 $35.00 $5,788.55

Rate $.55

Miles 461

11

Appendices
To Smarthinking tutors: My name is Christina Jorgensen and I am one of the math teachers here at IVL. I am evaluating the new Smarthinking tutor program that IVL is implementing this year. I am asking you to fill out the survey at the following link. This survey will help me to see what teachers are also tutors for Smarthinking and get your input on how you feel the new tutoring program is helping/hurting the students in your specific classes. Thank you for your help in this. http://www.surveymonkey.com/s/PZKQYBT

Sincerely, Christina Jorgensen

To Students in math and English: My name is Christina Jorgensen and I am one of the math teachers for IVL. I am evaluating the new Smarthinking tutoring program that we are using this year. I was wondering if you would be willing to feel out this survey. Please only feel it out if you have actually used the program. Thank you for your time and help. http://www.surveymonkey.com/s/PH8RL2N

Sincerely, Christina Jorgensen

12 Questions asked on the surveys for students and teachers/tutors. Students: 1. 2. 3. 4. 5. 6. Was the program easy to use and navigate through? Was your tutor helpful? Did the help you get from the tutor help you do better in your class? English students, did you like the new tools available for essays and papers? How many times this semester have you used the tutoring program? Feel free to write any additional comments.

Teachers: 1. Have you seen an improvement in grades by those that were getting help through the tutor center? 2. Did you feel that the new program has helped overall as an additional resource in your own classes? 3. English teachers, have you found an improvement in essays and papers turned in? 4. Is the program easy to use and navigate through? 5. How many students have you seen over the semester? 6. Did you work one hour or two? At what times and days did you work? 7. Feel free to write any additional comments.

13 Extra figures not included above.

Training

35% 65%

Yes No

Figure 5. Shows percentage of tutors that took training.

English Students use of new tools

40 30 20 10 0 Yes No Didn't use it English Students use of new tools

Figure 6. This graph shows whether the English students used the new tools available through Smarthinking.