Anda di halaman 1dari 18

Program Evaluation Tool

Paul A. Schalm Elementary School


Clawson Public Schools

Mrs. Bianca Hill


940 N Selfridge Blvd
Clawson, MI 48017-1033

Document Generated On June 14, 2017


TABLE OF CONTENTS

Introduction 1

Strategy/ Program/ Initiative Description 2

1) Readiness: What is the readiness for implementing the strategy/ program/ initiative? 4

2) Knowledge and Skills: Did staff and administrators have the knowledge and skills to implement the
strategy/ program/ initiative? 7

3) Opportunity: Was there opportunity for high quality implementation of the strategy/ program/
initiative? 9

4) Implementation with Fidelity: Was the strategy/ program/ initiative being implemented as intended?
12

Impact: What was the impact of the strategy/ program/ initiative on students? 14

Impact Conclusion 15

Report Summary 16
Program Evaluation Tool
Paul A. Schalm Elementary School

Introduction

Evaluation of strategies, programs, and initiatives to accelerate achievement and close achievement gaps is a key step in the continuous
school improvement process. In addition, all federal programs (Title I Part A, C, and D; Title II, and Title III) require annual evaluation,
especially when federal and/or state funds are used to support such efforts. More importantly, evaluation represents good practice and will
likely improve outcomes. The Program Evaluation Tool can be used both during implementation to make mid-course corrections as well as
following implementation to identify why results turned out as they did and how to improve implementation that will lead to increased student
achievement.

Page 1
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.
Program Evaluation Tool
Paul A. Schalm Elementary School

Strategy/ Program/ Initiative Description

What is the name of the strategy/ program/ initiative being evaluated?

Fountas and Pinnell's Leveled Literacy Intervention program which is used with students who have been identified as needing reading
support in grades 1-3.

Provide a detailed description of the strategy/ program/ initiative being evaluated.

The LLI program is being used in grades 1-3 for students eligible to receive intervention support. Students are identified as being eligible
using tri-annual screening data including (but not limited to) iReady, DRA, or F&P Benchmark assessment scores and who meet other
eligibility criteria (free/reduced lunch, English Language Learners, etc). LLI groups are delivered in small group settings (3-4 in grades 1 and
2, 5-6 in grade 3) by certified Reading Specialists. Students meet for intervention groups 4-5 times a week for 30 minutes. At the end of
every level students are assessed using a F&P Benchmark assessment.

What is the need being addressed by the strategy/ program/ initiative?

LLI is used with students whose literacy scores are low in vocabulary and/or comprehension as well as other basic literacy foundational skills
as identified by the online diagnostic iReady and other benchmark assessments.

Students who are selected for the program are one grade level below their peers as shown by initial screening and diagnostic data.

What is the reason for selecting the strategy/ program/ initiative including intended results?

In order to have 85% of students be proficient in Reading by 2022, LLI was chosen to support students who are not making adequate
progress in reading based upon benchmark assessments and who are not on target to meet proficiency levels on statewide assessments.
LLI is a widely used research-based program founded on the principles of Marie Clay.

Cite the research supporting the strategy/ program/ initiative, including a brief summary of research findings and targeted
population.

Leveled Literacy Intervention (LLI) consists of a series of planned lessons designed to provide supplementary instruction to kindergarten,
first, second and third grade children who are finding it difficult to learn to read and write. Teachers first use a systematic assessment to
determine children's instructional reading level, and then form groups of three children who are reading at approximately the same level.
Teachers then implement daily 30-minute lessons that incorporate a variety of instructional approaches, each of which is supported by
research (Fountas & Pinnell, 2003). In this way, it is possible to identify struggling readers and writers and intervene early rather than allow
the path of failure to continue (Juel, 1988). Over the last two decades, research has shown the need for and effectiveness of early
Page 2
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.
Program Evaluation Tool
Paul A. Schalm Elementary School

intervention in undercutting the costs of failure (Clay, 2007; Goldenburg, 1994; Hiebert & Taylor, 1994; Schmidt, Askew, Fountas, Lyons, &
Pinnell, 2005). Acquiring effective reading strategies early, along with having the opportunity to apply them to many texts over the years,
builds reading ability over time. Research has long supported the concept that readers learn by reading successfully; that process must be
right from the start (Stanovich, 1986).
http://www.heinemann.com/fountasandpinnell/research/lliresearchbase.pdf

Page 3
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.
Program Evaluation Tool
Paul A. Schalm Elementary School

1) Readiness: What is the readiness for implementing the strategy/ program/ initiative?

IN AN IDEAL STRATEGY/ PROGRAM/ INITIATIVE, stakeholders are well-prepared to implement the program. They have read and can
articulate the research foundation, and regularly use the terms in conversation with each other, students, and with parents. Staff, students
and parents express a high level of interest in, support for and commitment to the program. Specific concerns have been identified and
solutions have been planned/ implemented. Staff is able to seamlessly integrate the program within the context of other building/district
initiatives.
Overall Rating: 3.0
Statement or Question:a) What is the evidence regarding stakeholder (staff/ students/ parents) understanding of the need as well as
stakeholder ability to articulate the reason for the choice of the strategy/ program/ initiative?
Response:

Meeting agendas/minutes
School Improvement Plan elements
Professional development materials
Data collection plan; data analysis work

What does the evidence show regarding stakeholder (staff/ students/ parents) understanding of the need as well as stakeholder
ability to articulate the reason for the choice of the strategy /program/ initiative?

Teachers meet tri-annually in grade leveled data meetings to discuss which students need intervention as well as what type of intervention
would best meet various student needs. Staff have ample opportunities to ask questions about the program during data meetings, grade
level staff meetings, and can attend an LLI session at any time. Reading Specialists meet with parents at the fall parent-teacher conferences
about the program and the ways in which their child was chosen and what components of literacy are addressed during intervention.

Statement or Question:b) What is the evidence regarding stakeholders (staff/ students/ parents) having a shared vision and strong
commitment to the strategy/ program/ initiative?
Response:

Meeting agendas/minutes
School Improvement Plan elements
Data collection plan; data analysis work
School improvement team agendas

Parent compacts and information about the LLI program are sent home as students are added to intervention groups.

What does the evidence show regarding stakeholders (staff, parents, students) having a shared vision and strong commitment to
the strategy/ program/ initiative?

According to previous survey data, staff and parent support for LLI was positive. 83% of staff members stated they were satisfied with the
program. 100% of parents who responded also stated they were satisfied with the program. 100% of the responding parents indicated that
they read the take home books for 11-25 minutes each day.

Page 4
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.
Program Evaluation Tool
Paul A. Schalm Elementary School

Statement or Question:c) What is the evidence regarding how stakeholder (staff, parents, students) concerns were identified and
addressed?
Response:

Meeting agendas/minutes
Staff surveys
School Improvement Plan elements
Data collection plan; data analysis work
Stakeholder survey results

What is the evidence regarding how stakeholder (staff, parents, students) concerns were identified and addressed?

From our prior survey it was discovered that 83% of the teachers would like an opportunity to learn more about the program. The parent
survey showed that most students were happy to be included in the program.

Statement or Question:d) What is the evidence regarding the ability of staff and administrators to integrate the strategy/ program/ initiative
with existing work?
Response:

Meeting agendas/minutes
School Improvement Plan elements
Data collection plan; data analysis work

What does the evidence show regarding the ability of staff and administrators to integrate the strategy/ program/ initiative with
existing work?

Both Reading Specialists were trained to use the LLI program, have access to the MAISA units for Reading and Writing, and are familiar with
the Michigan Core Standards in Literacy. LLI is a stand-alone program addressing reading strategies that integrate across the curriculum by
linking intervention lessons with what occurs in the classroom.

Statement or Question Response Rating


Given the evidence you've assembled, choose Support and commitment were generally high, 3
one overall self-assessment of the readiness for but some concern or work remains.
implementing the strategy/ program/ initiative.

What action steps are needed to increase readiness to implement the strategy/ program/ initiative?

Page 5
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.
Program Evaluation Tool
Paul A. Schalm Elementary School

Use a teacher lab or instructional rounds model so that teachers can visit the intervention classes as LLI is being taught so they can better
understand the scope of the program. Parent workshops would also help to strengthen the home/school connection of the program. Next
school year we would like to schedule an initial meeting early in the school year to explain to parents what LLI is, why their child was chosen,
and what they can do to help at home. Parents will also be given the Parent Read at Home plan as designated by the new 3rd Grade
Reading Law.

Page 6
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.
Program Evaluation Tool
Paul A. Schalm Elementary School

2) Knowledge and Skills: Did staff and administrators have the knowledge and skills to implement
the strategy/ program/ initiative?

IN AN IDEAL STRATEGY/ PROGRAM/ INITIATIVE, personnel are able to clearly articulate what successful implementation looks and
sounds like and how specific practices will change as a result of its implementation. Staff and administrators can articulate specific outcomes
and specific criteria for evaluation. Personnel can demonstrate their ability to apply the knowledge and skills required to successfully
implement with fidelity, and professional learning opportunities are provided to address gaps in knowledge and skills.
Overall Rating: 4.0
Statement or Question:a) What is the evidence regarding staff and administrators' plan for how practice would change as a result of the
strategy/ program/ initiative?
Response:

Superintendent or administrator observations/walkthroughs


Program simulations, administrator observations

Reading strategies emphasized in the LLI program align with strategies used in the classroom.

What does the evidence show regarding staff and administrators' plan for how practice would change as a result of the strategy/
program/ initiative?

Administrator observations check to see that the program is delivered with fidelity and provide feedback to the Reading Specialists about
their lesson delivery. As we get further into use of the LLI program there is more cohesion between Reading Specialist use as well as links to
the classroom.

Statement or Question:b)What is the evidence regarding administrator knowledge of and ability to monitor and assess the effectiveness of
the strategy/ program/ initiative?
Response:

Superintendent or administrator observations/walkthroughs


Program simulations, administrator observations

What does the evidence show regarding administrator knowledge of and ability to monitor and assess the effectiveness of the
strategy/ program/ initiative?

The principals and Curriculum Director have studied extensively the effective reading strategies highlighted in the LLI program. In addition,
the Curriculum Director who monitors the fidelity of the LLI program is a former Reading Interventionist.

Statement or Question:c)What is the evidence regarding the sufficiency of opportunities for staff to learn the knowledge and skills identified
as essential (the non-negotiable or acceptable variations of the elements) to the strategy/ program/ initiative?
Response:

Page 7
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.
Program Evaluation Tool
Paul A. Schalm Elementary School

Staff surveys

What does the evidence show regarding the sufficiency of opportunities for staff to learn the knowledge and skills identified as
essential (the non-negotiable or acceptable variations of the elements) to the strategy/ program/ initiative?

Reading Interventionists were all trained to use LLI materials through a two day workshop provided by Heineman when the program was
adopted. Dialogue between Reading Interventionists and teachers regarding literacy skills is ongoing. Tri-annual data meetings, grade-level
meetings, monthly Student Support Team meetings, and informal conversations and conferences all provide opportunities for staff to learn
about essential reading skills and how they are addressed in intervention.

Statement or Question:d) What is the evidence regarding staff ability to apply the acquired knowledge and skills?
Response:

Self-assessment checklists
Staff surveys
Superintendent or administrator observations/walkthroughs
Program simulations, administrator observations

What does the evidence show regarding staff ability to apply the acquired knowledge and skills?

Year-end iReady and other benchmark data such as DRA or F&P Benchmark Assessment show growth for all students in overall reading
ability. The strategies used in LLI mirror the strategies used in the classroom.

Statement or Question Response Rating


Given the evidence you've assembled, choose Participants had sufficient knowledge and skills 4
one overall self-assessment of the participants' to succeed.
knowledge and skills to implement the strategy/
program/ initiative.

What action steps are needed to improve participants' knowledge and skills?

Cross check program fidelity by allowing time for the Reading Specialists to observe each other teaching a lesson. Build in time on the
MTSS calendar for regular meetings to collaborate and discuss intervention progress and needs.

Page 8
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.
Program Evaluation Tool
Paul A. Schalm Elementary School

3) Opportunity: Was there opportunity for high quality implementation of the strategy/ program/
initiative?

IN AN IDEAL STRATEGY/ PROGRAM/ INITIATIVE, building and district administrators provide significant support for project
implementation. Sufficient funds have been allocated and continue to be managed by building principal and or program director. Adequate
resources are available for full implementation including time for staff collaboration in various forms. Clearly defined structures/protocols are
in place to collect and review formative implementation data.
Overall Rating: 2.0
Statement or Question:a) What is the evidence regarding the sufficiency of administrative support to achieve the intended results?
Response:

Action plans
Email correspondence
Budget sheets
Logs, school schedules
Collaboration models (such as Professional Learning Communities, Collaborative Action Research, Lesson Study Teams)

What does the evidence show regarding the sufficiency of administrative support to achieve the intended results?

Administrators do their best to have a protected block of time in the schedule to pull students for intervention and continue to support tri-
annual grade level intervention meetings.

Statement or Question:b) What is the evidence regarding the sufficiency of opportunities for on-going professional learning, including
modeling and coaching?
Response:

Agendas/minutes
Collaboration models (such as Professional Learning Communities, Collaborative Action Research, Lesson Study Teams)

What does the evidence show regarding the sufficiency of opportunities for on-going professional learning, including modeling
and coaching?

An instructional coach was available for teachers to meet with and set up instructional rounds or teacher labs as needed.

Statement or Question:c) What is the evidence regarding the sufficiency of resources - including financial, time and personnel - to achieve
the intended results?
Response:

Budget sheets

Page 9
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.
Program Evaluation Tool
Paul A. Schalm Elementary School

Logs, school schedules

What does the evidence show regarding the sufficiency of resources - including financial, time, and personnel - to achieve the
intended results?

In 2015-2016 an additional level of LLI was purchased for students in grades 1-2 to share between both buildings. In the future it would be
helpful to have an additional kit (one per building). At Schalm Elementary, the Reading Specialists have indicated that they each would like
an additional F&P Benchmark assessment kit. Time was protected this year for intervention pull out. This year personnel was reduced
which affected the number of students who could benefit from the LLI program, particularly in upper grades.

Statement or Question:d) What is the evidence regarding the sufficiency of opportunities for staff collaboration to support implementation
of the strategy/program/initiative?

Response:

Email correspondence
Collaboration models (such as Professional Learning Communities, Collaborative Action Research, Lesson Study Teams)
Protocols for reviewing formative assessment

What does the evidence show regarding the sufficiency of opportunities for staff collaboration to support implementation of the
strategy/ program/ initiative?

At the beginning of the year the Reading Specialists collaborate and put together a year-long MTSS calendar. Throughout the year the
Reading Specialists collaborate by means of email, phone conversations, and 31a meetings, as well as shared documents on Google Docs.

Statement or Question:e) What is the evidence regarding structures being in place to collect and review implementation data?
Response:

Agendas/minutes
Email correspondence
Logs, school schedules
Protocols for reviewing formative assessment

What does the evidence show regarding structures being in place to collect and review implementation data?

Student growth data from the LLI program is collected and reviewed at tri-annual data meetings as part of our MTSS process. Students
receiving LLI intervention are assessed monthly/bi-monthly using F&P Benchmark Assessments. That data is shared with teachers on an
ongoing basis.

Page 10
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.
Program Evaluation Tool
Paul A. Schalm Elementary School

Statement or Question Response Rating


Given the evidence you've assembled, choose Basic resources and opportunities were 2
one overall self-assessment of the opportunity available, but significant gaps need to be filled.
for high quality implementation.

What action steps are needed to ensure opportunity for high quality implementation?

Purchase additional LLI kits (higher level) and F&P Benchmark Assessments
Increase personnel to expand opportunity to a greater number of students
Continue to protect intervention pull out time

Page 11
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.
Program Evaluation Tool
Paul A. Schalm Elementary School

4) Implementation with Fidelity: Was the strategy/ program/ initiative being implemented as
intended?

IN AN IDEAL STRATEGY/ PROGRAM/ INITIATIVE, all personnel involved in the program implement the strategies with fidelity according to
the research, carrying out responsibilities by their proposed timelines. They use clearly defined protocols to collect and review formative
implementation data to identify unintended consequences. Program leaders consider adjustments guided by implementation data while
maintaining the integrity of results.
Overall Rating: 4.0
Statement or Question:a)What is the evidence regarding a process being in place to monitor fidelity of implementation of the non-
negotiable or acceptable variations of the elements of the strategy/ program/ initiative, including timelines and responsibilities?
Response:

Principal's walkthroughs
Number of staff implementing with fidelity
Agendas and minutes of common planning time/meetings
Record of funds used
Collegial observations/visits
Program Time Line

What does the evidence show regarding the fidelity of implementation of the non-negotiable or acceptable variations of the
elements of the strategy/ program/ initiative, including timelines and responsibilities?

Program fidelity of instructional delivery is monitored through walk-throughs and observations from principals and the district Curriculum
Director. The LLI program has a fidelity checklist for self-evaluation of lessons and both Reading Specialists have had formal training in the
LLI program and its components. The program provides detailed lesson guides and also allows for teacher choice regarding areas of
emphasis according to student needs.

Statement or Question:b) What is the evidence regarding positive or negative unintended consequences that may have occurred, if any?
Response:

Agendas and minutes of common planning time/meetings

What does the evidence show regarding positive or negative unintended consequences that may have occurred, if any?

There were not unintended consequences as the program was delivered with fidelity.

Statement or Question:c) What do implementation data and student achievement results suggest for implementing/modifying the strategy/
program/ initiative?
Response:
Page 12
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.
Program Evaluation Tool
Paul A. Schalm Elementary School

Number of staff implementing with fidelity


Collegial observations/visits

How might these affect the integrity of the results?

Due to a reduction of personnel, fewer students throughout the district were able to receive support using LLI.

Statement or Question Response Rating


Given the evidence you've assembled, choose All research-based elements have been 4
one overall self-assessment of the fidelity of implemented with fidelity following the proposed
high quality implementation. timelines.

What action steps are needed to ensure faithful implementation of program plans?

Adequate time for collaboration, protected blocks in which to service students for intervention, and purchase of additional materials as
mentioned above.

Page 13
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.
Program Evaluation Tool
Paul A. Schalm Elementary School

Impact: What was the impact of the strategy/ program/ initiative on students?

IN AN IDEAL STRATEGY/ PROGRAM/ INITIATIVE, the schools achievement results on state or district wide assessments meet proficiency
standards. Achievement gaps between each of the relevant subgroups and their counterparts have been narrowed as proposed in the
School Improvement Plans measurable objectives. Interim assessment results indicate progress toward proficiency for all students to the
satisfaction of all stakeholders.

a) What is the evidence and what does it show regarding achievement of the measurable objective for all students when compared
to baseline state and local data?

Our measurable objective is that 85% of all students will demonstrate a proficiency in reading comprehension skills in English Language
Arts by 06/15/2022 on state assessments. At the time of this report student scores on state assessments were not available. Local
assessment data (iReady) shows that 61% of students in grades 1-3 were proficient in Reading at the end of the 2016-2017 school year.

b) What is the evidence and what does it show regarding achievement of the measurable objective for subgroups and their
counterparts when compared to baseline state and local data?

Subgroups made comparable results to the general population of the school.

c) What is the evidence and what does it show regarding stakeholder (staff/ students/ parents) satisfaction with the results?

According to previous survey data, both staff and parents indicated that the program was relevant and helpful for their students.

Statement or Question Response Rating


d) Were the objectives for this strategy/ Yes N/A
program/ initiative met?

Page 14
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.
Program Evaluation Tool
Paul A. Schalm Elementary School

Impact Conclusion

Statement or Question:Should the strategy/ program/ initiative be continued or institutionalized?


Response:

Yes

a)What is the evidence and what does it say regarding whether this was the right strategy/ program/ initiative to meet your needs?

88% of students at Kenwood who received services using LLI made at least one year's growth in reading levels. All three of the students
who did not make one year's growth were English Language Learners; one who was a newcomer in October. 95% of students at Schalm
made at least one year's growth.

b) What is the evidence and what does it say regarding whether the benefits of the strategy/ program/ initiative are sufficient to
justify the resources it requires?

See above data as evidence of the success of the program and the justification of the resources needed to implement it.

c) What adjustments, if any, might increase its impact while maintaining its integrity?

Slight changes in scheduling groups to ensure that the program's suggested block of instructional time is protected. Reducing group sizes as
well as adding more personnel to teach LLI would also increase the program's impact.

d) What is needed to maintain momentum?

Allocate funds to purchase more materials, increase intervention time and personnel, and provide opportunities for other staff and parents to
learn more about the program.

e) How might these results inform the School/District Improvement Plan?

Our SIP includes a strategy for intervention and support in Reading. LLI is part of this strategy.

Page 15
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.
Program Evaluation Tool
Paul A. Schalm Elementary School

Report Summary

Scores By Section
Section Score
1 2 3 4

1) Readiness: What is the readiness for


3
implementing the strategy/ progra...

2) Knowledge and Skills: Did staff and


4
administrators have the knowledge a...

3) Opportunity: Was there opportunity for


2
high quality implementation of the strateg...

4) Implementation with Fidelity: Was the


4
strategy/ program/ initiative bei...

Sections

Page 16
2017 Advance Education, Inc. All rights reserved unless otherwise granted by written agreement.

Anda mungkin juga menyukai