Administrator 2.

0
By Tammy Stephens

Last week the Stephens Group finished baseline evaluations for five EETT baseline evaluations involving
25 school districts, 315 teachers, 52 library media specialists, 77 teachers and 12,708 students.

One of the things I have been busy doing the last couple of months is traveling around and working with
administrators to help us gather data by conducting walk-throughs in their buildings.

The Stephens Group approaches evaluation and both a formative and summative process. The
evaluation process is a very collaborative process that will involve the grant evaluator, grant manager,
grant leaders, and teachers and administrators in the grant. “The role of local engagement,
collaboration, and feedback is paramount. Teachers and administrators at the local site should be
participants in, rather than recipients of the evaluation” (Means et.al., 2003, p. 6). Evaluation research
that is responsive to local concerns, constraints, and priorities can be structured and synthesized to
produce knowledge about effective uses of educational technology that has high face validity within
local communities and still informs wider research as well as practitioner and policy audiences” (Means
et.al.,p. 6).

Walk-through data of a consortium of school districts we worked with last year were able to track strong
gains in 21st century skills and ISTE NETS Standards (see graphs on next page). In order to gain inter-
rater reliability we view online videos of classrooms utilizing technology and rate them using the walk-
through checklist. We also work with administrators to develop coaching strategies to talk with
teachers about their observations to help them move to higher levels of technology integration. The
coaching model that we use is situational. Different coaching techniques are used depending on the
level of technology integration a teacher is at. Birman et al. (2001) also found that coaching is
responsive to the way teachers learn, “… and may have more influence on changing teaching practice”
(p. 921).

Continued on next page

 21st Century Skills
100%
% of classrooms observed

90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Priori Prod
Adap
tizing ucing
tabili Socia
Tech Infor Multi Glob Pers , Relev
Scien Econ Visua ty/M l and
Basic nolo mati cultu al Risk- onal Plan ant,
tific omic l anagi Creat Civic
Liter gical on ral Awar takin Resp ning, High
Liter Liter Liter ng ivity Resp
acy Liter Liter Liter enes g onsib and Quali
acy acy acy Com onsib
acy acy acy s ility Man ty
plexit ility
aging Prod
y
for… ucts
Pre 68% 15% 8% 67% 36% 30% 16% 17% 20% 29% 22% 29% 20% 64% 20%
Post 98% 65% 5% 100% 65% 85% 33% 25% 35% 75% 20% 75% 38% 40% 43%

ISTE NETS Standards

100%
% of classrooms observed

90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Creati Create Use Intera Comm Contri Locate
Identif Troubl
vity: origin model ct, unicat bute , Solve
y Collect eshoo
applyi al s and collab e to organi authe
trends , t
ng works simula orate, inform projec ze, ntic Exhibit
and proces syste
existin as a tions and ation t analyz real leader
foreca s, and ms
g means to publis and teams e, world ship
st analyz and
knowl of explor h with ideas to evalua proble
possib e data applic
edge perso e peers, effecti produ te, ms
ilities ations
to… nal… com… expe… vely… ce… synt…
Pre 20% 22% 10% 13% 9% 12% 14% 23% 10% 9% 23% 4%
Post 50% 45% 40% 23% 58% 55% 35% 55% 53% 38% 48% 8%
This year we are using a second walk-through checklist that measures ACOT Levels of technology
integration and can track changes in teacher practice over time with two consortiums.

Administrator Name:

District:

School:

Date of walk-through observation:

Number of classrooms observed:

Learning Environment Observed

Students have no interaction with other students
Students collaborate with peers
Students are provided opportunities to use higher order thinking skills
Technology access is adequate to meet lesson objectives
Students with special needs have access to appropriate hardware and
software

Technology Use: Who used the technology in the lesson or activity observed? Observed
All students
Some students
One student
The teacher

Teacher Role: What was the role of the teacher during the lesson or activity Observed
observed?
Leader
Facilitator
Observer
Lesson Implementation Observed Objective, Level
Instruction, or
Assessment
Technology use is not clearly related to lesson O Adoption
objectives
The lesson is focused on learning a technology I Adoption
skill
Traditional assessment methods including A Adoption
paper and pencil are used to measure student
outcomes
Technology use is somewhat related to lesson O Adoption
objectives
Technology is used for drill and practice, I Adaptation
tutorials, or as a free time activity
Productivity tools and courseware are used to I Adaptation
augment the lesson
Technology is used with little or no I Adaptation
management problems
Student outcomes are often measured using A Adaptation
teacher developed rubrics
Lesson Implementation Observed Objective, Level
Instruction, or
Assessment
Technology skills are learned in the context of O Appropriation
the lesson objectives
Would not be possible to meet lesson O Appropriation
objectives without the use of technology
Students use technology to engage in I Appropriation
authentic tasks
Lesson objectives including technology use O Invention
encourage student choice and planning to
complete assignments
Lesson objectives are designed to provide O Invention
students with opportunities to demonstrate
learning outcomes using technology
Lesson objectives permit students to initiate O Invention
technology use for learning and assessment
Technology use provides opportunities to I Invention
expand student interactions beyond the
classroom
Students are encouraged to seek new uses of I Invention
hardware and software for learning
Student products are assessed using student A Invention
developed rubrics and portfolios.
Comments:

This checklist was modified from the one created by Susan Brooks & Harvey Barnett (2002).
You can read more about the evaluations we are doing on our website on our research page at
http://www.thestephensgroup.com/research.html .

References:

Birman, B. F., Desimone, L., Garet, M. S., Porter, A. C., & Yoon, K. S. (2001). What makes professional
development effective? Results from a national sample of teachers. American Research Journal, 38(4),
915-945.

Means, B., Haertel, G., & Moses, L. (2003). Evaluating the Effects of Learning Technologies. In G. Haertel
& B. Means (Eds.), Evaluating educational technology: Effective research designs for improving learning.
New York: Teachers College Press.