Using Assessments for Systemic Reform

InspirED 2014

Jan 11, 2014

Vishnu Agnihotri
Educational Initiatives
For any queries, contact
vishnu@ei-india.com

Using Assessments for Systemic Reform

1. What are we trying to reform?
2. Making the problem visible and providing specific insights
3. Experiences in using assessments for systemic reform
4. Beyond student assessments of learning outcomes
5. Some technical challenges in assessments
6. Assessments for systemic reform- Points to keep in mind

Education for the Poor: Very Low Learning

Levels
Class 5

Less than 50% of class 5

children could write a sentence
on the picture

Class 4

Subtract 26 from 51

Correctly answered by only

40% students
Sample of about 10,000 students per class
Source: EI Student Learning Study 2008
3

EI- Wipro Study

on Student Learning in the
Metros

full report and a detailed question-wise analysis,

at: http://www.ei-india.com/whats-wrong-with-our-teaching/.

Private schools: Learning happens, but by

rote
Define density

Calculate the LCM of 12, 16 and 20.

If some ice is dropped into a glass of water,

what will it look like?

What IS LCM and where is it used?

Sample of about 8,000 students per class

Source: EI SLIMS 2006

Comparison with International Levels

A passage on
brushing
ones
teeth
and
what
works..

How
Indian
do?

did
states

Comparison with International Levels

TN 63%
HP 53%
PISA Avg 85%

PISA 2009+

Performance on a question on measurement

Questions that
check real

learning
77% of the
students chose
Option C while
only 11% chose
Option B
-Based on the SLIMS 2006
study results
8

Why do so
many students
struggle to
measure a
length or

compare
angles?

Teaching Vs Learning

Using Assessments for Systemic Reform

1. What are we trying to reform?
2. Making the problem visible and providing specific insights
3. Experiences in using assessments for systemic reform
4. Beyond student assessments of learning outcomes
5. Some technical challenges in assessments
6. Assessments for systemic reform- Points to keep in mind

Which of the following are examples of respiration?

1. Humans use oxygen and release carbon dioxide.
2. Plants use carbon dioxide and release oxygen.
3. Burning dry leaves uses oxygen and releases carbon dioxide.
A. only 1
B. only 2
C. only 1 and 2
D. 1, 2 and 3
Performance
in entire paper

Option A

Option D

Virtually no
change in
understanding
from 9.5 to 14.8
years!

What students at different ability levels can do?- Class 4

What students at different ability levels can do?...

Comparative performance across regions1

Comparative performance across regions2

Comparison across School Systems

State students typically score
3 to 8 percentage points
lower than Indian Private
Schools
However in some questions
involving familiarity in real
life, they score higher
Class 7

Social Science

The shaded dark grey part on this map shows the continent of
______________.
A.
B.
C.
D.

Asia
Australia
Antarctica
South America

Sample Question: This question tests the students

understanding and interpretation of geographical maps. 40% of
the tested students could answer this question correctly
compared to 67% of ASSET students.

Class 5

EVS

Here is a picture of grains of wheat. Manan buys some wheat

grains to make into bread, chapatis and biscuits.
To make these, the wheat grains should first be____________.
A.
B.
C.
D.

Put into the soil

Mixed with water
Cooked over fire
Made into powder

Sample Question: This question tests knowledge of the living

environment. 83% of the tested students could answer this
question correctly compared to 48% of ASSET students.

International Benchmarking
& other patterns
Question-level Diagnosis

Detailed, Question-wise Analysis Available for all

Teachers and Researchers

http://www.educationalinitiatives.com/GujaratDiagnostic/GD_StudyGraph_Display.php

Overall Summary Of
School Performance

Students Score Table

Recommendations

Skill Performance
Report & Questions
Found Difficult/Easy

Question Wise Comparison

Of Your School Versus All
Schools

Class Performance Table

Using Assessments for Systemic Reform

1. What are we trying to reform?
2. Making the problem visible and providing specific insights
3. Experiences in using assessments for systemic reform
4. Beyond student assessments of learning outcomes
5. Some technical challenges in assessments
6. Assessments for systemic reform- Points to keep in mind

Gujarat

Five year Comprehensive

Assessment driven Reform for
Educational Quality

All students of Classes 3 to 8

Teachers Self Assessment
Officers Assessment
Diagnostic Assessment
Student Interviews
Capacity Building Workshops
Monitoring Weak Schools

What is Gunotsav
Gunotsav means Celebrating Quality
Launched by Gujarat Government in 2009
Main objective was to ensure quality
education of students in Government
schools by
bringing awareness among teachers,
students, education administrators and the
society
addressing the gaps in learning improve
learning
outcomes
and
acknowledge
achievement through assessment
assessing the class room quality and
building an environment of accountability
at all levels
www.gunotsav.org

Education Dept., Government of Gujarat

Salient Features of Gunotsav

Largest school assessment program 53.5 lakh students in 34,000 Government
primary schools, 715 Ashramshalas and
593 granted schools across the state
every year since 2009 on the following
holistic parameters:

Learning Outcomes for each Class

Co-curricular Activities
Community Participation
Availability and Use of Infrastructure

Self evaluation of all schools by

teachers, further 25%(9000 Schools) also
externally
assessed
by
officers
(Ministers, IAS,IPS,IFS officers, ClassI &
II officers) every year since 2009

www.gunotsav.org

Education Dept., Government of Gujarat

Salient Features of Gunotsav

Self
Assessment

Officer
Assessment

Diagnostic
Assessment

Messaging to teacher and larger

educational community on importance
of learning outcomes & accountability
Provides teacher an understanding of the
achievement levels in classroom
Ensures enhancement of reading,
writing and numerical skills
Involvement of senior officers in
education and understanding key
ground level issues
Convergence with participation of
officers-staff from all departments
Actionable feedback of learning gaps,
common errors, misconceptions, strong &
weak competencies

Accountability framework which invests in teachers

& stakeholder participation with a comprehensive
view of quality

www.gunotsav.org

Education Dept., Government of Gujarat

Process Flow
PHASE 1 :
Self Assessment by ALL
Schools

PHASE 2 :
Assessment in 9000
schools by 3000 officers
3 day visit @1 school/day selected
randomly in assigned cluster

Done in September

Done in November -December

Remedial Extra Coaching Program

January March
Diagnostic Assessment
February
Monitoring by External Parties Unicef, Pratham & Educational Initiatives

www.gunotsav.org

Education Dept., Government of Gujarat

Transparency of the Initiative

Association of three well known organizations in the
field of education

Assessment of post Gunotsav Remedial class

programme by UNICEF
Evaluation of assessment processes during the
officer assessment-2011 by Pratham-ASER
Continuous support and analysis at all stages by
Educational Initiatives

www.gunotsav.org

Education Dept., Government of Gujarat

Innovative Use of Technology

Setup of special web portal
by iNDEXTb
Web based online data
entry- self evaluation &
officers evaluation
Scientific analysis
Data in public domain
Use of data for teachers
training needs
Monthly
LIVE
teachers
training
through
teleconference across state
Online report cards for
schools & teachers
www.gunotsav.org

Education Dept., Government of Gujarat

Bhutan
BHUTAN
5-year comprehensive improvement project

All students & teachers data

Easy to use queries to fetch information
Tracking of Learning Improvements

All teachers from PP-12

System Level Report
Individual Teacher & School Reports

All students of Classes 4,6,8

System Level Report
Individual Student & School Reports

Making Learning Levels Visible through a

Student Progress Tracking System

Visibility from the level of a child to that of the state

Availability of accurate data at
granular (student, school or block
level) as well as aggregated levels
based on authentication.

Note that drilling down to the level of

individual questions and individual
students is also possible as shown in
the images below and on the next page.
Starting from the consolidated national
performance, the performance of an
individual child on an individual
questions can be accessed in a matter
of clicks.
The report on the left is extremely useful
for class teachers as it shows them the
skills in which the class as a whole is
strong or weak, allowing for remedial
action.
The report on the right is the detailed
question-wise report for a particular
student.
Note that clicking the question number
shows the actual question.
For each question, the skill tested, the
students answer, the correct answer and
the percentage correct for all students of
that class nationwide, are shown.

Data can be drilled down

to the level of a specific
student and a specific
question or talent; schoollevel data can also be
stored and retrieved.

Student Progress Tracking System (SPTS)

The SPTS allows a school to
identify its strength and
weakness areas vis-a-vis all
other schools. This allows
specific resources to be
focused for those who need
them most.

SPTS allows action and remediation,

school level assessments, progress
monitoring, talent identification etc.
Over time, the system can enable
detection of educational learning
patterns that show the effectiveness of
specific
textbooks,
educational
methods or policies by providing
longitudinal data for comparison.

MCGM

Municipal Corporation
of Greater Mumbai

Three year Comprehensive

Assessment driven School
Excellence Reform

All students of Classes 3 and 6

Multiple Mediums of Instruction
System Level Report
Individual School and Ward Level
Reports
Dissemination Workshops

Using Large Scale Assessment Findings- Germany

PISA 2000 focused on
language literacy and
results shocked the
German nation. Germany
came well below the
average overall for all the
countries tested

Newspapers ran six-page

sections on the PISA
results. The news and
discussions of the results
were all over the radio
and television

Education became a
political agenda item, and
agendas of the right and
left were fused together
like never before

Several structural changes have since taken place in Germany:

Changing the school structure to reduce the influence of socio-economic background
on student achievement
Addressing language problems specifically for those students who do not speak
German at home, and ensuring that children of minority or immigrant parents attend
pre-school
The education system also had a real aversion to the use of empirical evidence and
rigorous analysis of data as the basis of educational decision making. Common
standards and national assessment at 3rd, 8th and 9th grade were instituted
At the school level, increased autonomy to schools, longer school hours and a strong
focus on improving teacher quality was instituted
Germany has now substantially improved its position in international rankings
Source: CSF presentation based on the report Strong Performers and Successful
Reformers in Education: Lessons from PISA for the United States, OECD 2011

Using Large Scale Assessment Findings- Brazil

Brazil lowest performing country on PISA 2000. Over 50% of the students scored at or
below level one and less than 1% scored at the top level
The national assessments SAEB and Prova Brazil ensure that schools,
municipalities and states receive data onhow their students fared on an examination
of the curriculum standards in Portuguese and mathematics set by the federal
government
In some states training is provided to teachers so that they can use the test results to
analyse individual classroom and school-wide performance and develop strategies to
improve learning
According to Brazilian officials, the real catalyst for school improvement has been the
federal government targets, set for every two years for schools based on each
schools trajectory on SAEB and Prova Brazil, which begins where the school was in
2005 and ends at its expected arrival at OECD PISA average performance by 2021
Source: CSF presentation based on the report Strong Performers and Successful
Reformers in Education: Lessons from PISA for the United States, OECD 2011

Using Assessments for Systemic Reform

1. What are we trying to reform?
2. Making the problem visible and providing specific insights
3. Experiences in using assessments for systemic reform
4. Beyond student assessments of learning outcomes
5. Some technical challenges in assessments
6. Assessments for systemic reform- Points to keep in mind

Quality Education Study

QES aims to find answers
to what constitutes
Quality education
A multi-year study to expand the meaning of
quality in education to include educational
outcomes beyond student performance in
subjects and study the attributes of quality
learning environments.
The study relates quality to a reflection of
outcomes in larger sense:
scholastic,
co-scholastic and
affective (specifically values and
attitude)

Assessing Attitudes
Class 4, 8

Civic Responsibility

People throw garbage out in the street, from the

bus or train, in the river, in empty buildings, in the
theatre, in parks and canals.

When do you think people could throw out

garbage in the street, from the bus or in the
canals?
A. when there are no garbage bins
B. when others do the same
C. never

Source: QES

Beliefs about discipline vs Learning outcomes

Source: Quality Education Study 2011

Beliefs about discipline vs Classroom Climate

Source: QES

Country ranking based on teacher beliefs

How do these beliefs link with learning outcomes?

Source: TALIS (OECD) + Quality Education Study 2011

In Progress- ABL Evaluation Study

Classroom observations- physical environment, classroom
processes & relationships
Teacher beliefs- purpose of learning, about learners, inclusion

The GRIT Scale

Consistency of interests &

Persistence through failure

Only 8 items!

Source: Angela Duckworth

Students assessing teachers

Students rating of teachers!

5 items most strongly associated with student learning- classroom control &
challenge
1. Students in this class treat the teacher with respect.
2. My classmates behave the way my teacher wants them to.
3. Our class stays busy and doesnt waste time.
4. In this class, we learn a lot almost every day.
5. In this class, we learn to correct our mistakes.
Source: Roland Ferguson, Harvard Economist

Using Assessments for Systemic Reform

1. What are we trying to reform?
2. Making the problem visible and providing specific insights
3. Experiences in using assessments for systemic reform
4. Beyond student assessments of learning outcomes
5. Some technical challenges in assessments
6. Using assessments for systemic reform- important points

Some Technical Challenges in Assessments

Sample Selection (sample or census?)

Control Group Selection

Test Development (item types; testing formats)
Test Administration (complexity, centralisation, fairness)
Points related to transparency (instrument, item-wise data)
Points related to standardisation (procedures)
Points in analysis and reporting (IRT, explaining unexpected
findings)

Using Assessments for Systemic Reform

1. What are we trying to reform?
2. Making the problem visible and providing specific insights
3. Experiences in using assessments for systemic reform
4. Beyond student assessments of learning outcomes
5. Some technical challenges in assessments
6. Assessments for systemic reform- Points to keep in mind

Evaluation & Assessment Framework- Key Points

Governance: Striking the right balance between central efforts and
local initiatives and between accountability and development
State vs District initiatives

Procedures: Designing the right instruments to ensure evaluation

and assessment contribute to improvement of teaching and learning
Training on what to assess, building diagnostic items

Capacity: Developing competencies within the state for evaluation

and for using feedback at all levels of the education system
Balance external assessments with teacher-led assessments
Balance and align Summative and Formative assessments (CCE)

Use of results: Organizing evaluative information for effective use;

avoiding misuse of E&A results
Differentiating between use for accountability and use for development
Reports designed to meet objectives
Based on OECD Paper

When Large Scale Assessments Fail to Create Impact

Actions
Regarded as a stand-alone
activity with little
connection to the Ministry

Inadequate involvement of
stakeholders in design and
implementation of an
assessment

Failure to communicate
findings to all in a position
to act

Lack of confidence in the

findings of a national
assessment

Political sensitivity to
making findings public

Failure to devise
appropriate action
following an assessment at
the level of general policies
or at the school level

Integrate assessment activity into

existing structures, policy, and
decision-making processes
Involve all relevant stakeholders in
design and implementation of an
assessment
Increase the likelihood of making
findings public by holding regular
stakeholder discussions
Make provision in the budget for
dissemination, plan activities, and
prepare a number of reports
tailored to user needs
Integrate national assessment
activity into policy and managerial
activities, and review findings to
determine implications and
strategies

Source: CSF presentation based on Using The Results Of A National

Assessment Of Educational Achievement, Greaney, Kellaghan & Murray, 2009

50

Thanks!
For any queries, contact
vishnu@ei-india.com

7. Higher classes / Board Exams - matter for Early

Learning
From a State Board Paper

From the CBSE Paper

From the IGCSE Paper

Initiatives
52
Source: Educational
Examples are
from Gujarat Board, CBSE and IGCSE Class 10 papers of 2009

9. The Science of Learning

Many initiatives fail because of capacity issues
While capacity is understood at level of teachers,
institutional and systemic capacity are often ignored
Need for Research based initiatives like a Science of
Learning Institute, Central Reading Research Institute,

Systemic
1Capacity
A body of
knowledge that
generates research
on learning and is
grounded in practice

Institutional
Capacity

Strong institutions
that develop,
enhance and
disseminate this
knowledge and are
not individual
dependent

People's
Capacity
Continuous
Professional
Development for
educational
functionaries,
teachers and others

The Plan of my Talk

Goal: Ensure Quality Learning for All Children
Government Schools

Private Schools

Strategies
Make the Problem Visible

Provide Insights / Diagnosis

Create New Tools to Solve

the Problem

Actions
Large Scale Assessments
Student Progress Tracking
Systems

Student Interviews
Diagnostic Sheets

Personalised Learning
Solution

Need for a New Science of Learning

School tests/ Board exams make us blind to

the consequences

School tests/ Board exams make us blind to

the consequences
Describe Evaporation
Vs

32% of nearly 10,000

students believe only
this is evaporation

Evaporation is the process of a liquid (like water) changing into a gaseous state. Which of the
following observations can be explained by the process of evaporation?
1. Water spilled on the floor dries up after some time.
2. Sweat from our skin disappears when we sit under a fan.
3. Water from the ocean changes into gas and forms clouds.

Option

Option

Percent Correct

only 1

15.1%

only 3

31.8%

both 1 and 3

23.4%

1, 2 and 3

27.6%

Source: ASSET Class 6

Board Exams in some progressive countries.

The more
progressive
countries are
moving towards
giving students
greater choice
be it in terms of
number of
subjects, levels,
or even attempts
allowed at the
exams

Source: Board Exam

Reform StudyGujarat

Board Exams in some progressive countries.

The more
progressive
countries are
moving towards
giving students
greater choice
be it in terms of
number of
subjects, levels,
or even attempts
allowed at the
exams

Source: Board Exam

Reform StudyGujarat

Different levels of depth/difficulty for a

subject. For example,
IB a Higher level paper and a Standard level
paper,
Finland offers advanced courses in certain
subjects.
Flexible choices in the same subject.
Alberta - a Pure Science Subject or a
Combined Science subject.
Hong Kong - Integrated Science or Combined
Science or Individual Science subjects such
as Physics, Chemistry and Biology.
More than one attempt to improve
scores
China, Finland - 2 attempts. The best attempt
is then counted towards total marks.
New Zealand unsuccessful can appear again
within the same academic year.

Board Exams in some progressive countries.

Source: Board Exam Reform Study- Gujarat

Board Exams in some progressive countries.

Source: Board Exam Reform Study- Gujarat

The case of Finland- No. 1 since 2000