Testing reading

comprehension

Evaluation and Assessment in Language

Education
Dr Kia Karavas

The reading process

Reading is a dynamic process in which the
reader interacts with the text to construct
meaning. Inherent in constructing meaning
is the reader's ability to activate prior
knowledge, use reading strategies and adapt
to the reading situation.

What can a skilled reader do?

A skilled reader rapidly and accurately decodes the
words, attaches the meaning to words and
sentences, connects text information to relevant
background knowledge, maintains a mental
representation of what he or she has already
read, forms hypotheses about upcoming
information and makes decisions based on his
or her purpose for reading all at the same
time. Carlisle and Rice, 2002

What influences reading

difficulty?
Reader variables
The readers background and subject/topic knowledge,
their cultural knowledge and their knowledge of the
language in which the target texts are written. The
readers ability to process printed information is clearly
crucial.
Text variables
Linguistic features of text clearly affect readability of text
and readers comprehension, and text type, organization
and so on as well as text topic influence how well readers
can process meaning. Many features of text need to be
considered in the design of tests of reading.

What should assessment of

reading focus on?
Alderson argues that we should focus our assessment of
reading on the target skills we want out students to
develop.
Major reading subskills include
Reading quickly to skim for gist, scan for specific details,
and establish overall organisation of the text
Reading carefully for main ideas, supporting details,
authors argument and purpose, relationship of
paragraphs, and fact vs opinion
Drawing inferences from both stated and implied content

Enabling reading skills include

Understanding at the sentence level (lexis,
syntax, cohesive markers)
Understanding at inter-sentence level (e.g.
anaphoric/catophoric reference,
recognizing discourse markers)
Understanding components of non-linear
texts (meaning of graph, chart labels etc)

Choosing texts
Many sources for reading texts: purpose written,
taken directly from authentic material or adapted.
Use a variety of text types; do not select texts of a
particular kind just because they are readily
available
Choose texts of appropriate length; detailed reading
can be assessed with texts consisting of a few
sentences. Texts of up to 2000 words may be used
for extensive reading at higher levels

Choosing texts

Choose texts with many discrete pieces of information

Choose texts that will interest the candidates but will not disturb or
excite them. Avoid texts with controversial or biased material (e.g.
abortion, international disputes, religion). These are not suitable for
assessment because they can upset candidates and affect the
reliability of results.
You should check the language of your reading texts. Ninety percent
(90%) of the words in a text should be known to students for good
comprehension (Nation 1990).
Choose topics within the experience of the test takers. However,
avoid texts that contain information which is part of the test takers
general background knowledge.

Developing test items/questions

The wording of the reading test questions should
not cause comprehension difficulties to learners.
It should always be within their capabilities and
less demanding than the text itself.
Responses to test items should make minimal
demands on writing ability.
The items should be in the same order as the
information in the text. Mixing up the order of
questions increases the difficulty level.

Assessing reading
comprehension: Three levels
The first level, literal comprehension, is
the most obvious. Comprehension at this
level involves surface meanings. At this
level, teachers can ask students to find
information and ideas that are explicitly
stated in the text.

Levels of comprehension
The second level is interpretive or referential
comprehension. At this level, students go beyond what
is said and read for deeper meanings. They must be
able to read critically and analyse carefully what they
have read. Students need to be able to see relationships
among ideas, for example, how ideas go together and
also see the implied meanings of these ideas. It is also
obvious that before our students can do this, they have
to first understand the ideas that are stated (literal
comprehension).

Referential Level of
comprehension
At this level, teachers can ask more challenging
questions such as asking students to do the following:
Re-arrange the ideas or topics discussed in the text.
Explain the author's purpose of writing the text.
Summarize the main idea when this is not explicitly
stated in the text.
Select conclusions which can be deduced from the
text they have read.

Levels of comprehension
Finally, the third level of comprehension is critical
reading whereby ideas and information are evaluated.
Critical evaluation occurs only after our students have
understood the ideas and information that the writer has
presented. At this level, students can be tested on the
following skills:
The ability to differentiate between facts and opinions.
The ability to recognize persuasive statements .
The ability to judge the accuracy of the information
given in the text.

Some tips on item writing

Do not write items for which the correct response can be
found without understanding the text.
Do not write items that some candidates are likely to
answer from general knowledge without reading the text.
Make the items independent of each other; do not make
the correct response on one item depend on another
item being responded to correctly.

Possible testing formats for reading tests:

Multiple choice
Multiple-choice questions are a common
device for testing students text
comprehension. They allow testers to
control the range of possible answers to
comprehension questions, and to some
extent to control students thought
processes when responding.

An example
1.1 Read the first part of the text on p. 2 (the left side column), and choose the best
answers (A, B, or C) for items 1-4.
1. This text
A.

is written in an academic register.

B.

has an impersonal tone.

is written as a narrative.

C.

2. One aim of this text is to

A.
B.
C.

argue for better housing for poor people.

describe the writers family life.
explain what is needed to become a writer.

8. What does the writer say about North Africa?

A Publishing there is of a high standard.
B Writers have achieved more freedom there.
C Publishing still has a long way to go there.

Matching
One alternative objective technique for the
testing of reading is multiple matching.
Here two sets of stimuli have to be
matched against each other as, for
example, matching headings for
paragraphs to their corresponding
paragraph, titles of books against extracts
from each book, and so on

Example
Read the following extracts (21-25) and decide in which publication they might appear. Use
each of the options below (A-H) only once. There is one option you do not need.
A.
C.
E.

Rules for playing a game

Instructions for paying tax
University brochure

B.
D.
F.

Written safety instructions

Reverse side of a books title page
Travel guide

21.

Exception for children under age 18. If you are planning to claim a return for your child, who was
under 18 at the end of 2009, and certain other conditions apply, you can include your childs income
on your form.

22.

All rights reserved. No part of this publication may be reprinted or reproduced

or utilised in any form or by any electronic, mechanical, or other means, now
known or hereafter invented, including photocopying and recording, or any
other information storage or retrieval system, without permission in writing
from the publishers.

23.

The inn is a little oasis amidst the narrow and bustling cobbled streets at the heart of the
historic city.

Ordering tasks (Strip stories)

Candidates are given a scrambled set of
words, sentences, paragraphs or texts and
have to put them into their correct order.
The ordering tasks can offer the possibility
of testing the ability to detect cohesion,
overall text organization.

Short-answer tests
Test-takers are simply asked a question which
requires a brief response, in a few words, not
just Yes/No or True/False. The justification for
this format is that it is possible to interpret
students responses to see if they have really
understood, whereas on multiple-choice items
students give no justification for the answer they
have selected and may have chosen one by
eliminating others.

The free-recall test

In free-recall tests (sometimes called
immediate-recall tests), students are asked
to read a text, to put it to one side, and then
to write down everything they can
remember from the text. The free-recall test
is often held to provide a purer measure of
comprehension, since test questions do not
intervene between the reader and the text.

The summary test

Students read a text and then are required to summarize
the main ideas, either of the whole text or of a part, or
those ideas in the text that deal with a given topic. It is
believed that students need to understand the main
ideas of the texts, to separate relevant from irrelevant
ideas, to organize their thoughts about the text and so
on, in order to be able to do the task satisfactorily.
(Example: Gapped Summary)
Scoring is problematic and can be subjective

Information-transfer test
The students task is to identify in the
target text the required information and
then to transfer it, often in some
transposed form, on to a table, map, figure
etc.

Cloze test
Cloze Tests are reading passages with the
blanks representing words that are deleted
from the original text; the blanks are to be
filled in by the reader

Types of cloze test

a. Fixed Ratio Cloze or Nth word deletion
Words are deleted systematically by counting off,
regardless of the part of speech.
b. Rational Deletion Cloze
Words are deleted by part of speech or content area
vocabulary.
c. Maze Technique
Three word choices are provided at each missing word
interval.
d. Limited Cloze
Word choices (one per blank) are provided all together in
a word bank at the top or bottom of the page.

Tips for developing reading

comprehension tests

Make sure your assessment matches your reading program. Test

the skills you have taught
Sample a range of reading subskills with different task types. Allow
4-10 items per task type
Choose a range of text types appropriate to your program.
Consider students background knowledge and interests in selecting
texts. Familiarity with the topic aids comprehension
Use authentic or adapted texts whenever possible
Exploit the entire text. Questions should cover all sections of a text
Assess inferencing and critical thinking. Include questions that
require students to think beyond what they see in print