How to assess writing
TESTING WRITING • Candidates should know just what is required of them and should not
___________________________
• Specify all possible content: In order to judge whether the tasks we
set are representative of the tasks that we expect students to be able
to perform, we have to be clear at the outset just what these tasks
are that they should be able to perform.
• Expressing: thanks, requirements, opinions, comment, attitude,
confirmation, apology, complaints, justifications, etc.
• Directing: Ordering, instructing, persuading, advising, warning
• Describing: Actions, events, objects, people, processes
• Eliciting: Information, directions, service, clarification, help,
permission
• Narration: Sequence of events
• Reporting: Description, comment, decisions.
• Types of text: Form, letter, message, fax, note, notice, postcard
• Addressees of texts, topics and dialect are unspecified.
____________________________________________
• If a test includes a wide ranging and representative sample of
specifications, the test is more likely to have a beneficial backwash
effect.
_____________________________________________
• Test only writing ability and nothing else:
• For the sake of validity, we should not set tasks which measure
creativity, imagination or even intelligence.
• Reading can interfere with the accurate measurement of writing
• One way of reducing on the candidates ability to read is to make use
of illustrations
__________________________________________
be allowed to go too far astray.
• A useful device is to provide information in the form of notes or
pictures.
• Full sentences are generally to be avoided.
BASIC CONSIDERATION IN ASSESSING WRITING
___________________________________________________
In choosing or designing a writing test, the logical place is by considering
what we plan to use the test for.
What is our purpose?
According to Bachman and Palmer (1996) there are two main purposes for
language test.
The primary purpose is to make inferences about language ability
The second purpose is to make is to make decisions based on those
inferences.
Let’s us consider three types of inferences that we can make on the basis of
language test: proficiency, diagnosis and achievement.
_____________________________________________________
The primary purpose of a language test is to make inferences about
language ability.
We need to be clear about how this ability this manifested in non-test
language use, on the other hand, and in language test, on the other.
We refer to the ability that we want to test as a CONSTRUCT.
The key to defining the construct of interest is determining what factors are
involved in real-world language use, and which of those factors are essential
to what we want to measure and what we do not.
Strategic competence
_________________
  Evaluating communicative situation or test task and engaging an
appropriate discourse domain
 Evaluating the correctness or appropriateness of the response
_________________
 Deciding how (and whether) to respond to the communicative
situation
_________________
 Deciding what elements of language knowledge are required to
reach the stablish goal.
_________________
 Retrieving or organising the appropriate elements of knowledge to
carry out the plan.
___________________________________________________
The term performance assessment is used to describe any assessment
procedure that involves either the observation of behaviour in the real world
of simulation of a real- life activity.
McNamara (1996) provides a distinction between a strong sense and weak
sense of performance assessment in language testing.
In the strong sense: the focus performance assessment is on the successful
completion of a given task that requires language use.
In weak sense of performance assessment: the focus of the assessment it is
on the language used, not on the fulfilment of the task.
_________________________________________
The most important consideration in designing and developing a language
test is the use for which it is intended.
Bachman and Palmer (1996) define test usefulness in terms of six qualities:
Reliability, construct validity, authenticity, interactiveness, impact, and
practicality.
DESIGNING WRITING ASSESSMENT TASK
Testing beyond the level of the individual classroom.
_________________________________________
These source emphasize that test development involves several stages,
which do not proceed in a strictly sequential or linear fashion, but inform each
other in an interactive fashion, that is feedback. It necessary to return to a
previous step to rectify a problem.
_________________________________________
Involves gathering information about such things as the test purpose,
characteristics of the target population and their real-word writing needs,
and available resources.
Test development involves gathering critical information and making
decisions that will guide the entire test development process.
Mandate: grows out of a perceived need on the part of various
stakeholders,. Such as administrative bodies or teachers, to measure
language ability for a particular purpose.
The theory based construct definitions are most useful in a situation where
test content be based on a specific curriculum.
Test of writing for low-profile foreign-language learners we may decide that
we are primarily interested in linguistic and textual knowledge, or knowledge
about grammar and vocabulary of the language and how sentences are
organized into text.
Give their options for construct definition with respect to topical knowledge:
a) Exclude topical knowledge from the construct definition.
b) Explicitly include topical knowledge as part of the construct definition
c) Define topical knowledge and language ability as separate
constructs.
The final point:
Is important to considere all aspects of test usefulness “ reliability, construct
validity, authenticity, instructiveness, impact, and practicality”
_________________________________________
Information from the design stage is used to create a test specifications, or
blueprints for the development of specific test tasks and complete test.
Test developers can use such questions to guide their test planning and to
evaluate draft test tasks, instruction and scoring procedures before there are
used operational
_________________________________________
The third stage in the test development process is the administration of test
tasks to examinees, both on a trial basis and operationally, and the concurrent
collection and analysis of test data and other relevant information about the
test procedures.
An important point to note is that the administration phase overlaps
significantly with the operationalization phase in the sense that test tasks need
to be tried out before a test can be administered operationally.
In the previous tests, several tasks are tested on a very good sample of
examinees to obtain preliminary information on various aspects of the task, for
example, if the instructions are clear, how long it takes for the examinees to
perform the task, etc.
The test, on the other hand, involves administering a full version of a test to a
larger sample to obtain statistical information. Although both stages may not
be feasible in all situations, it is important to perform at least some pre-tests to
perform them.
_________________________________________
White (1994) points out that most test developers consider at least the
following four minimum requirements for writing tasks: clarity, validity,
reliability, and interest.
Clarity is essential so that examinees can understand what is required of them
quickly and easily.
Validity White refers to the potential of the request to obtain written products
that cover the range of the capacity of interest among the examinees.
Reliability is essential because the qualification criteria must be applied
consistently to all responses, and readers must give similar, if not identical,
scores to the same items.
_________________________________________
___________________________________
At issue here is the question of what topic (content area) test takers should
write about, and what topics should be avoided.
the most important consideration here is accessibility to all test takers, since
everyone needs to have an equal chance of success.
the IELTS (International English Language Testing System) Handbook (1999)
provides the following list of topics for the general training writing module of
the test, in which test takers must write both a letter in response to a given
situation and an essay or report on a general topic .
 Travel.
 Accommodation.
 Current affairs.
 Shops and services.
 Health and welfare.
 Occupational health and safety.
 Recreation.
 Social and physical environment.
 The personal topics, in which the examinees write about their own
experiences, have the advantage that the examinees are committed
to the subject and, therefore, can perform better than they would.
 Personal issues also do not require any specialized knowledge and,
therefore, are accessible to most, if not all examined.
_________________________________________
 A warning message can include source materials, such as a reading
passage, a Brief quote, or a drawing. They provide content for the
examinees to write, or they can simply nominate a topic without any
additional stimulus material.
 There are several reasons for providing or not providing stimulus
material and, ultimately, the choice depends on the definition of the
construct and the considerations described in the previous section.
 If we are strictly interested in the capacity of language and we want to
explicitly exclude topical knowledge from the construct, stimulus
material such as a visual may be appropriate.
 In a foreign language class, for example, one could provide a series
of images that tell a story and ask the examinees to write the story that
the pictures tell. Providing stimulus material of this type provides the
content for the examinees to write, allowing them to focus primarily on
the linguistic aspects of the task.
• One of the first considerations is whether writing is being tested as
one skill within a larger battery of skills tests (as in a general
language proficiency test)) or as a skill on its own.
• In the former case, the amount of time devoted to writing will have to
be considered with respect with the total testing time available and
the degree of importance writing holds vis-á-vis other skills.
• For tasks that require reflection and planning such as academic
essays, a question of interest is the optimum time to provide test
 takers so they have enough time to plan, write, and revise their
writing.
___________________________________
• The amount of detail provided in instructions for writing tests can
have an effect on test scores.
• Guidelines for instructions:
• 1. They should be simple enough for test takers to understand.
• 2. They should be short enough not to take up too much of the test
administration time.
• 3. They should be sufficiently detailed for test takers to know exactly
what is expected of them.
___________________________________
• Giving writers a choice of task helps to ameliorate the effects of
background knowledge and interest: writers may choose to write on
the task that they feel they know most about or have the most
interest in. This may reduce anxiety and allow writers to perform their
best.
______________________________________
• With the increasing use of computers in education and testing, one
important question is whether to ask examinees to write by hand or
on a computer.
• There are 3 key issues that need to be considered:
• What extended are examinees already familiar with computers
• Whether there are consistent observable differences in either the
writing process or the end product when hand written and keyed
essays are compared
• An important consideration in deciding between hand written and
keyed essays is the implication for scoring, (handwritten essays tend
to be scored higher than keyed essays, because they are shorter
and handwritten essays show evidence of editing and revision).
References: Hughes. A (2003), Testing for a language assessment series;
Weigle (2002), Assessing writing. The Cambridge Language assessment
series.