Anda di halaman 1dari 63

WHAT IS EVALUATION?

Evaluation is assessing and judging the value of a piece of


work, an organisation or a service. Its main purpose is to help an organisation reflect on what it is trying to achieve,

assessing how far it is succeeding, and identify required


changes. Hutchinson & Waters (1987, pg: 97) stress: Evaluation is basically a matching process: matching needs to available solutions.

WE EVALUATE TO FIND OUT:


what our program has done; how well it has contributed to the goal, met the objectives and undertaken the strategies; what worked well and what didnt, and why;

whether there were any unintended outcomes; and


what can be learnt from the program to improve practice and inform other programs.

FORMATIVE EVALUATION

OUTCOME EVALUTION TYPES OF EVALUATION

SUMMATIVE EVALUTION

IMPACT EVALUTION

PROCESS EVALUTION

FORMATIVE EVALUATION
conducted early in the implementation of a program; aims to identify problems that arise during development and thus allowing modification.

SUMMATIVE EVALUATION

conducted at the end of the course;


focuses on effects or impacts; helps to decide what to do next.

PROCESS EVALUATION
focuses on how the program has been implemented; assesses whether activities have been conducted as planned.

IMPACT EVALUATION
focuses on the immediate effects of the program; judges how well the objectives have been met.

OUTCOME EVALUATION focuses on the longer term effects of the program;

judges how well the goal has been achieved.

EVALUATION:
Answers the question "How well did we do?" (Qualitative evaluation) Answers the question "How much did we do? (Quantitative evaluation)

Qualitative evaluation answers the questions below: What was learned? Are the learners using their new knowledge? If so, how? What do the learners think about the lessons, the teachers,

and the materials?


Do the students think that the courses made a difference in their lives? If yes, what kind of difference? Did the teachers do a good job of communicating the new information?

Did the teachers respect and support the learners ?

Do the learners like the materials? Do the students think the materials are appropriate for each group of learners? Do the materials communicate information they want to learn? Do the learners think that the program is successful?

Which activities do they think are good? Which activities do


they think are not good? Do people in the community think the results of the program are worth the cost and energy necessary to get the program started and to keep it going?

Quantitative evaluation answers the questions below:


How many people were in the target group? How many started the course? How many completed the course? How many dropped out? How many books were produced?

Are the test scores satisfactory?


Does the course worth to the money spent on it?

LEARNER ASSESSMENT

COURSE EVALUATION

LEARNER ASSESSMENT
It helps to assess students performance either at the beginning of the course or at the end.

COURSE EVALUATION
Course evaluation is the last, but not the least important stage. It helps to determine whether the course objectives are being met. Teachers should evaluate their courses to improve and promote their effectiveness. Evaluation of the course is a brave step for the teacher. He / she should be open-minded in hearing and implementing learners' comments.

Although these two different evaluation forms assess

different things on the surface, we can not seperate one


from the other. Evaluation of the learners does not only reflects the learners

performance but, to some extent, the effectiveness or noneffectiveness of the the course as well.

An ESP course is supposed to be successful in order to enable

particular learners to do particular things. If it fails to meet


its objectives, then something might be wrong with the course design, the analysis of the student competence or the

methodology.
We do not expect learners to indicate where the exact fault lies but this application will show us the possible lacks of our programme.

Hence, it can be concluded as both course and learner


evaluation have a similar function in providing the feedback for the ESP course; although each type has other purposes

and procedures.

Although having admitted the importance of evaluation,


there is a lack of guidance how the objectives of the course be tested.

The lack of testing procedures can be attributed to prejudices


against the participating linguists and other practitioners. Yet this does not imply that there are no tests available.

For instance; The London Chamber of Commerce and Industry The Associated Examination Board (AEB) Pitman Examinations Institute Secretarial and Commercial English

Cambridge University

English for Business and English for Science


Technical English

The City and Guilds of London Institute

What is lacking is the sound theoretical or emprical basis for

ESP testing.
There are three basic types of assessment: Placement Tests Achievement Tests Proficiency Tests

PLACEMENT TESTS:
This type of test is used to assess learners academic ability in a certain field so that he or she can be placed in the appropriate courses. It determines whether the learner needs the course or what form of course the learner should take. If a learner is fully qualified in the required skills, no further course is necessary. So it can be concluded as placement tests sometimes act as proficiency tests.

A placement test is diagnostic as it indicates how far the


learner fails to meet the proficiency level. The placement test has a formative value as the result of the test used in forming

the content and the nature of the course.

How Can A Placement Test Be Trustworthy?


According to Alderson and Hughes (1981), there are some doubts about the ability to diagnose learners needs accurately through tests. Nevertheless, in the absence of no more accurate instruments, the course designer has quite a few options. So these tests can only be a guide for us. A good placement test should indicate not only what the learner lacks, but also what potential the learner has for learning.

ACHIEVEMENT TESTS:
Achievement tests are standardized measures of knowledge, information, or procedural learning such as how to do something. Achievement tests may assess general academic skill areas, such as reading, writing, or mathematics, or they may test for content knowledge in a specic academic subject such as biology or American history.

Achievement tests are used by school systems to provide a


standard measure of individual student performance and provide an aggregated measure of performance that enabling

school systems to evaluate their effectiveness. Achievement


tests are also used as part of the diagnostic assessment of individuals to determine whether they have a learning

disability, and therefore qualify for special education services.

Basic principles that should be followed for constructing an


achievement test along with other tests: Test what the students have learnt. This may not be in exact correlation with what you have taught during the lesson. Test what you actually want to test.

Do not write test items demanding special knowledge on


one subject area.

PROFICIENCY TESTS:

Its an exercise whose aims is to evaluate the background of a


person in a given branch of knowledge with or without regard to specific academic learning.

According to Davies and West (1984), the primary purpose of


language testing is to assess whether the students will be able to perform the language tasks required of them. They say such tests are criterion-referenced. With this application there is no fail / pass distinction.

Instead, there is a scale of degrees of proficiency in the task.


Here is an example of a scale used for the British Councils English Language Testing Service (ELTS).

1. NON-USER: Unable to use the language or doesnt provide relevant evidence of language competence forassessment. 2. INTERMITTENT USER: No real communication possible although single-word messages may be conveyed and

understood.
3. EXTREMELY LIMITED USER: Below level of functional competence; although general meaning can be conveyed and understood in simple situations; there are repeated breakdowns in communication. 4. LIMITED USER: Basic functional competence limited to familiar situations, but frequent problems in understanding and fluency can make communication a constant effort.

5. MODEST USER: Partian command of the language coping with overall meaning in most situations although some misunderstanding and lack of fluency could block fluency. 6. COMPETENT USER: Generally effective command of the

language, although occasional misunderstanding and lack of


fluency could interfere with communication. 7. GOOD USER: Operational command of the language;

occasional inaccurasies and misunderstandings in some situations. 8. VERY GOOD USER: Fully operational command of the language; occasional minor inaccurasies or misunderstandings in some situations.

9. EXPERT USER: Fully operational command of the language;

appropriate,accurate
understanding.

and

fluent

with

complete

The proficiency tests requiring students to perform certain


tasks is closely related with the concept of ESP. Proficiency tests for specific purposes should be able to determine whether a student can perform the required skills or not via their own topic of interests rather than a general topic.

Although

specific

language

proficiency

tests

are

an

extension of the ESP principle, there are still some problems about it. The problems are:

Proficiency

tests are criterion-referenced. What should the

criteria be then? What skills and knowledge enable someone to perform particular tasks? Although there is a huge development in our knowledge of language use, we still have lacks on what makes communicative performance possible.

How specific is specific? Can a test in Engineering be attributed to all branches of Engineering hoping it to be a valid indicator?

So we should consider proficiency tests as a guide until a more valid technique is found out.

As a result; the value of tests primarily depends on how they


are used. Both teachers and learners must have a positive attitude towards them until they disappear from our lives.

All the tests we have encountered should be seen as a


performance task in the teaching learning process. Even if a grade is given to a student, the importance of the grade lies under its meaning.

Evaluation has two aims:


assessment feedback Although assessment can be seen as a means of measuring what the learner knows, it also provides a positive feedback to

inform teachers and learners about what is still unknown.

As an ESP course needs to satisfy a particular need, evaluation helps to see how well the course is fulfilling the need. It helps to determine whether it meets its objectives. The result of this evaluation forms your starting point for any necessary revisions, helps to design your other related courses and guides to your teaching methodology. It plays a social role involving teachers, learners, sponsors etc. in evaluation. There are four basic questions to be asked in

course evaluation.

What should be evaluated? How can ESP courses be evaluated? Who should be involved in the evaluation? When (how often) should evaluation take place?

WHAT SHOULD BE EVALUATED?


Your ability to collect information & Your ability to use information once it has been collected.

When considered ESP, the main aims are to meet the


learners needs as language learners and the learners needs as language users. The success of an ESP course is achieved if

the course satisfies both kinds of needs. So we should ask the


following questions:

Is the course fulfilling the learners language learning needs? Has the course fulfilled the learners language using needs?

Is the course fulfilling

If you get a YES to any of these questions, then just go

ahead! You are on the right way!


However, assuming that you might have only an ORDINARY YES instead of an outright NO, the next question should be asked.

What areas of need are not being / have not been fulfilled? If we know where the problematic areas are, we can try to find out the source(s) of the problems. Here are some other relevant questions on what to evaluate on course evaluation.

Were the unfulfilled needs identified during the course design project? If not, why not?
How can the course design process be improved to avoid this problem in future?

Taking these needs into account, how can the course be changed?
If the needs were identified during the course design process, is the fault in: a. the syllabus(es), b. the applied materials, c. the teaching and learning techniques, d. the testing procedures, e. logistical / administrative arrangements, f. the course evaluation system?

HOW CAN ESP COURSES BE EVALUATED?

Evaluation can be done in two different ways: implicitly and


explicitly. Implicit evaluation takes place during the semester; students grades, participation, and motivation give clues to the teacher on how their learning is going on. Explicit evaluation may take place at the end of the course or after students have experienced it. Using questionnaires, surveys, talks, etc. teachers ask the students to express their attitude towards the subject matter, instructional methods, activities,

teacher's role and so on.

Although there are lots of techniques in evaluating ESP


course, most common ones are the followings. It is not necessary to use only one of the following techniques at a

time. More than one technique can also be used at the same
time. TEST RESULTS QUESTIONNAIRES DISCUSSIONS

INTERVIEWS

There is not a strict limit on which technique


choose. It depends on your own situation.

you should

Yet having got the evaluation information is only the

beginning in this process. The information you got must be


gathered together and needs to be discussed with all the participant parties (teacher learner administrator sponsoretc) and some conclusions be drawn.

This course evaluation report should be used as a basis for


decision making. They are valuable suggestions to redesign your course and correct the problems.

EVALUATION INVOLVES

ASKING QUESTIONS

ANSWERING QUESTIONS

ANALYSING, REFLECTING ON & JUDGING THE EVIDENCE

MAKE CHANGES!

WHO SHOULD BE INVOLVED IN THE EVALUATION?

Whoever will involve in the evaluation may vary, but the


very closely related participants will be the ESP teachers, the learners, and the course sponsors as well as the former

students.

Evaluation is primarily concerned with peoples perceptions


of value; therefore their evaluation may also vary. The teacher can have difficulty in getting the exact view as the students may be reluctant to criticise authority for fear of their own assessment.

The students can also be indifferent thinking that, this


evaluation will be beneficial to the future students but them.

However, according to Waters (1985), the best basis for


promoting frank and useful feedback can only be achieved through an open and trusting relationship.

Who you ask and how you ask also affect the results. Thats
why, the methodology of evaluation process should be taken into a great consideration. You should explicitly note how you obtain the information, from whom you got it and by what means.

WHEN & HOW OFTEN SHOULD EVALUATION TAKE PLACE?

It is difficult to mention how frequently an evaluation should


take place. Whether we apply this experience too often or rare, there exists danger in both.

The most important frequencies of the evaluation should be:


in the first week of the course at regular intervals throughout the course at the end of the course after the course (if possible)

The evaluation made after the class is potentially the most


valuable one as the learners will be in a position to judge how well the course was prepared for the target situation

they are in now.

Evaluation can be time-consuming, complex and frustrating. There might be no obvious solution to the problem.

Feedback from one party may contradict feedback from


another. Learners might take it as an incompetence. Criticising a course might be seen as a lack of respect to the authority.

STUDENT ANALYSIS

FORMULATION OF GOALS AND OBJECTIVES

CONTENT

SELECTION OF TEACHING MATERIALS

PLANNING THE COURSE

COURSE EVALUATION

REFERENCES
Hutchinson, T. & Waters, A. (1987). English For Specific Purposes: a learning-centered appproach. Cambridge: Cambridge University Press . Spencer, L.(2003). Quality in Qualitive Evaluation: A framework for assessing research evidence. The Brazilian ESP Project. Ed. Holmes, J. (1988)

Caado, Mara Luisa Prez & Esteban, Ana Almagro. (2005). Authenticity in the Teaching of ESP: An Evaluation Proposal