Accreditation & Outcome

Based Approach
Prof Megat Johari Megat Mohd Noor
UTM QRIM@KL & MJIIT

Lahore/Faisalabad/Jamshoro, Pakistan

September 2016
Introduction
We are joining the Mutual Recognition
Train (MRT) !

3
EDUCATION ACCORDS PRACTICE AGREEMENTS

IPEA
International Professional Engineers Agreement
WASHINGTON (ENGINEERS MOBILITY FORUM)

ACCORD 4 YEARS
APEC ENGINEER

SYDNEY IETA
ACCORD 3 YEARS International Engineering Technologist Agreement
(ENGINEERING TECHNOLOGISTS MOBILITY FORUM)

DUBLIN AIET
ACCORD 2 YEARS
Agreement of International Engineering Technician

FEANI / EUR-ACE / ENAEE 3+2

(EUROPE) YEARS INTERNATIONAL
ENGINEERING
NABEEA ALLIANCE (IEA)
(ASIA) (INTERNATIONAL
ENGINEERING MEETING, IEM) 4
 Development of International
Engineering Alliance

Development Development
New Accords
WA signed by 6 of formal of graduate
and
organisations peer review attribute
Agreements
processes exemplars

1990s 2001
28 Sep 1989 1997-2015
onwards onwards

IEA Established in 2007

5
WASHINGTON ACCORD FULL SIGNATORY Provisional Status
1. Australia - Engineers Australia (1989)
19. Bangladesh
2. New Zealand - Institution of Professional Engineers NZ (1989)
3. Canada - Engineers Canada (1989) 20. Pakistan
4. United States - Accreditation Board for Engineering and 21. Phillippines
Technology (1989)
5. United Kingdom - Engineering Council UK (1989) 22. Peru
6. Ireland - Engineers Ireland (1989) 23. Costa Rica
7. Hong Kong China - The Hong Kong Institution of Engineers (1995)
8. South Africa - Engineering Council of South Africa (1999) 24. Mexico
9. Japan - Japan Accreditation Board for Engineering Education
(2005)
10. Singapore - Institution of Engineers Singapore (2006)
11. Chinese Taipei - Institute of Engineering Education Taiwan (2007)
12. Korea - Accreditation Board for Engineering Education of Korea
(2007)
13. Malaysia - Board of Engineers Malaysia (2009) Potential Applicants
14. Turkey - MUDEK (2011)
25. Thailand
15. Russia - Association for Engineering Education of Russia (2012)
16. India - National Board of Accreditation (2014) 26. Indonesia
17. Sri Lanka - Institution of Engineers Sri Lanka (2014)
6
18. China - CAST (2016)
Accreditation

7
Importance of Accreditation to Institutions of Higher
Learning

• Recognises institutional missions and goals

• Involves faculty/staff in evaluation and planning
• Assists institutions in determining the
acceptability of transfer credits
• Promotes “best practices” in education
• Increases visibility and reputation of the
institution
• Aids engineering schools to identify required
operational resources to institution
management
8
Pakistan Washington Accord Route (2011 – 2016)

Nominator
• Nominator (EME, GIKI) Prof Abang (MAL)
Prof Megat (MAL)
Prof Lock (SIN)
Mentor
• Mentor (Islamabad, Topi, Risalpur,
Prof Megat (MAL)
Faisalabad, Peshawar, Karachi, Ir Azlan (MAL)
Lahore) Prof Lock (SIN)
Reviewer?
• 1a Reviewer (Universities?) Kim (Korea)
Collins (UK)
• 1b Reviewer (ADM) Basil (NZ)

9
 Constituents in Accreditation
Representatives of
Universities DECIDE
ENDORSE

Representatives of ENGINEERING PAKISTAN

Industries ACCREDITATION ENGINEERING
BOARD (EAB) COUNCIL (PEC)
Representatives of
Professional
Societies ACCREDITATION
DEPARTMENT
EAB (AD)
Representatives of
EVALUATION
Government
PANEL
FACILITATE
RECOMMEND

10
 EAB Manual
• From input based to outcome based

20.. 2014 11
 Students
(…..)

Academic Academic &

Curriculum Support
(…..) Programme Staff (…..)
Objectives (…..)
& Outcomes
(…..)

Facilities (…..) Industry

Linkage (…..)
12
Programme Evaluators (PEVs)

 Chair (Criteria of appointment)

 Two members (Criteria of
appointment)
one member with extensive academic
- knowledgeable
experience and one member
- trained with
extensive industrial experience
- independent
13
 Challenges
• Paradigm Shift – Outcome & Quality
• Maintain Fundamentals while Encourage
Inclusion of Latest Technology Advancement in
the Curriculum
• Allow Academic Innovation and Creativity
• Avoid Side-tracked
• Variety of Modes of Delivery
 Depth of Knowledge Required
Complex Broadly Defined Well defined
Problems Problems Problems

In-depth knowledge Solved using

Knowledge of
that allows a limited theoretical
principles and
fundamentals- knowledge, but
applied
based first normally requires
procedures or
principles analytical extensive practical
methodologies
approach knowledge

15
Engineering & Technology Domain
Engineers

Supervision &
Research & Design Work
Maintenance
Technologists

Education
Strong in Appropriate
Mathematics, Mathematics,
Engineering Engineering
Sciences, Sciences,
Professional Professional
courses courses
(Theoretical) (Practical)
Engineering Technology
Breadth & Depth Breadth & Depth
of Curricula of Curricula
16
 Educational Process & Stakeholders Internal Stakeholders
Teachers
Students
Pull Programme EO / O Development/ Review University
Specification

factor
External Stakeholders
Course O / Content Potential Employers / Industry
Alumni
Development / Review Regulatory Body
1, 2, 3 ……
Formative / Summative

Course Implementation Internal Stakeholders

1, 2, 3 …… Teachers

Course Assessment Internal Stakeholders

1, 2, 3 ……
Teachers
Teacher – Knowledge, Skills, Affective Technicians
Students – Teaching Students
Teacher – Descriptive Self Assessment
on Cohort’s Achievement
Internal Stakeholders
Programme Evaluation Teachers
Students
Summative

Summative - direct
Exit Survey - indirect
Industry Survey - indirect
Alumni Survey - indirect External Stakeholders
External – direct Potential Employers / Industry
Accreditation - direct Alumni
Regulatory Body
External Assessor 17
 Academic IQA Practices in Perspective

(8) Leadership, governance

STAKEHOLDER NEEDS
AND INSTITUTIONAL

and administration
MISSION

(1) Vision,mission and (7) Programme

Monitoring and

STAKEHOLDER’S SATISFACTION
learning outcomes
Review

(6) Educational (2a) Curriculum (5) Academic

resources Design Staff

INPUT (STUDENTS) (3) Student OUTPUT

(2b) Curriculum Assessments (GRADUATES)
(4a) Selection of Students Delivery
FEEDBACKS FROM
STAKEHOLDERS

(4b) Supporting Services

(9) Total Continual Quality

Improvements
18
 Directed & Coherent Curriculum
Graduate Relevant to Industry
Programme Objective
(after 3-5 Years)
Programme Outcome
(at Exit)
Course/Unit/Learning Outcome
(Abilities & Intentional)

19
Characteristics of OBE curricula

• Have programme objectives,

programme outcomes, course
outcomes and performance
indicators.
• Stated objectives and outcomes can be
assessed and evaluated.
• Centered around the needs of the
students and the stakeholders.
20
 Characteristics of OBE curricula
• Learning outcomes are intentional and
assessed using suitable performance
indicators.
• Programme objectives address the
graduates attainment in their career within
3-5 years after their graduation.
• Programme outcomes (abilities attained
by students before they graduate) are
formulated based on the programme
objectives – TOP DOWN.
21
 Characteristics of OBE curricula
• Programme outcomes address Knowledge,
Skills and Attitudes to be attained by
students.
• Course outcomes must satisfy the stated
programme outcomes. There is no need for
ANY (individual) course to address all
programme outcomes.
• Teaching/ Learning method may have to be
integrated to include different delivery
methods to complement the traditional
Lecturing method.
22
 OBE in a nut shell

 What do you want the students to have or

able to do?  Knowledge, Skill, Affective

 How can you best help students achieve

it?  Student Centred Delivery

 How will you know what they have

achieved it?  Assessment

 How do you close the loop

 PDCA 23
Plan, Do, Check & Act (PDCA)

24
 Strategy of OBE
• Top down curricula design
• Appropriate Teaching & Learning Methods
• Appropriate Assessment & Evaluation
Methods

25
 Different Levels of Outcomes

Programme Educational Objectives Few years after

Graduation – 3 to 5 years

Programme Outcomes Upon graduation

Course/subject Outcomes Upon subject completion

Weekly/Topic Outcomes Upon weekly/topic completion

26
Accreditation
 Preparation for Accreditation
• Comprehend the EAB Manual
• Prepare the SAR
• Address previous accreditation report
• Arrange the evidence
• Complete the Checklist
• Assign key persons according to accreditation
schedule

28
SAR & Evidence

29
Azlan

or
a man sitting
down in a
garden and a
lady passing
by?
30
 Issues from
• Approval Report
• First Year Visit Report
• External Examiner’s Report
CQI
• Stakeholders’ Recommendations
• Recent Accreditation Visit
Documents & Records
EVIDENCE BASED Interviews
Observations
 Exercise
Self Assessment Report
• Prepare a checklist of questions based on the following
criteria in preparing a SAR:
– Programme Objectives
– Programme Outcomes
– Curriculum and Learning Process
– Students
– Faculty
– Support staff
– Industry stakeholders
– Alumni
– Facilities and Infrastructure
– Institutional and Financial Support
– Continual Quality Improvement
– Industry Linkage
 Exercise

Industrial Linkage
Using the Clause 9 of the PEC Manual, evaluate
the relevant part of the submitted SAR.
 3.2.9 Criterion 9: Industrial Linkages
Students are expected to undertake assignments
from industry to provide solutions to complex
engineering problems. Students and faculty should
be encouraged to establish collaboration for R&D
and product development related projects, with
due regard to environmental and societal impact.
Feedback from the industry and employers is
crucial and an essential part of curriculum review
process used to evaluate attainment of the
program objectives.
 4.1.9 Industrial Linkages
4.1.9.1
Discuss the involvement of
industry in discussions and forums,
professional practice exposure, and
collaborative projects / research
for the solutions to engineering
problems.
 Exercise

List down HEI’s representatives (parties) that

will be involved in an accreditation visit

36
 Exercise

Item/Criteria List down the involved parties

Accreditation Planning
Opening Meeting
Programme Educational Objectives
(……)
Programme Outcomes (…….)
Curriculum (…….)
Students (……)
Academic & Support Staff (……)
Facilities (……)
Quality Management System (……)
Exit Meeting

37
 PROGRAMME EVALUATOR’S APPROACHES

• Sensible questioning
• Check records
• Observing processes
• Analyse inputs and outputs
• Organised using tables, matrices,
flowcharts and checklists
38
 Quiz

What are the six typical words that Programme

Evaluators (PEVs) would usually begin with,
when questioning?

Programme Evaluators’ (PEVs’) Best friend –

39
 Quiz

What are the methods/techniques employed by

Programme Evaluators (PEVs) when conducting
an accreditation exercise?

40
 Cause for concerns at
Accreditation Decision Meeting
• Phases of OBE
– Planning
– Implementation
– Effectiveness
• CQI
• List of concerns
• Breadth & depth (taxonomy & complex problem)
• Staffing
• Industrial Training
• Commitment to change
• System failure
• Stagnant (no improvement)
• Repeat offender
• Safety 41
Rubrics for New Programme, New
Cycle & Continuing

42
 What WA will be observing?
• Adherence to EAB document
• EAB PEV’s aplomb and decorum
• Probing questions (not interrogative)
• Discussion level
• Clarity of reports
• Graduate outcomes
• Health & safety at HEIs
• Equivalency of practice

43
 Exercise

Opening Meeting
You are the Dean/HoD with three programmes
to be evaluated; Mechanical Engineering, Civil
Engineering and Electrical Engineering, for the
third cycle. Prepare a list of talk points to
address the Programme Evaluators (PEVs) at the
Opening Meeting.

44
 OPENING MEETING
• Welcome Evaluation Team
• Introduce team members
• Corrective & Preventive
Actions from previous
accreditation
• Short presentation on 10 minutes
Faculty/Dept/Prog strengths
• Fill up with the latest (within
a specified timeframe) if any

45
 PEC Manual 2014
Programme Educational Objectives

46
 Exercise
List down potential stakeholders
• Major

• Minor
 Programme Educational Objectives

• Broad statements: What graduates are

expected to achieve (BE) a few years after
graduation.
• Linked to programme outcomes
• Include feedback from employers, alumni,
academics and other stakeholders

48
Programme Educational
PROGRAM EDUCATIONAL Objectives
OBJECTIVE (PEO)

 Limit number of statements (manageable)

 No restatement of outcomes
 Forward looking and challenging
 Distinctive/unique features/having own
niche
 Specific, Measurable, Achievable, Result
oriented, and having a Time frame (SMART)
49
 Exercise

Write down your evaluation on the following

PEO statements

50
 Exercise

Write down your evaluation on the following

PEO statements

51
 PEC Manual 2012
Programme Outcomes

52
 Programme Outcomes (PO)

What the graduates are expected to know and

able to perform or attain by the time of
graduation (skills, knowledge and
behaviour/attitude)
There must be a clear linkage between
Programme Objective and Outcomes

Need to distribute the outcomes throughout the

programme, and not one/two courses only addressing a
particular outcome 53
 Exercise

Programme Outcomes
• Discuss on HEI’s possible approaches or
methods to demonstrate implementation of
the 12 programme outcomes
• Discuss on the possible models to show
attainment of the 12 programme outcomes

54
 PEO
WHAT YOU WANT YOUR GRADUATES TO BE IN 3 - 5 YEARS
WA3 WA9
DESIGN IND & TEAM

WA 1

UNIVERSITY EXPERIENCE
WA10
WA5
EXTRA-CURRICULAR

ENGINEERING COMMUNICAT-
KNOWLEDGE MODERN TOOLS
ION

WA 2
PROBLEM
4 YEARS WA11
WA6 ENGR & SOC
ANALYSIS PROJ MGMT &
WA7 ENV & SUST
FINANCE
WA8 ETHICS

WA4 WA12
INVESTIGATION LIFE LONG
 Problem Organised Project Work
or POPBL (Project Oriented Problem Based
Learning)
Literature Lectures Group Studies

Problem Analysis Problem Solving Report

Tutorials Field Work Experiment

57
 POPBL Requirements
• High degree of supervision
• Office space
• Lectures to be constantly changing or
renewed
• Flexibility in the distribution of resources

58
 Instructors/Supervisors

• Pedagogical skills
• Scientific skills
• Time management
• Project based on staff research

59
 Graduates’ Strength

AALBORG UNIV DENMARK

• Strong in problem TECHNICAL UNIV
solving
• Specialist knowledge
• Communication
• Technical
• Cooperation methodology
• General technical
knowledge

60
 Typical questions on PEO/PO
• What are the PEOs/POs?
• Who were involved in the development of the
PEOs/POs?
• How were they developed/improved?
• To what extent the stakeholders were
involved?
• How their attainment were determined?
• What were the improvements introduced?
61
 Exercise

Curriculum
• Discuss on the possible relationship between
taxonomy levels and the different knowledge
profile with consideration to the 12 programme
outcomes
• What are typical probing questions in
ascertaining that student’s POs have been
attained?
• What are typical probing questions in
ascertaining that student’s COs have been
attained?
62
 ASSESSMENT:
Processes that identify, collect, use and
prepare data for evaluation of achievement
of programme outcomes or educational
objectives.
EVALUATION:
Processes for interpretation of data and
evidence from assessment practices that
determine the programme outcomes are
achieved or result in actions to improve
programme. 63
 Outcome-Based Assessment
Implementation Assessment Data
Strategy Strategy Sources/Assessment
instruments
Industrial project Exams, interview, Reports, interview
Improve student survey, observe, schedule, survey,
competence in assess skill level, observation records,
communication, monitor grades of exams and
teamwork, and project development of projects, exit skill
management skills checklist
Design course Assessment by List of assessment
Address industry industry, and criteria, observation,
needs lecturers reports, interview,
students evaluation,
exams, exit skill
checklist 64
 Assessment

– drives learning (necessary evil!)

– is formative or/and summative; to
demonstrate student’s competence in
demonstrating a specific outcome
– is the process that identify, collect, use
and prepare data that can be used to
evaluate attainment.
65
 Assessment
• Do not assess those that have not
been taught

66
 What Assessment?
• Assessing Student/Cohort (Course Outcome)
• Assessing Student/Cohort & Faculty
(Programme Outcome)

67
 Assessment Process

–Anecdotal vs. measured results

–Reliance on course grades only
–Over-reliance on indirect assessment
(survey)

68
MEASURE & EVALUATE

University Assessment & Evaluation

69
How will you know what they
have achieved it?
 Formative Assessment
 Summative Assessment
 Course Assessment
 Program Assessment
 Assessment Tools
 Direct and Indirect Assessment
70
How do you close the loop ?

• Assessment Plan
• Who is doing what and when
• Stakeholder participation
• CQI in place

71
 Programme Outcome Assessment
Matrix
Outcome indicators PO 1 PO 2
& core courses
Project Report A B
Course 1 B B
Course 2 C B

A: slightly, B: moderately, C:substantively - base on a

review of course materials (syllabus, learning objectives,
tests, other assessment…..)
Outcome 1: ability to …..

Outcome 2: ability to ….. 72

 Course Assessment Matrix
Outcome-related PO 1 PO 2
learning objectives
Explain A C
Perform calculation B B
Identify B B
Solve B C

A: slightly, B: moderately, C: substantively

Outcome 1: ability to …..

Outcome 2: ability to …..

73
 Rubric Adopted from G.Rogers

4 – Exceeds 3 – Meets Criteria 2 - Progressing 1 - Below

Criteria to Criteria Expectations
Content Provides ample Provides adequate Some details but Inconsistent or few
supporting detail supporting detail may include details that may
to support solution/ to support solution/ extraneous interfere with the
argument argument. or loosely meaning of the text.
related material.
Organization Organizational Organizational Little completeness Little evidence of
pattern is logical & pattern is logical & & wholeness, organization or any
conveys conveys completeness though organization sense of wholeness
completeness & wholeness attempted. & completeness.
& wholeness. with few lapses.
Style Uses effective Uses effective Limited & Limited or
language; makes language & predictable inappropriate
engaging, appropriate vocabulary, perhaps vocabulary for the
appropriate word word choices not appropriate for intended audience
choices for for intended audience intended audience & purpose.
audience & purpose. & purpose.
& purpose.
Consistently Generally follows Generally does not Does not follow the
follows the rules for standard follow the rules of rules of standard
the rules of English. standard English. English. 74
 Performance Criteria/ Indicators -
Good Teamwork
Students are able to demonstrate

1. Positive contribution to the team project (minutes of

meeting)
2. Well prepared and participate in discussion (observation)

3. Volunteer to take responsibility

4. Prompt and sufficient attendance

5. Aplomb and decorum

75
 Quiz
Courses PO1 PO2 PO9 PO10
C1 3 2 1 1
C2 2 1 2 2
C3 3 0 3 2
C4 2 1 3 1
Discuss on the potential problems, if any, where 3, 2, 1, and 0
refer to High, Moderate, Low, and No emphasis, respectively.
C1..4 refer to the courses, whereas POs 1,2,9 and 10 refer to
Programme Outcomes.

How would cohort POs attainment be obtained?

76
 Quiz
PO1 PO2 PO9 PO10
C1-CO1 + +
C1-CO2 + + +
C1-CO3 + + +
C1-CO4 + +

How would you design the assessment for

the above matrix?
CO: Course Outcomes +: There is assessment
77
 Quiz
Table 1 PO 1 Discuss on the
Q1 CO1 + attainment of COs and
Q2 CO2 + POs for both Tables 1&2,
Q3 CO3 + where Qs are questions
Q4 CO4 + set to address the COs

Table 2 PO 1 PO9
Q1 CO1 + CO2 +
Q2 CO2 + CO3 +
Q3 CO3 + CO4 +
Q4 CO4 +
78
 Quiz
Delivery Assessment
Lecture
Laboratory
PBL
Case Method
Project Based

Identify suitable assessment techniques for the

different delivery modes.
79
 Lessons learnt from accreditation
activities related to assessment
 Do not know the teaching plan
 Done without referring to the plan
 Do not know how to translate plan into
assessment
 Assessing at low-medium level (not challenging)
 No feedback to students except at end of
semester
 Do not know how to relate assessment to
expected outcomes
 Repetition
 Bulk marking
 Traditional assessments
80
 AssessmentBig PictureAlignment
– Constructive

Programme or
Student
Improvement ?

Selective
Culminating
Hybrid

Taxonomy Level (Average, From, Up To)

81
 Curricula Models
Distribution of Knowledge, Skills & Attitude
elements throughout the 4 years

S&A
Yr. 4 S&A 30%
30%

Yr. 3 K 70% K 70% K 70% K 70%

Yr. 2
S&A
30%
Yr. 1 S&A
30%

A B C D 82
Evaluation of Outcomes at Programme Level
ECV3092 Civil Engineering Design
MyOBE
(Capstone Design Course) Software

83
 Evaluation of Outcomes at Programme Level
ECV3092 Civil Engineering Design
MyOBE Process
(Capstone Design Course) Module

Teaching and Assessment

Programme Outcomes Course Assessment
Plan
Attributes that are Course Assessment Mark
Planning of course outcome
expected to be attained by Course Assessment
the students Planning of course
Summary
assessment

Programme Evaluation
Programme Objectives
Programme Outcomes Evaluation
Summative
Stakeholders’ Survey
Trend Analysis

84
Evaluation of Outcomes at Programme Level
ECV3092 Civil Engineering Design
MyOBE
(Capstone Design Course) Snapshots

Lecturers’ Module:
Enter all course
assessment marks

85
Evaluation of Outcomes at Programme Level
ECV3092 Civil Engineering Design
MyOBE
(Capstone Design Course) Snapshots

86
 PO Attainment

Final Year
Final Year Project Final Year Courses
Design Project

Third Year Courses

Second Year Courses

First Year Courses

87
 What constitutes strength?
 Exceeds the minimum standard set by the EAC
Engineering Accreditation Manual.
 Extensive benchmarking (not only via the external
examiners path) with more established
programmes/institutions.
 The curriculum is built on strong fundamentals
(engineering sciences) and appropriate engineering
knowledge according to the discipline, which transcend
national boundaries.
 Generic attributes (professional and/or interpersonal
skills) should also be evident to prepare graduates for the
advanced part of their career.
88
 What constitutes strength? Cont…
• A curriculum with clear (measurable) objective(s)
and outcomes (that satisfies the (12) EAC stipulated
outcomes)
• Involved stakeholders, both internal and external,
extensively
• An appropriate working load for students
determined through extensive consultation with the
academics (Usually a 15 – 16 credit per semester
loading)
• Blend of delivery methods
89
 What constitutes strength? Cont…
 Programme challenges students to achieve greater
heights than just satisfying the minimum standard
 Attain competency in the open-ended project based
and problem oriented courses
 Majority of the staff has PhD qualification and the
number available indicates a low staff-student ratio
(that enables greater contact with students)
 The academic staffs also conduct research that
permeates/contributes to teaching and learning.
90
 What constitutes strength? Cont…
• Over and above Industrial Training (extensive &
distributed professional exposure) that does not
compromise on the cognitive domain
• Ergonomics is taken seriously by the institution to
reduce occupational hazard
• Safety culture
• Show that they have the plan and the completion of
the quality cycles is widespread
• Monitoring of the QMS also indicates strength.
91
 What constitutes strength? Cont…
• Students’ ability to give opinion and articulate
with substance
• Students are clear of their goals upon graduation
and highly motivated during their course of
study (“constructive criticisms”)
• Widespread involvement of students in co-
curricular activities (not forced as part of
curriculum nor limited to small group of
students).

92
What constitutes strength? Cont…
• Academic staff with Professional Engineer
status
• Academic staff are actively participating in
professional activities (not merely members)
• Design courses are taught by experienced
academics (with consultancy experience or
professional engineers).

93
What constitutes strength? Cont…
• Up to date facilities are made available and
they exceed the recommended student-
equipment ratio appropriate to the relevant
discipline.
• Extensive electronics publications for life long
learning, project based courses and the final
year project

94
Taxonomy & Course Outcome
 Bloom’s Taxonomy
• Knowledge (list)
• Comprehension (explain)
• Application (calculate, solve, determine)
• Analysis (classify, predict, model,derived)
• Synthesis (design, improve)
• Evaluation (judge, select, critique)

96
97
lower order Intermediate Higher order 98
lower order Intermediate Higher order 99
 Why are course outcomes important?

• Define the type and depth of learning students are

expected to achieve
• Provide an objective benchmark for formative,
summative, and prior learning assessment
• Clearly communicate expectations to learners
• Clearly communicate graduates’ skills to the
stakeholders
• Define coherent units of learning that can be further
subdivided or modularized for classroom or for other
delivery modes.
• Guide and organize the instructor and the learner.
100
Three components of a learning outcome
Reaching the “Standard” (criteria of acceptable level of
performance)
• describe the principles used in designing X.(Verb)
• orally describe the principles used in designing X. (Verb &
Condition)
• orally describe the five principles used in designing X. (Verb &
Condition & Standard)

• design a beam. (Verb)

• design a beam using Microsoft Excel design template . (Verb &
Condition)
• design a beam using Microsoft Excel design template based on
BS 5950:Part 1. (Verb & Condition & Standard)
101
Learning outcomes by adding a condition and standard

Poor
• Students should be able to design research.

Better
• Students should be able to independently design
and carry out experimental and correlational
research.
Best
• Students should be able to independently design
and carry out experimental and correlational
research that yields valid results.
Source: Bergen, R. 2000. A Program Guideline for Outcomes Assessment at Geneva College
102
WA Knowledge Profile (WK)

103
 WK1 WA Knowledge Profile WK5
natural sciences engineering
(WK) design

WK2
mathematics,
numerical
WK6
analysis,
engineering
statistics,
practice
computer and
information
science
4 YEARS WK7
WK3 engineering in
engineering society
fundamentals

WK4 WK8
engineering research
specialist literature
knowledge 104
 WA Knowledge Profile (Curriculum)
Theory-based natural sciences WK1
Conceptually-based mathematics, numerical WK2
analysis, statistics and formal aspects of
computer and information science to
support analysis and modelling
Theory-based engineering fundamentals WK3
Engineering specialist knowledge that WK4
provides theoretical frameworks and bodies
of knowledge for the practice areas; much is
105
forefront
 WA Knowledge Profile
Knowledge that supports Engineering design in WK5
the practice areas
Knowledge of Engineering practice WK6
(technology) in the practice areas
Comprehension of the role of Engineering in WK7
society and identified issues in engineering
practice: ethics and professional responsibility
of an engineer to public safety; the impact of
engineering activity: economic, social,
cultural, environmental and sustainability
Engagement with selected knowledge in the WK8
Research literature 106
WA Programme Outcome or
Graduate Attributes (WA)

107
 Washington Accord Graduate Attributes
PROGRAMME OUTCOMES
WA1 Engineering Knowledge Breadth & depth of knowledge
WA2 Problem Analysis Complexity of analysis
WA3 Design/Development of Breadth & uniqueness of engineering problems i.e. the extent to
Solutions which problems are original and to which solutions have
previously been identified and coded
WA4 Investigation Breadth & depth of investigation and experimentation
WA5 Modern Tool Usage Level of understanding of the appropriateness of the tool
WA6 The Engineer and Society Level of knowledge and responsibility
WA7 Environment and Type of solutions
Sustainability
WA8 Ethics Understanding and level of practice
WA9 Individual and Team Work Role in and diversity of team

WA10 Communication Level of communication according to type of activities performed

WA11 Project Management and Level of management required for differing types of activity
Finance
WA12 Life-long Learning Preparation for and depth of continuing learning 108
 PROGRAMME OUTCOME
Engineering Knowledge
(WA1) Apply knowledge of mathematics, natural
science, engineering fundamentals and an
engineering specialisation to the solution of
complex engineering problems; (WK1 to WK4)

WA = Programme Outcome
WK = Knowledge Profile = Curriculum 109
 PROGRAMME OUTCOME
Problem Analysis - Complexity of analysis
(WA2) Identify, formulate, research literature
and analyse complex engineering problems
reaching substantiated conclusions using first
principles of mathematics, natural sciences and
engineering sciences (WK1 – WK4)

110
 PROGRAMME OUTCOME
Design/Development of Solutions – Breadth and
uniqueness of engineering problems i.e. the extent
to which problems are original and to which
solutions have previously been identified or codified
(WA3) Design solutions for complex engineering
problems and design systems, components or
processes that meet specified needs with appropriate
consideration for public health and safety, cultural,
societal, and environmental considerations (WK5)
111
 PROGRAMME OUTCOME
Investigation - Breadth & Depth of Investigation
& Experimentation
(WA4) Conduct investigation of complex problems
using research based knowledge (WK8) and
research methods including design of
experiments, analysis and interpretation of data,
and synthesis of information to provide valid
conclusions 112
 PROGRAMME OUTCOME
Modern Tool Usage - Level of understanding of the
appropriateness of the tool
(WA5) Create, select and apply appropriate
techniques, resources, and modern engineering
and IT tools, including prediction and modelling, to
complex engineering problems, with an
understanding of the limitations. (WK6)
113
 PROGRAMME OUTCOME
The Engineer and Society - Level of knowledge
and responsibility
(WA6) Apply reasoning informed by contextual
knowledge to assess societal, health, safety, legal
and cultural issues and the consequent
responsibilities relevant to professional
engineering practice and solutions to complex
engineering problems. (WK7) 114
 PROGRAMME OUTCOME
Environment and Sustainability - Type of solutions
(WA7) Understand and evaluate the sustainabilty
and impact of professional engineering work in the
solutions of complex engineering problems in
societal and environmental contexts (demonstrate
knowledge of and need for sustainable
development) (WK7)
115
 PROGRAMME OUTCOME
Ethics - Understanding and level of practice
(WA8) Apply ethical principles and commit to
professional ethics and responsibilities and norms
of engineering practice. (WK7)

116
 PROGRAMME OUTCOME
Individual and Team Work – Role in and diversity
of team
(WA9) Function effectively as an individual, and as
a member or leader in diverse teams and in multi-
disciplinary settings

117
 PROGRAMME OUTCOME
Communication – Level of communication
according to type of activities performed
(WA10) Communicate effectively on complex
engineering activities with the engineering
community and with society at large, such as being
able to comprehend and write effective reports
and design documentation, make effective
presentations, and give and receive clear
instructions 118
 PROGRAMME OUTCOME
Project Management and Finance – Level of
management required for differing types of
activity
(WA11) Demonstrate knowledge and
understanding of engineering and management
principles and economic decision-making and
apply these to one’s own work, as a member and
leader in a team, to manage projects and in
multidisciplinary environments 119
 PROGRAMME OUTCOME
Life-long Learning – Preparation for and depth of
continuing learning
(WA12) Recognise the need for, and have the
preparation and ability to engage in independent
and life-long learning in the broadest context of
technological change
120
 WK1 WA9 WK5 WA3
natural sciences
IND & TEAM
engineering DESIGN
design
WK2
mathematics,
WA1 numerical WA10
WK6 WA5
ENGINEERING analysis,
engineering
COMMUNICAT-ION MODERN TOOLS
KNOWLEDGE statistics,
practice
computer and
information
WA2
PROBLEM
4 YEARS
science
WA6 ENGR & SOC
ANALYSIS WA11 WK7 WA7 ENV & SUST
WK3MGMT &engineering
PROJ FINANCE in
engineering society WA8 ETHICS
fundamentals

WK4 WA12 WK8 WA4

engineering
LIFE LONGresearch INVESTIGATION
specialist literature
knowledge
121
WA Complex Problem (WP)

122
 Complex Problem

Need to think broadly and systematically

and see the big picture

Difficult Decision
Uncertain Strategy
Confusing Idea
Contentious Product
Intractable Change
123
 Difficulty & Uncertainty
• Complexity – the problem contains a large
number of diverse, dynamic and
interdependent elements
• Measurement – it is difficult or practically
unfeasible to get good qualitative data
• Novelty – there is a new solution evolving
or an innovative design is needed

124
Scientific/Technical
Problems A
can combine to Complex Problem
form

125
 Limited
Explanation,
Explanation,
Prediction,
Prediction,
Control
Control
Isolatable
Unbounded Results in an
Systems, Results in a
Systems, No educated
Controlled Covering Law
Experiment guest
Experiment

Complex

Complex Simple causal

? f(x,y,z)
causal Chains Chains
Technical

A limited
All the Salient
number of
Difficult to features are
Measurable features are
measure captured by
captured by
the Model
the Model
Operating with Operating with
scare adequate
resources resources 126
 Characteristics
Technical Problems
• Isolatable boundable problem
• Universally similar type
• Stable and/or predictable
problem parameters
• Multiple low-risk experiments are
possible
• Limited set of alternative
solutions
• Involve few or homogeneous
stakeholders
• Single optimal and testable
solutions
• Single optimal solution can be
clearly recognised
Complex Problems
• No definitive problem boundary
• Relatively unique or unprecedented
• Unstable and/or unpredictable
problem parameters
• Multiple experiments are not
possible
• No bounded set of alternative
solutions
• Multiple stakeholders with different
views or interest
• No single optimal and/or objectively
testable solution
• No clear stopping point
127
Complex Problems (Need High Taxonomy Level)
Complex Engineering Problems have characteristic WP1 and some or all of WP2 to WP7, EP1 and EP2, that
can be resolved with in-depth forefront knowledge

WP1 Depth of Knowledge Resolved with forefront in-depth engineering

required knowledge (WK3, WK4, WK5, WK6 or WK8) which
allows a fundamentals-based, first principles analytical
approach
WP2 Range of conflicting Involve wide-ranging or conflicting technical,
requirements engineering and other issues.
WP3 Depth of analysis required Have no obvious solution and require abstract thinking,
originality in analysis to formulate suitable models.
WP4 Familiarity of issues Involve infrequently encountered issues
WP5 Extent of applicable codes Beyond codes of practice
WP6 Extent of stakeholder Involve diverse groups of stakeholders with widely
involvement and level of varying needs.
conflicting requirements
WP7 Interdependence Are high level problems including many component
parts or sub-problems.
EP1 Consequences Have significant consequences in a range of contexts.
EP2 128
Judgement Require judgement in decision making
 Problem Oriented, Team-Based Project Work as a
Learning/Teaching Device

1. Problem-oriented project-organized education deals with

the solution of theoretical problems through the use of any
relevant knowledge, whatever discipline the knowledge
derives from. We are dealing with KNOW WHY (Research
Problems).
2. In design-oriented project work, the students deal with
KNOW HOW problems that can be solved by theories and
knowledge they have acquired in their previous lectures.
(Design Problems).

129
 Complex Engineering Activities (Project based)
Preamble Complex activities means (engineering) activities or
projects that have some or all of the following
characteristics listed below
Range of Diverse resources (people, money, equipment,
resources materials, information and technologies).
Level of Require resolution of significant problems arising
interaction from interactions between wide ranging or
conflicting technical, engineering or other issues.
Innovation Involve creative use of engineering principles and
research-based knowledge in novel ways
Consequences to Have significant consequences in a range of
society and contexts, characterised by difficulty of prediction
the environment and mitigation.
Familiarity Can extend beyond previous experiences by
130
applying principles-based approaches.
 WA – WK – WP Relationships
WA1 – Engineering Knowledge
(Science, Mathematics & Engineering)
(WK1, WK2, WK3, WK4)
to solve
Complex Engineering Problems
WP1 – Depth of Knowledge
required:
Resolved with forefront in-depth
engineering knowledge
(WK3, WK4, WK5, WK6 or WK8)
which allows a fundamentals-based,
first principles analytical approach
WK1 - natural sciences (WA1) (know what)
WK2 - mathematics, numerical analysis,
statistics, computer and information science
(WA1)
WK3 - engineering fundamentals (WA1)
WK4 - engineering specialist knowledge
(WA1)
WK5 - engineering design (know how)
WA3 - Design
WK6 - engineering practice (know how)
WA5 - Modern Tools
WK8 - research literature (know why)
WA4 - Investigation
 to solve
WK1 - natural sciences (WA1)
Complex Engineering Problems
WK2 - mathematics, numerical analysis,
statistics, computer and information science (WA1)

WP1 – Depth of Knowledge WK3 - engineering fundamentals (WA1)

required:
Resolved with forefront in-depth WK4 - engineering specialist knowledge
engineering knowledge (WA1)
(WK3, WK4, WK5, WK6 or WK8)
WK5 - engineering design
which allows a fundamentals-based,
WA3 - Design
first principles analytical approach
WK6 - engineering practice
WP2 Range of conflicting requirements
WA5 - Modern Tools
WP3 Depth of analysis required
WP4 Familiarity of issues WK8 - research literature
WA4 - Investigation
WP5 Extent of applicable codes
WP6 Extent of stakeholder involvement and level
of conflicting requirements Some or all
WP7 Interdependence WP2 – WP7, EP1 & EP2
EP1 Consequences
EP2 Judgement
 to solve
WK1 - natural sciences (WA1)
Complex Engineering Problems
WK2 - mathematics, numerical analysis,
statistics, computer and information science (WA1)

WP1 – Depth of Knowledge WK3 - engineering fundamentals (WA1)

required:
Resolved with forefront in-depth WK4 - engineering specialist knowledge
engineering knowledge (WA1)
(WK3, WK4, WK5, WK6 or WK8)
WK5 - engineering design
which allows a fundamentals-based,
WA3 - Design
first principles analytical approach
WK6 - engineering practice
WP2 Range of conflicting requirements
WA5 - Modern Tools
WP3 Depth of analysis required
WP4 Familiarity of issues WK8 - research literature
WA4 - Investigation
WP5 Extent of applicable codes
WP6 Extent of stakeholder involvement and level
of conflicting requirements
WK7 - engineering in society
WA6 - engineer & society
WP7 Interdependence
WA7 - environment & sustainability
EP1 Consequences WA8 - ethics
EP2 Judgement
Breadth
 WK1 - natural sciences (WA1)

Design Course WK2 - mathematics, numerical analysis,

statistics, computer and information science (WA1)

WP1 – Depth of Knowledge WK3 - engineering fundamentals (WA1)

required: WK4 - engineering specialist knowledge (WA1)
Resolved with forefront in-depth
engineering knowledge WK5 - engineering design
(WK3, WK4, WK5, WK6 or WK8) WA3 - Design
which allows a fundamentals-based, WK6 - engineering practice
first principles analytical approach WA5 - Modern Tools
WP2 Range of conflicting requirements
WK8 - research literature
WP3 Depth of analysis required (WA2)
WA4 - Investigation
WP4 Familiarity of issues
WP5 Extent of applicable codes WK7 - engineering in society
WP6 Extent of stakeholder involvement
WA6 - engineer & society (WK7)
and level of conflicting WA7 - environment & sustainability (WK7)
requirements WK7 (WA6, WA7, WA8 – ethics (WK7)
WA8) WA2 - Problem Analysis (WK 1-4)
WP7 Interdependence WA9 - Individual and Team Work
WA10 - Communication
EP1 Consequences WA11 - Project Management and Finance
EP2 Judgement WA12 - Life-long Learning
 Example 1: Complex Problem Solving
• Two villages in Timbuktu are separated from each other
by a valley, at its deepest section about 30 metres.
• The valley is dry all the year around, except for the four
months, from October to December each year, where
torrential rainfall can flood major parts of the valley to a
depth of over 12 metres in some site.
• The soil is generally lateritic with firm bedrock
underneath. A bridge connecting the two villages is in a
state of disrepair and has to be replaced.
• Write a project brief on how would you approach to
design for the replacement bridge.
• You are limited to the use of locally available building
materials.
• Heavy equipment is not available for the construction. 135
Aspects
• Economics
• Social
• Environment
• Ethics
• Management
• Technology
• Analysis
• Evaluation
136
Thinking
• Site condition
• Weather
• Available technology
• Building materials
• Design
• Costing
• Scheduling
137
 Solutions?
• Problem solving skills
• Formulate the problem
• Literature
• Experiment?

138
 Assessment
• Report – style and content (flow)
• Display – attractive ?
• Viva / Articulation
• Teamwork
• Management – scheduling

139
Example 2: Complex Problem Solving
River
Spring

Fissured Rocks
Sandy soil

Clayey soil

Igneous rock

140
How does complexity relates to
curriculum?
• General Subjects
• Industrial Placement
• Core & Specialist (Engineering) Subjects –
Complex Problem Solving
• Elective Subjects – Complex Problem Solving
• Design Project – Complex Engineering
Activities
• Final Year Project – Complex Problem Solving 141
Closing Remarks

142
Thank You
Appendix
144
 Complex Problem Solving (CPS)
• Dynamic, because early actions determine the
environment in which subsequent decision must
be made, and features of the task environment
may change independently of the solver’s actions;
• Time- dependent, because decisions must be
made at the correct moment in relation to
environmental demands;
• Complex, in the sense that most variables are not
related to each other in a one-to-one manner
145
 Microworld CPS Model
• The problem requires not one decision, but a
long series, in which early decisions condition
later ones.
• For a task that is changing continuously, the
same action can be definitive at moment t1
and useless at moment t2.
• Include novel solutions to an old dilemma in
general science (external validity vs.
experimental control)
146
 Expert-novice CPS Model
• Expert-novice approach most of the time
produces conclusions that are crystal-clear.
• It almost guarantees statistically significant
results, because the groups compared (expert
and novices) are very different and tend to
perform very differently when confronted with
similar experimental situations (Sternberg
1995).

147
Naturalistic decision making (NDM)
• Naturalistic decision making (NDM) (e.g.,
Zsambok and Klein 1997, Salas and Klein
2001)
• ‘real-world’ task
• Example interviewing firefighters after
putting out a fire or a surgeon after she has
decided in real time what to do with a
patient.

148
 Dynamic decision making DDM
• Dynamic decision making (DDM) (Brehmer
1992, Sterman 1994).
• Discrete dynamic decision tasks that change
only when the participant introduces a new
set of inputs.
• Variables like time pressure have been
successfully integrated in models like
Busemeyer and Townsend’s (1993) decision
field theory
149
 Implicit learning in system control
• This tradition has used tasks like the sugar
factory (Berry and Broadbent 1984) or the
transportation task (Broadbent et al. 1986), that
are governed by comparatively simple
equations.
• The theorization and computational modeling in
this branch of CPS are extremely rich. Models
are based on exemplar learning, rule learning,
and both (e.g., Dienes and Fahey 1995, Gibson
et al. 1997, Lebiere et al. 1998).
150
European complex problem solving (CPS)

• Initiated by Dörner (Dörner and Scholkopf

1991, Dörner and Wearing 1995)
• A large number of tasks that have been
considered complex problem solving are
nowadays affordable for theory development
and computer modeling (e.g. Putz-Osterloh
1993, Vollmeyer et al. 1996, Burns and
Vollmeyer 2002, Schoppek 2002)
• Transport real-life complexity to the lab in a
way that can be partly controlled
151
 Time related
• Time variant – time invariant (dynamic vs.
static systems)
• Continuous time – discrete time.
• Degree of time pressure – decision has to be
made quickly

152
 Variable related
• Number and type (discrete/continuous) of
variables
• Number and pattern of relationships
between variables
• Non-Linear - Linear

153
 System behaviour related
• Opaque - transparent.
• Stochastic - deterministic
• Delayed feedback - immediate feedback.

154
 Delivery
• Knowledge-lean vs. knowledge-intensive
• Skill based vs planning based (reactive vs
predictive
• Learning vs. no learning during problem
solving
• Understanding-based vs. search-based
problems
• Ill-defined vs. well-defined
155
 Conclusion
• Problem solving has been traditionally a
task-centered field. VanLehn (1989) think
that ‘task’ and ‘problem’ are virtually
synonymous.

156
157
The author would like to thank the contributors of the clip arts
used in this presentation

158