Anda di halaman 1dari 61

TUGAS ASSESMENT PEMBELAJARAN PTK

REVIEW ARTIKEL PENELITIAN

LANUIHSAN
162052007001

PROGRAM PASCASARJANA
JURUSAN PENDIDIKAN TEKNOLOGI KEJURUAN
UNIVERSITAS NEGERI MAKASSAR
2017
REVIEW JURNAL EVALUASI PENDIDIKAN

1. JURNAL OF INDIA
Judul Design of an assessment system for collaborative problem solving in
STEM Education
Penulis Kuen-Yi Lin, Kuang-Chao Yu, Hsien-Sheng Hsiao, Yih-Hsien Chu,
Yu-Shan Chang, Yu-Hung Chien
Tahun Terbit 2015
Penerbit Journal Computer Education 2(3):301-322
DOI 10.1007/S40692-015-0038-x
Reviewer Lanuihsan, S.Pd, Gr
Tanggal Review 9 Juni 2016
HASIL REVIEW
Tujuan Penelitian ini mengembangkan sistem penilaian untuk mengevaluasi SMP
kemampuan pemecahan masalah kolaboratif siswa (CPS) dalam konteks
sains, pendidikan teknologi, teknik, dan matematika (STEM).
Subyek Penelitian Pelajar SMP di Taiwan sebanyak 222 orang
Usia 13-15 tahun
Metode Analisis statistik
Definisi Operasional Untuk menilai kemampuan pemecahan masalah kolaboratif siswa, tiga
Variabel Independen besar keterampilan pemecahan masalah kolaboratif diadopsi di sini, dan
terkait-silang dengan empat proses pemecahan masalah utama untuk
membentuk matriks keterampilan pemecahan masalah kolaboratif (lihat
OECD 2013). Tiga keterampilan dan empat proses tercantum dalam Tabel
1. Untuk memiliki keterampilan membangun dan mempertahankan
pemahaman bersama, siswa harus memiliki kemampuan untuk
mengidentifikasi pengetahuan timbal balik, untuk mengidentifikasi
perspektif dari agen lain, dan untuk membangun visi bersama dari masalah
negara (Dillenbourg dan Traum 2006; OECD 2013). Untuk keterampilan
mengambil tindakan yang tepat untuk memecahkan masalah, siswa harus
dapat memahami kendala masalah, membuat sasaran tim menuju solusi,
mengambil tindakan pada tugas, dan memantau hasil dalam kaitannya
dengan grup dan sasaran-sasaran masalah ( OECD 2013). Untuk
membangun dan mempertahankan organisasi kelompok, siswa harus dapat
memahami peran mereka sendiri dan peran dari agen lain, mengikuti aturan
keterlibatan untuk peran mereka, memantau organisasi kelompok, dan
memfasilitasi perubahan yang dibutuhkan untuk menangani hambatan
terhadap masalah (OECD 2013)
Cara & Alat Ukur Sebanyak 222 siswa SMP Taiwan berusia 13–15 tahun berpartisipasi dalam
Variabel pelajaran ini. Mereka berasal dari sembilan kelas di dua sekolah yang
berbeda, dan menghabiskan tiga minggu menyelesaikan delapan tugas
dalam sistem berbasis web CPS. Pada minggu pertama, para guru
menjelaskan pengoperasian sistem penilaian, dan para siswa diharapkan
untuk menyelesaikan dua tugas. Pada minggu kedua dan ketiga, siswa
harus menyelesaikan tiga tugas per minggu. Sehubungan dengan format
sistem CPS berbasis web, jumlah agen komputer yang berinteraksi dengan
siswa bervariasi. Siswa bekerja dengan satu agen (awalnya selama 4
modul) dan kemudian dua agen (selanjutnya, untuk 4 modul lainnya).
Hasil Penelitian Menurut data dan analisis yang disajikan di atas, sistem penilaian yang
dikembangkan dalam penelitian ini terdiri dari item dengan diskriminasi
yang sulit diterima dan memuaskan, dan validitas konten dan validitas
terkait kriteria juga dinilai baik. Secara keseluruhan, sistem penilaian untuk
pemecahan masalah kolaboratif dalam pendidikan STEM yang disajikan
oleh penelitian ini efektif dalam menilai keterampilan pemecahan masalah
kolaboratif siswa. Meskipun mayoritas interkorelasi yang digunakan
sebagai bukti konsistensi hasil di delapan tugas dalam keterampilan CPS
yang diberikan secara keseluruhan sangat signifikan secara statistik dan
juga dalam banyak kasus nilai korelasi yang cukup tinggi menjadi dianggap
kuat.
2. JURNAL OF CANADIAN
Judul Design, implementation, and evaluation of an ePortfolio approach to
support Faculty development in vocational education
Penulis Annemarieke Hoekstra, Jocelyn R. Crocker
Tahun Terbit 2015
Penerbit Journal Studies in Educational Evaluation 562 (2015):xxx-xxx
DOI 10.1016/j.stueduc.2015.03.007 0191-491x
Elsevier
Reviewer Lanuihsan, S.Pd, Gr
Tanggal Review 9 Juni 2016
HASIL REVIEW
Tujuan Makalah ini menjelaskan desain, implementasi, dan evaluasi dari inisiatif
pengembangan fakultas yang disebut pendekatan ePortfolio, yang
dirancang untuk mendorong pembelajaran yang terjadi secara alami di
tempat kerja.
Subyek Penelitian Lokasi penelitian adalah institut kejuruan pasca-sekolah menengah Kanada
Barat yang menawarkan pelatihan pemagangan, program diploma dua
tahun, program gelar, dan sertifikat profesional di bidang perdagangan,
konstruksi bangunan dan desain, teknik dan ilmu terapan, kesehatan dan
keselamatan, TI dan elektronik, bisnis dan administrasi, serta perhotelan
dan seni kuliner. Lembaga ini memiliki sekitar 1100 fakultas dan 80.000
mahasiswa penuh waktu dan paruh waktu
Metode Metode survey dan wawancara
Definisi Operasional Berdasarkan perbandingan ini, kami membuat kategori di seluruh peserta
Variabel Independen serta hitungan berapa banyak peserta yang masuk dalam kategori ini.
Misalnya, adopsi tema pendekatan ePortfolio termasuk tiga kategori: (1)
tidak ada portofolio yang dibuat, (2) halaman portofolio dengan beberapa
dokumen yang diunggah, dan (3) ePortfolio yang lebih terperinci.
Cara & Alat Ukur Pendekatan ePortfolio diimplementasikan dalam dua pilot.
Variabel Setelah masing-masing percontohan, baik survei dan wawancara data
dikumpulkan untuk menentukan adopsi, persepsi fakultas, dan persepsi
fakultas tentang dampak pendekatan ePortfolio
Pendekatan ePortfolio diujicobakan dalam dua fase. Pertama-tama, para
dekan dari lima fakultas di lembaga diminta untuk mencari kursi-kursi
departemen yang bersedia untuk mencoba pendekatan baru dalam (salah
satu dari mereka) program-program yang lebih kecil. Yang kedua, para
dekan diminta untuk mencari kursi departemen dari dua hingga tiga
program yang lebih besar. Yang pertama pilot dilakukan dari September –
Desember 2011 dengan 32 peserta (termasuk 5 kursi dan 27 anggota
fakultas) dari lima program dalam perdagangan, ilmu kesehatan, bisnis, dan
media dan desain. Percobaan kedua dilakukan dari September – Desember
2012 dengan 75 peserta dari 12 program di 9 departemen 2 termasuk
program dalam perdagangan, teknik, ilmu terapan, dan kesehatan. Satu
program berpartisipasi dalam kedua pilot. Jumlah total departemen yang
berpartisipasi adalah 13: empat di pilot pertama, delapan di bagian kedua,
dan satu yang berpartisipasi dalam keduanya.
Evaluasi difokuskan pada dampak pendekatan ePortfolio dan
pelaksanaannya, termasuk rincian mengenai sesi orientasi yang diadakan,
survei evaluasi siswa baru, survei baru yang digunakan oleh kursi untuk
mengumpulkan umpan balik mengenai kinerja mereka, tingkat respons
siswa, dan alat umpan balik.
Software pengolah data menggunakan Mahara version 1.4.6
Hasil Penelitian Tabulasi silang yang serupa menunjukkan bahwa semua responden
yang mengumpulkan umpan balik pada aspek-aspek tertentu dari pekerjaan
mereka juga menunjukkan bahwa aspek-aspek tersebut sebenarnya
merupakan bagian dari pekerjaan mereka.
Tabel 2 menunjukkan bahwa mayoritas peserta (69%) mengumpulkan
lebih banyak dan / atau jenis umpan balik yang berbeda dalam pilot
daripada sebelum pilot. Dari lima komentar untuk pertanyaan ini, tiga
menunjukkan perbedaannya adalah pengumpulan umpan balik rekan.
Komentar lain menyatakan '‘tidak ada perbedaan.’ ’Hanya 31% yang
melaporkan bahwa mereka tidak mengumpulkan umpan balik yang lebih
banyak atau berbeda dari sebelumnya.
Wawancara mengkonfirmasi temuan survei, bahwa mayoritas (9 dari
12) orang yang diwawancarai mengumpulkan umpan balik. Tujuh dari 12
orang yang diwawancara menciptakan portofolio, meskipun hanya dua
yang menciptakan portofolio yang lebih rumit. Dari wawancara dengan
fakultas dan kursi, kami belajar bahwa setiap departemen mengadopsi
pendekatan ePortfolio ke tingkat yang berbeda selama uji coba pertama.
Dua kursi tidak mendiskusikan dan tidak berencana membahas portofolio
dengan anggota fakultas yang melapor kepada mereka. Tiga kursi
mengadakan diskusi dengan beberapa anggota fakultas dan berencana
untuk duduk dengan anggota fakultas yang tersisa
3. JURNAL OF AUSTRALIA
Judul Two dimensional efficiency measurements in vocational education
Penulis Peter Fieger, Renato Andrin Villano, John Rice and Ray Cooksey
Tahun Terbit 2015
Penerbit International Journal of Productivity and Performance Management Vol.66
No. 2. 2017 pp.196-215
DOI 10.1108/IJPPM-09-2015-0139
Reviewer Lanuihsan, S.Pd, Gr
Tanggal Review 9 Juni 2016
HASIL REVIEW
Tujuan Tulisan ini berusaha memberikan analisis seperti itu melalui penerapan
pendekatan integrative untuk pengukuran efisiensi untuk mengeksplorasi
aktivitas dan efisiensi pengiriman (beban mengajar) dan efisiensi hasil
(hasil kerja
Subyek Penelitian sektor pendidikan dan pelatihan kejuruan (VET) Australia
Metode Dalam penelitian ini penulis menggunakan data yang berkaitan dengan
demografi kohort siswa, karakteristik kelembagaan dan data hasil
pendidikan, sementara menggunakan analisis stokastik, untuk
mengembangkan dua langkah dan model efisiensi yang berbeda.
Definisi Operasional Dalam studi ini, kami akan menetapkan dua jenis efisiensi yang berbeda di
Variabel Independen sektor TAFE Australia dan menggunakan analisis parametrik stokastik
stikastik (SFA) untuk menentukan efisiensi masing-masing lembaga.
Model empiris pertama dirancang untuk memperkirakan efisiensi dalam
transformasi sumber daya keuangan menjadi jam mengajar (dari sini
disebut "model beban mengajar"). Model kedua memperkirakan efisiensi
transformasi sumber daya pengajaran menjadi hasil pasca-studi, yaitu,
tingkat kerja lulusan TAFE (dari sini disebut "model hasil pekerjaan")
Cara & Alat Ukur Untuk memperkirakan efisiensi "jam pengajaran" dan efisiensi "hasil
Variabel pekerjaan" berdasarkan metodologi stochastic frontier yang dikembangkan
oleh Aigner et al. (1977). Kontribusi utama dari para penulis ini adalah
pengenalan pendekatan baru untuk spesifikasi istilah kesalahan, yaitu,
pemisahannya ke dalam istilah "noise" normal dan satu sisi istilah
inefisiensi. Fungsi produksi frontier stokastik merupakan perluasan fungsi
Cobb-Douglas (1928) klasik
Ini akan mencakup grafik dari kedua komponen efisiensi dan analisis
klaster untuk menentukan "kelompok efisiensi". Akhirnya, mereka akan
menggunakan analisis diskriminan kanonik dengan tujuan mengembangkan
tipologi lembaga yang efisien
Hasil Penelitian di kedua model yang telah diamati beberapa ketidakefisiensian yang jelas.
Inefisiensi ini terutama terkait dengan tingkat keterpencilan dan
karakteristik siswa, yang keduanya dapat dilihat sebagai eksogen bagi
TAFE itu sendiri. Sebagai contoh, lembaga TAFE yang paling tidak efisien
lebih mungkin ditemukan di lokasi terpencil, memiliki persentase laki-laki
yang lebih tinggi, dan proporsi yang lebih besar dari orang-orang dari latar
belakang yang tidak berbahasa Inggris.
inefisiensi ini didorong oleh kombinasi faktor-faktor yang saling terkait,
termasuk lokasi geografis, infrastruktur yang tersedia dan tidak adanya
jenjang profesi lulusan
J. Comput. Educ. (2015) 2(3):301–322
DOI 10.1007/s40692-015-0038-x

Design of an assessment system for collaborative


problem solving in STEM education

Kuen-Yi Lin1 • Kuang-Chao Yu1 • Hsien-Sheng Hsiao1 •

Yih-Hsien Chu1 • Yu-Shan Chang1 • Yu-Hung Chien1

Received: 30 November 2014 / Revised: 15 June 2015 / Accepted: 18 June 2015 /


Published online: 11 July 2015
Ó Beijing Normal University 2015

Abstract This study developed an assessment system for evaluating junior high
students’ collaborative problem-solving skills (CPS) in the context of science,
technology, engineering, and mathematics (STEM) education. The main theoretical
basis for designing the CPS learning goals of this assessment system is the matrix of
collaborative problem solving proposed by the Organisation for Economic Co-
operation Development (OECD). In this assessment system, there are eight
assessment modules in STEM education developed by four groups of researchers
who participated in this study. The modules were delivered as a web-based learning
platform, where the computer was programmed to act as the agent of collaboration
(rather than human-to-human collaboration), thus making this exploratory study
more unique. The eight modules included problem tasks such as designing shelves,
using a microwave oven, construct a house, etc. In order to ensure the validity of the
assessment system, including the eight assessment modules in STEM education,
three specialists were invited to examine the eight modules’ content validity.
Additionally, in order to ensure criterion-related validity of the effectiveness of the
modules, they were tested among 222 Taiwanese junior high students. Discrimi-
nation and difficulty indices were obtained for student CPS performance on each of
the eight modules. Furthermore, intercorrelations were obtained for the students’
performance on each of the eight assessment modules in relation to overall and each
of three collaborative problem-solving skills, i.e., (1) Establishing and maintaining
shared understandings, (2) Taking appropriate action to solve the problem, and (3)
Establishing and maintaining team organization. The following conclusions are
drawn: (1) there was an evidence that the assessment system items had acceptable
difficulty and satisfactory discrimination; (2) the intercorrelations of the students’

& Kuen-Yi Lin


linkuenyi@ntnu.edu.tw
1
Department of Technology Application and Human Resource Development, National Taiwan
Normal University, 162, Heping East Road Sect. 1, Taipei, Taiwan

123
302 J. Comput. Educ. (2015) 2(3):301–322

performance on the eight tasks in relation to overall and each of the three collab-
orative problem-solving skill dimensions were sufficiently significant, in most cases,
to conclude that the eight tasks are consistently producing the same performance
outcomes for overall and each of the three skills; that is, this assessment system was
deemed effective for evaluating junior high students’ collaborative problem-solving
skills in STEM education; and (3) with respect to some of the challenges in eval-
uating students’ collaborative problem-solving skills, one of the most demanding for
us was designing a series of tasks and questions in evaluating students’ performance
in establishing and maintaining team organization. Hence, we recommend that
further research on students’ performance in establishing and maintaining team
organization should be the focus of future studies.

Keywords Assessment system  Collaborative problem solving  Computer agent 


Junior high students  STEM education

Introduction

Background literature

Both in the educational environment at various levels and in the workplace,


traditional models of skill development have been changing; one area still ripe for
reform is the development of skills related to teamwork and problem solving. In this
regard, collaborative problem solving (CPS) is a more promising area of
investigation than individual problem solving, since the former includes, among
others, the following potential advantages: effective division of labor; the ability to
draw on multiple people’s knowledge, viewpoints, and experiences; and the
possibility to improve solutions through mutual feedback (Organisation for
Economic Co-operation and Development [OECD] 2013). Further, with the recent
developments in web technologies for information sharing and communication,
collaborative problem solving online is likely to become a more important means of
delivery in many fields. Davis et al. (2011) identified virtual collaboration as one of
the ten key skills for future workplaces; and 94 % of 921 industries in North
America and Europe utilize or plan to utilize web-based technologies, including
e-mail, videoconferencing, instant messaging, and others, to facilitate collaborative
problem solving (OECD 2013). Therefore, facility with web-based CPS skills will
be an important advantage for those applying for a job in future workplaces.
More and more countries, including for example Singapore and Israel, are
becoming concerned about fostering key skills for the twenty-first century and are in
the process of conducting curriculum reform to address this issue; collaborative
problem-solving skills have been one of their major concerns (Darling-Hammond
2011). Greene (2011) believes that the emphasis for the past 40 or 50 years on
teaching students how to solve questions rooted in a particular subject matter
domain is mistaken, and that the focus should not be on reaching a solution per se,
but on the process of problem solving, using a collaborative approach. This, Greene
says, will lead to various changes in the education system; for example, the

123
J. Comput. Educ. (2015) 2(3):301–322 303

traditional role of subject matter will change, and as far as pedagogy, the most
important thing will be not how teachers ‘‘teach’’ their students, but how they
‘‘communicate with’’ their students and try to help them solve problems
collaboratively.
Since the collaborative problem-solving skills are important to our students
(Darling-Hammond 2011; Greene 2011), how to explore students’ performance in
collaborative problem-solving skills is an important issue. With respect to the
development of a system to assess collaborative problem-solving skills, two
important issues are worthy of further exploration as follows: (1) What are the
merits of using a computer agent (rather than a human agent)? Many scholars (e.g.,
Johnson and Valente 2008; Rosen and Tager 2013; VanLehn et al. 2007) used a
computer agent in training or evaluating, but further exploration is needed on the
best way for using a computer agent in assessing students’ collaborative problem-
solving skills. (2) There is a need for a sound theoretical basis for the design of the
collaborative tasks. The OECD developed a matrix of collaborative problem solving
(OECD 2013) that is promising and we apply it here, but recent evidence is that it
has been difficult for researchers in designing collaborative tasks and assessing
students’ performances using the matrix of collaborative problem solving. For
example, Kuo (2014) developed a collaborative problem-solving system, but the
tasks were simple, and there were difficulties in assessing the students’ performance
based on skills derived from the matrix of collaborative problem solving. That is, a
more effective design of the collaborative tasks is needed for further exploration. In
addition to these cited challenges in assessing collaborative problem-solving skills,
Kuo and Wu (2013) also found that only 19 out of 66 computer-based assessments
took advantage of dynamic and interactive media for item presentations. Therefore,
in light of these issues and challenges, our goal was to explore more fully on how to
design and assess a good collaborative task, while using the computer as the agent
during collaboration, including a dynamic and interactive environment, to assess
students’ collaborative problem-solving skills.

Rationale for the study

In order to construct a dynamic and interactive environment, this study included the
design of a series of collaborative problem-solving tasks that are related to junior
high school students’ daily life experiences, instead of adopting traditional subject
matter-based problems. By incorporating tasks more grounded in daily life
experiences, we hope to increase the motivation and engagement of the students
during the CPS tasks. In order to solve these real-life problems collaboratively,
students have to use science, technology, engineering, and mathematics (STEM)
knowledge, as recommended in modern STEM education educational systems for
the twenty-first century (Bybee 2013). To sum up, this study develops and applies
an assessment system for evaluating students’ collaborative problem-solving skills
in STEM education using a web-based delivery system and the computer as the
collaborative agent. The study employs the collaborative problem-solving frame-
work proposed by the Organisation for Economic Co-operation and Development
(2013) as a theoretical basis for the design of the CPS tasks and CPS skills included

123
304 J. Comput. Educ. (2015) 2(3):301–322

in the assessment system, and we have analyzed its effectiveness when applied
among 222 junior high students in Taiwan. The assessment system has eight
modules each based on a different situation in daily life; and a computer agent (or
two agents) that interact with students (simulating human kinds of verbal
interactions) in order to assess their collaborative problem-solving skills.

Research questions

Two research questions guided the study: (1) Is there evidence that the eight
modules have good validity in assessing junior high school students’ collaborative
problem-solving skills? (2) What are some of the difficulties in developing an
assessment system for collaborative problem solving in STEM education?

Related work

Collaborative problem solving

Scifres et al. (1998) explored the effectiveness of different web-based technologies


for the development of students’ collaborative problem-solving skills; the results
showed that although participants showed greater skill gain than with traditional
learning, participants were not satisfied with their team members and that most felt
they each made more effort than the other team members. Findings such as this led
other studies to conclude that supporting collaborative learning activities and
procedures requires specialized tools instead of just standard, web-based technolo-
gies (Harasim 1999; Isenhour et al. 2000).
In recent years, more and more studies have tried to use computer-supported
approaches to help students’ learning, in particular their attention and their attitudes
toward collaboration (Roschelle et al. 2010; Warwick et al. 2010). For example,
some studies have focused on developing systems to solve open assignments by
supporting collaboration (Looi et al. 2010; Nussbaum et al. 2009), while others have
adopted interactive whiteboard technology to try to facilitate student interaction and
collaboration in order to help them solve problems actively (Warwick et al. 2010;
Wood and Ashfield 2007). Although these approaches have had some good results,
Alvarez et al. (2013) argue that further experiments and assessments are still
needed. In a similar spirit, the work of Ding (2009) tried to visualize the sequential
process of knowledge elaboration in computer-supported collaborative problem
solving, and the results showed that there were three different collaboration patterns
that emerged in terms of joint and individual knowledge elaboration. Because the
number of participants in that study was limited (only six), at least two issues are
likely to be important for further exploration: (1) more patterns or mechanisms may
have been revealed if more participants were included, and (2) with a stronger data
set, the correlation of the elaboration patterns with the learning performance could
have been explored. Kuo et al. (2012) utilized a cognitive apprenticeship approach
to facilitate students’ collaborative problem solving, and Witken’s field dependence
theory to explore their performance. Kuo et al.’s (2012) results showed that learners

123
J. Comput. Educ. (2015) 2(3):301–322 305

with a field independence cognitive style had better learning performance in a


collaborative cognitive apprenticeship. Kuo (2014) also developed a collaborative
problem-solving system in assessing junior high students’ collaborative problem-
solving skills, but the tasks in that system were focused on students’ school life, e.g.,
students had to make an exercise plan in their physical education. However, students
were not able to explore their performances in relation to the matrix of collaborative
problem-solving skills because the design of the tasks was too simple in Kuo’s
(2014) collaborative problem-solving system.
Based on the insights from these previous studies, and some of the needs they
revealed for improved designs, our study aimed to develop a web-based collaborative
problem-solving system that is more clearly situated in theory based on the OECD
matrix, and includes a sufficient range of tasks, with a much larger number of
participants, to potentially provide a more robust assessment of the web-based CPS
learning experience. The use of the computer as agent adds an additional novel
component that we hope will provide further insights into the potential for this kind of
computer-based affordance in helping students learn and reflect upon their CPS skills.

Computer agent

A computer agent is a computer-simulated participant with the ability to propose


goals, execute assignments, communicate messages, respond to other participants’
messages, detect and adapt to environments, and to learn (Franklin and Graesser
1996). There are many systems using computer agents for training or evaluating,
across fields such as using computer agents to find solutions collaboratively (VanLehn
et al. 2007); developing a reading, writing and communicating system (McNamara
et al. 2007; Johnson and Valente 2008); developing an inferential problem-solving
system (Azevedo et al. 2010; Biswas et al. 2010; Cai et al. 2011); and so on.
However, additional insights about how to apply computer agents in the design of
an inferential problem-solving system to assess and develop students’ collaborative
problem-solving skills is a topic worthy of further investigation, especially given the
limited foci of some current approaches that lacked computer agents. For example,
Azevedo et al. (2010) measured learners’ cognition and meta-cognition in the
learning process using multimedia, and Biswas et al. (2010) used a social interaction
approach to measure skill at self-regulation of learning. Cai et al. (2011) focused on
integrating the assessment of users’ input into an intelligent teaching system. None
of these previous studies applied computer agents in web-based collaborative
problem solving, per se, and the design of a dynamic and interactive environment
for evaluating students’ collaborative problem-solving skills by applying computer
agents that meet Franklin and Graesser’s (1996) criteria is a crucial matter.
In addition, the effects of different collaborative approaches (e.g., learner–
learner, or learner–computer agent) on students’ performance in collaborative
problem solving are also worthy of exploration. Rosen and Tager (2013) compared
the effects of different collaborative approaches for students’ collaborative problem-
solving skills, and found that students had somewhat better performance in
collaboration with computer agents than their collaborative interactions with other
learners, but the difference was not significant. To sum up, collaboration between

123
306 J. Comput. Educ. (2015) 2(3):301–322

learners and computer agents has been shown to be useful for developing students’
collaborative problem-solving skills, but more in-depth studies are needed.
Consequently, this study utilizes a computer agent method instead of a learner–
learner collaborative method. Moreover, this study incorporated a new assessment
system for collaborative problem solving in STEM education that encouraged
students to engage with practical problems during the CPS that are relevant to their
daily life; and this is an additional difference compared to other collaborative
problem-solving systems.

Theoretical framework

Web-based collaborative problem solving

Collaborative problem-solving skill is the capacity of an individual to effectively


engage in a problem-solving process wherein two or more agents are working
together by sharing the understanding and effort required to come to a solution, and
integrating their knowledge, skills, and efforts to reach that solution (OECD 2013).
As for web-based collaborative problem solving, the core concept is of course the
effective use of web-based technologies in collaborative problem solving. However,
it has been demonstrated that normal web-based tools, such as e-mail, videocon-
ferencing, newsgroups, and so on, are inappropriate for use in learning, since the
original design of these tools is not for educational purposes (Harasim 1999).
Therefore, more and more researchers have come to believe that a successful
web-based, collaborative problem-solving assessment system must consist of at
least two elements: normal web-based tools, such as the planning of group space
and personal space on the Web, and tools that specifically support learning activities
and procedures, such as a multimedia interface that integrates learners’ ideas into
solutions (Harasim 1999; Isenhour et al. 2000).
According to the definition of collaborative problem solving proposed by the
OECD (2013), the most important issue is the design of the computer agent; this is
also reflected in the principles set forth by Franklin and Graesser (1996). The
computer agent should be utilizable in different situations, including tutoring,
collaborative learning, knowledge construction, and other ones; and should be
included in the collaborative problem-solving skill assessment system (OECD
2013). This study included a computer agent in eight assessment modules in STEM
education. Students must collaborate with the computer agent in establishing and
maintaining shared understanding, taking appropriate action to solve the problem,
and establishing and maintaining team organization. There were no human–human
collaborative interactions. In order to more effectively realize the goal of ensuring
collaboration of students and computer agents in an authentic STEM learning
experience, STEM specialists were invited to participate in developing the scripts
for the eight assessment modules used in this study. That is, within this design for a
web-based CPS system, the computer agents communicate with students using the
scripts composed by the STEM specialists, and the way students respond to the

123
J. Comput. Educ. (2015) 2(3):301–322 307

scripted prompts from the computer agent is how we assess the effectiveness of the
CPS system in promoting students’ learning of collaborative problem-solving skills.

Matrix of collaborative problem-solving skills

In order to assess students’ collaborative problem-solving skills, three major


collaborative problem-solving skills are adopted here, and cross-related with four
major problem-solving processes to form a matrix of collaborative problem-solving
skills (see OECD 2013). The three skills and four processes are listed in Table 1.
To possess the skill of establishing and maintaining shared understanding,
students must have the ability to identify mutual knowledge, to identify the
perspectives of other agents, and to establish a shared vision of the problem state
(Dillenbourg and Traum 2006; OECD 2013). For the skill of taking appropriate
action to solve the problem, students must be able to understand the problem
constraints, create team goals toward a solution, take action on the tasks, and
monitor the results in relation to the group and to the problem goals (OECD 2013).
For establishing and maintaining group organization, students must be able to
understand their own role and the roles of the other agents, follow the rules of
engagement for their role, monitor group organization, and facilitate changes
needed to handle obstacles to the problem (OECD 2013).
Adopting these definitions of the three skills, this study designed a collaborative
problem-solving system for use in assessing students’ collaborative problem-solving

Table 1 Matrix of collaborative problem solving including the specific CPS skills (A1–D3) that were
assessed
Skills process (1) Establishing and (2) Taking appropriate (3) Establishing and
maintaining shared action to solve the problem maintaining team
understanding organization

(A) Exploring (A1) Discovering (A2) Discovering the types (A3) Understanding roles
and perspectives and of collaborative interaction to solve the problem
Understanding abilities of team needed to solve the
members problem, along with the
goals
(B) Representing (B1) Building a shared (B2) Identifying and (B3) Describing roles and
and representation and describing tasks to be team organization
Formulating negotiating the completed (communication
meaning of the protocols/rules of
problem engagement)
(C) Planning and (C1) Communicating (C2) Enacting plans (C3) Following rules of
Executing with team members engagement, (e.g.,
about the actions to prompting other team
be/being performed members to perform their
tasks)
(D) Monitoring (D1) Monitoring and (D2) Monitoring results of (D3) Monitoring,
and Reflecting repairing the shared actions and evaluating providing feedback, and
understanding success in solving the adapting team
problem organization and roles

Source OECD (2013, p. 11)

123
308 J. Comput. Educ. (2015) 2(3):301–322

skills. As for the approach used in assessing students’ collaborative problem-solving


skills, see the detailed explanation in the Evaluation section introduction.

CPS modules for applying STEM content

Because some previous collaborative problem-solving studies included limited use


of skills relevant to students’ school life (e.g., Kuo 2014), and it is becoming
increasingly clear that real-life situations help to promote engagement and
motivation, we have developed a series of eight CPS modules in our assessment
of students’ realistic collaborative problem-solving skills. In addition, we have
included STEM themes as a major part of the eight modules. Raju and Clayson
(2010) analyzed two national reports and proposed that the development of STEM
talents is a major trend in the United States. However, the traditional models of
education in these fields are inadequate, because they focused on knowledge
transmission instead of offering different learning opportunities that engage the
students in solving daily problems by applying what they learned in the school
(Johnson 1989). Consequently, many researchers recommend that the learning
scenarios should be focused on solving practical problems in the students’ daily life
to more effectively assess authentic STEM collaborative problem-solving skills
(Blackwell and Henkin 1989; Daugherty and Wicklein 1993; Martin-Kniep et al.
1995). Therefore, when considering the best ways to develop and assess students’
web-based collaborative problem-solving skills, the inclusion of content relevant to
students’ daily life, should be a matter of first priority.
In order to explore and improve students’ problem-solving skills, many
researchers have utilized a variety of different instructional methods (Barak and
Mesika 2007; Hong et al. 2013; Kirschner et al. 2011). However, most of these
studies have focused on developing students’ skill at solving structured and subject-
based problems with designated solutions in terms of the theoretical learning
content of scientific fields such as physics, biology, and chemistry. As a result,
students are not given an opportunity to collaboratively develop their problem-
solving skills, while engaged with practical, daily life problems that typically have
no predefined solutions (Dixon and Brown 2012; Sternberg 2001). It is self-evident,
that the problem-solving skills students need in order to arrive at solutions
appropriate to daily life are different from those they generally need in solving less
practically situated school-based problems; and teachers need to adopt different
strategies to instill these skills (Johnson et al. 2011).

Implementation

System framework

The design of the assessment system is centered on eight assessment modules for
applying STEM content relevant in daily life for the participating Taiwanese
students (see Fig. 1). The modules include eight practical problem-solving tasks:
building shelves, using a microwave oven, defusing a bomb, interior design, two

123
J. Comput. Educ. (2015) 2(3):301–322 309

An Assessment System for Collaborative Problem-Solving in STEM Education

Fig. 1 Assessment modules including eight STEM tasks largely centered on practical type problems:
Building Shelves, Using a Microwave Oven, Defusing a Bomb, Interior Design, Two days’ Travel in
Kaohsiung, Buying a Cell Phone, Building a House, and Designing a Desk Tidy

days’ travel in Kaohsiung, buying a cell phone, building a house, and designing a
desk tidy. With respect to computer agents, students were required to work with the
agent(s) to solve problems collaboratively; they communicate with the computer
agents by responding using the keyboard to insert their responses, or selecting an
item onscreen (see Fig. 2). Once the students input what they want to say, the
computer agent searches the item bank and responds to the students (see Fig. 3). In
order to evaluate students’ collaborative problem-solving skills, many possible
reactions to the student’s input during the problem-solving process are included in
this system, these include one where wrong guidance is offered by the computer
agent, in order to evaluate students’ CPS performance in relation to these
challenging situations.

Web-based CPS system planning

The design elements of task characteristics, problem scenario, medium, and team
composition were considered in the process of developing the CPS system (OECD
2013). With regard to task characteristics, all the tasks in this study were related to
students’ daily life, and the students and the computer agents had to play different
roles and take different responsibilities to solve the problems. With regard to the
problem scenario, as mentioned above, tasks that could plausibly appear in the daily
lives of these Taiwanese students were adopted. As for medium, students were
provided with sufficient information to make decisions by the computer agents as
they worked together, but students had to determine the correctness of the
information offered by the agent. Finally, for team composition, students worked
with either one or two computer agents and were required to play different roles in
the different scenarios.

123
310 J. Comput. Educ. (2015) 2(3):301–322

Time Return to main menu Topic Show previous question

Dialogue Input Items

Fig. 2 Assessment system interface that the student uses to collaboratively interact with computer agents

Client Server (assessment system)

assessment module

Communication module
Data input

Agent
Students Data analysis module Search
Item
Reaction bank
(Return the data) Reaction module Reaction

Fig. 3 Assessment system diagram showing the information flow and interactions of student and
computer agents, including the role of the assessment module and its embedded agent where student input
data are analyzed, and appropriate items are retrieved from the Item bank to provide a reaction to the
students’ input

Evaluation

As mentioned above, in order to assess students’ collaborative problem-solving


skills, this study employed the collaborative problem-solving matrix proposed by the
OECD (2013), composed of three skills. In the eight assessment modules, this study
included a series of questions in assessing students’ performance in the matrix of
collaborative problem solving. Take the task of building shelves for example, there
are 13 questions for evaluating students’ performance in establishing and maintain-
ing shared understanding, 19 questions for evaluating students’ performance in
taking appropriate action to solve the problem, and 7 questions for evaluating

123
J. Comput. Educ. (2015) 2(3):301–322 311

Fig. 4 Table of students’ original performance scores on questions A1–D3 derived from the OECD
collaborative problem-solving matrix. For example, the original scores for the shelf construction task
(line 2), include the scores for sample questions C1 and C2 (1.00 and 0.60, respectively): These two
questions are: (C1) Do you think that a small and simple set of plastic shelves is a good choice for solving
this problem?; (C2) Do you know the installation process of building shelves?

students’ performance in establishing and maintaining team organization during the


collaborative problem-solving process (see sample questions and score table in
Fig. 4). Students must answer these questions according to their interaction with the
computer agent as they progress through the collaborative problem-solving tasks
included in the eight assessment modules. For the 12 elements in the matrix of
collaborative problem solving (A1–D3), the overall score assigned depended on the
students’ percentage of correct responses to the relevant questions related to each
element. If the students answered all questions related to an element correctly, their
responses were assigned a value of 1.0 (100 % correct, see Fig. 4).
To obtain a performance assessment of the three skills (columns 1–3, Table 1),
the mean values of A1–D1, A2–D2, and A3–D3, were calculated, respectively. For
a composite assessment of the collaborative problem-solving skills, the OECD
(2013) suggests the following criteria of percent correct responses, (1) establishing
and maintaining shared understanding should account for between 40 and 50 % of
overall skill performance, (2) taking appropriate action to solve the problem,
between 20 and 30 % of overall performance, and (3) establishing and maintaining
team organization, between 30 and 35 % of overall skill performance. Therefore to
be concise, we adopted a criterion level of 40, 30, and 30 % for these, respectively.
With regard to validity, three STEM education or collaborative problem-solving
specialists reviewed the content validity of the eight assessment modules according
to the OECD criteria of the 12 skills listed in Table 1 and concurred that they were
adequate. Furthermore, to examine the criterion- related validity of our assessment
of the eight modules, Pearson product-moment intercorrelations of the students’
performance on the eight modules were obtained in relation to overall and each of
three skill areas as follows. For example, for the overall CPS skill, the assessment
values for each of the eight CPS module scenarios were intercorrelated to determine
if they were consistently rendering the same result (see Table 4 in Results for this
example). Likewise, intercorrelations were obtained for each of the eight CPS
modules in relation to skill 1 (Table 1); i.e., ‘‘Establishing and maintaining shared
understanding’’ (Table 5, Results). Also, intercorrelations were obtained for skill 2:
‘‘Taking appropriate action to solve the problem’’ (Table 6, Results); and similarly
for skill 3: ‘‘Establishing and maintaining team organization’’ (Table 7, Results).

123
312 J. Comput. Educ. (2015) 2(3):301–322

The higher the intercorrelations in each table, the more likely the eight scenarios
were consistent, yielding the same results for the particular identified skill, hence
supporting the validity of the design.

Participants and procedures

A total of 222 Taiwanese junior high students aged 13–15 years old participated in
this study. They came from nine classes in two different schools, and spent three
weeks finishing the eight tasks in the CPS web-based system. In the first week, the
teachers explained the operation of the assessment system, and the students were
expected to complete two tasks. In the second and third weeks, students had to finish
three tasks per week. With respect to the format of the web-based CPS system, the
number of computer agents that the student interacted with varied. The student
worked with one agent (initially during 4 modules) and later two agents
(subsequently, for 4 other modules).

Findings for each research question

Research Question 1 Is there evidence that the eight modules have good validity in
assessing junior high school students’ collaborative problem-solving skills?

Students’ performance on the assessment modules

Average student performance using the assessment system was about 0.7 (out of a full
mark of 1.0; this is equal to 70 % correct in the test, see Table 2). Students tended to
have lower CPS performance if the problem scenario was related to students’ hands-on
experiences (that is, making shelves or designing and tidying a desk). Based on the

Table 2 Means (M) and standard deviations (SD) for Students’ performance on the assessment modules
Tasks Establishing and Taking Establishing and Collaborative
maintaining shared appropriate maintaining team problem solving
understanding action to solve organization
the problem

M SD M SD M SD M SD

Shelves .65 .24 .49 .19 .74 .18 .63 .17


Microwave oven .74 .18 .76 .18 .76 .23 .75 .17
Defuse a bomb .75 .14 .71 .13 .72 .21 .73 .13
Interior design .84 .15 .75 .20 .72 .23 .78 .18
Travel in Kaohsiung .74 .21 .73 .22 .77 .24 .74 .18
Buy a cell phone .78 .19 .63 .16 .71 .15 .71 .14
Construct a house .71 .22 .82 .15 .88 .18 .79 .15
Design and tidy desk .62 .20 .56 .27 .54 .27 .58 .23

N = 222; the maximum possible score for each item was 1.0

123
J. Comput. Educ. (2015) 2(3):301–322 313

results, it is hard to find related studies for comparison to our finding. Because, many
prior studies focused on different dimensions to be assessed than the ones we used,
such as using the computer agent in designing reading, writing, and communicating
systems (McNamara et al. 2007; Johnson and Valente 2008), developing an inferential
problem-solving system (Azevedo et al. 2010; Biswas et al. 2010; Cai et al. 2011); and
so on. In our assessment system, the computer agent played an important role as a
technology specialist, and was designed to offer as much valuable information as
possible to the students, but the results were not as expected. Among possible reasons
for this discrepancy, the student’s prior experiences with the particular kinds of hands-
on tasks we included in the CPS system may have been too limited. If the students had
very little prior experience with some of the practical kinds of tasks we included, it may
have been difficult for them to work effectively with the kind of information that the
computer agent was providing. That is, students’ prior knowledge in the specific
domain could be an important factor in influencing their effective interaction with the
information provided by the computer agent in the collaborative STEM problem-
solving scenarios. Further issues with limitations that may have arisen due to the kind
of information and its organization used for the items by the agent in responding to the
student are also examined in the Discussion.

Discrimination and difficulty of the assessment modules

In order to explore the discrimination and difficulty of the eight assessment


modules, the analysis of the discrimination and difficulty indexes of these eight
assessment modules is shown in Table 3. In order to simplify the entries in the table,
the average results of three dimensions are shown instead of all items in these eight
assessment modules. The discrimination indexes of the CPS assessment modules in
this study were between 0.34 and 0.70 with an average of 0.49, and the difficulty
indexes were between 0.57 and 0.92 with an average of 0.80 (details shown in
Table 3). According to Ebel and Frisbie’s (1986) viewpoints, an item should be
removed if its discrimination index (D) is below 0.19 and is acceptable but needs to
be modified if D is between 0.20 and 0.29. All the discrimination indexes of the CPS
assessment modules are larger than 0.34, so the discrimination of the eight
assessment modules was deemed to be good. As for the difficulty of the eight
assessment modules, Kuo (1996) believed that the optimal difficulty index (P) is
between 0.40 and 0.80. However, the average difficulty indexes is 0.80 and some
items are larger than 0.80. Overall, the difficulty of the eight assessment modules is
acceptable, but some items require revision to make the difficulty more acceptable.

The correlations of students’ CPS performance across eight different


problem scenarios

In addition to the content validity that was reported by the panel of expert reviewers,
the criterion-related validity was also examined through Pearson intercorrelations as
explained in the methods section on Evaluation. The use of this kind of evaluation
criterion is difficult to find in publications for the past few years. The only system of
a similar kind was developed by Kuo (2014). The core ideas of Kuo’s system are

123
314 J. Comput. Educ. (2015) 2(3):301–322

Table 3 Discrimination and difficulty indexes of the assessment modules


Assessment modules CPS Difficulty Correct ratio of Correct ratio of Discrimination
index high-score group low-score group index

Shelves 1 0.86 0.97 0.34 0.63


2 0.57 0.75 0.26 0.39
3 0.80 0.97 0.51 0.46
Microwave oven 1 0.80 0.94 0.50 0.44
2 0.82 0.96 0.52 0.44
3 0.88 0.98 0.44 0.54
Defuse a bomb 1 0.83 0.91 0.55 0.36
2 0.75 0.89 0.55 0.34
3 0.80 1.00 0.46 0.54
Interior design 1 0.88 1.00 0.64 0.36
2 0.83 0.95 0.46 0.49
3 0.80 0.95 0.40 0.55
Travel in Kaohsiung 1 0.83 0.96 0.43 0.53
2 0.82 0.97 0.43 0.54
3 0.85 1.00 0.43 0.57
Buy a cell phone 1 0.86 1.00 0.50 0.50
2 0.68 0.83 0.42 0.41
3 0.77 0.87 0.50 0.37
Construct a house 1 0.72 0.88 0.37 0.51
2 0.69 0.90 0.20 0.70
3 0.67 0.89 0.20 0.69
Design and tidy desk 1 0.79 0.97 0.41 0.56
2 0.87 0.98 0.61 0.37
3 0.92 1.00 0.64 0.36

Note CPS column entries are as follows: 1 represents ‘‘Establishing and maintaining shared under-
standing,’’ 2 represents ‘‘Taking appropriate action to solve the problem,’’ 3 represents ‘‘Establishing and
maintaining team organization.’’

similar to this study, but the difference is that Kuo’s system is much simpler and
students can finish one test in a few steps (e.g., five steps). The eight assessment
modules used in our study were developed by four small groups (including one
professor, one graduate student, and one information technology engineer); thus, a
question can be raised about how consistent the eight modules were in yielding
similar evaluation results. Therefore, our rationale was to examine how consistently
the eight modules yielded similar results for overall and each of three CPS skills. If
the intercorrelations of the results for each of the eight modules were large within
any one CPS skill, this would support a conclusion that they were consistently
producing similar results. Accordingly, the results of the intercorrelations across the
eight task modules in relation to overall CPS performance are presented in Table 4.
The correlation values of the students’ collaborative problem-solving skills, across
eight different problem scenarios, all reached a statistically significant threshold

123
J. Comput. Educ. (2015) 2(3):301–322 315

Table 4 Correlations of students’ overall CPS performance across eight different problem scenarios
Tasks 1 2 3 4 5 6 7 8

1. Shelves 1 .51** .53** .39** .52** .44** .35** .50**


2. Microwave oven – 1 .45** .33** .44** .41** .32** .40**
3. Defuse a bomb – – 1 .39** .51** .49** .44** .60**
4. Interior design – – – 1 .47** .39** .34** .45**
5. Travel in Kaohsiung – – – – 1 .56** .46** .57**
6. Buy a cell phone – – – – – 1 .55** .58**
7. Construct a house – – – – – – 1 .56**
8. Design and tidy desk – – – – – – – 1

* p \ .05, ** p \ .01

(p B 0.05); although the correlation values ranged considerably from 0.33 to 0.60.
That is, the eight tasks yielded sufficiently similar results for the assessment of the
students’ CPS performance to be considered adequately consistent.

The correlations of establishing and maintaining shared understanding


across eight different problem scenarios

According to the intercorrelation results in Table 5, the correlations of students’


performance establishing and maintaining shared understanding across the eight
problem scenarios all reached a significant level, indicating an adequate level of
consistency within this task dimension. The correlation values of ‘‘microwave oven and
interior design,’’ ‘‘interior design and construct a house,’’ though statistically significant,
are below 0.2, and suggest less confidence that the eight tasks are substantially yielding
similar results. The correlations account for less than 4 % of the variance, even though
they are statistically significant with the large N of 222 students. That is, the assessment
of the students’ performance in the skill of ‘‘Establishing and maintaining shared
understanding’’ may differ depending on which of the eight tasks they are performing. It
is not clear why there are these differences. However, for the tasks 1–4, students worked
with one computer agent, and for tasks 5–8, students worked with two computer agents.
Therefore, the possible reason for the lack of consistency in results (0.19 for item 2 in
relation to item 4, and 0.18 for item 4 in relation to item 7) is not likely due to the number
of computer agents, because the number of agents was different between these two sets,
yet the results were low for both.

The correlations of students’ taking appropriate action to solve the problem


across eight different problem scenarios

According to the results in Table 6, the correlations of students’ performance on the


skill of ‘‘Taking appropriate action to solve the problem,’’ across the eight different
problem scenarios, all reached a significant level. That is, the eight tasks
consistently yield similar outcome patterns. If we check the correlations for three
comparisons that are low, i.e., (1) task 2 ‘‘Microwave oven’’ and task 6 ‘‘Buy a cell

123
316 J. Comput. Educ. (2015) 2(3):301–322

Table 5 Correlations of ‘‘Establishing and maintaining shared understandings’’ across eight different
problem scenarios
Tasks 1 2 3 4 5 6 7 8

1. Shelves 1 .43** .57** .24** .48** .41** .30** .51**


2. Microwave oven – 1 .42** .19** .36** .35** .22** .35**
3. Defuse a bomb – – 1 .26** .58** .44** .36** .57**
4. Interior design – – – 1 .24** .31** .18** .27**
5. Travel in Kaohsiung – – – – 1 .50** .32** .51**
6. Buy a cell phone – – – – – 1 .37** .46**
7. Construct a house – – – – – – 1 .55**
8. Design and tidy desk – – – – – – – 1

* p \ .05, ** p \ .01

Table 6 Correlations of students’ ‘‘Taking appropriate action to solve the problem’’ across eight dif-
ferent problem scenarios
Tasks 1 2 3 4 5 6 7 8

1. Shelves 1 .22** .22** .28** .26** .21** .20** .31**


2. Microwave oven – 1 .30** .32** .23** .14* .20** .29**
3. Defuse a bomb – – 1 .32** .28** .30** .32** .42**
4. Interior design – – – 1 .48** .24** .38** .45**
5. Travel in Kaohsiung – – – – 1 .36** .33** .48**
6. Buy a cell phone – – – – – 1 .35** .41**
7. Construct a house – – – – – – 1 .37**
8. Design and tidy desk – – – – – – – 1

* p \ .05, ** p \ .01

phone;’’ (2) for task 1 ‘‘Shelves’’ and task 7 ‘‘Construct a house,’’ and (3) task 2
‘‘Microwave oven and task 7 ‘‘construct a house,’’ we note that all values are less
than, or equal, to 0.2. Among the four tasks analyzed in these three comparisons the
number of computer agents varied. There are two computer agents in the tasks ‘‘Buy
a cell phone’’ and ‘‘Construct a house.’’ In these two module tasks, the two
computer agents will offer different choices to the students, and some of the choices
are designed to be wrong choices as explained in the foregoing methods section on
System framework. So the students may face more difficulties in taking appropriate
action to solve the problems in these two task modules, and this could be one of the
possible reasons for the low correlations of the outcomes for these two tasks.

The correlations of students’ establishing and maintaining team


organization across eight different problem scenarios

According to the analysis results in Table 7, the correlations of students’


establishing and maintaining team organization performance across eight different

123
J. Comput. Educ. (2015) 2(3):301–322 317

Table 7 Correlations of students’ ‘‘Establishing and maintaining team organization’’ across eight dif-
ferent problem task scenarios
Tasks 1 2 3 4 5 6 7 8

1. Shelves 1 .29** .24** .29** .15* .19** .12 .24**


2. Microwave oven – 1 .31** .38** .33** .35** .25** .31**
3. Defuse a bomb – – 1 .35** .31** .33** .15* .44**
4. Interior design – – – 1 .38** .33** .34** .45**
5. Travel in Kaohsiung – – – – 1 .30** .30** .33**
6. Buy a cell phone – – – – – 1 .26** .43**
7. Construct a house – – – – – – 1 .18**
8. Design and tidy desk – – – – – – – 1

* p \ .05, ** p \ .01

problem scenarios all reached a significant level, but the shelves and construct a
house are an exception. That is, there are six tasks at least that can assess students’
establishment and maintenance of team organization effectively. However, some
correlations were low, and these findings should therefore be taken cautiously and
re-checked by different studies and from different angles. If we check the
correlations of ‘‘shelves and construct a house,’’ ‘‘shelves and travel in Kaohsiung,’’
‘‘shelves and buy a cell phone,’’ ‘‘defuse a bomb and construct a house,’’ ‘‘construct
a house and design tidy desk,’’ we note that the values are below 0.2. The possible
reason for the low correlations with the shelves task in assessing students’
establishing and maintaining team organization is that we had a different design in
the shelves task. Students had to communicate with the computer agent by inputting
what they wanted to say using the keyboard, and the computer agent searched the
keyed-in input by accessing the database and responded to the students. In addition,
students also could indicate their response by selecting from among a set of choices.
According to the analysis of students’ online record, some students did not want to
communicate with the computer agent by keying in their responses and preferred to
select the choice. Therefore, if the task is largely designed to be most effective using
the keyboard communication mode, but students preferentially elect to use the
simpler choice mode, this may be a source of differential performance in
establishing and maintaining team organization, hence contributing to lower
intercorrelations across tasks.
Research Question 2 What are some of the difficulties in developing an
assessment system for collaborative problem solving in STEM education?
This study was admittedly complex. Our goal was to provide some of the first
assessment-based evidence for a CPS web-based system that was theoretically based
(OECD matrix), that included computer agents (rather than human-to-human agents
during CPS), and incorporated problem tasks that were situated in every day kinds
of issues that junior high school students may encounter. Weaving all three of these
components into a novel system of this kind, that also used broad STEM content,
was challenging; especially given the lack of a similar ambitious study in the

123
318 J. Comput. Educ. (2015) 2(3):301–322

literature to help guide us. One of the issues concerns the challenge of creating a
computer agent system that is sufficiently responsive to the varied student inputs to
be reasonably natural and sufficiently contingent to encourage and reinforce student
engagement in a way that leads to consistent gains across the eight different task
modules. In this exploratory study, a team of experts designed a limited number of
potential items that the ‘‘agent’’ could retrieve in responding to student input.
Consequently, some lack of coherence is likely to occur; and it is difficult in every
case to develop an algorithm that would provide appropriate decision making by the
‘‘agent’’ to respond in both a logical and human-based natural way. More
sophisticated agent systems are clearly needed, and this study offers some insights
on the prospects for success as well as the challenges.
Another challenge emerged for the skill of ‘‘Establishing and maintaining team
organization.’’ This also was likely influenced partially by the lack of a more ‘‘human-
like’’ response of the computer agent as well as the limited number of options
programmed in the agent submodule in the server system (Fig. 3). Unless the student
has a sense that the collaborator is responding in a contingent and logical way, it is
difficult to see how a truly coordinated collaborative relationship can develop.

Discussion and conclusions

According to the data and analysis presented above, the assessment system developed
in this study consisted of the items with acceptable difficulty and satisfactory
discrimination, and the content validity and criterion-related validity were also judged
to be good. Overall, the assessment system for collaborative problem solving in
STEM education presented by this study was effective in assessing students’
collaborative problem-solving skills. Although the majority of intercorrelations used
as evidence of consistency of outcomes across the eight tasks within a given CPS skill
(Tables 4, 5, 6, and 7) on the whole were highly statistically significant and also in
many cases of sufficiently high correlation values to be considered strong, there were
a few that were low. These were highlighted in the Findings section and the reasons
for these discrepant findings remain somewhat enigmatic as to why the consistencies
were so low. Among these, the pairs of items that had low correlations: task 2
‘‘Microwave oven’’ and task 6 ‘‘Buy a cell phone;’’ task 1 ‘‘Shelves’’ and task 7
‘‘Construct a house,’’ and task 2 ‘‘Microwave oven and task 7 ‘‘construct a house,’’
there are no immediate clues as to why these tasks of different demands were so
discrepant. These findings may offer an opportunity for further research to determine
what task attributes in relation to CPS skills are most likely to be challenging for
students at different grade levels and prior levels of knowledge.
As explained in the results for Research question 2, one skill that requires further
attention, ‘‘Establishing and maintaining team organization,’’ is especially chal-
lenging with respect to the design of a naturalistic computer agent. It proved hard to
design a problem scenario to assess this skill, mostly because it is difficult to always
design a response format for the computer agents that is sufficiently ‘‘human-like;’’
rather, they act according to a more limited set of item bank options developed by
our research team members. The complexities that emerge when the mode of

123
J. Comput. Educ. (2015) 2(3):301–322 319

interaction of the learner with the agent varies (such as keying in versus choosing
options) may also contribute to differences in performance assessment outcomes
across varied tasks. In Ding’s (2009) study, three different collaboration patterns in
terms of joint and individual knowledge elaboration were shown to be effective, but
only one collaboration pattern was employed in the assessment system for
collaborative problem solving, because the students had to follow the script provided
in that study. In general, the limitations of a script-based computer agent are a major
problem for Ding’s study, and likely for others given our current state of development
in the field. It was also a major issue in our study, when in addition to the complexities
of the OECD framework, we also included natural every day problems to assess
students’ performance of collaborative problem-solving skills in STEM education.
That is, if students have different collaboration patterns (Ding 2009), and the
computer agent contributes yet another variation on the complexity of the patterns,
then the students may not achieve the level of sophistication that they may achieve in
a more natural human-to-human-based experience. Besides, if the task is designed as
a keyboard response mode, where the text may be complex and not easily parsed by
the software, it may be hard for the system to capture the students’ real thoughts; this
may be especially difficult due to the limitation of our techniques in Chinese latent
semantic analysis. If we want to improve the system and reduce these mistakes, it
could be possible to offer the students multiple choices in responding, rather than a
single option provided by the computer agent (e.g., Kuo 2014), but it is difficult in this
case to prevent the students from merely choosing an answer instead of keying in an
answer that is more clearly what they want to say, in addition to the problem of less
consistency across different tasks, as discussed above.
In addition, there is another important difficulty that needs to be addressed. In
this study, we tried to assess 12 collaborative problem-solving skills in each
problem scenario; this large number of assessments became a drawback due to the
complexity of the problem scenarios. In some cases, there were just one or two
questions used to assess a given skill (see Table 1) and this may also be the reason
why the average difficulty of the eight assessment modules was deemed only
‘‘acceptable’’ according to Kuo’s (1996) criteria. If in future work, we include fewer
of the 12 skills in one problem scenario to try to obtain a more robust item analysis
of the CPS responses, we face another limitation, i.e., it may be difficult to get a
comprehensive sense of students’ real collaborative problem-solving skills.
Moreover, this study utilized a criterion-related validity method of examining the
intercorrelations of the results for each of the eight tasks in relation to overall and
each of three CPS dimensions as a way of gaining evidence of consistency and a form
of validity for the eight modules in the assessment CPS system. Even if a correlation
value reached the significant level (p B 0.05), we found in some instances that the
amount of variance accounted for was low; thus there is still a limitation that one or
more of the eight assessment modules could have by chance missed the core objective
of the collaborative problem-solving task that we intended to achieve.
To sum up, this study proposed a novel, and somewhat complex, assessment
system for collaborative problem solving in STEM education that can be used to
assess junior high students’ collaborative problem-solving skills. Although we used
a computer agent, it is likely that with appropriate online data capture systems, it

123
320 J. Comput. Educ. (2015) 2(3):301–322

could be adapted for assessing collaborative learning where human-to-human


collaborations were used. Even with some of the limitations of our current capacity
to create a more human-like computer agent or agents, the results for many of the
assessments in this study were encouraging. We hope the reflections and
suggestions provided can help future researchers develop more effective collabo-
rative problem-solving systems or inspire others to conduct related studies.

Acknowledgments This research was funded by the Ministry of Science and Technology of the
Republic of China under Contract numbers NSC 102-2511-S-003-059-MY2. The findings and
recommendations contained in this article of those of the authors and do not necessarily reflect those
of the Ministry of Science and Technology. We are extremely grateful to Professor O. Roger Anderson
and Ying-Tien Wu for their mentoring efforts, the reviewers for their helpful comments, and the teachers
and students who participated in this study.

References
Alvarez, C., Salavati, S., Nussbaum, M., & Milrad, M. (2013). Collboard: Fostering new media literacies
in the classroom through collaborative problem solving supported by digital pens and interactive
whiteboards. Computers & Education, 63(1), 368–379.
Azevedo, R., Moos, D., Johnson, A., & Chauncey, A. (2010). Measuring cognitive and metacognitive
regulatory processes used during hypermedia learning: Issues and challenges. Educational
Psychologist, 45, 210–223.
Barak, M., & Mesika, P. (2007). Teaching methods for inventive problem-solving in junior high school.
Thinking Skills and Creativity, 2(1), 19–29.
Biswas, G., Jeong, H., Kinnebrew, J., Sulcer, B., & Roscoe, R. (2010). Measuring self-regulated learning
skills through social interactions in a teachable agent environment. Research and Practice in
Technology-Enhanced Learning, 5(2), 123–152.
Blackwell, D., & Henkin, L. (1989). Mathematics: Report of the Project 2061 Phase I mathematics panel.
Washington, D. C.: American Association for the Advancement of Science.
Bybee, R. W. (2013). The case for STEM education: Challenge and opportunities. Arlington, CA:
National Science Teachers Association Press.
Cai, Z., Graesser, A. C., Forsyth, C., Burkett, C., Millis, K., Wallace, P., et al. (2011). Trialog in ARIES:
User input assessment in an intelligent tutoring system. In W. Chen & S. Li (Eds.), Proceedings of
the 3rd IEEE international conference on intelligent computing and intelligent systems (pp.
429–433). Guangzhou: IEEE Press.
Darling-Hammond, L. (2011). Policy frameworks for new assessments. In P. Griffin, B. McGaw, & E.
Care (Eds.), Assessment and teaching 21st century skills. Heidelberg: Springer.
Daugherty, M., & Wicklein, R. (1993). Mathematics, science, and technology teachers’ perceptions of
technology education. Journal of Technology Education, 4(2), 28–43.
Davis, A., Fidler, D., & Gorbis, M. (2011). Future work skills 2020. Palo Alto, CA: Institute for the
Future.
Dillenbourg, P., & Traum, D. (2006). Sharing solutions: Persistence and grounding in multi-modal
collaborative problem solving. The Journal of the Learning Sciences, 15, 121–151.
Ding, N. (2009). Visualizing the sequential process of knowledge elaboration in computer-supported
collaborative problem solving. Computers & Education, 52, 509–519.
Dixon, R. A., & Brown, R. A. (2012). Transfer of learning: Connecting concepts during problem solving.
Journal of Technology Education, 24(1), 2–17.
Ebel, R. L., & Frisbie, D. A. (1986). Essentials of education measurement. Englewood Cliffs, NJ:
Prentice Hall.
Franklin, S., & Graesser, A. C. (1996). Is it an agent or just a program? A taxonomy for autonomous
agents. In Proceedings of the third international workshop on agent theories, architectures, and
languages. Springer, Berlin.

123
J. Comput. Educ. (2015) 2(3):301–322 321

Greene, R. W. (2011). Collaborative problem solving can transform school discipline. Phi Delta Kappan,
93(2), 25–28.
Harasim, L. (1999). A framework for online learning: The virtual-U. Computer, 32(9), 44–49.
Hong, J. C., Hwang, M. Y., & Tai, K. H. (2013). Applying the BaGua to revitalize the creative problem
solving process during a goal oriented contest. Thinking Skills and Creativity, 9, 123–128.
Isenhour, P. L., Carroll, J. M., Neale, D. C., Rosson, M. B., & Dunlap, D. R. (2000). The virtual school:
An integrated collaborative environment for the classroom. Educational Technology and Society,
3(3), 74–86.
Johnson, J. R. (1989). Technology: Report of the project 2061 phase I technology panel. Washington, D.
C.: American Association for the Advancement of Science.
Johnson, S. D., Dixon, R., Daugherty, J., & Lawanto, O. (2011). General versus specific intellectual
competencies: The question of learning transfer. In M. Barak & M. Hacker (Eds.), Fostering human
development through engineering and technology education (pp. 55–74). Rotterdam: Sense
Publishers.
Johnson, L. W., & Valente, A. (2008). Tactical language and culture training systems: Using artificial
intelligence to teach foreign languages and cultures. In M. Goker & K. Haigh (Eds.), Proceedings of
the twentieth conference on innovative application of artificial intelligence (pp. 1632–1639). Menlo
Park, CA: AAAI Press.
Kirschner, F., Paas, F., Kirschner, P. A., & Janssen, J. (2011). Differential effects of problem-solving
demands on individual and collaborative learning outcomes. Learning and Instruction, 21(4),
587–599.
Kuo, S.-Y. (1996). Psychological and educational testing. Taipei: Jing-Hua Bookstore.
Kuo, B.-C. (2014). Teachers’ collaborative problem solving teaching competency project. Retrieved on
March 7, 2015, from https://sites.google.com/site/cpswebsite2014/
Kuo, F.-R., Hwang, G.-J., Chen, S.-C., & Chen, S.-Y. (2012). A cognitive apprenticeship approach to
facilitating web-based collaborative problem solving. Educational Technology & Society, 15(4),
319–331.
Kuo, C.-Y., & Wu, H.-K. (2013). Toward an integrated model for designing assessment systems: An
analysis of the current status of computer-based assessments in science. Computers & Education,
68, 388–403.
Looi, C.-K., Chen, W., & Ng, F.-K. (2010). Collaborative activities enabled by GroupScribbles (GS): an
exploratory study of learning effectiveness. Computers & Education, 54(1), 14–26.
Martin-Kniep, G., Feige, D., & Soodak, L. (1995). Curriculum integration: An expanded view of an
abused idea. Journal of Curriculum and Supervision, 10(3), 227–249.
McNamara, D. S., O’Reilly, T., Rowe, M., Boonthum, C., & Levinstein, I. B. (2007). iSTART: A web-
based tutor that teaches self-explanation and metacognitive reading strategies. In D. S. McNamara
(Ed.), Reading comprehension strategies: Theories, interventions, and technologies (pp. 397–421).
Mahwah, NJ: Erlbaum.
Nussbaum, M., Alvarez, C., McFarlane, A., Gomez, F., Claro, S., & Radovic, D. (2009). Technology as
small group face-to-face collaborative scaffolding. Computers & Education, 52(1), 147–153.
Organisation for Economic Co-operation and Development (2013). Draft collaborative problem solving
framework. Unpublished manuscript, OECD.
Raju, P. K., & Clayson, A. (2010). The future of STEM education: An analysis of two national reports.
Journal of STEM Education, 11(5&6), 25–28.
Roschelle, J., Rafanan, K., Estrella, G., Nussbaum, M., & Claro, S. (2010). From handheld collaborative
tool to effective classroom module: Embedding CSCL in a broader design framework. Computers &
Education, 55(3), 1018–1026.
Rosen, Y., & Tager, M. (2013). Computer-based assessment of collaborative problem-solving skills:
Human-to-agent versus human-to-human approach. Retrieved on June 2, 2014 from http://
researchnetwork.pearson.com/wp-content/uploads/CollaborativeProblemSolvingResearchReport.pdf
Scifres, E. L., Gundersen, D. E., & Behara, R. S. (1998). An empirical investigation of electronic groups
in the classroom. Journal of Education for Business, 73, 247–250.
Sternberg, R. J. (2001). Teaching problem solving as a way of life. In A. L. Costa (Ed.), Developing
minds: A resource book for teaching thinking (3rd ed., pp. 451–454). Alexandria, VA: Association
for Supervision and Curriculum Development.
VanLehn, K., Graesser, A. C., Jackson, G. T., Jordan, P., Olney, A., & Rose, C. P. (2007). When are
tutorial dialogues more effective than reading? Cognitive Science, 31, 3–62.

123
322 J. Comput. Educ. (2015) 2(3):301–322

Warwick, P., Mercer, N., Kershner, R., & Staarman, J. K. (2010). In the mind and in the technology: The
vicarious presence of the teacher in pupil’s learning of science in collaborative group activity at the
interactive whiteboard. Computers & Education, 55(1), 350–362.
Wood, R., & Ashfield, J. (2007). The use of the interactive whiteboard for creative teaching and learning
in literacy and mathematics: A case study. British Journal of Educational Technology, 39(1), 84–96.

Kuen-Yi Lin is an Associate professor in technology education at National Taiwan Normal University,
Taiwan. His research interests deal largely with hands-on learning, STEM education, and technology
education.

Kuang-Chao Yu is a Professor at National Taiwan Normal University, Taiwan. His research interests
deal largely with concept and knowledge construction related to technology and engineering education.

Hsien-Sheng Hsiao is a Professor and department head at National Taiwan Normal University, Taiwan.
His research interests deal largely with database management system, computer networks, e-learning, and
cloud computing.

Yih-Hsien Chu is an Associate Professor at National Taiwan Normal University, Taiwan. His research
interests deal largely with technology education, human resources education, human resources
development and manufacturing technology.

Yu-Shan Chang is a Professor at National Taiwan Normal University, Taiwan. His research interests
deal largely with technology education, technological creativity and multimedia design.

Yu-Hung Chien is an Associate Professor at National Taiwan Normal University, Taiwan. His research
interests deal largely with interior design, computer aided design and product design.

123
G Model
JSEE-562; No. of Pages 13

Studies in Educational Evaluation xxx (2015) xxx–xxx

Contents lists available at ScienceDirect

Studies in Educational Evaluation


journal homepage: www.elsevier.com/stueduc

Design, implementation, and evaluation of an ePortfolio approach


to support faculty development in vocational education
Annemarieke Hoekstra *, Jocelyn R. Crocker *
Northern Alberta Institute of Technology, 11762-106 Street NW, Edmonton, AB, Canada T5G 2R1

A R T I C L E I N F O A B S T R A C T

Article history: This article provides an account of the design, implementation, and evaluation of an ePortfolio approach
Received 15 May 2014 to faculty development and performance evaluation at a Canadian post-secondary vocational education
Received in revised form 18 February 2015 institute. The approach was piloted in two phases in 13 departments. Survey and interview data were
Accepted 26 March 2015
collected and analyzed to determine adoption, reception by faculty, and impact of the approach on
Available online xxx
faculty development. While adoption of the approach in the pilots was limited, participants who adopted
the approach reported collecting more and different feedback, developing increased awareness of areas
Keywords:
for improvement, and planning their professional learning more explicitly. Further studies are needed to
Faculty development
Professional development
determine what design elements of the portfolio optimally support professional development and
Professional learning performance evaluation.
Electronic portfolio ß 2015 Elsevier Ltd. All rights reserved.
Faculty evaluation
Vocational education

Introduction conference attendance and the creation and delivery of in-house


workshops.
As in many professions, the need for ongoing professional Professional development literature argues that behaviours
development exists for teaching professionals in post-secondary learned during workshops and courses should be reinforced in the
education. In Canadian post-secondary vocational education, this workplace (Blume, Ford, Baldwin, & Huang, 2010; Kirkpatrick,
necessity stems from ongoing changes in the vocations being 2007). Post-secondary faculty development initiatives have aimed
taught and an increasingly diverse student population (Bye, to incorporate workplace practices as a source of learning that
Pushkar, and Conway, 2007; Darwin, 2007; Harris et al., 2001; uses, for instance, peer review or action research (Amundsen &
Statistics Canada, 2010). For instance, in an effort to further Wilson, 2012; Stes, Min-Leliveld, Gijbels, & Van Petegem, 2010).
contribute to economic development, Canadian provincial gov- However, in her review of professional development literature,
ernments now encourage post-secondary vocational education Webster-Wright (2009) maintains that professional developers
institutes to engage in applied research in support of business and educational leaders should move beyond an intervention-
development (e.g., Polytechnics Canada, 2014). In addition, transfer-impact paradigm and focus on how professionals
changes in educational approaches, such as a move to more naturally learn at work.
authentic forms of student assessment, may require faculty to This paper describes the design, implementation, and evalua-
think differently about supporting student learning (e.g., Stiehl & tion of a faculty development initiative called the ePortfolio
Lewchuk, 2008). Finally, vocational educators may also be approach, which was designed to foster learning that occurs
required to serve on committees and in leadership roles. These naturally in the workplace. In 2009, senior leaders at a Canadian
changes and additional duties require vocational educators to post-secondary institute for vocational education mandated a
keep learning during their careers. To address this learning need, taskforce to design a comprehensive approach to faculty develop-
post-secondary institutes have traditionally invested in course or ment and performance evaluation. The paper describes how, based
on a literature review and several rounds of faculty engagement, the
taskforce designed the ePortfolio approach. This approach consisted
of a comprehensive process for faculty to (1) collect feedback from
* Corresponding authors. Tel.: +1 780 471 7862/+1 780 378 5197.
E-mail addresses: annemarh@nait.ca, anne_marieke@yahoo.com (A. Hoekstra), multiple sources on multiple aspects of their work; (2) compile the
jocelync@nait.ca (J.R. Crocker). feedback in their ePortfolio, reflect on it, and make changes to their

http://dx.doi.org/10.1016/j.stueduc.2015.03.007
0191-491X/ß 2015 Elsevier Ltd. All rights reserved.

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13

2 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

teaching practice if necessary; (3) formulate goals for future growth formal student feedback is a mandatory part of college or
based on their feedback and reflection; and (4) share their ePortfolio university workplaces, and many post-secondary teachers also
with their supervisor to collaboratively assess their performance seek informal feedback from their students. However, as Halonen
and determine goals for the next year. The paper also describes how and Ellenberg (2006) highlight, students might provide a one-sided
the ePortfolio approach was implemented in two consecutive pilots view of faculty performance or allow their feedback to be
and the survey and interview data collected in order to evaluate the influenced by bias. Feedback from additional sources such as
pilots. The research questions addressed in this paper are as follows: colleagues or faculty support staff might provide a more
comprehensive picture of professional performance (Millis,
(1) To what extent was the ePortfolio adopted in the participating 2006; Jimerson, 2014; Seijts et al., 1998; Wayman & Jimerson,
departments in the first and second pilot? 2014). Therefore, the taskforce established that feedback collection
(2) To what extent did faculty in the first and second pilot value the from multiple sources should be encouraged.
ePortfolio approach as a whole? Wayman and Jimerson (2014) also found that for feedback to be
(3) What was the impact of the ePortfolio approach on the relevant to teachers and students, it needs to be timely (see also
professional development of participating faculty members? Halonen & Ellenberg, 2006) so faculty members can discuss the
feedback with their students and adjust their teaching to
Literature review accommodate students’ needs. Therefore, the third parameter
for the ePortfolio approach was that student feedback should be
We define faculty development in accordance with Taylor and sought 30–50% of the way through the semester with a quick
Rege-Colet (2009) who describe it as development across ‘‘the full turnaround of the feedback.
spectrum of academic work’’ (p. 143). Our view of faculty Educational development literature describes learning with and
development also aligns with Steinert (2000), who broadly defines from colleagues as a workplace learning strategy, using, for
it as any type of activity that is aimed at renewing or assisting instance, faculty learning communities (Cox & Richlin, 2004) or
faculty in their roles. Our definition is wider in scope than simply mentoring (Johnson, 2006). Faculty might also initiate informal
instructional development (Stes et al., 2010) and educational learning activities such as seeking feedback on course materials or
development (Taylor & Rege-Colet, 2009), which both refer to the discussing experiences during their break (Hoekstra et al., 2009;
development of the faculty member in the areas of teaching and Eraut, 2004). Based on these findings, the taskforce determined
student learning. Our definition of faculty development includes that faculty members would be encouraged to share their
professional learning in all aspects of the role of the faculty portfolios with colleagues.
member including, for example, teaching, research, and adminis-
trative tasks. Professional development plans and self-directed learning
Research on professional learning generally suggests that such
learning is deeply embedded in practice (Opfer & Pedder, 2011; Literature on workplace learning, teacher learning, and faculty
Schön, 1983) and informed by the way people within and outside development all stress that conditions in the workplace impact on
of the organization conduct and understand their work (Bound, the quality of professional learning at work (Amundsen & Wilson,
2011; Engeström, 2011; Lave & Wenger, 1991). Billett (2002) 2012; Clarke & Hollingsworth, 2002; Fuller & Unwin, 2011; Harris
proposes that learning at work occurs through participating in et al., 2001; Opfer & Pedder, 2011; Sambrook, 2005). It can thus be
work-related activities. These activities play a central role in our expected that enhancing faculty development requires faculty
definition of professional learning. Most conceptualizations of motivation to engage in learning activities; it also needs the
learning imply a relatively lasting change in behaviour or capacity workplace culture, structure, and systems to encourage and
for behaviour (Shuell, 1986). In the present study, professional nurture engagement in these activities (Fuller & Unwin, 2011).
learning is defined as engaging in activities that lead to improved One strategy to encourage more planning and regulation of
professional practice or the capacity to behave in improved ways. professional development adopted in several professions, includ-
Such learning may be intentional or unintentional and conscious or ing K-12 teaching, is the use of a personal development plan (PDP),
beyond the learner’s awareness (Hoekstra, Beijaard, Brekelmans, & which aims to foster self-directed professional learning (Beausaert,
Korthagen, 2007; Eraut, 2004; Marsick & Watkins, 1990) and Segers, Gouarge, & Gijselaers, 2013). According to Roberson and
involve formal and/or informal learning activities (e.g., Hoekstra, Merriam (2005), self-directed learning refers to ‘‘intentional and
Brekelmans, Beijaard, & Korthagen, 2009; Lohman & Woolf, 2001). self-planned learning, where the individual is responsible for and
in control of the learning’’ (p. 270). Beausaert et al. (2013) reviewed
Reflective practice and feedback 54 studies and reported that attempts to implement PDPs have
varying degrees of success. In their own study, Beausaert et al.
Teacher educators, educational developers, and scholars who compared the number of learning activities reported by PDP users
study workplace learning all stress the importance of reflective with the number of activities of non-PDP users, concluding that,
practice as part of ongoing professional development (Hoekstra & retrospectively, PDPs encourage participation in learning activi-
Korthagen, 2011; Amundsen & Wilson, 2012; Brookfield, 1995; ties. Sharing their learning plans and assessment of their
Cross & Steadman, 1996; Van Woerkom & Croon, 2008; Zeichner & performance with the supervisor allows him or her to collaborate
Liu, 2010). While reflection can occur individually based on input with the faculty member and become an accountability partner for
from individual experience, reflection can be greatly enhanced the faculty member’s own development (Higgerson, 2006; Murray,
through incorporating feedback from others into the individual’s 1997). Based on this literature, the taskforce determined that the
own reflective process (Day, 1999; McGovern, 2006). The taskforce ePortfolio approach should encourage faculty members to set their
thus determined that the ePortfolio approach should include own learning goals and encourage department chairs to provide
student feedback as well as self-reflection on feedback (Hubball, sufficient modes of support and guidance for achieving these goals.
Pratt, & Collins, 2006; McGovern, 2006; Seijts, Taylor, & Latham,
1998). Performance evaluation and ePortfolios
Student feedback is a common component of faculty assess-
ment in universities (e.g., Pallett, 2006) and two-year colleges (e.g., The taskforce intended to design a process that would not only
Cross & Steadman, 1996). In many post-secondary institutions, foster faculty development but also support the assessment of

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13

A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx 3

faculty performance in collaboration with the department chair. diploma programs, degree programs, and professional certificates
Peter Seldin’s book ‘‘Evaluating Faculty Performance’’ (Seldin, in fields including trades, building construction and design,
2006) provides a strong case for the use of portfolios to assist in this engineering and applied sciences, health and safety, IT and
process (see also Freeman, Harkness, & Urbackzewski, 2009). electronics, business and administration, and hospitality and
Portfolios can include a collection of an individual’s accomplish- culinary arts. The institute has approximately 1100 faculty and
ments, plans, reflections, observations, and work samples (Gran- 80,000 full-time and part-time students.
berg, 2010). One advantage of ePortfolios as opposed to the Historically, the institute’s faculty members were required to
traditional format is they can also incorporate digital files such as create a performance management plan in which they outlined
simulations and videos (Butler, 2006). Smith and Tillema (2003) professional goals for the next year and annually discussed with
identify four types of portfolios including the dossier portfolio (a their supervisors the extent to which they achieved their goals.
record of achievements and work samples for promotional There is no promotion and tenure process at the institute; in other
purposes), the training portfolio (a required collection of efforts words, faculty members are directly hired into salaried positions
accumulated during a program), the reflective portfolio (a personal and remain faculty until retirement or job change. Professional
array of work providing evidence of growth and accomplishments development at the institute was most often conceptualized as
for promotion or admission), and the personal development portfolio attending conferences, courses, and workshops either on teaching
(a reflective account of professional growth over time). Portfolios or one’s trade/discipline. The institute had a process for collecting
can be used as both formative assessment to support professional student feedback once per semester, but this feedback was not
learning and summative assessment for promotion or certification necessarily part of the performance management review cycle.
(Smith & Tillema, 2003). When the focus is on formative assessment,
portfolios can be used to motivate faculty to reflect on priorities, ePortfolio design
accomplishments, successes, and challenges and plan their learning
accordingly (Devanas, 2006). Portfolios can thus easily integrate In 2009, a taskforce was formed to create a comprehensive
aspects of the personal development plan, including activities such approach to faculty development and performance evaluation.
as goal setting and collecting feedback specific to those learning Previously, instructors who scored low on student feedback
goals. The 2013 study by Beauseart et al. suggests that including such surveys had been sent to the faculty support office for coaching,
activities in a portfolio approach would likely contribute to leading to the impression that faculty development was remedial
enhanced participation in learning activities. (see also Cross & Steadman, 1996) rather than a necessity for all
While portfolios are widely used in teacher education to faculty to stay current and up to date. In addition, there was a need
support the development of student teachers (e.g., Granberg, 2010) to review and update the existing student feedback questionnaires
and used increasingly with post-secondary students of various and processes. The approach the taskforce was asked to design was
disciplines including nursing (e.g., Jasper & Fulton, 2005), medicine not intended to replace existing human resources processes
(e.g., Carraccio & Englander, 2004) and business (e.g., Flanigan, regarding personnel decisions; rather, it was intended to encour-
2012), the use of portfolios in support of faculty development and age faculty to collect comprehensive feedback and share it with
evaluation has received much less attention (Butler, 2006; their chair so as to collaboratively and continually work toward
McColgan & Blackwood, 2009). With respect to universities, improving their practice.
ePortfolios have been found to support and enhance professional Based on literature on leading by engagement (Axelrod, 2010),
development of faculty within the university promotion and the taskforce held a series of town hall meetings and focus groups
tenure process (Blair, 2001; Freeman et al., 2009). However, little is with faculty and other staff to engage them in the design and
known about the effect or use of portfolios in general and implementation stage. During these sessions, members of the
ePortfolios specifically in support of faculty development. Con- academic staff union indicated that the role of faculty members
ceptually, ePortfolios have the potential to enhance and promote had significantly expanded beyond teaching and that faculty
self-directed learning (Goliath, 2009) and self-regulated learning development and performance evaluation should cover this
(Lamont, 2007). Based on this literature, the taskforce determined expanding array of faculty responsibilities; these views aligned
that the best way to satisfy the parameters was through the closely with the findings of the literature review (e.g., O’Meara &
creation and sharing of professional ePortfolios. While not Braskamp, 2005). The taskforce then used focus groups with
intended for purposes of tenure and promotion, the professional institutional leadership, faculty members, and executive members
ePortfolio approach designed by the taskforce would cut across of the academic staff union to collaboratively identify eight aspects
multiple types of portfolios as identified by Smith and Tillema of the role of a faculty member: teaching, formal leadership, formal
(2003). Faculty would be invited to include a list of professional administrative duties, professionalism, formal curriculum devel-
development sessions attended (as in a training portfolio), opment, applied research, scholarship and professional develop-
evidence of feedback collected, and documented reflections on ment, and corporate citizenship. See page 4 of Online supplement
their feedback (as in a reflective portfolio). The compilation of this 3 for a description of each of these aspects. During these focus
evidence would serve to inform reflections on personal profes- groups, faculty members also indicated that feedback collected
sional growth over time (as in a personal development portfolio). only from students might not be a reliable method to assess their
performance. Taskforce members agreed with this point of view,
Methods based on the literature described earlier in this paper. Stakeholders
and taskforce members collaboratively decided to identify the
The ePortfolio approach was implemented in two pilots. following sources of evidence of performance: self, students, peers,
Following each pilot, both survey and interview data were and ‘‘others,’’ the latter including externals and faculty support
collected to determine adoption, faculty perception, and faculty personnel. In further faculty engagement sessions, a matrix was
perception of the impact of the ePortfolio approach. created with the eight aspects as rows and the four sources as
columns. Participants in the sessions collaboratively determined
Study context which sources would be able to provide feedback on which aspects
of the faculty role.
The site of study is a Western Canadian post-secondary Faculty support staff then proceeded with developing feedback
vocational institute that offers apprenticeship training, two-year tools for those cells in the matrix that were identified as good

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13

4 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

sources for feedback on certain roles. These tools were presented mance assessment and coaching. Specific attention was paid to the
together as a feedback toolkit and made available through the idea that performance evaluation should happen on an ongoing
institute’s intranet. The toolkit included informal student feedback basis with the purpose of supporting professional growth. It was
tools, a standardized student survey, self-reflection resources, also highlighted that human resource processes around disciplin-
peer-review tools including classroom observation templates, and ary measures were outside the scope of the ePortfolio approach.
a standardized leadership survey. See Online supplement 1 for an A follow-up session in a computer lab introduced participants
overview of the feedback tools in the toolkit. Thus, based on the to the online ePortfolio software Mahara. A mandatory part of both
literature and faculty engagement sessions, the taskforce designed pilots included the automatic collection and reporting of student
the ePortfolio approach as a faculty development and performance feedback through a newly developed online electronic student
evaluation initiative that encouraged faculty to (1) collect feedback survey. In addition, supports were offered through a website,
from multiple sources on multiple aspects of their role; (2) compile online feedback toolkit (see Online supplement 1), reminder
this feedback in their ePortfolio, reflect on it, and make changes to emails, and additional workshops in which participants could
their teaching practice, if necessary; (3) formulate goals for future practice the use of the ePortfolio tool. These additional workshops
growth based on their feedback and reflection; and (4) share their were offered to departments who requested extra help.
ePortfolio with their supervisor to collaboratively assess their Based on the findings from the first pilot, a number of changes
performance and determine goals for the next year. were made to the implementation of the second pilot. The student
and leadership surveys were refined, more diverse feedback tools
The ePortfolio software were developed, and it was decided to provide a customized
orientation and workshop to each participating department
Mahara (mahara.org) is an open-source software program separately. Between the first and second pilot, the institute
chosen as the web-based ePortfolio software tool because it is initiated a drastic academic change that involved major curriculum
intuitive in use and works within any browser. Mahara permits development for all credit programs, a new credit system, the
password protection and allows the faculty member to customize introduction of common courses, and a profound revision of the
different versions of the portfolio for different audiences, such as academic administration software used by the institute. This
peer groups or supervisors. The faculty member can have full change caused a great increase in workload of faculty.
control over who sees which page(s) and can set limits for the
duration intended audiences can access those pages. Mahara Instruments
version 1.4.6 was slightly customized for use within the institute
by creating a login page with the institute’s colours and logo. The evaluation focused on the impact of the ePortfolio approach
and its implementation, including details regarding the orientation
Pilot implementation sessions held, the new student evaluation survey, the new survey
used by chairs to collect feedback on their performance, student
The ePortfolio approach was piloted in two phases. In the first, response rates, and the feedback tools. This paper focuses on those
the deans of all five faculties in the institute were requested to seek questions pertaining to adoption of the approach, how faculty
department chairs willing to try out the new approach in (one of experienced the ePortfolio as a whole, as well as the impact of the
their1) smaller programs. In the second, the deans were requested approach on their professional learning activities. Data regarding
to seek department chairs of two to three larger programs. The first the orientation sessions and the composition and usefulness of
pilot was carried out from September–December 2011 with 32 individual feedback tools are not included in this paper.
participants (including 5 chairs and 27 faculty members) from five In collaboration with the taskforce, the researchers created a
programs in trades, health sciences, business, and media and survey and interview guide aimed at collecting feedback on the
design. The second pilot was carried out from September– orientation to the ePortfolio, the feedback tools developed, and the
December 2012 with 75 participants from 12 programs in ePortfolio approach as a whole. An interview guide was developed
9 departments2 including programs in trades, engineering, applied to collect more detailed feedback on these elements and obtain a
sciences, and health. One program participated in both pilots. The more in-depth understanding of the impact of the ePortfolio
total number of participating departments was 13: four in the first approach on faculty learning. Based on the responses from the first
pilot, eight in the second, and one that participated in both. pilot, we also evaluated our survey questions. The survey used for
In the first pilot, all participants, including the department the first pilot had many open-ended questions. The answers to
chairs, attended the same orientation to the approach during two those questions were not always easy to interpret, and not all
half-days in September of interactive introductory sessions participants answered. For instance, one question read as follows:
organized by the taskforce. The approach was just as new to the How do you feel about the whole experience of collecting feedback,
department chairs as to the instructors. During the orientation, an reflecting on it, and documenting it? One participant answered
open conversation between a faculty member and a chair with ‘‘Informative.’’ Another wrote: ‘‘We are here because we are
regarding feedback of the faculty member was modeled. In the good at our trade. This generally means we are not all great
second pilot, departments received an orientation within their writers.’’ The first answer is so general that it’s hard to interpret
own program. In the orientation of both pilots, participants were what exactly this participant found informative; the second
introduced to the purpose of the ePortfolio approach: to support response most likely points to a reaction to the encouragement
faculty development and performance evaluation. They were to include written reflections. This question was amended to
presented with an overview of the philosophy behind the include more closed-ended questions, such as (1) I found the
approach, including the expanding role of faculty, eight aspects ePortfolio approach helpful for my professional development. (2)
of the faculty role, the reasons why feedback should be collected The ePortfolio approach is a useful way to set learning goals. While
from multiple sources, the value of reflective practice, and the role these amended questions yielded better data for evaluating the
the department chair could play in supporting formative perfor- second pilot, these changes prevented us from comparing the pilot
1 survey data with the pilot 2 survey data. The second pilot survey
1
Some department chairs oversee more than one program.
also included more closed-ended questions pertaining to concerns
2
One participating department included instructors who teach in three separate the taskforce had heard about anecdotally, such as a lack of trust in
programs. the security of the Online data. The interview questions were the

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13

A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx 5

same as for the first pilot and are listed in Online supplement 2. The perceived value of ePortfolio approach, (3) impact of adoption of
survey questions are presented in Online supplement 3. ePortfolio approach. Interviews from the first pilot were coded by
two researchers, in most cases reaching a minimum of 85%
Data collection agreement, and discrepancies were discussed. Interviews from the
second pilot were analyzed by one researcher, using the same
The first pilot was carried out from September–December 2011 themes. For the analysis of interview data from both pilots, a
with 32 participants from five programs, including programs in matrix was created with each column representing a theme and
trades, health sciences, business, and media and design. Before the each row, a participant. Interview excerpts were copied into the
start of data collection, research ethics approval was obtained from cells or, in the case of very large excerpts, summarized in the cells.
the institute’s research ethics board. This board verifies that This tool allowed the researchers to compare interview excerpts
proposed recruitment and data collection procedures are in relating to one theme across the participants. Based on these
accordance with In Canadian Tri-Council Framework for Research comparisons, we created categories across participants as well as a
involving Human Subjects. Data collection took place in early count of how many participants would fall into this category. For
2012. To invite faculty to the survey and to select interview instance, the theme adoption of ePortfolio approach included three
participants, the chairs of the participating programs were categories: (1) no portfolio created, (2) portfolio page with some
approached to provide a list of their participating faculty members. documents uploaded, and (3) more elaborate ePortfolio. See Online
From this list, 1–2 faculty members per participating program supplement 4 for a list of coding categories and the number of
were selected to be interviewed using a random number generator. interviewees categories apply to. Because only two interviewees in
In programs with more than six faculty members, two participants the second pilot created more than one page in their portfolio, we
were selected to be interviewed. In the first pilot, we interviewed limited the analysis of the second pilot interviews to the first two
seven randomly selected participants: one from each of the three research questions on adoption and perception of value of the
smaller programs (<6 faculty members) and two from the two ePortfolio approach. In the next section, we answer the research
larger programs (6 members). We also interviewed each of the questions by presenting survey and interview data. We describe
five department chairs. We thus had 12 interview participants in the categories developed in our interview analysis, how many
pilot 1: seven faculty members and five department chairs. Of all participants this category applied to, and examples of interview
32 pilot 1 participants who were invited to anonymously respond excerpts.
to an online survey, 26 responded. The decision to collect
anonymous feedback was made in an attempt to increase the Findings
chances of receiving honest feedback. Because a number of
participating programs were small (3 faculty members) the survey This section presents the findings in the order of our research
did not ask participants which program they were part of since that questions: (1) To what extent was the ePortfolio adopted in the
information might identify them. programs participating in the first and second pilot? (2) To what
The second pilot was carried out from September–December extent did faculty in the first and second pilot value the ePortfolio
2012 with 75 participants from 12 programs in 9 departments3 approach as a whole? (3) What was the impact of the ePortfolio
including programs in trades, engineering, applied sciences, and approach on the professional development of participating faculty
health. One program participated in both pilots. Data collection members?
took place in early 2013. We used the same sampling methods as in
pilot 1 to select interviewees. Three of the selected faculty The extent of ePortfolio adoption in pilot 1
members declined to participate. Research ethics guidelines
stipulate that those approached to participate in the study may Using survey and interview data, the extent of the adoption of
‘‘opt out’’ of the data collection without having to provide a reason. the ePortfolio approach was assessed in terms of whether or not
We were therefore unable to establish their reasons for declining. the participants reported collecting feedback and/or compiling
Using a random number generator, alternative participants were that feedback into their ePortfolio, reflecting on it, sharing it with
selected from the remaining faculty members in those depart- their supervisor, and making changes if necessary. Most of the first
ments. Unfortunately, in one small program, the freedom to opt out pilot participants reported collecting some type of feedback.
meant that we could not interview anyone. As a result, 14 Table 1 provides the frequency overviews related to two survey
participants from 11 programs from the second pilot were questions: (1) Is this aspect4 part of your job? (2) Who did you
interviewed. Time constraints prevented us from interviewing collect feedback from regarding this aspect of your job? These
the department chairs. All 75 pilot 2 participants were invited to results show that the majority of survey respondents (22 out of 26)
fill out the online pilot 2 survey. A reminder was sent two weeks were teaching during the pilot; cross-tabulation confirms that 20
after the survey was distributed, resulting in a total of of these 22 respondents collected student feedback on their
42 responses. teaching. Examination of the raw data file indicates that one survey
respondent chose not to fill out any of the questions listed in
Data analysis Table 1. Similar cross-tabulations show that all respondents who
collected feedback on certain aspects of their job also indicated
The responses to closed survey questions were summarized that those aspects were in fact part of their job. Table 1
into frequency overviews. The responses to open survey questions demonstrates that quite a few respondents collected feedback
were categorized and summarized. This paper only reports the from peers on job aspects that applied to them. For instance, 13 out
results from the survey questions relating to the adoption of of 23 respondents to whom Scholarship/Professional Development
the ePortfolio approach and faculty members’ perception of the applied sought peer feedback.
value and impact of the approach. Table 2 shows that the majority of participants (69%) collected
Interviews were transcribed verbatim. Interview excerpts from more and/or different types of feedback in the pilot than they did
the first pilot were organized according to the research questions before the pilot. Of the five comments to this question, three
of this paper using the themes (1) adoption of ePortfolio, (2)
4
These aspects were based on the eight aspects of the role of a faculty member as
3
One department offered three separate programs. identified in the stakeholder consultation, which is described in the Method section.

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13

6 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

Table 1
Aspects part of respondents’ job and who they collected feedback from regarding that aspect.

Part of job? Feedback on this aspect collected from:

Yes No Peersa Studentsa Supervisora Othera Nobody Not Applicable

Teaching 79% (22) 11% (3) 21% (6) 71% (20) 14% (4) 4% (1) 0% (0) 7% (2)
Professionalism 86% (24) 4% (1) 50% (14) 21% (6) 39% (11) 7% (2) 4% (1) 4% (1)
Corporate citizenship 68% (19) 18% (5) 14% (4) 4% (1) 11% (3) 4% (1) 29% (8) 14% (4)
Scholarship/professional development 82% (23) 7% (2) 46% (13) 0% (0) 18% (5) 4% (1) 29% (8) 0% (0)
Applied research 14% (4) 64% (18) 4% (1) 0% (0) 0% (0) 0% (0) 7% (2) 46% (13)
Formal curriculum development 64% (18) 21% (6) 29% (8) 7% (2) 25% (7) 0% (0) 18% (5) 11% (3)
Formal administrative duties 36% (10) 46% (13) 18% (5) 4% (1) 11% (3) 0% (0) 7% (2) 25% (7)
Formal leadership 36% (10) 46% (13) 29% (8) 7% (2) 14% (4) 0% (0) 4% (1) 32% (9)
a
Respondents were able to select more than one response regarding data source for each aspect of their job.

Table 2 commented: ‘‘I wasn’t comfortable enough with the software,


Responses to the question: As a result of the pilot, did you collect more/different
right. [. . .] I guess it just seemed like it was a lot of effort that I
feedback than you used to?
wasn’t prepared to put in, to be honest with you.’’ When asked
More feedback Different feedback whether s/he discussed the ePortfolios in a supervisory role with
Yes No Total her/his staff, Addy replied: ‘‘No, because I don’t think their
response to it was any better than mine. [. . .] I spoke to them. I did
Yes 35% (9) 27% (7) 16
No 8% (2) 31% (8) 10 speak to them about it and ask them, and nobody had [made an
ePortfolio].’’ In some programs, the interviews with multiple
Total 11 15 26 faculty from the same programs revealed that the extent to which
the approach was adopted differed from one faculty member to
the next.
indicated the difference was the collecting of peer feedback. The
other comments stated ‘‘no difference.’’ Only 31% reported that ePortfolio adoption in pilot 2
they collected neither more nor different feedback than before.
As for the documentation of feedback, growth, and develop- The second pilot included 75 participants, 42 of which
ment, most pilot participants created an ePortfolio page using responded to the survey. Table 5 shows that of the 37 participants
Mahara during a 1-h workshop in the computer lab. Table 3 shows who chose to respond to this question, half (49%) spent 2 h or less
that 77% (35% + 42%) of respondents spent 5 h or less on on the ePortfolio approach. This likely represents the time spent in
documenting growth and development, with nine of them the orientation and Mahara workshop. The other respondents
spending less than 1 h. spent more time, but only 16% spent more than 5 h on the
Participants were also asked whether they included their ePortfolio approach.
feedback, reflections, and evidence of accomplishments in Mahara. Table 6 shows participants’ answers to the question, Do you
Table 4 shows the majority included only some or none in their believe you sufficiently understand the purpose and principles
ePortfolios. Two respondents chose to include comments to this underlying the ePortfolio approach? Only half of the participants
question; one indicated insufficient time, and the other asserted (48%) claimed to understand the purpose of the ePortfolio
that building the ePortfolio seemed to be too much work. approach.
The interviews confirm the survey findings, in that the majority When asked whether respondents used Mahara as an ePortfolio
(9 out of 12) of interviewees collected feedback. Seven out of tool, 22 (52%) responded ‘‘yes,’’ 20 (48%) responded ‘‘no.’’ Those
12 interviewees created a portfolio, although only two created a who responded ‘‘yes’’ were directed to the question of whether
more elaborate portfolio. From the interviews with faculty and they found Mahara easy to use. Of those who used Mahara, eight
chairs, we learned that each department adopted the ePortfolio (36%) found it easy to use whereas the remaining 14 (64%) did not
approach to a different degree during the first pilot. Two chairs did find it easy to use. Ten out of the 22 respondents (46%) who used
not discuss and were not planning to discuss the portfolio with the Mahara said that they did not feel confident that their pages were
faculty members who report to them. Three chairs had these private. None of the pilot 1 participants had voiced these concerns.
discussions with some faculty members and were planning to sit In the five written comments to this question, one respondent
down with the remaining faculty members. commented that every server can be hacked. Two comments also
The five programs differed in the extent to which the ePortfolio indicated concerns about uploading personal information online.
approach was adopted. For example, one program had all its One respondent did not see the point of broadcasting professional
faculty members create and share their ePortfolios with one achievements online; a fifth did not see a need for another online
another. These ePortfolios included student and leadership social media tool.
feedback. Chair Kim5, for instance, reported that, regarding the Cross-tabulation of the question regarding understanding of the
ePortfolios, ‘‘We’ve had discussions as a group; we’ve had purpose and principles of the ePortfolio approach with the
discussions as individuals’’ as s/he met with her/his staff to view question of whether respondents used Mahara (see Table 6)
the contents of their ePortfolios. On the other hand, in one indicates no apparent relation between perceived understanding
program, all faculty had tried out the student survey and the of the approach and the use of Mahara. A cross-tabulation of the
leadership survey, no one in the program had been able to create an same question with the question of whether participants found the
ePortfolio. They also did not have supervisor-staff conversations or approach helpful for professional development (PD)(see Table 6)
a staff meeting about the ePortfolio. Addy, the chair of this program indicated that those who expressed uncertainty in understanding
showed a higher percentage of respondents who were undecided
5
To protect the identity of participants, all names used in this paper are gender- regarding its helpfulness. Other than that, understanding of the
neutral pseudonyms. purpose and principles does not seem related to whether or not

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13

A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx 7

Table 3
Responses to the question: How much time did you spend on the following aspects of the ePortfolio approach?

<1 h 1–5 h 5–10 h 10–20 h >20 h

Learning to use Mahara 20% (5) 72% (18) 4% (1) 4% (1) 0% (0)
Collecting and reflecting on feedback 16% (4) 48% (12) 28% (7) 8% (2) 0% (0)
Documenting your growth and development 35% (9) 42% (11) 8% (2) 15% (4) 0% (0)
(including writing reflections, uploading in Mahara, etc.)

participants found the approach helpful. The participant who your plans for the department.’’ Four participants thought that the
indicated not understanding the ePortfolio approach did not ePortfolio approach was a good opportunity to reflect on
respond to the question regarding its helpfulness. performance for the purpose of identifying any gaps or areas for
The interviews provided more insight into the reasons for the future growth. Three participants felt the ePortfolio approach was
poor adoption of the approach in pilot 2. Fourteen participants helpful in making supervisors aware of the work their faculty
were interviewed; all reported placing a high value on receiving members are involved in. For example, Jamie expressed:
and learning from student feedback, yet only five of the 14 created
And that’s where I do see the point where the supervisor might
an ePortfolio. Of the nine participants who did not build an
see that, Oh, wow. You’ve done this. Have you done this? Have
ePortfolio, two missed the orientation, and one stated lack of
you done this? You know they had forgotten, or they did not
interest due to imminent retirement. The other six indicated
know because they’re busy managing their department.
limited time and did not see the creation of an ePortfolio as a
priority. Three interviewees put a few documents into Mahara, Three participants indicated it was nice to get good perfor-
such as an ePortfolio page and some teaching materials. The mance reaffirmed. Addy: ‘‘It reassured me that I was doing some
remaining two interviewees built a more comprehensive ePortfo- things right.’’ After receiving feedback using the department
lio, including some documentation of achievements and projects survey for chairs, chair MacKenzie expressed that the ePortfolio
they were working on. None of the interviewees included student process was a ‘‘good opportunity for the leaders to ask for
feedback in their ePortfolio. These findings suggest that adoption assistance’’ from their own supervisors.
of the approach in pilot 2 was less than during pilot 1. A sub-theme of perceived value of ePortfolio approach was the
view of Mahara, the online software used by the participants to
Faculty perception of value of the ePortfolio approach in pilot 1 create their ePortfolios. One participant did not access or use
Mahara at all and therefore did not express any views on the
In response to the question of whether participants found the software. Two participants felt that Mahara was difficult to learn.
ePortfolio approach helpful for their professional development Neither of these participants used Mahara to create an ePortfolio
(see Table 7), 47% found the approach helpful or very helpful; 46% because neither felt comfortable with the software. Addy explains:
found the approach not so helpful; 8% found the approach not
helpful at all. Back to Mahara, I guess I’m not a social media person. I don’t use
Nineteen out of the 26 respondents provided an answer to the Facebook. I don’t use MyPage or any of that, so I really wasn’t
open-ended question: What did you find helpful and/or not helpful invested in it. If I was applying for a different job, I could see it,
about the ePortfolio approach? Seven indicated they found it right. I wasn’t comfortable enough to use it for, like, a
helpful that the approach encouraged reflective practice. [performance review] presentation to my boss.
Responses in this category included ‘‘making reflection and action Four participants did not see much added value in Mahara. Two
more deliberate and focused was helpful,’’ and ‘‘allows you to participants did not like the process of uploading paper
reflect on what you are doing.’’ An additional three respondents documentation to Mahara’s web site. Alex explains: ‘‘I think it
indicated they appreciated how the ePortfolio allowed them to would be just much easier just to have a folder and have a file in
collate relevant documents where the supervisor could see them; your hard drive just to deposit stuff.’’ Other participants felt that
it was ‘‘helpful that everything could be in one place.’’ Four of the Mahara was too much work to learn. For example, Jamie
19 comments indicated that participants found the approach expressed: ‘‘It did more than I needed it to do, and it was one
unhelpful, redundant, or cumbersome though none provided a more piece of software that I need to learn.’’ Despite these
rationale. The remaining five responses commented on the use of concerns, three of these participants created an ePortfolio using the
Mahara and the need for more training or practice. The interview software. Finally, five participants thought Mahara was easy to
data provided more in-depth insight into the faculty experiences. learn. For example, Tracy stated: ‘‘I think the tool was easy.’’
From the 12 participants interviewed, three did not see any
benefit or added value in the ePortfolio approach, either because Faculty perception of value of ePortfolio approach in pilot 2
they were not interested in learning due to imminent retirement or
because the approach was similar to the learning processes they The participants in the second pilot saw less value in the
usually engaged in. Nine participants saw added value in the ePortfolio approach, and they raised a number of concerns that
ePortfolio approach. Five of those nine thought the ePortfolio was a were absent in the first pilot. Table 8 shows that approximately a
good opportunity to store feedback, reflections, and achievements quarter of the participants found the ePortfolio approach helpful
in one place. Leslie liked the ePortfolio because ‘‘It is good to have a for professional development, setting learning goals, assessing
place to put all that kind of stuff down, your goals and even just performance, and receiving coaching and support from their
supervisor. The majority of respondents were undecided or did not
Table 4 find the approach useful or helpful at all.
Responses to the question: Did you include your feedback, reflections, and evidence Ten participants took the opportunity to provide written
of accomplishments in Mahara? comments to the questions in Table 8. They indicated that the
Response Yes Most of it Some of it No ePortfolio was introduced at the wrong time (five comments) and
that they were uncomfortable with posting information online
Yes 8% (2) 19% (5) 46% (12) 27% (7)
(one comment). Three commented that the ePortfolio is not a good

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13

8 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

Table 5
Time spent on ePortfolio approach in pilot 2.

<1 h 1–2 h 3–5 h 6–10 h >10 h

How much time did you spend on the ePortfolio approach 27% (10) 22% (8) 35% (13) 5% (2) 11% (4)
as a whole, including setting learning goals, collecting
feedback, reflecting on feedback, and documenting these
in Mahara?

Table 6
Cross-tabulation of participants’ perception of their understanding of the purpose and principles underlying the ePortfolio approach, and the question: ‘‘Did you use Mahara
as an ePortfolio tool?’’.

Response Used Mahara

Yes No Total Strongly Agree Undecided Disagree Strongly Disagree N/Aa


Agree

I’m sure I understand the purpose 24% (10) 24% (10) 48% (20) 11% (2) 21% (4) 16% (3) 16% (3) 32% (6) 5% (1)
and principles entirely
I’m not sure I entirely understand 29% (12) 21% (9) 50% (21) 10% (2) 5% (1) 35% (7) 15% (3) 20% (4) 15% (3)
the purpose and principles
I don’t understand the purpose 0% (0) 2% (1) 2% (1) (0) (0) (0) (0) (0) (0)
and principles of the ePortfolio
approach

Total 52% (22) 48% (20) 100% (42) 10% (4) 13% (5) 26% (10) 15% (6) 26% (10) 10% (4)
a
N/A = Not Applicable, SA = Strongly Agree, A = Agree, U = Undecided, D = Disagree, SD = Strongly Disagree.

use of time because only the format and not the content was expressed mostly concerns with the approach. Francis explains
discussed with the chair (one comment) or because they prefer to how s/he normally goes about improving practice and how an
use a different tool for professional development (two comments). ePortfolio would help in this process:
The remaining comment questioned whether the ePortfolio
I’m usually quite cognizant of getting feedback and reflecting.
approach would actually add to faculty accountability or not:
[. . .]. So I generally do it quite often, and I do it verbally quite a
‘‘We can choose what our supervisor sees, so we can just pick the
bit. [. . .] Sometimes I have the students kind of just write on a
best survey [results].’’
little sticky note what worked, what didn’t. [. . .] I kind of run
Of the 14 interview participants, only one interviewee, Dallas,
some exercises by some colleagues before I try them in the class
expressed seeing a clear benefit to the approach. S/he believed in
to see if it would work or not, [. . .] And then I create lesson plans
the ePortfolio’s abilities to create an overview of accomplishments
where I have little [. . .] At the bottom I just have a little area for
and support reflection:
comments[. . .] And the purpose is eventually to be able to [. . .]
I think, you know, a few years down the road it will be a nice kind of put that into some type of portfolio as part of the
timeline. Oh, I was working on this, [. . .] I’ve worked more into – reflection. Hasn’t happened yet, yeah.
I’ve been helping with some research here and this here, you
While s/he values reflection, Francis did not create an ePortfolio
know. So I think it will be an excellent tool for me to reflect on
online:
what I’ve been doing.
So I – like I said, the student feedback came in. I looked at it, I
Dallas wished for more time to work on the ePortfolio and was
addressed it personally, and I addressed it in the classroom.
eager to continue using the approach. The other four interview
[. . .]. I didn’t do anything with that feedback with the ePortfolio,
participants who had created at least one ePortfolio page
unfortunately. I kind of had a bit of an – I guess a reservation

Table 7
Responses to the question about helpfulness of the ePortfolio approach.

Very helpful Helpful Not so helpful Not helpful at all

To what extent did you find the ePortfolio approach helpful 12% (3) 35% (9) 46% (12) 8% (2)
for your own professional development?

Table 8
Responses to the question: Please indicate to what extent you agree with the following statements.

Strongly Agree Undecided Disagree Strongly Not


Agree disagree applicable

I like the ePortfolio approach 10% (4) 18% (7) 33% (13) 8% (3) 26% (10) 5% (2)
I found the ePortfolio approach helpful for my professional 10% (4) 13% (5) 26% (10) 15% (6) 26% (10) 10% (4)
development
The ePortfolio approach is a useful way to set learning goals 3% (1) 21% (8) 28% (11) 13% (5) 28% (11) 8% (3)
The ePortfolio approach is a useful way to assess performance 3% (1) 18% (7) 23% (9) 25% (10) 25% (10) 8% (3)
The ePortfolio is a useful way to receive coaching and support 3% (1) 20% (8) 30% (12) 15% (6) 25% (10) 8% (3)
from my supervisor

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13

A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx 9

about the ePortfolio about the fact that it’s online. I’m not really And the feedback I got was very positive. Nothing that surprised
comfortable with putting information online. me because I talked to my students all the time, and I know
where they’re at. So I didn’t personally find that feedback to be
The interviewees who did not create an ePortfolio were not
particularly of value to me.
asked what they found helpful about the approach. One of the
interviewees volunteered his expectation that it the approach Of the 12 pilot 1 participants interviewed, two indicated they
would be useful to keep a record. did not adopt the approach, experienced no impact, and did not
A total of six of the interviewees reported concern over putting intend to use the approach. A second group of two participants
their information online. Robin, one of these six, also expressed reported that, among other things, the approach helped make their
discomfort with having to learn a new software tool. Quinn would chair aware of all the work they did. For instance, Jamie reported:
rather the information be stored on a server hosted at the institute:
And there’s a reason why I’m putting all this stuff in. Because I’m
‘‘And then the other thing is who’s going to see my information in
pretty sure our supervisor [the department chair] has a pretty
the long-term? How secure is it? Will my students have – be able to
good idea of what we’re doing, but sometimes he/she might not
hack into it or access it or whatever?’’ Harper wondered whether
remember just all the work that all of us staff do.
the information might be accessed by upper management who
might use it for salary decisions. While the taskforce had not Six participants reported their participation in the ePortfolio
wanted to be prescriptive in what participants should include in approach has helped them become aware of areas of improvement
their ePortfolio, three interviewees expressed they were unsure to their practice. Chair MacKenzie reported, for instance, ‘‘It helped
what an ePortfolio might look like and expressed a desire for me pinpoint things that I need to work on for sure, yeah.’’ Addy,
direction. Kelly indicated: ‘‘For sure, templates would help and, also a chair, described an improvement in practice as a result of
you know, without knowing the full details on ePortfolios, but feedback received:
templates and what’s expected throughout the whole and what
So I may talk to one associate chair and not the other. So one
you expect the finished product to look like.’’ Robin expressed
hand doesn’t know what the other hand is doing. So with that in
questions about ways to provide evidence of interpersonal skills:
mind, then I try to be more inclusive now [that I received that
And is it a true evaluation of my skills, I guess, is the other thing. feedback] because I really wasn’t that conscious of it.
I just kind of feel – I think I bring something unique to the
Three participants reported that the ePortfolio approach made
classroom. I think I’m really personable with my students. I care
them think more explicitly about actively planning their profes-
a lot about them. I personally talk to them a lot. I end up being a
sional development. For instance, Jean indicated: ‘‘I think the one
counsellor most days. But, again, I don’t know how that – how
thing it did for me was, it made me think, ‘‘Do I document enough?’’
those two mesh.
Leslie commented:
Interviewer: So how can you show what your strengths are if But, yeah, it does help me to outline my goals. Definitely it
they’re not in an actual project that you can post? helped me to identify new ideas and kind of have a place to put
them down and things that I want to do with the program to,
Robin: Right. Exactly. you know – in terms of organization of the second year, clinical
experience, different ways that we can [keep in touch with
Perceived impact of ePortfolio approach in pilot 1 students]– because our students are so far away.
Finally, Jordan indicated:
After pilot 1, when asked whether they used the feedback they
collected per aspect to improve their performance, more than 75% Now I’ve got things that I’m thinking, I could do that to get
of respondents to whom this aspect applied answered affirmative- better feedback, or I could do this. So it helped me think about
ly, with the exception of corporate citizenship and applied research what I’m going to do in the future for my professional
(see Table 9). development, definitely. It also identifies what I could probably
Although respondents who replied ‘‘no’’ did not provide an use more training on. So it kind of helps me plan.
explanation, at least one participant, Jean, indicated the feedback
was all positive and therefore not used to improve work: Perceived impact of ePortfolio approach in pilot 2

As indicated previously, the adoption of the ePortfolio approach


Table 9 in the second pilot was limited. Because only one of the
Responses to the question: Did you use the feedbackb to improve your work? interviewees in pilot 2 had actually developed an ePortfolio, we
Yesa No Not applicable did not analyze the interviews for impact of the ePortfolio on
professional development. Our survey data (Table 10) show that
Teaching 81% (17) 19% (4) (2)
Professionalism 85% (17) 15% (3) (2)
similar to the first pilot, a majority of participants reported the
Corporate citizenship 56% (5) 44% (4) (11) feedback they collected impacted their practice.
Scholarship/professional 86% (12) 14% (2) (7)
development
Applied research 33% (1) 67% (2) (13)
Discussion
Formal curriculum 91% (10) 9% (1) (6)
development This paper reports on the design, implementation, and
Formal administrative 75% (6) 25% (2) (8) evaluation of two pilot studies of a faculty development and
duties
performance evaluation initiative called the ePortfolio approach at
Formal leadership 89% (8) 11% (1) (9)
a Western Canadian institute for vocational education. In a
a
Respondents could only select one option: yes, no, or not applicable. One response to calls for more detailed description of faculty
respondent did not provide any answer while some did not provide an answer for
some of these aspects.
development initiatives (Steinert et al., 2006; Stes et al., 2010)
b
This question was attached to the questions in Table 1; thus ‘the feedback’ design principles were described as well as the stakeholder
refers to the type of feedback the participants collected as indicated in Table 1. consultation process used. The purpose of the ePortfolio approach

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13

10 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

Table 10
Responses to the question: Please indicate to what extent you agree with the following statements.

Strongly agree Agree Undecided Disagree Strongly disagree Not applicable

The feedback I received in the summary student 7% (2) 57% (16) 21% (6) 14% (4) 0% (0) 0% (0)
feedback report influenced the way I teach
The feedback collected through the chair survey 0% (0) 57% (4) 29% (2) 0% (0) 14% (1) 0% (0)
influenced the way I supervise

was to foster naturally occurring professional learning activities of portfolio was a key contextual factor affecting the acceptance and
participating faculty members. uptake of portfolios by professionals. Goliath (2009) studied
The ePortfolio approach was piloted in two rounds. The adoption of an ePortfolio by physicians. Similar to our findings, she
research questions addressed in this paper were: (1) To what described that individual factors such as comfort level with
extent was the ePortfolio adopted in the programs participating in technology influenced participants’ adoption of the ePortfolio
the first and second pilot? (2) To what extent did faculty in the first technology. Another reason for limited adoption that was
and second pilot value the ePortfolio approach as a whole? (3) mentioned by many participants in the second pilot was a lack
What was the impact of the ePortfolio approach on the of time. During the second pilot, an institute-wide curriculum
professional development of participating faculty members? This redevelopment process was implemented. This has, at least in part,
section discusses our main findings, lessons learned, and implica- accounted for the fact that adoption was greater in the first pilot
tions for research. than in the second.
Aside from a lack of time or interest, the participants provided
Added value of feedback other reasons for why they chose not to build an ePortfolio. A
number of them found it cumbersome to learn the software,
In terms of the adoption of the ePortfolio approach, the majority whereas others expressed concerns about the privacy and security
of participants in both pilots indicated that they collected of Mahara, a lack of trust in the organization where online
feedback, reflected on that feedback, and used it to improve their information may be accessed and used for purposes other than
practice. They most likely would have collected some type of formative assessment, and a lack of clarity about what kind of
feedback if they had not been participating in the pilot; however, information could be included in an ePortfolio. These concerns
the results suggest that due to the pilot, they collected more and/or could have been alleviated with additional preparation and
different types of feedback than they did previously. The support sessions. The need for advance preparation is echoed by
participating department chairs reported finding value in gather- Blair (2001), who identified separate workshops on using teaching
ing feedback by surveying the faculty in their department portfolios for professional development and creating an electronic
regarding their responsibilities as a chair, a practice that previously portfolio as being key to the success of an ePortfolio as a faculty
had not been common throughout the institute. All participants development tool. Smith and Tillema (2003) also pointed to the
interviewed explained that they greatly valued learning from importance of clarity of expectations: ‘‘The trust in the portfolio
feedback, and many described ways in which they collected and was strengthened when there were explicit criteria for the content
discussed feedback with students in order to improve the students’ of the portfolio, how to go about it and of how the portfolio was
learning experience. This concurs with similar findings by Smith going to affect their professional status’’ (p. 644). This finding also
and Tillema, who concluded: confirms the need for clear performance expectations (e.g.,
Higgerson, 2006).
Another aspect which attracted much attention in the face-to-
Despite the challenges with the tool, participants reported that
face interviews was the importance of feedback during the
creating an ePortfolio was an opportunity for them to:
portfolio collection process. In three out of the four settings
(dossier, training and reflective portfolios) external feedback is
(1) Identify issues and ask assistance from supervisor in dealing
seen as a positive and essential part of the process. (p. 646)
with these issues.
By encouraging faculty to collect more and different types of (2) Collect more and different types of feedback than before.
feedback, the ePortfolio approach did indeed foster a naturally (3) Store learning goals, achievements and other information
occurring professional learning process of faculty. relevant to their development in one place.
(4) Store feedback and accomplishments over time, which allows
Creating an ePortfolio one to look back and reflect on performance and also allows for
the formulation of goals for future improvement.
In both pilots, less than half of the participants who responded (5) Show the supervisor all the different tasks one is involved in.
to the survey actually documented their feedback and reflections
in the online ePortfolio tool Mahara, though the adoption of the The above benefits reported by our participants are consistent
ePortfolio approach was greater in the first pilot than in the second. with Granberg’s (2010) findings, which established that student
Our participants’ weariness of the portfolio software and the fact teachers expressed benefits of using ePortfolios as: portfolios for
that it exists online conflicts with Freeman et al. (2009) who report learning (opportunities 1 & 2 discussed above), portfolios as an
the majority of their study participants had positive perceptions of archive (3 & 4), and portfolios for assessment (5).
their ePortfolio technology. This difference in reception could be
explained by the fact that the participants in the pilot Freeman Impact of the ePortfolio approach
et al. conducted all volunteered to use the ePortfolio software. Our
participants, on the other hand, were included in the pilot because One perceived impact of the approach involved an increased
the department chairs agreed for the faculty in an entire program awareness by participants of areas of practice that could use
to participate in the pilot. This concurs with Smith and Tillema’s improvement. Participants also thought more explicitly about
(2003) findings that the mandatory or voluntary nature of a planning their professional development. These findings suggest

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13

A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx 11

that if adopted fully, the ePortfolio approach has the potential to Fourthly, a four-month period might be too short to really try
enhance self-directed learning. This concurs with the findings of out the ePortfolio approach. Since the approach assumes a yearly
Goliath (2009) who found that all nine physicians in her study cycle, and given that in the first cycle participants and department
scored higher on the readiness scale for self-directed learning as a chairs are still learning the process, a two-year pilot might be more
result of creating an ePortfolio. It also concurs with the impact useful and could include an initial orientation and a number of
Smith and Tillema (2003) found in that the personal development follow up sessions. Regular formative feedback from the pilot
type of portfolio gave ‘‘participants space and opportunity for self- participants on the approach itself, as well as the support sessions,
directed learning, which was valued as contributing to professional could allow the faculty development support staff and the
growth’’ (p. 645). More research is needed to study whether these department chairs to address concerns as they arise.
impacts can be sustained over time.
Limitations
Design and implementation, lessons learned
The study of the impact of the ePortfolio approach was limited
Many faculty development initiatives serve a limited number of to self-reported outcomes described in interviews. The findings
faculty members over a limited amount of time (e.g., Stes et al., regarding the impact of the approach are limited because few
2010). Because of its potential to enhance reflective practice and participants actually adopted the approach in the way it was
self-directed learning (e.g., Goliath, 2009), an initiative like the intended. The interview guide and survey used were designed to
ePortfolio approach could be considered as a way to systematically directly inform practice and required data collection, analysis, and
support the professional learning of all faculty members and over a reporting to be completed within three months. Studying the
prolonged period of time, possibly throughout their entire career at participants’ ePortfolios themselves could have increased our
the institute. However, upon the evaluation of the two pilots of the insight into the impact of the approach on professional learning,
ePortfolio approach, questions remain about whether the ePortfo- but this was considered to create an artificial environment where
lio would be suitable for institute-wide implementation and, if so, participants might feel the need to create an ePortfolio just for the
how it should be implemented. researchers. In addition, it might have created undue stress in
First of all, based on the findings, the question arises whether participants as they might be afraid the researchers would evaluate
the ePortfolio approach should be maintained in its current form. their performance rather than the impact of the ePortfolio on their
Participants indicated learning the ePortfolio software, Mahara, is professional learning. Other limitations of our methods include the
cumbersome. In addition, they were wary of posting information fact that the pilot 2 survey included improved survey questions,
online. Sharing portfolio documentation with their department which allowed the taskforce to gain more detailed insight into
chair does not require the use of ePortfolio software, nor does it participants’ experiences, but unfortunately prevented the
require online storage. The literature suggests that composing an researchers from comparing survey data across pilots. Finally,
ePortfolio allows for enhanced reflective practice (e.g., Lamont, our recruitment method for the survey in both pilots, and the
2007), but the question is whether the extent to which this interviews in the second pilot, may have contributed to sample
reflection is enhanced by the use of the software warrants the costs bias. For the second pilot, we know that non-response was most
involved in using that software. These costs include online storage likely due to lack of time. However, we do not have conclusive
fees for third party storage provider, cost of technical support, cost information about this, as research ethics procedures stipulate that
of development and delivery of faculty workshops, as well as the survey and interview participation is voluntary, and no reason has
time spent by faculty to learn to use the software. Our pilot data did to be provided for not responding.
not suggest that the software added to the quality of reflection but
that it was the act of compiling documents that encouraged Implications for research
reflection on and formulating goals for professional development.
A simplified, less costly version of an ePortfolio could potentially While the use of portfolios for student assessment and student
serve the same purpose. This simplified version could, for instance, learning is taking hold in several disciplines, its use in supporting
include a template with questions for reflection and performance ongoing professional learning of post-secondary educators so far
assessment to be filled out by faculty, along with a number of has been limited. Yet, the literature suggests it could be a valuable
appendices including feedback, reflections, teaching materials, and approach to supporting faculty learning throughout their careers.
documentation of achievements and other records of performance. Further studies are needed to determine the organizational
Faculty could compile and store these documents on their own conditions facilitating the implementation of the ePortfolio
computers and show them to their chair during a performance approach, including issues of power and human resources
assessment meeting. processes. In addition, studies are needed to determine the design
Secondly, the findings of both pilots point to lack of time. Wider requirements for the ePortfolio approach that optimally facilitates
implementation of the ePortfolio approach would need to be self-reflection and self-directed learning, requires minimal learn-
carefully timed and not coincide with other drastic institute-wide ing on the part of the faculty, and facilitates performance
innovations. assessment. Carefully designed studies investigating the impact
Thirdly, our findings also highlight how the supervisors and of a well-designed ePortfolio tool accompanied by ePortfolio
their understanding of the approach played a role in determining assessment processes should be able to shed light on the impacts of
the need of their faculty for support and in setting expectations for such an approach on faculty development and ultimately on
their department members. Because of the important role they students’ learning experience.
play, department chairs might need more guidance and prepara-
tion before the approach is introduced in their departments. In Conclusion
general, department chairs might play a larger role in fostering
faculty development: for instance, by encouraging transfer of Our study suggests that an ePortfolio approach to faculty
learning in courses and workshops into the workplace. (See, for development and performance evaluation has the potential to
instance, Lancaster, Di Milia, & Cameron (2012). Chairs would need support collection of various types of feedback, increase awareness
to set clear expectations for both the content of the ePortfolio as of areas of improvement, and encourage more explicit thought
well as how this content will be assessed. about planning and monitoring of professional development.

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13

12 A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx

When adopted by a whole organization, the approach could Flanigan, E. J. (2012). ePortfolios and technology: Customized for careers. International
Journal of Information and Communication Technology Education, 8(4), 29–37.
integrate professional development plans, student feedback Freeman, L., Harkness, M., & Urbaczewski, A. (2009). International Journal of Global
practices and other feedback on performance, as well as Management Studies, 1(2), 31–46.
performance review processes. With careful implementation, a Fuller, A., & Unwin, L. (2011). Workplace learning and the organization. In M. Malloch,
L. Cairns, K. Evans, & B. N. O’Connor (Eds.), The SAGE handbook of workplace learning
well-designed ePortfolio process potentially constitutes an effec- (pp. 46–59). Los Angeles, CA: SAGE.
tive learner-centred approach to faculty development and perfor- Granberg, C. (2010). E-portfolios in teacher education 2002–2009: The social construc-
mance evaluation. tion of discourse, design and dissemination. European Journal of Teacher Education,
33(3), 309–322.
Goliath, C. L. (2009). Diffusion of an e-Portfolio to assist in the self-directed learning of
Acknowledgements physicians: An exploratory study. University of Akron (Dissertation).
Halonen, J. S., & Ellenberg, G. B. (2006). Teaching evaluation follies: Misperception and
misbehavior in student evaluation of teachers. In P. Seldin (Ed.), Evaluating faculty
A heartfelt thanks to the pilot participants who provided their performance: A practical guide to assessing teaching, research, and service (pp. 150–
thoughts and feedback in our interviews and surveys. Many thanks 165). San Francisco, CA: Jossey-Bass.
to the taskforce members for their collaboration with the research Harris, R., Simons, M., Hill, D., Smith, E., Pearce, R., et al. (2001). The changing role of staff
development for teachers and trainers in vocational education and training. Leabrook,
team. Thanks to the students in the Captioning and Court Reporting Australia: The National Centre for Vocational Education Research (NCVER) Ltd.
program for their assistance with interview transcription. We are Retrieved from hhttp://www.ncver.edu.au/publications/595.htmli.
grateful for the thoughtful input and editing assistance from Susan Higgerson, M. L. (2006). Building a climate for faculty evaluation that improves
teaching. In P. Seldin (Ed.), Evaluating faculty performance: A practical guide to
Chaudoir, Jeffrey Kuntz, and Jennifer Semchuk in the final stages of assessing teaching, research, and service (pp. 35–49). San Francisco, CA: Jossey-Bass.
writing. And finally, thanks to the blind reviewers for their helpful Hoekstra, A., Beijaard, D., Brekelmans, J. M. G., & Korthagen, F. A. J. (2007). Experienced
comments. teachers’ informal learning from classroom teaching. Teachers and Teaching: Theory
and Practice, 13(2), 189–206. http://dx.doi.org/10.1080/13540600601152546
Hoekstra, A., Brekelmans, M., Beijaard, D., & Korthagen, F. (2009). Experienced teachers’
Appendix A. Supplementary data informal learning: Learning activities and changes in behaviour and cognition.
Teaching and Teacher Education, 25(5), 663–673. http://dx.doi.org/10.1016/
Supplementary data associated with this article can be found, in the j.tate.2008.12.007
Hoekstra, A., & Korthagen, F. (2011). Teacher learning in a context of educational
online version, at http://dx.doi.org/10.1016/j.stueduc.2015.03.007. change: Informal learning versus systematically supported learning. Journal of
Teacher Education, 62(1), 76–92. http://dx.doi.org/10.1177/0022487110382917
Hubball, H. T., Pratt, D. D., & Collins, J. B. (2006). Enhancing reflective teaching practices:
References Implications for faculty development. The Canadian Journal of Higher Education,
35(3), 57–81.
Amundsen, C., & Wilson, M. (2012). Are we asking the right questions? A conceptual Jasper, M., & Fulton, J. (2005). Marking criteria for assessing practice-based portfolios at
review of the educational development literature in higher education. Review of masters’ level. Nurse Education Today, 25(5), 377–389.
Educational Research, 82(1), 90–126. http://dx.doi.org/10.3102/0034654312438409 Jimerson, J. B. (2014). Thinking about data: Exploring the development of mental
Axelrod, R. H. (2010). Terms of engagement: New ways of leading and changing organiza- models for ‘‘data use’’ among teachers and school leaders. Studies in Educational
tions (2nd ed.). San Francisco, CA: Berrett-Koehler Publishers. Evaluation, 42, 5–14. http://dx.doi.org/10.1016/j.stueduc.2013.10.01
Beausaert, S., Segers, M., Fouarge, D., & Gijselaers, W. (2013). Effect of using a personal Johnson, B. W. (2006). On being a mentor: A guide for higher education faculty. Mahwah,
development plan on learning and development. Journal of Workplace Learning, NJ: Lawrence Erlbaum Associates.
25(3), 145–158. Kirkpatrick, D. (2007). Four levels of evaluation: Measurement and evaluation. Pewaukee,
Billett, S. (2002). Critiquing workplace learning discourses: Participation and continu- WI: American Society for Training and Development.
ity at work. Studies in the Education of Adults, 34, 56–68. Lamont, M. (2007). What are the features of e-portfolio implementation that can
Blair, K. (2001). Electronic portfolios in tenure and promotion decisions: Making a enhance learning and promote self-regulation? Paper presented at the European
virtual case. In Retrieved from ERIC database (ED 466 138). Institute for E-Learning (EIfEL) conference.
Blume, B. D., Ford, J. K., Baldwin, T. T., & Huang, J. L. (2010). Transfer of training: A meta- Lancaster, S., Di Milia, L., & Cameron, R. (2012). Supervisor behaviours that facilitate
analytic review. Journal of Management, 36, 1065–1105. http://dx.doi.org/10.1177/ training transfer. Journal of Workplace Learning, 25(1), 6–22.
0149206309352880 Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation.
Bound, H. (2011). Vocational education and training teacher professional develop- Cambridge: Cambridge University Press.
ment: Tensions and context. Studies in Continuing Education, 33(2), 107–119. Lohman, M. C., & Woolf, N. H. (2001). Self-initiated learning activities of experienced
http://dx.doi.org/10.1080/0158037X.2011.554176 public school teachers: Methods, sources, and relevant organizational influences.
Brookfield, S. D. (1995). Becoming a critically reflective teacher. San Francisco, CA: Jossey- Teachers and Teaching: Theory and Practice, 7(1), 59–74.
Bass. Marsick, V. J., & Watkins, K. E. (1990). Informal and incidental learning in the workplace.
Butler, P. (2006). A review of the literature on portfolios and electronic portfolios. London/New York: Routledge.
Palmerston North: Massey University College of Education Retrieved from McColgan, K., & Blackwood, B. (2009). A systematic review protocol on the use of
hhttp://www.eportfoliopracticequt.edu.au/docs/Butler%20-%20Review%20o- teaching portfolios for educators in further and higher education. Journal of
f%20lit%20on%20ePortfolio%20research%20-%20NZOct%202006.pdfi. Advanced Nursing, 65(12), 2500–2507.
Bye, B., Pushkar, D., & Conway, M. (2007). Motivation, interest, and positive affect in McGovern, T. G. (2006). Self-evaluation: Composing an academic life narrative. In P.
traditional and nontraditional undergraduate students. Adult Education Quarterly, Seldin (Ed.), Evaluating faculty performance: A practical guide to assessing teaching,
57(2), 141–158. research, and service (pp. 96–110). San Francisco, CA: Jossey-Bass.
Carraccio, C., & Englander, R. (2004). Evaluating competence using a portfolio: A Millis, B. J. (2006). Peer observations as a catalyst for faculty development. In P. Seldin
literature review and web-based application to the ACGME competencies. Teaching (Ed.), Evaluating faculty performance: A practical guide to assessing teaching, research,
and Learning in Medicine, 16(4), 381–387. and service (pp. 82–95). San Francisco, CA: Jossey-Bass.
Clarke, D., & Hollingsworth, H. (2002). Elaborating a model of teacher professional Murray, J. P. (1997). Successful faculty development and evaluation: The complete
growth. Teaching and Teacher Education, 18, 947–967. http://dx.doi.org/10.1016/ teaching portfolio. In ERIC Document Reproduction Service No. ED405759. Washing-
S0742-051X(02)0005307 ton, DC: Graduate School of Education and Human Development, The George
Cox, M. D., & Richlin, L. (Eds.). (2004). Building faculty learning communities: New Washington University.
directions for teaching and learning. San Francisco, CA: Jossey-Bass. O’Meara, K. A., & Braskamp, L. (2005). Aligning faculty reward systems and develop-
Cross, K. P., & Steadman, M. H. (1996). Classroom research: Implementing the scholarship ment to promote faculty and student growth. National Association of Student
of teaching. San Francisco, CA: Jossey-Bass. Personnel Administrators Journal, 42(2), 223–240.
Darwin, S. (2007). The changing contexts of vocational education: Implications for Opfer, V. D., & Pedder, D. (2011). Conceptualizing teacher professional learning.
institutional vocational learning. International Journal of Training and Research, 5(1), Review of Educational Research, 81(3), 376–407. http://dx.doi.org/10.3102/
55–71. 0034654311413609
Day, C. (1999). Developing teachers: The challenges of lifelong learning. London: Falmer Pallett, W. (2006). Uses and abuses of student ratings. In P. Seldin (Ed.), Evaluating
Press. faculty performance: A practical guide to assessing teaching, research, and service (pp.
Devanas, M. A. (2006). Teaching portfolios. In P. Seldin (Ed.), Evaluating faculty perfor- 50–65). San Francisco, CA: Jossey-Bass.
mance: A practical guide to assessing teaching, research, and service (pp. 111–130). Polytechnics Canada (2014). Research and innovation. Retrieved from hhttp://www.po-
San Francisco, CA: Jossey-Bass. lytechnicscanada.ca/polytechnic_advantage/applied_research_commercialization/
Engeström, Y. (2011). Activity theory and learning at work. In M. Malloch, L. Cairns, K. ProgramsCourses/FacultiesSchoolsCentres/TeachingLearning/Departments/
Evans, & B. N. O’Connor (Eds.), The SAGE handbook of workplace learning (pp. 86– Academici.
104). Los Angeles, CA: SAGE. Roberson, D. N., & Merriam, S. B. (2005). The self-directed learning process of older,
Eraut, M. (2004). Informal learning in the workplace. Studies in Continuing Education, rural adults. Adult Education Quarterly, 55(4), 269–287. http://dx.doi.org/10.1177/
26(2), 247–273. http://dx.doi.org/10.1080/158037042000225245 0741713605277372

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
G Model
JSEE-562; No. of Pages 13

A. Hoekstra, J.R. Crocker / Studies in Educational Evaluation xxx (2015) xxx–xxx 13

Sambrook, S. (2005). Factors influencing the context and process of work-related learning: Saroyan & M. Frenay (Eds.), Building teaching capacities in higher education: A
Synthesizing findings from two research projects. Human Resource Development comprehensive international model. Sterling, VA: Stylus Publishing.
International, 8(1), 101–119. http://dx.doi.org/10.1080/1367886052000352591 Van Woerkom, M., & Croon, M. (2008). Operationalising critically reflective work
Schön, D. (1983). The reflective practitioner: How professionals think in action. New York, behavior. Personnel Review, 37, 317–331.
NY: Basic Books. Wayman, J. C., & Jimerson, J. B. (2014). Teacher needs for data-related professional
Seijts, G., Taylor, L., & Latham, G. (1998). Enhancing teaching performance through goal learning. Studies in Educational Evaluation, 42, 25–34. http://dx.doi.org/10.1016/
setting, implementation and seeking feedback. International Journal for Academic j.stueduc.2013.11.001
Development, 3(2), 156–167. Webster-Wright, A. (2009). Reframing professional development through understand-
Seldin, P. (2006). Essential operating principles and key guidelines. In P. Seldin (Ed.), ing authentic professional learning. Review of Educational Research, 79, 702–739.
Evaluating faculty performance: A practical guide to assessing teaching, research, and http://dx.doi.org/10.3102/0034654308330970
service (pp. 20–34). San Francisco, CA: Jossey-Bass. Zeichner, K., & Liu, Y. (2010). A critical analysis of reflection as a goal for teacher
Shuell, T. J. (1986). Cognitive conceptions of learning. Review of Educational Research, education. In N. Lyons (Ed.), Handbook of reflective inquiry. New York, NY: Springer.
56, 411–436.
Smith, K., & Tillema, H. (2003). Clarifying different types of portfolio use. Assessment
and Evaluation in Higher Education, 28(6), 625–648.
Annemarieke Hoekstra Ph.D., is a Teaching and Learning Specialist at the Northern
Statistics Canada (2010). Projections of the diversity of the Canadian population, 2006 to
Alberta Institute of Technology (NAIT). Her research interest is in teacher professional
2031. Ottawa, ON: Author.
learning. She currently holds a grant from the Canadian Social Sciences and Research
Steinert, Y. (2000). Faculty development in the new millennium: Key challenges and
Council to study the professional learning of vocational educators. She obtained her
future directions. Medical Teacher, 22(1), 44–50.
Ph.D. in Education from Utrecht University in 2007. Her dissertation focused on
Steinert, Y., Mann, K., Centeno, A., Dolmans, D., Spencer, J., Gelula, M., et al. (2006). A
informal teacher learning in the workplace. From 2007 until 2009, she worked as
systematic review of faculty development initiatives designed to improve teaching
an instructor in the JR Shaw School of Business at NAIT.
effectiveness in medical education: BEME Guide No. Medical Teacher, 28(8), 497–
526.
Stes, A., Min-Leliveld, M., Gijbels, D., & Van Petegem, P. (2010). The impact of
instructional development in higher education: The state-of-the-art of the re- Jocelyn R. Crocker M.Ed., has been an instructor with the department of Biological
search. Educational Research Review, 5(1), 25–49. http://dx.doi.org/10.1016/ Sciences at the Northern Alberta Institute of Technology (NAIT) since 2004. She has
j.edurev.2009.07.001 received multiple teaching awards including the NAIT Instructional Excellence Award.
Stiehl, R., & Lewchuk, L. (2008). The outcomes primer: Reconstructing the college She obtained a Master of Education from the Department of Secondary Education at the
curriculum. Corvallis, OR: The Learning Organization. University of Alberta in 2010; her thesis focused on the experiences of foreign-trained
Taylor, L., & Rege-Colet, N. (2009). Making the shift from faculty development to professional women in adult upgrading. Her current research interests are in meta-
educational development: A conceptual framework grounded in practice. In A. cognition and teacher professional learning.

Please cite this article in press as: A. Hoekstra, J.R. Crocker. Design, implementation, and evaluation of an ePortfolio approach to support
faculty development in vocational education. Studies in Educational Evaluation (2015), http://dx.doi.org/10.1016/j.stueduc.2015.03.007
The current issue and full text archive of this journal is available on Emerald Insight at:
www.emeraldinsight.com/1741-0401.htm

IJPPM
66,2 Two dimensional
efficiency measurements
in vocational education
196 Evidence from Australia
Received 24 September 2015
Revised 15 February 2016
Peter Fieger, Renato Andrin Villano, John Rice and Ray Cooksey
4 April 2016 University of New England, Armidale, Australia
Accepted 5 May 2016

Abstract
Purpose – In Australia, the vocational education and training (VET) sector accounts for approximately A$8
billion of public spending, of which around A$6.6 billion is spent on government providers that include
Technical and Further Education (TAFE) institutes. The TAFE institutes in Australia are large, public VET
providers, generally funded and managed by state government. Measuring the efficiency and effectiveness of
TAFE institutes is of great interest to policy makers, regulators, consumers and to the institutions
themselves. The paper aims to discuss these issues.
Design/methodology/approach – In this study the authors use data relating to student cohort
demographics, institutional characteristics and educational outcome data, while employing stochastic frontier
analysis, to develop two distinct efficiency measures and models. The first model examines institutional
efficiency in the transformation of financial resources into teaching loads. The second model evaluates efficiency
in the transformation of institutional resources into post-study employment outcomes. K-means cluster analysis
is used to establish groupings of similar institutes and subsequent canonical discriminant analysis is employed
to develop a typology of these clusters.
Findings – In both models the authors find significant inefficiencies in the Australian TAFE system. The
relationship between both efficiency measures is then assessed. While there is no direct linear relationship, a
distinct pattern could be detected. Finally the authors develop a typology of efficient institutions.
Originality/value – This study contributes to the existing research by defining efficiency in vocational
education in two distinct ways and by the utilisation of the derived efficiencies in the development of a
typology of efficient institutes. In doing so, this research makes an original contribution to the understanding
of the drivers of efficiency in vocational education.
Keywords Efficiency, Productivity, Effectiveness, Input/output analysis
Paper type Research paper

1. Introduction
In Australia, the vocational education and training (VET) sector accounts for approximately
A$8 billion of public spending, of which around A$6.6 billion is spent on government
providers that include Technical and Further Education (TAFE) institutes (NCVER
Financial Information, 2014). The TAFE institutes in Australia are large, public VET
providers, generally funded and managed by state government. They are geographically
dispersed, providing important access to tertiary education in regional and remote areas, as
well as in urban locations.
In Australia’s Federal system of government, education and training is a shared activity
of the states and the central (Federal) government. The VET element of education, however,
has traditionally been dominated by state-funded and managed providers although the
Federal government has taken an active role in its regulation and funding over recent
decades (Burke, 2015). Unfortunately, this surplus of regulatory and funding attention has
International Journal of led to a VET system that is arguably in crisis, exhibiting all of the characteristics of
Productivity and Performance
Management over-bureaucratisation, inefficiency, waste and ineffectiveness (Garrick, 2011). VET is thus a
Vol. 66 No. 2, 2017
pp. 196-215
politically contested domain, dominated by vertical duplication between states and Federal
© Emerald Publishing Limited
1741-0401
agencies, and significant variance between states in relation to public spending rates per
DOI 10.1108/IJPPM-09-2015-0139 capita and teaching quality (Billett, 2014).
To further complicate matters, various Australian jurisdictions (at both the national and Two
state levels) have been promoting a variety of market-based reforms to increase competition dimensional
and anticipated efficiency. Generally these deregulatory reforms have favoured private efficiency
education providers to the detriment of established, government owned training providers
like TAFE institutes. These changes have thus caused considerable uncertainty in the measurements
provision of VET in Australia (Toner, 2014). Even aside from these exogenous pressures,
finite governmental resources, and the demand for greater accountability in vocational and 197
higher education training spending, have led to increased scrutiny in relation to training
outcomes and measuring returns on investment in public training expenditure.
Quite clearly, while the skills generated by vocational education are important for the
economy and society, the overall efficacy of the vocational education system in generating
those skills is regularly questioned.
In this dynamic and politicised policy context, measuring the efficiency and effectiveness
of TAFE institutes is of great interest to policy makers, regulators, consumers and to the
institutions themselves. It is a matter of significant conjecture whether the market-based
reforms are indeed improving systemic efficiency and effectiveness (Harris, 2015). A key
problem in making this determination relates to the manner in which efficiency is
determined in publicly funded VET institutes like TAFEs. Knowledge about institutional
efficiency may aid government agencies in allocating funds and in assessing the impact of
funding decisions. Furthermore, institutions themselves may use information about their
own efficiency to benchmark themselves against other institutions and to make adjustments
to their own resource allocation.
Compounding the problems of measurement are the longstanding problems with VET’s
status within the Australian educational sector. Generally considered the “poor cousin” to
university education (Harris, 2015), the outcomes from VET are often difficult to determine
due to its student body’s lower socio-economic status and generally lower academic
credentials. As such, any appropriate set of performance measures must investigate both
educational processes/activities and also educational outcomes.
This paper seeks to provide such an analysis through the application of an integrative
approach to efficiency measurement to explore both activity and delivery efficiency
(teaching load) and outcome efficiency (employment outcomes).
In this study we will define two different types of efficiencies in the Australian TAFE
sector and employ parametric stochastic frontier analysis (SFA) to determine the respective
efficiencies of individual institutes. The first empirical model is designed to estimate
efficiency in the transformation of financial resources into teaching hours (from here on
termed “teaching load model”). The second model estimates the efficiency of the
transformation of teaching resources into post-study outcomes, namely, the employment
rate of TAFE graduates (from here on termed “employment outcome model”).
Once both institutional efficiencies for each institute have been established we will
analyse whether there is a relationship between both types of efficiencies, and whether a
typology of efficient institutes can be developed. We will proceed in the following manner:
first, we will review the theoretical underpinnings of the technique used and identify and
describe the appropriate variables and data that are going to be used in the analysis. Then,
we will operationalise the models, discuss the resulting estimates, and establish groups that
share similar patterns of efficiency. A canonical discriminant analysis will follow to
determine which variables are related to membership in different groups of efficiency.
Finally we will consider what practical relevance the research results have and whether
concrete policy implications could emerge from our findings.
In the next section a review of the literature in relation to the analysis of productivity in
public institutions is provided, with a special focus on SFA. This is followed by a narrower
review of the extant empirical literature exploring efficiency in the Australian vocational
IJPPM education sector. The paper then proceeds in Section 3 to present an overview of the method
66,2 employed in the analysis and an outline of the data utilised to explore institute-level
efficiency. Section 4 presents the results of the analysis, drawing on a variety of
visualisations to illustrate the various drivers of efficiency, with Section 5 discussing the
practical implications of or findings for the Australian VET sector.

198 2. Review of Literature


The contemporary approach to analyse the productivity of public institutions is based on
the work done by Farrell (1957). In his seminal paper, he argued that the measurement of
efficiency is necessary to ascertain whether additional inputs are needed to increase desired
outputs or if such outputs can be increased by raising efficiency alone. Farrell also
developed a generalisable production function which enabled the computation of efficiency
measurements under multiple input scenarios.
Two distinctly different methodologies to determine production frontier have emerged since
the 1970s. The first followed from Aigner et al. (1977). They formulated the stochastic frontier
model, a parametric maximum likelihood technique. This method overcame the previous
limitations of frontier estimation by introducing a new approach to the specification of the error
term, namely, its separation into a normal “noise” term and a one sided inefficiency term.
Almost at the same time, Charnes et al. (1978) published their work on a non-parametric linear
programming method entitled data envelopment analysis (DEA). This method focusses on the
scalar measure of the efficiency of each unit under consideration which is obtained after the
determination of weights for the observed data for inputs and outputs.
The main application of both methods has been the efficiency analysis of public
institutions and government owned entities. In such contexts, the appropriate measures of
inputs and outputs can often difficult to capture. Furthermore, traditional accounting
methods for the measurement of inputs and outputs are often inadequate to measure the
complex nature of varying inputs and outcomes. In the educational context, for example,
simple measures of graduations that do not take into account quality and occupational
outcomes are not appropriate measures of performance. The spectrum of sectors analysed
has varied across a wide field of institutional units, ranging from hospitals, public transport,
public utilities and prisons, to numerous applications of educational contexts.

2.1 Efficiency in the vocational education sector


Efficiency analysis utilising SFA or DEA has been applied frequently in educational
contexts (Bayraktar et al., 2013; Grosskopf et al., 2014). Such analyses have proven
particularly important in decomposing institutional effects from those effects driven by
student cohort capabilities. The derived information has proven useful in determining the
most important inputs and processes of providing efficacious educational arrangements.
There is, however, a paucity of econometric frontier analysis, utilising SFA or DEA, in the
Australian educational context. That which does exist tends to focus on Australia’s
universities, leaving the very large vocational education sector relatively under-researched.
Among the former work in universities, Avkiran (2001) applied DEA and used 1995 data of
Australian universities to determine universities’ productivity in respect to the delivery of
educational services and fee paying enrolments. Other DEA studies examining cross-sectional
university performance were performed by Abbott and Doucouliagos (2003), Carrington et al.
(2005) and Worthington and Lee (2008). Horne and Hu (2008) and Abbott and Doucouliagos
(2009) published SFA research of Australian and New Zealand and Australian universities.
The vocational education sector in Australia generally services a student cohort with lower
levels of academic preparation and capacity and generally lower educational and career
aspirations (Chesters and Smith, 2015). This tends to complexify notions of efficiency and
performance measurement in the VET context, especially as compared to the university context.
Lower academic preparation can predispose students to poorer academic achievement and often Two
lower career aspirations and outcomes. Generally, specific communities (in Australia, e.g. the dimensional
indigenous community) are over-represented in VET but are also over-represented among the efficiency
unemployed. Measuring the success and efficiency of VET is thus challenging, as even a low
level of educational and occupational outcome may indeed be far superior to the outcome that measurements
may have been achieved in the absence of VET.
Only a small number of studies involving Australian TAFEs and other VET providers 199
could be identified. These were notably the research by Abbott and Doucouliagos (2002)
that performed DEA applications utilising data from Victorian institutes only and one
nationwide DEA study by Fieger (2010). Significant research that could be considered
case-based has been undertaken (Toner, 2014; Pillay et al., 2013). Much of this research,
which is informed to a greater or lesser degree by institution-level data, tends to explore the
often complex arrangements by which public funding supports vocational education
arrangements. Characteristically, this research does not provide robust empirical evidence
relating to the drivers and outcomes of institute-level (in) efficiencies. Specifically in relation
to this research, there has been no previous published efficiency analysis of the Australian
TAFE sector which utilised the stochastic frontier approach.

3. Method of analysis
We will be estimating “teaching hours” efficiency and “employment outcome” efficiency
based on the stochastic frontier methodology developed by Aigner et al. (1977). The main
contribution of these authors was the introduction of a new approach to the specification of
the error term, namely, its separation into a normal “noise” term and a one sided inefficiency
term. Stochastic frontier production functions are an extension to the classic Cobb-Douglas
(1928) function which can generally be expressed in this form:
b b
Y ¼ eb0 X 1 1 X 2 2 . . . X bnn ee (1)
This model can then be transformed by taking the log of both sides and the error term then
be disaggregated into the statistical noise portion v, and the non-negative technical
efficiency component u which is distributed independently from v. The technical efficiency
TEi of individual DMUs of ui can then be determined by:
TEi ¼ eui (2)
Once we have estimated the institutional technical efficiencies for “teaching hours” and
“employment outcome” we will analyse the potential relationship between both efficiencies.
This will include the graphing of both efficiency components and a cluster analysis to
determine “efficiency clusters”. Finally, we will employ canonical discriminant analysis with
the aim of developing a typology of efficient institutions.

3.1 Data characteristics and preparation


One of the aims of this study is to ascertain the efficiency of Australian TAFE institutes via
SFA and to determine which exogenous variables drive the calculated efficiencies.
When deciding on an approach to undertake efficiency frontier analysis of TAFE institutes
one has to take into account some specific circumstances that are unique to the VET sector.
Similar efficiency frontier analyses involving universities or secondary schools can often
rely on data such as the number of full time staff, staff qualifications, number of graduates,
test scores, grades, research outputs such as publications and conference presentations,
successful grant applications, and others. Data comparable to the aforementioned are
difficult to obtain for TAFE institutes. There is obviously a scarcity of research and
research-related inputs and outputs that relate to TAFEs. Many TAFEs employ a large
IJPPM percentage of part time lecturers, and this proportion differs from institution to institution
66,2 such that reliable data about this proportion is difficult to obtain. Furthermore, TAFEs do
not consistently award grades in the same way for some or all of their courses through
“competency based” assessments.
It is therefore clear that there are some circumstances that encumber the specification of
frontier efficiency models for TAFE providers. The majority of those circumstances can be
200 categorised into three groups: the absence of functional data for the entire sector (e.g. staff
qualification data were not reported in a standardised way by institutions); partial data only
available for a subset of TAFEs (e.g. certain financial data); and data that is too dissimilar in
nature due to the lack of a comprehensive national reporting standard (e.g. assessment
beyond competency-based assessment).
Despite the aforementioned difficulties we have been able to assemble and derive a data
set containing adequate information to undertake the course of research set out in earlier
paragraphs. The data used in this study came from several sources. These sources included
institutional annual reports, information on institutional websites, personal requests to
institutional administrators and state regulators, the Australian TAFE Student Outcome
Survey (SOS), and the Australian TAFE Students and Courses database maintained by the
National Centre for Vocational Education Research.
Of significance was the choice of year(s) for which data should be obtained.
It was intended to assemble a panel of data comprising a number of years in an effort to:
maximise the number of data points and enable analysis of changes in efficiency over a
given period. However, data collection was more difficult than anticipated as institutes do
not publish financial data in a uniform pattern. Specifically the collecting of several
consecutive years of financial data appeared to be difficult. It was thus decided to focus on
one particular year with the following stipulation: the year had to be as recent as possible, it
had to be an augmented SOS year[1] to enable the use of the most robust institutional data,
and the chosen year had to have the maximum of available data points. Taking these
considerations into account 2011 was chosen as the year of analysis.
The initial plan was to include all 69 Australian TAFE and TAFE like institutions[2] in
this analysis. However, this intention was impeded by a number of factors. In addition to
those institutes that did not provide data, some institutions proved to be too specialised to
be compared on an equal footing with the majority of TAFE institutes. Some of the TAFE
units of universities did not have delineated financial data for their TAFE division available.
After considering availability of data for the remaining institutes it was decided to include
those units in the final data set that had data for the total expenditure variable in 2011
available. This yielded 56 TAFEs for inclusion in the analysis.
In addition to financial expenditure data the “teaching hours” variable used in the efficiency
analysis was sourced from the Students and Courses database. This variable indicates the
number of student contact hours by institution. A number of further items were sourced
predominantly from the 2011 SOS. These included the proportion of students by institute in
terms of sex, student type (module completers/graduates), indigenous students, students who
used a language other than English at home, and disabled students. Other variables included
were the average age of the student body at individual institutions, and an average institutional
remoteness score derived from the Australian Bureau of Statistics’ ARIA variable. We also
used the SOS to determine the number of different courses offered by each institution which
had at least one student enroled. A categorical variable indicating size was derived from the
total expenditure variable. The categories created were based on the intention to achieve a
reasonably uniform distribution of institutions across categories and comprised “very large”,
signifying total expenditure in excess of $120,000,000, large ($70,000,000-$120,000,000),
medium ($45,000,000-$69,999,999), small ($25,000,000-$44,999,999), and very small with total
expenditure of less than $25,000,000.
4. Results Two
4.1 Teaching load model dimensional
The first model in this study aimed to evaluate the teaching load efficiency of a number of efficiency
TAFE institutes. Our interest was in determining institutional efficiency based on basic
financial expenditure and administrative input and the produced output as measured by measurements
teaching contact hours. The starting point to operationalise our efficiency model was in the
form of a production function as expressed by a Cobb-Douglas equation: 201
b2 e
T¼e E C eb0 b1
(3)
where T denotes the output in teaching hours, E the total expenditure, and C the number of
courses offered by a given TAFE. C was included as it is an indicator of the complexity of
college administration. Taking the natural logarithm of (3) and accounting for the SFA
specific error component consistent with Kumbhakar and Lovell (2000) resolves to:
lnðT i Þ ¼ b0 þb1 lnðE i Þþb2 lnðC i Þ þvi ui (4)
Descriptive statistics for variables used in estimating this model can be found in Table I.
In addition to the frontier production function (4) we intended to investigate which
exogenous variables may be influencing technical efficiency. To address this question, we
estimate the relationship between the inefficiency measure and various exogenous
variables. There are several authors that have adopted these approach, including
Kumbhakar et al. (1991), Reifschneider and Stevenson (1991), Huang and Liu (1994) and
Battese and Coelli (1993, 1995). We therefore specified a second component in which we
included some variables which were hypothesised to influence efficiency.
In general, the inefficiency effects model in cross-sectional form is given as:
X
K
ui ¼ d 0 þ d i zi (5)
i¼1

where, z represents the hypothesised K predictors of efficiency and δ the parameters that
needed to be estimated. In our model we hypothesised that predominantly demographic
factors influence efficiency, as these factors may require administrative adjustments to
TAFE operations. We therefore entered the variables with institutional indicators for
English as a second language, disability, remoteness, age and sex, into our efficiency model
(for descriptive statistics see Table II). Equations (4) and (5) enable us to estimate the

Variable n Mean SD Minimum Maximum


Table I.
Teaching hours 56 5,521,177.5 4,174,682.5 473,279 22,346,943 Descriptive statistics
Total expenditure 56 79,966,968.0 53,563,163.2 12,324,312 288,974,000 teaching load
Number of courses offered 56 172.6 83.3 32 439 efficiency SFA model

Variable n Mean SD Minimum Maximum

English second language 56 16.3 9.8 4.6 40.2


Students with disability 56 9.4 2.9 4.4 18.5 Table II.
Remoteness (ARIA) 56 2.1 1.0 1.1 4.7 Descriptive statistics
Student age 56 33.0 2.2 27.6 37.1 teaching load
Proportion of males 56 57.2 10.7 32.8 96.6 inefficiency model
IJPPM marginal effects of the total expenditure and the number of courses offered by TAFE on the
66,2 teaching hours by taking into account the institute-specific variables. A negative coefficient
of the exogenous variable in Equation (2) indicates that the institutes with larger values of
the variables tend to have lower level of inefficiency, thus they are more efficient.
This two component scenario would have originally been estimated in a two step
approach, where the first step specifies the stochastic production frontier and leads to the
202 estimation of efficiency scores and the second step is to estimate the relationship between
efficiency scores and efficiency predictors. Wang and Schmidt (2002) have demonstrated
that this two step procedure is biased and that instead stochastic frontier models and the
way in which efficiency ui depends on predictors can and should be estimated in one single
step using maximum likelihood estimation.
Analysis by Waldman (1982) has shown that for the specification of a stochastic frontier
model it is beneficial to examine the third moments of the least squares residual. If this quantity
is positive, then the least squares slope estimates and λ ¼ 0 represent a local maximum of the
likelihood. Conversely, if the third moment is negative, the likelihood has a greater value at some
other point where λ ¼ 0. This means that negative skewness of the residuals of the OLS
regression indicates that maximum likelihood estimation is indeed the appropriate procedure to
estimate the production frontier. We thus began our analysis with the formulation of a linear
regression model identical to our proposed SFA model. The results can be seen in Table III
(Model 1). The third moment based on the OLS residuals was estimated to be −0.63, thus
indicating it to be a satisfactory prerequisite for the maximum likelihood estimation of the
stochastic frontier. While the estimates of the OLS model only have limited usefulness, they
provide a meaningful starting point for the maximum likelihood estimation (Cullinane and Song,
2006). The R2 estimate of the OLS was, at 0.91, fairly substantial and indicated that most of the
variation in teaching hours can be explained by total expenditure and number of courses offered
by institute. The two independent variables themselves are highly significant and both exhibit
the sign that would be expected, e.g. higher expenditure and increasing number of courses tend
to be associated with a rise in teaching hours.
We could then estimate our basic stochastic frontier model, using the same variables
(Table III, Model 2). While coefficients and intercepts have the same sign as in OLS regression,

OLS MLE
Model1 Model2 Model 3
Variables Est PW|t| Est P W|z| Est PW |z|

Stochastic frontier model


Constant −4.221 o0.001 −4.022 o 0.001 −2.730 o0.001
Total expenditure 0.926 o0.001 0.989 o 0.001 0.968 o0.001
Number of courses offered 0.553 o0.001 0.345 o 0.001 0.134 0.025
Inefficiency model
Constant −17.631 0.001
English second language 0.129 0.027
Students with disability 0.053 0.726
Remoteness (ARIA) 2.708 o0.001
Student age −0.074 0.768
Proportion of males 0.112 0.048
R2 0.913
Wald χ2 385.4 o 0.001 983.5 o0.001
Table III. σv 0.126 o 0.001 0.127 o0.001
Estimates for OLS σu 0.387 o 0.001 0.201 o0.001
and SFA models – λ 3.073 o 0.001 2.048 o0.050
teaching load model γ 0.904 0.715
along with similar magnitude and strong significance, the real interest here were the estimated Two
variance parameters. The strong significance of the Wald test indicates that the coefficient(s) dimensional
were significantly different from 0 and thus confirmed the model’s explanatory power. efficiency
The estimate of the variance of the standard deviations of the inefficiency component σu and
idiosyncratic component σv were both significant. This suggests the statistical significance of measurements
the inefficiency and random error components of the model.
The estimate for the ratio of the variance of the inefficiency component to the variance of 203
the idiosyncratic component γ at 0.9 was quite high and denoted that 90 per cent of the
variability in delivered teaching hours could be attributed to technical inefficiencies.
The closeness of γ to 1 pointed towards the existence of a deterministic production frontier
(Parsons, 2004). The significance of γ and λ affirmed the preponderance of inefficiency in the
composite error term and also validated SFA as the appropriate tool for this specific
analysis (Chen, 2007). Additionally a test was performed to determine whether the units
investigated by our Cobb-Douglas model exhibit constant returns to scale.
The test of this hypothesis determined whether the sum of the coefficients in the model
was statistically different from 1. The sum of the coefficients for “total expenditure” and
“number of courses” was calculated as 1.33 and the test for equality to 1 yielded a χ2 value of
6.54 ( p ¼ 0.0106), so that we could reject the hypothesis of constant returns to scale
technology and assume an increasing returns to scale setting. In the scenario considered,
this meant that outputs would increase disproportionally when inputs are increased.
Having gained insights into the characteristics of our basic frontier model we could
proceed to specify the SFA model that included explanatory variables for the technical
inefficiency variance function (Table III, Model 3). First we noted that parameters and
significance of the frontier function were comparable to the model without the inefficiency
terms. The Wald χ2 value and the variance component of the random error term of the whole
model were also significant and of similar magnitude. The main items of interest in model
three were thus the inefficiency effects. We note that the proportion of students with a
disability and the institutional mean age of the student body were not related to institutional
inefficiency. The strong significance of remoteness pointed to inefficiency being a function
of remoteness. This result confirmed the findings of Fieger (2010), who found remoteness to
be the key variable associated with inefficiency. This finding may be partially attributed to
Australia’s unique geography and related issues of infrastructure and demographics,
however, it must also be noted that “remoteness” acts also as a proxy for institution size as
many urban institutes tend to be significantly larger than rural institutes.
Internationally, remoteness is rarely identified as driver of inefficiency, although Izadi
et al. (2002) found some incidental relationship between remoteness and inefficiency.
In Model 3 we found further, albeit weaker, positive associations between the proportion of
males and inefficiency, and the proportion of students with English as a second language
and inefficiency. Possible explanations here may be that males tend to be engaged at higher
rates in apprenticeships, which require larger administrative and financial efforts on the
part of the institution. An assessment of the correlation between the proportion of males and
the proportion of apprentices and trainees in 2011 revealed an overall correlation of 0.44
( p o0.001), thus supporting this explanation.
Greater financial, educational and administrative efforts may also be at play when
considering the relationship between increasing inefficiency and higher rates of non-native
English speakers. Larger proportions of students with English as a second language may
necessitate more intensive teaching modes, such as lower teacher/student ratios, or other
remedial programmes, which may in turn explain some variation in institutional inefficiency
in respect to the percentage of non-native English speakers.
After verifying the suitability of our model and discussing the interpretation of model
statistics and coefficients we were interested in the actual estimated efficiencies of
IJPPM individual institutions. The inefficiency term of a stochastic frontier model can be assumed
66,2 to follow several distributions, such as half-normal, exponential, truncated normal or γ
forms. It has been suggested that it is reasonable to assume that in empirical work efficiency
terms follow a half-normal distribution (Kumbhakar and Lovell, 2000). The efficiencies
follow from (5) and specifically for the half-normal production model are derived by:
 
1F sn mni =sn
nexpðmni þ 1=2sn Þ
2
204 TE ¼   (6)
1F mni =sn
where Φ signifies the cumulative distribution of the normal distribution and m*i and σ* are
defined as:
mni ¼ Ei s2u =s2s (7)
and:
sn ¼ su sv =ss (8)
The calculated efficiencies for Model 3 can be found in Table AI.

4.2 Employment outcome model


Our second frontier model was designed to assess the efficiency of institutions in the
transformation of resources into positive labour market outcomes for their graduates. The
dependent variable in the model was the “employment outcome”. This variable was created
via a hierarchical regression model which produced an employment score for each institute
(Fieger, 2016). The purpose of this method was to produce an employment outcome measure
which enabled the comparability between institutes after covariates such as demographic
composition of the student body and local labour market conditions were taken into account.
The mean of this employment outcome variable was 0, with increasing values indicating
better employment outcomes. Predictor variables for employment were funding per
teaching hour (in A$), institutional completion rate for qualifications (in per cent), proportion
of students enroled in Certificate III or higher qualifications (in per cent), proportion of
graduates (in per cent) and the size of the respective institute.
Our hypothesis was that increased per hour funding for teaching would be related to
improved employment outcomes. All other predictors were also thought to impact on the
outcome and added to the model to adjust for those variables. Descriptive statistics of all
dependent and independent variables in the employment outcome efficiency model can
be found in Table IV.
As in the teaching load efficiency model, we were interested in how a number of
extraneous variables related to the inefficiencies that may become apparent in the model.

Variable n Mean/(%) SD Min. Max.

Employment outcome 56 0.0 0.1 −0.3 0.3


Funding per hour 56 17.9 11.8 8.9 87.3
Completion rate 56 27.3 11.8 4.6 72.2
Certificate III or higher 56 82.1 8.7 52.3 96.7
Group 56 38.9 14.3 15.7 76.2
Institute size (%)
Very large 23
Table IV. Large 23
Descriptive statistics Medium 21
employment outcome Small 25
model Very small 7
Here we added the variables age, sex (proportion of males), degree of remoteness of the Two
individual institute (1 indicated “urban” to 5 indicated “very remote”), proportion of students dimensional
with a disability (in per cent), proportion of students with English as a second language (in efficiency
per cent), and the average pass rate for individual modules by institute (in per cent) into the
inefficiency component of the model. Descriptive variable statistics can be found in Table V. measurements
The starting point for the employment outcome model was again an OLS regression model
(Table VI, Model 1). The R2 value for the OLS employment model was 0.30, a value 205
considerably smaller than in the “teaching load efficiency” model. Coefficients of the predictor
variables displayed some unexpected properties. Only the proportion of graduates was
significant at the 95 per cent level. A higher proportion of graduates was associated with a
lower employment score. Another interesting result was that funding per teaching hour was
not related to employment outcomes. With respect to institutional size, compared to very large
institutions, medium and smaller institutions had strong to marginally significant superior
employment outcomes. We calculated the third moment of the residuals of the OLS model as
−0.54. This negative skewness validated the intended SFA approach.
Model 2 (Table VI) represented the basic SFA model without inefficiency effects. Variances
of the idiosyncratic (σv) and inefficiency (σu) components were significantly different from 0.
The γ value of 0.92 pointed to the existence of a deterministic frontier and the significance of λ
denoted the presence of inefficiency. The test for the hypothesis of constant returns to scale
technology was performed by determining the sum of the coefficients. This summation yielded
0.24 ( χ2 for difference from one was 16.43 ( po0.001)) which suggested that TAFEs under this
model operated under a decreasing returns to scale environment. This can be interpreted as if
inputs were increased under this scenario, outputs would increase at a lower rate than inputs.
The full SFA model including inefficiency effects can be found as Model 3 in Table VI.
Parameter estimates and slope signs of this model were comparable to the basic SFA model,
although the proportion of graduates was not negatively associated with employment
outcomes anymore. The inefficiency component of the model indicated that remoteness was
strongly associated with inefficiency. This replicated the main result of the “teaching load
efficiency” model, which also ascertained remoteness as a key predictor of inefficiency.
Two additional inefficiency predictors exhibited marginal significance[3]. These included
the proportion of students with a disability, and average age of the student body. Students
with disabilities may have greater difficulty in obtaining post-study employment which
could contribute to lower employment outcomes and thus explain why higher proportions
of them appear to be associated with lower employment efficiency. The average age of the
student body was negatively related to inefficiency. We speculate that this result may be
due to the generally poorer employment outcomes for younger age groups.

4.3 Relationship between “teaching load” and “employment outcome” efficiency


To investigate a possible relationship between teaching hours efficiency and employment
outcome efficiency we graphed the two measures in a scatterplot (Figure 1).

Variable n Mean SD Min. Max.

Age 56 33.0 2.2 27.6 37.1


Sex 56 57.2 10.7 32.8 96.6
Remoteness (ARIA) 56 2.1 1.0 1.1 4.7 Table V.
Disability 56 9.4 2.9 4.4 18.5 Descriptive statistics
English 2nd language 56 16.3 9.8 4.6 40.2 employment outcome
Load pass rate 56 81.6 6.6 57.0 94.3 model
IJPPM OLS MLE
66,2 Model1 Model2 Model 3
Variables Est PW|t| Est PW |z| Est PW |z|

Stochastic frontier
Constant −0.167 0.8 0.285 0.651 0.228 0.836
Funding per hour 0.01 0.828 0.001 0.976 0.018 0.711
206 Completion rate −0.025 0.477 −0.03 0.309 −0.008 0.923
Cert III or higher 0.19 0.154 0.107 0.384 0.03 0.872
Graduates −0.092 0.024 −0.077 0.014 −0.008 0.794
Very large – – – – – –
Large 0.046 0.227 0.051 0.1 0.052 0.172
Medium 0.078 0.053 0.089 0.005 0.152 0.003
Small 0.073 0.084 0.077 0.014 0.052 0.103
Very small 0.05 0.401 0.023 0.638 0.007 0.917
Inefficiency model
Constant −1.61 0.871
English second language 0.078 0.112
Students with disability 0.416 0.061
Remoteness (ARIA) 2.233 0.004
Student age −0.493 0.076
Proportion of males −0.003 0.944
Funding per hour −0.044 0.331
Completion rate 0.048 0.495
Cert III or higher 0.041 0.642
Graduates 0.017 0.684
Load pass rate −0.025 0.735
Very large – –
Large 1.333 0.272
Medium 1.32 0.286
Small −0.689 0.685
Very small −4.861 0.275
R2 0.302
Wald χ2 28.08 o0.001 22.87 o0.001
Table VI. σv 0.037 0.001 0.043 o0.001
Estimates for OLS σu 0.131 o0.001 0.122 o0.001
and SFA models – σ2 0.018 o0.001 0.165 o0.010
employment outcome λ 3.51 o0.001 2.838 o0.010
model γ 0.925 0.739

1 38 36 1 18 33 343049
7727
37
71 4
50 65
5 286651 46
15
513
217
55 31
16
142924
1110 43 44 64 20
19
26 35 74748 70
23
45
3222
53
Teaching hours efficiency

0.8
52 57

40
0.6

56
110
0.4 60
Figure 1. 58
Location of institutes
in teaching hours and 0.2 74
employment outcome
efficiency graph 0.6 0.7 0.8 0.9 1
Employment outcome Efficiency
An interesting pattern became evident from this graph. There appeared to be three major Two
constellations: some institutes scored relatively low on “teaching hours” efficiency and high dimensional
on employment outcome efficiency, whereas others attained a high teaching hours efficiency efficiency
and low employment outcome efficiency, and the remainder rated relatively high on both
efficiencies. Interestingly, there were no institutions that displayed low scores on both types measurements
of efficiencies examined in this study. It was of interest to statistically separate these three
possible combinations of teaching hours and employment outcome efficiency (e.g. high/high, 207
high/low, and low/high) and to evaluate the institutions that constituted the pattern in
Figure 1 with respect to possible observable characteristics, including demographic,
educational and environmental variables as determinants of group membership thereof.
We performed a partition cluster analysis, using the k-means method with three target
clusters. This technique involved an iteration process in which each institute was initially
randomly assigned to a cluster, and then subsequently was allocated to the cluster with the
closest mean, as calculated using the Euclidean distance method. After this, new cluster
means were determined and the process iteratively continued until no institute changed
groups. The resulting clusters can be seen in Table VII.
The location allocation following from the clusters in Table VII can be seen in Figure 2.
We then employed canonical discriminant analysis to examine the extent to which several
covariates could be utilised to statistically differentiate between locations 1, 2, and 3. The
covariates entered into the discriminant function were age, completion rate, load pass rate,
disability (per cent), remoteness, graduates (per cent), age, male gender (per cent), satisfaction,
salary, indigeneity (per cent), SES, Certificate III or higher (per cent), English as a second
language (per cent), Australian born (per cent), the percentage of apprentices and trainees, and
the size of the institution as measured by the number of student delivery hours. The essential
statistics for the two resulting discriminant functions can be found in Table VIII.

Location Institutes

Location 1 40, 56, 58, 60, 74, 110


Location 2 4, 10, 11, 14, 24, 29, 37, 38, 45, 50, 71
1, 5, 7, 13, 15, 16, 17, 18, 19, 20, 22, 23, 25, Table VII.
Location 3 26, 27, 28, 30, 31, 32, 33, 34, 35, 36, 43, 44, Institutions by cluster
46, 48, 49, 51, 52, 53, 55, 57, 64, 65, 66, 70, 77 location

1 38 36 1 5 28 1833 34
51 49
30
77
4627
71 4 14 24 43 6566 15
37 50 44 64 20
191713
25
55 31
16
1110 29 26 23 35 74748 70
45
Location 2 3222
53
0.8
Teaching hours efficiency

Location 3
52 57

40
0.6

56
110
0.4 60
58
Location 1
0.2 74 Figure 2.
Institutes by cluster
0.6 0.7 0.8 0.9 1 location
Employment outcome Efficiency
IJPPM It could be seen that both discriminant functions were significant, but that the first
66,2 discriminant function captured 79 per cent of the variance. The discriminating ability of the
covariates was then be assessed by the evaluation of the standardised canonical
discriminant function coefficients (Table IX).
Generally, values close to 0 indicated diminishing discriminating ability to separate the
three locations. The percentage of disabled students, for instance, had thus a negligible
208 contribution to the separability of the three efficiency locations. The discriminant function
coefficients were graphed for easier interpretation (Figure 3). Variables near the origin of

Discriminant Canonical Eigenvalue Cumulative Likelihood


Function Correlation Variance ratio F PrW|F|
Table VIII.
Canonical 1 0.864 2.937 0.787 0.141 3.937 o0.001
discriminant functions 2 0.665 0.794 1.000 0.558 2.064 0.035

Function 1 Function 2

Load pass rate 0.222 0.021


Completion rate −0.297 −0.601
Disability (%) 0.477 −0.163
Remoteness −0.927 −0.037
Graduates (%) 0.136 −0.070
Age 0.300 1.100
Male (%) −0.350 −0.110
Satisfaction 0.113 −0.151
Salary −0.268 0.001
Indigenous (%) −1.117 −0.632
SES 0.007 0.629
Cert III or higher (%) 0.141 −0.004
Table IX. English 2nd language (%) 0.591 0.313
Standardised Australian born (%) 1.043 0.470
canonical discriminant Apprentices and trainees (%) 0.436 0.794
function coefficients Institute size (in mill delivery hours) −0.177 −0.422

Standardized discriminant function loadings

Age
1
Standardized discriminant function 2

Apprentices and trainees

SES
0.5 Austr born

English 2nd language

Salary Cert 3+ Load pass rate


0 Remoteness
Graduates
Sex Satisfaction
Disability
Inst.Size (mill del. hours)
Figure 3. –0.5 Completion rate
Standardised Indignous
discriminant function
–1 –0.5 0 0.5 1
loadings
Standardized discriminant function 1
this graph, such as load pass rate, Certificate III or higher, student satisfaction, and Two
percentage of graduates provided little discriminating ability. The location of the remaining dimensional
variables signified their contribution to the discriminant function, with age, remoteness, and efficiency
percentage indigenous and Australian born students and apprentices and trainees were
having the strongest impact. measurements
Finally, we examined the confusion matrix (Table X) and the discriminant function
plot (Figure 4) to assess how well the covariates are able to separate the three 209
efficiency locations.
Table X illustrates how many institutions were correctly classified into their location using
the two significant discriminant functions. Overall 52 of the 56 institutes (92.9 per cent) were
accurately classified. Locations 2 and 3 appeared to have more misclassifications, implying
that these two locations were harder to separate. Examination of the discriminant function
score plot (Figure 4) confirmed that location 1 was fairly well separated from the others, while
there was some notable overlap between locations 2 and 3.
Finally, we calculated the means of the covariates of the canonical discriminant analysis
and performed a one way analysis of variance including a Bonferroni multiple comparison
test. The results can be found in Table XI
The table confirmed that differences were more prominent between location 1 vs 2
and 3 rather than between locations 2 and 3. Completion rates stood out as being

Location 1 2 3
True Classified Total

1 6 0 0 6
100 0 0 100
2 0 8 3 11
0 72.7 27.3 100
3 0 1 38 39
0 2.56 97.4 100
Total 6 9 41 56
10.7 16.1 73.2 100 Table X.
Priors 0.11 0.20 0.70 100 Confusion matrix

Discriminant function scores


4

3
2 33 3
1 3
Discriminant score 2

. 3 33
1 3 33 3
3 Centroid 3 3
Centroid 1 1 3 3333 .
3 2 33 3 3 3
0 3 3 3 3 3
1 3 3 3
1 3
2 . 2 .
1 2 3
3 2 3 2 2
2
–2 Centroid 2 2
.

2
–4
2 Figure 4.
Discriminant function
–6 –4 –2 0 2
score plot
Discriminant score 1
IJPPM Location means Location differences PW |t|
66,2 1 2 3 1v2 1v3 2v3 PW |F|

Load pass rate 78.6 79.3 82.7 1.000 0.471 0.420 0.169
Completion rate 15.5 36.2 26.6 0.001 0.061 0.033 0.001
Disability (%) 7.9 10.5 9.3 0.231 0.801 0.660 0.195
Remoteness 4.0 1.8 1.9 o0.001 o0.001 1.000 o0.001
210 Graduates (%) 27.2 49.6 37.7 0.004 0.218 0.031 0.004
Age 34.2 31.5 33.2 0.046 0.855 0.071 0.028
Male (%) 63.9 51.6 57.7 0.068 0.521 0.274 0.063
Satisfaction 4.3 4.2 4.2 0.036 0.188 0.481 0.041
Salary 68,814 53,225 55,990 o0.001 o0.001 0.442 o0.001
Indigenous (%) 24.3 6.3 3.0 0.002 o0.001 1.000 o0.001
SES 2.4 2.9 3.0 0.592 0.134 1.000 0.124
Cert III or higher (%) 73.3 81.7 83.5 0.154 0.020 1.000 0.024
English 2nd language (%) 14.2 20.0 15.6 0.732 1.000 0.567 0.359
Table XI. Australian born (%) 84.3 77.5 79.7 0.538 0.878 1.000 0.402
Location means and Apprentices and trainees (%) 18.0 15.5 17.3 1.000 1.000 1.000 0.736
comparison tests Institute size (in million teaching hours) 0.7 8.0 5.6 0.001 0.013 0.210 0.002

statistically different between all three locations, with location 2 exhibiting the highest
completion rate. While discriminant function loadings (Table IX and Figure 3) indicated
the strongest discriminating ability for remoteness, average age, and the percentage of
indigenous and Australian born students, in terms of significant differences between their
location means these categories were unremarkable. It is further worth reflecting that
while institutes in location 1 displayed several traits that may be considered to have a
negative connotation (such as the lowest completion rate, lowest percentage of graduates,
and lowest percentage of students enroled in Certificate III or higher courses), in respect of
some outcomes these institutes scored exceedingly well. For instance, graduates of
location 1 institutes had higher satisfaction rates than students from other locations, and
attained significantly higher post-training salaries. The lack of a coherent association
between the demographic, institutional, and environmental variables on one side and
combined institutional efficiency (e.g. “teaching hours” efficiency and “employment
outcome” efficiency) could be associated with other unobservable variables, that could the
institutes scores on both types of efficiencies. This means that, in the practical evaluation
of the productivity in the vocational education sector it should thus be kept in mind that
TAFE efficiency is a multi-dimensional concept and its results depend on carefully
defined input and output measures.
Efficiencies should be defined carefully depending on the specific property that is
intended to be evaluated. In our study we defined two separate types of efficiency
and created rankings for the TAFE institutes under examination. We found that
efficiencies calculated under one definition are not necessarily an indicator for efficiencies
obtained via alternative definitions. It therefore seems prudent to conclude that any
results stemming from the efficiency analysis of Australian TAFE institutes, and by
extension the efficiency of any group of public institutions, should always be accompanied
by a carefully phrased explanation on how efficiency was specifically defined.
The validity of the research results presented in this paper is somewhat limited by the
small number of institutions that we were able to include in our models. While the number of
possible inclusions is naturally limited to the number of TAFE institutes in Australia, some
institutes did not contribute data to this analysis. To overcome this we did consider
assembling a panel data set covering a number of periods. However, recent frequent
changes in the Australian TAFE sector, including institutional split-ups, amalgamations, Two
formations and mergers with universities suggested that this approach would not yield a dimensional
superior data set. efficiency
measurements
5. Conclusion
In this study we have applied stochastic frontier models to estimate two types of efficiencies
of Australian TAFE institutes, focussing on the transformation of financial and 211
administrative inputs into teaching load outcomes on one hand, and the transformation
of institutional resources into employment outcomes on the other.
Noting the abovementioned caveats in relation to how efficiency has been determined in
this study, in both models we have observed some clear inefficiencies. These inefficiencies
were mainly related to the degree of remoteness and student characteristics, both of which
could be seen as exogenous to the TAFEs themselves. For example, the least efficient TAFE
institutes were more likely to be found in remote locations, had a higher percentage of
males, and a larger proportion of individuals from non-English speaking backgrounds.
We speculate these inefficiencies were driven by a combination of interrelated factors,
including geographic location, available infrastructure and the absence of occupational
diversity of graduates.
As TAFE institutes serve a student constituency that is generally of lower socio-
economic status, an acknowledgement of the higher costs and lower efficiencies inherent in
providing educational activities to TAFE’s traditional student body is an important finding.
In the policy context where private, for-profit providers tend to “cherry pick” the most
academically able and most readily geographically serviceable students, TAFEs tend to
provide an “educator of last resort” service for students who would otherwise be
marginalised from the tertiary education sector. This clearly drives issues associated with
institutional efficiency.
In the second part of this paper we analysed the association between the institutional
efficiencies estimated earlier. While there was no linear relationship we could detect a
distinct pattern of efficiencies. We further demonstrated that a typology could be developed
that predicted the institutional membership in distinct groups of efficiency.
Our two types of efficiencies have been specifically defined for this study. Theoretically,
it is possible to define an almost infinite number of other efficiencies. We showed in this
paper that different types of efficiencies of the same institutes are not necessarily linearly
related. For policy makers it is therefore necessary to take a multi-dimensional approach
that takes into account the various aspects of different approaches to the concept of
efficiency when making policy decisions.
The TAFE sector in Australia has a long history of providing education to low SES
students, to students in regional areas and to students with low prior academic achievement.
Measures of efficiency must take into account a variety of internal and external outcome
measures in determining the efficiency of the institutions.
The wider, but related, notion of institutional effectiveness would integrate questions of
economic efficiency and also consider the next best alternative at the student, community
and national level that would flow from the absence of the TAFE institutions. In many
instances, the alternative to TAFE education and a skilled job may well be unemployment,
social and economic marginalisation and welfare dependency.
This emphasises that in the efficiency analysis of educational institutions it is necessary
that any efficiency model needs to be specified with a clear purpose in respect to which
particular aspect of institutional efficiency is going to be investigated, and these efficiency
aspects need to be considered in wide context that investigates both internal operational
elements and also the social benefits that may flow from the upskilling of an archetypical
TAFE student.
IJPPM Notes
66,2 1. Odd years feature an augmented sample of the SOS, containing about 300,000 questionnaires, of
which about one third receives a response. In these years the SOS is designed to enable estimates
at an institutional level. In even years the SOS sample contains about 100,000 questionnaires, and
the focus of estimates is the state level.
2. In the context of this study, the term “TAFE and TAFE like institute” refers to TAFE institutes,
212 TAFE divisions of a university, Skills Institutes and Polytechnics. From here on only referred
to as “TAFE”.
3. In this paper, we consider a p-value of 0.05op o 0.10 “marginal”.

References
Abbott, M. and Doucouliagos, C. (2002), “A data envelopment analysis of the efficiency of Victorian
TAFE institutes”, Australian Economic Review, Vol. 35 No. 1, pp. 55-69.
Abbott, M. and Doucouliagos, C. (2003), “The efficiency of Australian universities: a data envelopment
analysis”, Economics of Education Review, Vol. 22 No. 1, pp. 89-97.
Abbott, M. and Doucouliagos, C. (2009), “Competition and efficiency: overseas students and technical
efficiency in Australian and New Zealand universities”, Education Economics, Vol. 17 No. 1,
pp. 31-57.
Aigner, D., Lovell, C.A. and Schmidt, P. (1977), “Formulation and estimation of stochastic frontier
production function models”, Journal of Econometrics, Vol. 6 No. 1, pp. 21-37.
Avkiran, N.K. (2001), “Investigating technical and scale efficiencies of Australian universities through
data envelopment analysis”, Socio-Economic Planning Sciences, Vol. 35 No. 1, pp. 57-80.
Battese, G.E. and Coelli, T.J. (1993), “A stochastic frontier production function incorporating a model for
technical inefficiency effects”, Working Papers No. 69, Econometrics and Applied Statistics,
University of New England, Armidale.
Battese, G.E. and Coelli, T.J. (1995), “A model for technical inefficiency effects in a stochastic frontier
production function for panel data”, Empirical Economics, Vol. 20 No. 2, pp. 325-332.
Bayraktar, E., Tatoglu, E. and Zaim, S. (2013), “Measuring the relative efficiency of quality
management practices in Turkish public and private universities”, Journal of the Operational
Research Society, Vol. 64 No. 12, pp. 1810-1830.
Billett, S. (2014), “The standing of vocational education: sources of its societal esteem and implications
for its enactment”, Journal of Vocational Education & Training, Vol. 66 No. 1, pp. 1-21.
Burke, G. (2015), “Australia’s funding schemes in postsecondary education and disadvantaged
students”, Journal of Educational Planning and Administration, Vol. 29 No. 1, pp. 5-27.
Carrington, R., Coelli, T. and Rao, P. (2005), “The performance of Australian universities: conceptual
issues, and preliminary results”, Economic Papers, Vol. 24 No. 2, pp. 145-163.
Charnes, A., Cooper, W.W. and Rhodes, E. (1978), “Measuring the efficiency of decision making units”,
European Journal of Operational Research, Vol. 2 No. 6, pp. 429-444.
Chen, C.F. (2007), “Applying the stochastic frontier approach to measure hotel managerial efficiency in
Taiwan”, Tourism Management, Vol. 28 No. 3, pp. 696-702.
Chesters, J. and Smith, J. (2015), “Social capital and aspirations for educational attainment: a cross-national
comparison of Australia and Germany”, Journal of Youth Studies, Vol. 18 No. 7, pp. 932-949.
Cobb, C.W. and Douglas, P.H. (1928), “A theory of production”, The American Economic Review, Vol. 18
No. 1, pp. 139-165.
Cullinane, K. and Song, D.W. (2006), “Estimating the relative efficiency of European container
ports: a stochastic frontier analysis”, Research in Transportation Economics, Vol. 16 No. 1,
pp. 85-115.
Farrell, M.J. (1957), “The measurement of productive efficiency”, Journal of the Royal Statistical Society. Two
Series A (General), Vol. 120 No. 3, pp. 253-290. dimensional
Fieger, P. (2010), “An investigation of TAFE efficiency”, 2010 AVETRA Conference, Surfers Paradise, efficiency
8-9 April.
measurements
Fieger, P. (2016), “Efficiency and effectiveness in the Australian technical and further and education
system”, doctoral thesis, University of New England, Armidale.
Garrick, B. (2011), “The crisis discourse of a wicked policy problem: vocational skills training in 213
Australia”, The Australian Educational Researcher, Vol. 38 No. 4, pp. 401-416.
Grosskopf, S., Hayes, K.J. and Taylor, L.L. (2014), “Efficiency in education: research and implications”,
Applied Economic Perspectives and Policy, Vol. 36 No. 2, pp. 175-210.
Harris, R. (2015), “Quality in the Australian VET sector: what has been happening?”, International
Journal of Training Research, Vol. 13 No. 1 pp. 16-34.
Horne, J. and Hu, B. (2008), “Estimation of cost efficiency of Australian universities”, Mathematics and
Computers in Simulation, Vol. 78 No. 2, pp. 266-275.
Huang, C. and Liu, J.T. (1994), “Estimation of non-neutral stochastic frontier production function”,
Journal of Productivity Analysis, Vol. 5 No. 2, pp. 171-180.
Izadi, H., Johnes, G., Oskrochi, R. and Crouchley, R. (2002), “Stochastic frontier estimation of a CES cost
function: the case of higher education in Britain”, Economics of Education Review, Vol. 21 No. 1,
pp. 63-71.
Kumbhakar, S. and Lovell, C. (2000), Stochastic Frontier Analysis, Cambridge University Press,
Cambridge.
Kumbhakar, S., Gosh, S. and McGuckin, J.T. (1991), “A generalised production frontier approach for
estimating determinants of inefficiency in US dairy farms”, Journal of Business and Economic
Statistics, Vol. 9 No. 3, pp. 279-286.
NCVER Financial Information (2014), Australian Vocational Education and Training Series, NCVER,
Adelaide.
Parsons, L.J. (2004), Measuring Performance Using Stochastic Frontier Analysis: An Industrial Sales
force Illustration, Institute for the Study of Business Markets, The Pennsylvania State
University, PA.
Pillay, H., Watters, J.J. and Hoff, L. (2013), “Critical attributes of public-private partnerships: a case
study in vocational education”, International Journal of Adult Vocational Education and
Technology, Vol. 4 No. 1, pp. 31-45.
Reifschneider, D. and Stevenson, R. (1991), “Systematic departures from the frontier: a framework
for the analysis of firm inefficiency”, International Economic Review, Vol. 32 No. 3, pp. 715-723.
Toner, P. (2014), “Contracting out publicly funded vocational education: a transaction cost critique”,
The Economic and Labour Relations Review, Vol. 25 No. 2, pp. 222-239.
Waldman, D.M. (1982), “A stationary point for the stochastic frontier likelihood”, Journal of
Econometrics, Vol. 18 No. 2, pp. 275-279.
Wang, H.J. and Schmidt, P. (2002), “One-step and two-step estimation of the effects of exogenous
variables on technical efficiency levels”, Journal of Productivity Analysis, Vol. 18 No. 2,
pp. 129-144.
Worthington, A.C. and Lee, B.L. (2008), “Efficiency, technology and productivity change in Australian
universities, 1998-2003”, Economics of Education Review, Vol. 27 No. 3, pp. 285-298.

(The Appendix follows overleaf.)


IJPPM Appendix
66,2

Technical efficiency
Institute Teaching load efficiency Employment outcome efficiency

214 1
4
0.984
0.977
0.909
0.820
5 0.973 0.927
7 0.932 0.950
10 0.953 0.870
11 0.943 0.860
13 0.971 0.989
14 0.953 0.859
15 0.978 0.982
16 0.966 0.991
17 0.953 0.981
18 0.986 0.944
19 0.960 0.973
20 0.963 0.973
22 0.862 0.976
23 0.921 0.924
24 0.964 0.878
25 0.968 0.980
26 0.908 0.896
27 0.985 0.990
28 0.973 0.939
29 0.959 0.871
30 0.987 0.978
31 0.967 0.992
32 0.866 0.968
33 0.982 0.956
34 0.996 0.973
35 0.920 0.929
36 0.986 0.891
37 0.979 0.719
38 0.991 0.780
40 0.621 0.946
43 0.960 0.926
44 0.946 0.941
45 0.893 0.669
46 0.980 0.983
47 0.916 0.955
48 0.927 0.963
49 0.992 0.983
50 0.972 0.819
51 0.981 0.954
52 0.739 0.938
53 0.840 0.995
55 0.967 0.978
56 0.474 0.932
57 0.723 0.997
58 0.327 0.953
Table AI. 60 0.389 0.995
Teaching load 64 0.948 0.969
efficiency and
employment outcome
efficiency by institute (continued )
Two
Technical efficiency dimensional
Institute Teaching load efficiency Employment outcome efficiency
efficiency
65 0.977 0.940 measurements
66 0.979 0.947
70 0.918 0.986
71 0.978 0.724 215
74 0.198 0.885
77 0.983 0.983
110 0.423 0.994
Mean 0.888 0.929
SD 0.182 0.074 Table AI.

Corresponding author
Peter Fieger can be contacted at: pfieger2@une.edu.au

For instructions on how to order reprints of this article, please visit our website:
www.emeraldgrouppublishing.com/licensing/reprints.htm
Or contact us for further details: permissions@emeraldinsight.com
Reproduced with permission of the copyright owner. Further reproduction prohibited without
permission.