Anda di halaman 1dari 311

TOOLS AND EVALUATION TECHNIQUES FOR

COLLABORATIVE E-LEARNING

COMMUNITIES

Niki Lambropoulos (Lampropoulou)

Submitted for Examination of Doctor of Philosophy

Center for Interactive Systems Engineering


London South Bank University
London
United Kingdom
Tools and Evaluation Techniques for Collaborative E-Learning Communities

To my friends, family

and

the Greek teachers

ii
Ph.D. Thesis Centre for Interactive Systems Engineering LSBU
Tools and Evaluation Techniques for Collaborative E-Learning Communities

Is the sand other than the rocks?


That is, is the sand perhaps nothing but a greater number of very tiny
stones?
Is the moon a greater rock?
If we understood rocks, would we also understand the sand and the moon?
Is the wind a sloshing of the air analogous to the sloshing motion of the
water in the sea?
What common features do different movements have?
What is common to different kinds of sound?
How many different colours are there?

And so on. In this way we try gradually to analyse all things, to put together
things which at first sight look different, with the hope that we may be able
to reduce the number of different things and thereby understand them better.

Richard P. Feynman, Six easy Pieces. 1995, p. 23-24.

iii
Ph.D. Thesis Centre for Interactive Systems Engineering LSBU
Tools and Evaluation Techniques for Collaborative E-Learning Communities

Acknowledgments

First, I want to express my thanks to Xristine Faulkner and Fintan Culwin, my


Supervisors, for all they have done to support me in my studies. Both have given
generously of their time and talents, and it is my (possibly biased) view that they provide
a model of PhD supervising at its best. I cannot thank them enough. In addition, Louise
Campbell and Chung Lam from the Research Office for helping all research students at
LSBU on an individual basis.

Many other people have also provided valued input to my research through discussions,
participating in empirical work, or commenting on written work (or more than one of
these). I particularly wish to thank Sophi Danis, Sara BenIsaac, my Yoga mates,
especially Ilana Isserow, Jennifer Pearl and Lesley Todd, who have been through the
course with me; John Henderson for all the fun and support; also Catherine Spiro, Betty
Shane, Ben Daniel, Martha Christopoulou and last but not least, Mariza Smirli for their
help and support.

Special thanks to my Dad, Konstantinos, my Mum, Aphrodite and my Sister, Georgia for
being such a tolerant and supportive family.

Most of my work has been funded by the Greek Ministry of Education and Religious
Affairs given as three years of educational paid leave of absence. For this, I need to
thank Theodoros Birbas, Socratis Papathanasiou, Epaminondas Georgopoulos, and
Panayiotis Zevlas. In addition, many thanks to the Greek School Network and in
particular, Michael Paraskevas and Vangelis Grigoropoulos for their help and providing
access to research space.

I need to thank my mates in the Dream-e-Team, Marianna Vivitsou, Dimitris Konetas,


Alexander Gkikas, Sofia Papadimitriou, Panayiotis Kampylis, Elias Economakos, and
especially Nikos Minaoglou, for the unimaginable creative collaboration all these years.

Great thanks to Alexander Muir, Jenny Preece and Ben Shneiderman for the continuous
inspiration as well as their insights and energy that enabled me to overcome obstacles.

Finally, special thanks to my examiners, Dr Judy Ramsay and Professor Stephen Lerman
for the detailed comments and insights that improved this thesis.

iv
Ph.D. Thesis Centre for Interactive Systems Engineering LSBU
Tools and Evaluation Techniques for Collaborative E-Learning Communities

Declaration

I grant powers of discretion to the University Librarian to allow this thesis to


be copied in whole or in part without further reference to me. This permission covers
only single copies made for study purposes, subject to normal conditions of
acknowledgment.

v
Ph.D. Thesis Centre for Interactive Systems Engineering LSBU
Tools and Evaluation Techniques for Collaborative E-Learning Communities

Abstract

This study provides new multidisciplinary approaches for tools and evaluation
techniques to ensure quality in collaborative e-learning communities. The research
problem was the Greek teachers’ absence of participation in e-learning discussions
for 3 years. Three conceptual frameworks were used to understand and evaluate the
situation: passive and active participation, the collaborative e-learning episode, and
the sense of e-learning community index. Two interventions were made, collaborative
e-learning and the introduction of associated software-based tools: participation
graphs and avatars, MessageTag, a tool to depict the levels of critical thinking in
collaborative e-learning, and social network analysis tools, the visualisation
interactions nodes and centrality. Ethnotechnology was the research design
triangulating quantitative and qualitative data as well as social network analysis.

The originality of this study lies in the investigation of the processes on the social and
learning aspects of collaborative e-learning and associated tools.
The results indicated that the suggested frameworks and tools can be useful in
supporting collaborative e-learning communities.

The wider implications from the findings emphasise the need for: the organisations’
e-learning readiness; the e-learners’ prior knowledge and ability to interact; the
facilitation of e-learners’ social awareness; the change of teaching and learning
approaches for different levels and types of interactions, participation, and critical
thinking; the development of collaborative e-learning communities; the use of tools
anchored in learner-centred design and solid pedagogical frameworks. If all
components are present in an online course then e-learning quality can be ensured.

vi
Ph.D. Thesis Centre for Interactive Systems Engineering LSBU
Tools and Evaluation Techniques for Collaborative E-Learning Communities

Περίληψη

Αυτή η έρευνα έχει βασιστεί σε διεπιστημονικές προσεγγίσεις για εργαλεία και


τεχνικές αξιολόγησης με σκοπό να εξασφαλίσει την ποιότητα στις συνεργατικές
διαδικτυακές μαθησιακές κοινότητες. Το ερευνητικό πρόβλημα στηρίχτηκε στην
απουσία συμετοχής των Ελλήνων εκπαιδευτικών σε συζητήσεις διαδικτυακών
μαθημάτων για 3 χρόνια. Τρία εννοιολογικά πλαίσια χρησιμοποιήθηκαν για να
κατανοηθεί και να αξιολογηθεί η περίπτωση: παθητική και ενεργητική συμμετοχή, το
επεισόδιο συνεργατικής μάθησης, και ο δείκτης αισθήματος ‘ανήκειν’ στη διαδικτυακή
μαθησιακή κοινότητα. Έγιναν δυο παρεμβάσεις, η συνεργατική μάθηση και η
εισαγωγή σχετικών εφαρμογών: αυτά είναι γραφήματα και άβαταρ συμμετοχής, το
εργαλείο MessageTag για την αναπαράσταση των επιπέδων κριτικής σκέψης στη
συνεργατική διαδικτυακή μάθηση, και τα εργαλεία ανάλυσης κοινωνικών δικτύων για
την απεικόνηση των διαδράσεων με βάση τους διαδραστικούς κόμβους και την
κεντρικότητα. Η ερευνητική μεθοδολογία ήταν η Εθνοτεχνολογία και
χρησιμοποιήθηκε για το συνδυασμό ποσοτικής και ποιοτικής ανάλυσης και ανάλυσης
κοινωνικών δικτύων με σκοπό την εγκυρότητα των αποτελεσμάτων.

Η πρωτοτυπία αυτής της έρευνας έγκειται στην ερευνητική διαδικασία για την
κοινωνική και μαθησιακή διάσταση της συνεργατικής μάθησης και των σχετικών
εργαλείων. Τα αποτελέσματα έδειξαν ότι τα προτεινόμενα πλαίσια και εργαλεία
μπορεί να είναι χρήσιμα για την υποστήριξη των διαδικτυακών μαθησιακών
κοινοτήτων.

Οι ευρύτερες επιπτώσεις από τα αποτελέσματα δείχνουν την ανάγκη για: την


ετοιμότητα των οργανισμών όσον αφορά τη διαδικτυακή εκπαίδευση, την
προηγούμενη γνώση και ικανότητα των εκπαιδευομένων για διάδραση, τη
διευκόλυνσή τους για κοινωνική συνειδητοποίηση, την αλλαγή που έχει υπάρξει στις
μεθόδους διδασκαλίας και μάθησης για διαφορετικά επίπεδα και τύπους διάδρασης,
συμμετοχής και κριτικής σκέψης, την ανάπτυξη διαδικτυακών μαθησιακών
κοινοτήτων, και τέλος τη χρήση εργαλείων βασισμένων στο χρηστο-κεντρικό
σχεδιασμό και ισχυρές παιδαγωγικές προσεγγίσεις. Εάν όλα τα στοιχεία είναι
υπαρκτά σε ένα διαδικτυακό μάθημα τότε η ποιότητα της διαδικτυακής εκπαίδευσης
μπορεί να εγγυηθεί.

vii
Ph.D. Thesis Centre for Interactive Systems Engineering LSBU
Tools and Evaluation Techniques for Collaborative E-Learning Communities

Table of Contents
Acknowledgments …………………………………………………………… iv
Declaration ……………………………………………………………………. v
Abstract ……………………………………………………………………….. vi
Περίληψη (Abstract in Greek) ……………………………………………… vii
Table of Contents …………………………………………………………… viii
Index of Tables ……………………………………………………………… xi
Index of Figures …………………………………………………………….. xiii
Index of Graphs …………………………………………………………….. xiv
Index of Appendices ……………………………………………………….. xv

Chapter 1: Introduction ……………………………………………………. 1


1.1 INTRODUCTION ………………………………………………………….. 2
1.2. EDUCATORS’ ONLINE TRAINING ………………………………….... 3
1.2.1 Greek teachers’ online training ……………………………… 5
1.3. QUALITY IN ONLINE EDUCATION …………………………………… 7
1.4. AIMS AND RESEARCH QUESTIONS ………………………………… 9
1.5. OVERVIEW OF THE CHAPTERS …………………………………….. 11
REFERENCES …………………………………………………………………. 12

Chapter 2: The Literature Review ……………………………………… 14


2.1 INTRODUCTION ……………………………………………………….. 15
2.2. COLLABORATIVE LEARNING ……………………………………... 15
2.2.1 From the Individual to the Community: a Brief History … 18
2.3 ONLINE & E-LEARNING COMMUNITIES …………………………… 24
2.3.1. Passive Participation in Online Communities …......... 25
2.3.2 Passive Participation in e-Learning Communities ….. 27
2.4 DESIGN FOR LEARNERS AS USERS & USERS AS LEARNERS .. 31
2.4.1 Socio-Technical & User-Centred Design …………………. 31
2.4.1.1 Towards a Learner-Centred Design ……………. … 33
2.4.2 Instructional Design & Engineering ………………………... 34
2.5. TOOLS FOR COLLABORATIVE E-LEARNING COMMUNITIES ... 38
2.5.1. When the Online Community Met E-Learning ………... 41
2.5.1.1. Design Principles for e-Learning Communities …... 42
2.5.1.2. E-Learning Tools & Interface Design …………....... 44
2.5.2. Tools to Support Collaborative e-Learning
Communities ……………………………………………. 45
2.5.2.1. Tools to Structure Dialogical Sequences ……....... 46
2.6 PROPOSITIONS FOR DESIGN ………………………………………. 51
2.6.1. Collaborative e-Learning Episodes …………………… 51
2.6.2. Levels of Participation ………………………………….. 53
2.6.3. The Sense of e-Learning Community Index …………. 55
REFERENCES ………………………………………………………………… 56

viii
Ph.D. Thesis Centre for Interactive Systems Engineering LSBU
Tools and Evaluation Techniques for Collaborative E-Learning Communities

Chapter 3: The Research Design ………………………………………. 67


3.1 INTRODUCTION ……………………………………………………….. 68
3.1.1. E-Learning Research ………………………………………… 68
3.2 RESEARCH DESIGN ……………………………………………………. 70
3.2.1. Examining the Research Context ……………………………. 70
3.2.1.1 Ethnography …………………………………..………. 71
3.2.1.2 Ethnotechnology: the virtual ethnography …............. 72
3.2.1.3 Ethnotechnological methods …………………………. 73
3.2.2. In the Search for Quality: Human-Human Interaction
Analysis …………………………………………………….. 76
3.2.2.1 Posts: Thematic analysis ……………………………… 76
3.2.2.2 Interactions: Social network analysis ………………… 79
3.2.3. Questionnaire Design ………………………………………….. 81
3.2.4. E-Research Coordination: Time-short series design ………… 83
3.2.5. Research Constraints ………………………………………….. 86
REFERENCES ………………………………………………………………….. 87

Chapter 4: The Research Context ………………………………………. 92


4.1 INTRODUCTION ………………………………………………………… 93
4.1.1. The Native’s Point of View: background and
Characteristics ……………………………………………. 93
4.2 IDENTIFYING INTENTIONS …………………………………………… 96
4.3 PLANNING ………………………………………………………………. 99
4.3.1. Moodle@GSN: Moodle at the Greek School Network .. …. 99
4.3.1.1 Users and activity at Moodle@GSN ……………. 100
REFERENCES ………………………………………………………………… 107

Chapter 5: Tools & Evaluation Techniques ……………………….. 108


5.1 INTRODUCTION ………………………………………………………... 109
5.2 COLLABORATIVE E-LEARNING TOOLS …………………………… 110
5.2.1. Design for initial design: Create prototype for
testing by the e-learners …………………………………… 110
5.2.1.1. Presence & Co-Presence Awareness:
Visualisations Interactions Tools ………………… 111
5.2.1.2. Participation Awareness:
Evaluation Participation Tools …………………… 112
5.2.1.3. Structuring Collaborative E-Learning:
MessageTag ……………………………………… 113
5.2.2. The E-mmersion Block ……………………………………… 115
5.2.2.1. Tools Evaluation Pool ………………………………. 118
5.2.2.2. Application of guidelines and heuristics from
feedback in design …………………………….. 125
5.2.2.3. Implications for Research design ………………… 127
5.2.3. Greek teachers Moodle developers ………………………. 130
5.2.3.1. Application of guidelines and heuristics from
feedback in design …………………………….. 133
REFERENCES ………………………………………………………………… 136

ix
Ph.D. Thesis Centre for Interactive Systems Engineering LSBU
Tools and Evaluation Techniques for Collaborative E-Learning Communities

Chapter 6: Main Study …………………………………………………. 137


6.1 INTRODUCTION ……………………………………………………….. 138
6.1.1. Initial activities ……………………………………………….. 139
6.2 DESCRIPTION OF THE RESEARCH CONTEXT ………………….. 141
6.2.1. Who are the Greek teachers? ……………………………… 141
6.2.2. Conditions of working and learning online ………………… 142
6.2.3. Previous Knowledge of collaborative e-learning techniques
& participation ……………………………………………. ….. 144
6.3 TRACING PARTICIPATION …………………………………………… 147
6.3.1. Frequency of visits …………………………………………… 147
6.3.2. On Participation ………………………………………………. 151
6.3.2.1. Active and passive participants ……………………. 151
6.3.2.2. Active participation levels …………………………… 152
6.3.2.3. Passive participation levels …………………………. 153
6.4 THE COLLABORATIVE E-LEARNING EPISODES ………………… 158
6.4.1. Quantitative variables …………………………………….. … 158
6.4.2. Qualitative variables …………………………………………. 160
6.5 THE SENSE OF THE E-LEARNING COMMUNITY INDEX ……….. 166
6.5.1. Community evolution ………………………………………… 167
6.5.2. Sense of belonging ……………………………………….. … 172
6.5.3. Empathy ………………………………………………………. 174
6.5.4. Trust …………………………………………………………… 177
6.5.5. Intensity ……………………………………………………….. 178
6.5.6. E-learning quality …………………………………………….. 182
6.5.7. Global Social Network Analysis …………………………….. 184
6.5.7.1. Global cohesion ……………………………………… 186
6.5.7.2. Global centrality ……………………………………… 192
6.5.7.3. Local nodes and centrality in real time ……………. 197
6.6 PEDAGOGICAL USABILITY ....................………………………….... 201
6.6.1. Pedagogical Usability ............................…………………… 201
6.6.2. Correlations & crosstabulations ……………………………… 204
6.7 INTERVENTION ANALYSIS ………………………………………...... 209
REFERENCES ………………………………………………………………… 215

Chapter 7: Conclusions………………………………………………... 218


7.1 INTRODUCTION ………………………………………………………… 219
7.2 SCOPE OF FINDINGS …………………….…………………………… 219
7.3 CONTRIBUTIONS ………………………………………………………. 222
7.3.1. Key contributions …………………………………………….. 222
7.3.2. Secondary contributions …………………………………….. 223
7.4 THESIS LIMITATIONS ………………………………………………… 225
7.5 CONCLUSIONS ………………………………………………………… 225
REFERENCE .…………………………………………………………………. 232

Appendices ……………………………………………………………………… 234

Glossary …………………………………………………………………………… 285

x
Ph.D. Thesis Centre for Interactive Systems Engineering LSBU
Tools and Evaluation Techniques for Collaborative E-Learning Communities

Index of Tables
Chapter 1
Table 1.5-1. Chapters Overview ……………………………………………………… 11
Chapter 2
Table 2.2.1-1. The Socially-Shared Cognition Approach ………………………….. 21
Table 2.3.2-1. Reasons for lurking in e-learning communities ................................ 29
Table 2.3.2-2. Strategies for enhancing activity in e-learning .................................. 29
Table 2.4.2-1. Instructional Design problems …………………………………………. 36
Table2.5.1.1-1. Design principles for online and e-learning communities ................ 42
Table 2.5.2.1-1. Learning and Instructional Activities ……………………………….. 46
Table 2.5.2.1-2. MessageForum Collaborative Learning Attributes ………………… 48
Table 2.5.2.1-3. Similarities and differences between MessageForum & InterLock 50
Table 2.6.2-1. Levels of participation measurement …………………………………. 53
Table 2.6.2-2. Levels of activity measurement ………………………………………... 54
Chapter 3
Table 3.2-1. Research Design ………………………………………………………….. 70
Table 3.2.2.1-1. Collaborative e-Learning Episodes Coding Matrix ………………… 78
Table 3.2.4-1. Observations, interventions and evaluation ………………………… 84
Table 3.2.4-2. Limitations and strengths in Time-Short Series …………………….. 84
Table 3.2.4-1. Research methodology ……………………………............................ 85
Chapter 4
Table 4.3.1.1-1. Courses categories and number of e-learners ………………… 100
Table 4.3.1.1-2. e-Learners’ posts and views in the 6 active courses …………….. 103
Table 4.3.1.1-3. e-Learners’ replies and dates ………………………………………. 104
Table 4.3.1.1-4. E-Learning Engineering for Moodle@GSN ……………………….. 106
Chapter 5
Table 5.2.-1. Iterative Design Blocks ………………………………………………… 110
Table 5.2.2.1-1. Use of MessageTag …………………………………………………. 123
Table 5.2.2.3-1. Research Design …………………………………………………… 129
Table 5.2.3-1. Demographics for the three Greek Teachers / Moodle Developers 130
Table 5.2.3-2. Questionnaire open questions …………………………………….…. 131
Table 5.2.3-3. Tools Pedagogical Usability Scores …………………………………. 132
Chapter 6
Table 6.1.1-1. The questionnaires’ selection process ……………………………… 139
Table 6.2.3-1. Greek teachers’ knowledge and attitudes on collaboration
and participation ……………………………………………………… 145
Table 6.3.1-1. Moodle@GSN forums and users view log files ………………….. 147
Table 6.3.1-2. Forums and users view log files in the research pool …………… 148
Table 6.3.1-3. Total number of posts ……………………………………………… 148
Table 6.3.1-4. Temporal overview of all activities ………………………………… 149
Table 6.3.1-5. Temporal overview of posted messages (add post/forum) …….. 150
Table 6.3.2.1-1. Number of active and passive participants …………………….. 151
Table 6.3.2.2-1. Active participation levels (Initial proposition) ………………….. 152
Table 6.3.2.2-2. Active participation levels (Second proposition) ………………. 153
Table 6.3.2.3-1. Passive Participation Levels …………………………………….. 154

xi
Ph.D. Thesis Centre for Interactive Systems Engineering LSBU
Tools and Evaluation Techniques for Collaborative E-Learning Communities

Table 6.3.2.3-2. Posts from passive participants ………………………………… 155


Table 6.4.2-1: Collaborative E-Learning Episodes Overview ………………….. 160
Table 6.4.2-2. Collaborative e-Learning Episodes selected for analysis ……… 161
Table 6.4.2-3. Collaborative e-Learning Episodes temporal overview ………… 162
Table 6.4.2-4. CeLEs: e-learners and e-tutors’ contributions ………………….. 163
Table 6.5.1-1. Knowing the community …………………………………………… 169
Table 6.5.3-1. Empathy factors ……………………………………………………. 174
Table 6.5.3-2. Correlations for Empathy factors …………………………………. 176
Table 6.5.4-1. Trust levels …………………………………………………………… 177
Table 6.5.4-2. Trust development towards individuals ………………………….. 177
Table 6.5.5-2. Persistence in Moodle@GSN …………………………………….. 179
Table 6.5.5-3. Persistence in the research pool …………………………………. 181
Table 6.5.7.1-1. Group Network Cohesion: Density & Reciprocity …………….. 186
Table 6.5.7.1-2. Cliques …………………………………………………………….. 188
Table 6.5.7.1-3. Cliques in Structural Equivalence ………………………………. 192
Table 6.5.7.2-1. Group Centrality …………………………………………………… 193
Table 6.5.7.2-2. Top 10 Scorers in Out-Degree Centrality ………………………. 194
Table 6.5.7.2-3. Top 10 Scorers in In-Degree Centrality ………………………… 195
Table 6.6.1-1. Overall scores for the collaborative tools’ usability and utility ….. 201
Table 6.6.1-2. Use of MessageTag in the Research pool ……………………….. 202
Table 6.6.2-1. Time using LMS * Frequency of Use ……………………………… 206
Table 6.6.2-2. Training on LMS * Frequency of Use ……………………………… 207
Table 6.7-1. Intervention Analysis ………………………………………………….. 209
Table 6.7-2. E-learning engineering ……………………………………………….. 210
Table 6.7-3. The Sense of e-Learning Community Index Checklist ……………. 212

xii
Ph.D. Thesis Centre for Interactive Systems Engineering LSBU
Tools and Evaluation Techniques for Collaborative E-Learning Communities

Index of Figures
Chapter 2
Figure 2.2.1-1 The Eyeball of Participation ………………………………………… 22
Figure 2.5-1. Google trend history for online community and e-learning …………. 39
Figure 2.5-2. PLATO III ………………………………………………………………… 40
Figure 2.5.2-1: Categorisation of collaborative learning tools ................................. 45
Figure 2.5.2.1-1. InterLock Interface ………………………………………………..…. 47
Figure 2.5.2.1-2. MessageForum Attributes in a Discussion Topic ………………… 49
Figure 2.6.1-1. The Collaborative E-learning Episode (CeLE) ……………………….52
Figure 2.6.2-1. Participation Levels in Collaborative e-Learning Communities …… 54
Chapter 3
Figure 3.2.1.3-1 Ethnotechnology and methods ……………………………………... 75
Figure 3.2.3-1. Questionnaire Design Methodology ………………………………… 82
Chapter 4
Figure 4.2-1. Organisation of the Education System in Greece 2003/04 …………. 96
Figure 4.3.1-1. Moodle@GSN research context …………………………………….. 99
Chapter 5
Figure 5.2.1.1-1. Visualisation Interaction Tools (VIT) Nodes & Centrality …........ 111
Figure 5.2.1.1-2. Visualisation Interactions Tools (VIT) production line ………….. 111
Figure 5.2.1.2-1. Course and individual participation levels graph ………………. 112
Figure 5.2.1.2-2. Participation evaluation graphs production line …………………. 113
Figure 5.2.1.2-1. Initial Design for Message Tagging ………………………………. 114
Figure 5.2.2-1 Total Codes Network in the Web Design pool ............................ 117
Figure 5.2.2.1-1. Participants and number of messages (ATLAS.ti) ……………… 118
Figure 5.2.2.1-2. Location of VIT on the discussion forum …………………………. 119
Figure 5.2.2.1-3. VIT Nodes …………………………………………………………… 119
Figure 5.2.2.1-4. VIT Centrality ………………………………………………………... 119
Figure 5.2.2.1-5. MessageTag …………………………………………………………. 122
Figure 5.2.2.2-1. Lurkers overall view in VIT Centrality (right) ……………………. 124
Figure 5.2.2.2-2. Redesign of participation graphs …………………………………. 126
Figure 5.2.2.2-3 CeLE MessageTag tool ……………………………………………… 127
Figure 5.2.3.1-1. Discussion on tools’ Greek names ……………………………….. 134
Figure 5.2.3.1-2. Pedagogical Usability Attributes ................................................ 135
Chapter 6
Figure 6.1.1-1. The e-tutors in the online course ……………………………………. 138
Figure 6.1.1-2. Normality overview for tools pedagogical usability and utility
in HCE ……………………………………………………………….. 140
Figure 6.2.1-1. Participants’ location in Greece …………………………………… 142
Figure 6.5.7-1. GSN adjacency matrix in UCINET ……………………………….. 185
Figure 6.6.2-1. Correlations analysis in HCE 3.0 ………………………………… 204

xiii
Ph.D. Thesis Centre for Interactive Systems Engineering LSBU
Tools and Evaluation Techniques for Collaborative E-Learning Communities

Index of Graphs
Chapter 4
Graph 4.3.1.1-1. Online course categories in Moodle@GSN ……………………… 101
Graph 4.3.1.1-2. Comparison between number of messages and replies ............. 102
Graph 4.3.1.1-3. Activity in the online course …….…………………………………. 104
Chapter 5
Graph 5.2.2-1. Participating countries ……………………………………………….. 115
Graph 5.2.2.1-1. 31/03/2006 – 5/28 e-learners ……………………………………… 121
Graph 5.2.2.1-2. 04/04/2006 – 10/28 e-learners …………………………………….. 121
Graph 5.2.2.1-3. 08/04/2006 – 10/28 e-learners …………………………………….. 121
Chapter 6
Graph 6.2.2-1. Correlations between time in education, use of computers,
and Learning Management Systems (LMS) …………………….. 143
Graph 6.2.3-1. Communication with the educational authorities …………………. 146
Graph 6.3.1-1. Comparison between sent messages and
messages for analysis ................................................................. 149
Graph 6.3.1-2. Logs of overall activity VS posting ………………………………… 150
Graph 6.3.2.3-1. Active & Passive Participation locus
from the same participants ....................................................... 155
Graph 6.4.1-1. Comparison between messages for analysis, richness of text,
And discussion depth ……………………………………………. 159
Graph 6.4.2-1. CeLEs factors’ comparison graph …………………………………. 162
Graph 6.4.2-2. Comparison for number of words posted by e-learners
and e-tutors ……………………………………………………….. 164
Graph 6.5.1-1. Community evolution elements ……………………………………. 167
Graph 6.5.1-2. New members’ contributions ………………………………………… 170
Graph 6.5.1-3. Roles in the e-learning community ……………………………….. 172
Graph 6.5.2-1. Participants’ opinions on e-learning community elements …….. 173
Graph 6.5.2-2. Comparison of themes and community elements ………………. 174
Graph 6.5.3-1. Scatter plot for empathy …………………………………………… 175
Graph 6.5.5-1. Passive & Active Participation Process …………………………. 178
Graph 6.5.6-1. Elements that show community evolution: e-learning …………. 182
Graph 6.5.6-3. The e-learning facilitators …………………………………………. 183
Graph 6.5.6-2. Correlations between codes on collaborative e-learning quality 184
Graph 6.5.7.1-1. Reciprocal ties in GSN (a) & the research pool (b) ………….. 187
Graph 6.5.7.1-2. Structural equivalence dendrogrammes in GSN (all) ……….. 170
Graph 6.5.7.1-3. Structural equivalence dendrogrammes in the
research pool (all) …………………………………………… 170
Graph 6.5.7.1-4. Structural equivalence dendrogramme in GSN (e-learners) .. 191
Graph 6.5.7.1-5. Structural equivalence dendrogramme in the
research pool (e-learners) ………………………………… 191
Graph 6.5.7.3-1: VIT Nodes in CeLE IX ………………………………………….. 198
Graph 6.5.7.3-2. VIT Centrality in CeLE IX ………………………………………. 199

xiv
Ph.D. Thesis Centre for Interactive Systems Engineering LSBU
Tools and Evaluation Techniques for Collaborative E-Learning Communities

Index of Appendixes

Appendix I: Online Community Management …………………………………… 234


A_I_1. Moderator’s Responsibilities - Interactivity Management ........ 234
A_I_2. Suggestions for Writing Online Messages ............................... 235
Appendix II: Risk Management …………………………………………………… 236
Appendix III: Initial Questionnaire (sample) ……………………………………. 239
A_III_1. The usability section in the Initial Questionnaire ................... 239
Appendix IV: Questionnaire Main study I (sample) ……………………………... 240
A_IV_1. Demographic data .................................................................... 240
Appendix V: Questionnaire Main study II ………………………………………... 242
Appendix VI: Participants’ Documents ……………………………………………. 250
A_VI_1. Invitation to the study ................................................................ 250
A_VI_2. Netiquette ................................................................................. 250
A_VI_3. Instructions of use Moodle and the Research Pool ............... 251
Appendix VII: The E-mmersion Data Analysis …………………………………….. 252
A_VII_1. Demographics ........................................................................... 252
A_VII_2. Crosstabulation: Internet Use in class *
Use for educational purposes ......................................... 253
A_VII_3. Crosstabulation: Moodle Use * Time using LMS ..................... 253
A_VII_4. Moodle Usability ....................................................................... 253
A_VII_5. Training in Educational Technologies .................................... 253
A_VII_6. Reasons for participating in e-learning communities ............ 254
A_VII_7. Use of e-learning tools .............................................................. 254
A_VII_8. Messages Quantitative Analysis ............................................... 254
A_VII_9. Forums and users view log files ............................................... 255
A_VII_10. Participation in e-learning Communities ............................... 255
A_VII_11. Online course experience ...................................................... 256
A_VII_12. Messages Analysis: Collaborative E-Learning Episode I ... 257
A_VII_13. CeLE 1 (AIa-1:stanzas1-25) .................................................. 258
A_VII_14. Participant SP1: Thought Processes ..................................... 262
Appendix VIII: Conditions of working and learning online ……………………. 263
Appendix IX: Thematic Analysis in the Main Study ……………………………… 264
A_IX_1. Post-retreat opinions on participation ..................................... 264
A_IX_2. Post-retreat opinions on e-learners’ participation
in the project .................................................................... 264
A_IX_3. Post-retreat opinions on new members’ contribution ............ 265
A_IX_4. Post-retreat opinions on communities ..................................... 266
A_IX_5. Post-retreat opinions on other communities ........................... 266
A_IX_6. Post-retreat opinions on new members’ contribution ............ 267
A_IX_7. Post-retreat opinions on learning ........................................... 267
Appendix X: Collaborative e-Learning Episodes (Examples from
the Main Study ……….. ………………………………………………. 268
A_X_1. Collaborative e-Learning Episode III .......................................... 268
A_X_2. CeLE-III Locus (CeLE-CIII: Stanzas 4-23) .................................. 269
A_X_3. CeLE-III Analysis ........................................................................ 270
A_X_4. CeLE-III Code Network ............................................................... 270
A_X_5. Internalisation and externalisation thought process ............... 271
A_X_6. Collaborative e-Learning Episode IX ......................................... 271

xv
Ph.D. Thesis Centre for Interactive Systems Engineering LSBU
Tools and Evaluation Techniques for Collaborative E-Learning Communities

A_X_7. Internalisation and externalisation thought process ................. 276


A_X_8. CeLE-IX Locus (CeLE-GIX: Stanzas 2-84) .................................. 276
A_X_9. CeLE-IX Analysis .......................................................................... 277
A_X_10. CeLE-IX Code Network .............................................................. 278
Appendix XI: Messages Quantitative Analysis …………………………………….. 280
A_XI_1. Messages Quantitative Analysis in Moodle@GSN .................. 280
A_XI_2. Messages Quantitative Analysis in the Research Pool .......... 281
Appendix XII: Main Study Data & Reports …………………………………………. 282
A_XII_1. Pedagogical Usability – Utility results ………………………… 282
A_XII_2. Correlations in SPSS …………… ……………………………… 285
A_XII_3. Most important correlations in the
Hierarchical Clustering Explorer ….................................…....... 286
A_XII_4. The best thing in the project ……………………………………… 288
Appendix XIII: Recommendations ………………………………………………….. 289

xvi
Ph.D. Thesis Centre for Interactive Systems Engineering LSBU
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 1: Introduction

Introduction:

1 The problem of e-learning quality related to socio-cultural


learning and participation in e-learning communities

Key Topics Covered in This Chapter:

• What the study is about


• Why educators’ online training is important for their professional
development
• The problem of quality in online education as a distinction between
provision of information and acquisition of knowledge
• Aims and research questions
• Overall structure of the thesis

Chapter 1 introduces the research context, the research problem and the aims and
objectives of this study. Because socio-cultural learning theories and e-learning
design techniques evolved separately, they lack convergence. This is one of the
causes of quality problems in e-learning. An example is presented from the Greek
teachers’ e-learning community project aimed at their professional training and
development. An initial study showed that the mere provision of information did not
facilitate new-knowledge construction. Consequently, this study targets the
development of evaluation techniques and associated tools to enhance participation
in collaborative e-learning communities.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 1


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 1: Introduction

1.1 INTRODUCTION

For the past 50 years two main trends have been observed in education, the
socio-cultural focus and the use of technology. However, these two trends have
evolved almost separately. Socio-technical design and user-centred design were
planning approaches aiming to acknowledge that the development of interactive
technologies increasingly relies on an appreciation of the social circumstances in
which systems are used. Educational or instructional design is the systematic
processing of activities to solve an instructional problem with the aid of technologies.
Nonetheless, educational design and in particular e-learning design neglected the
dual and situated persona of the learner; she acts as both a user and a learner. In
addition, the e-learning systems were found to be information-based mainly
supporting monologue instead of being communication-based towards dialogue. For
this reason they fail to support e-learners’ transition between internalisation to
externalisation and becoming active participants. Thus, mere provision of information
points to poor e-learning quality. So, if educational design could understand the
technology of collaborative practice, e-learning quality could be improved.

This chapter introduces the concept of quality in e-learning and examines its
relationship to socio-cultural collaborative learning and associated design. The
research context is the Greek teachers’ e-learning community, started in 2003 as part
of a project for online teachers’ training and aimed at enabling teachers to acquire
new competencies. However, these aims were not met because of passive
participation and this implies that information acquisition may not be automatically
related to collaborative learning. It appears that learning within e-learning
communities is not always successful.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 2


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 1: Introduction

1.2 EDUCATORS’ ONLINE TRAINING

Education is generally acknowledged as one of the crucial components of


personal and professional development. The integration of Information and
Communication Technology (ICT) in education as well as the social and collaborative
nature of the Internet provided another medium for communication and training;
however, despite the advantages, the problem of e-learning quality is evident
worldwide.
In his foreword for the United Nations Educational, Scientific and Cultural
Organization (UNESCO) report (2002), Daniels said that within a short time ICT has
become one of the basic building blocks of the modern society. Furthermore, the
current shift occurring in the Web from a static content environment where end users
are the recipients of information—defined as Web 1.0—to one where they are active
content creators—defined as Web 2.0—can be described as a transition to a more
distributed, participatory, and collaborative environment (Delich, 2006). Web 2.0 is a
platform where “knowledge-working is no longer thought of as the gathering and
accumulation of facts, but rather, the riding of waves in a dynamic environment”
(Downes, 2005). To Berners-Lee (2007), the Web is not only a technological tool but
also a social phenomenon that enables collaboration and creativity.

ICT is the backbone of the knowledge economy and has been recognised as an
effective tool for promoting economic growth and development (World Bank Report,
Chen & Kee, 2005). Despite the expansion of ICT, general access to ICT varies
across continents, and countries even within the same continent, indicating a digital
divide (Reddy & Manjulika, 2002). Organisations, educational institutions and
business have been investing in the use of ICT in Education, or in what ESRC now
calls Technology Enhanced Learning (TEL) (ESRC, 2006). E-Learning is a
component of TEL and describes learning via the Internet, intranet, and extranet (WR
Hambrecht and Co, 2000:8). The freedom that e-learning offers and the increasing
number of online courses provided by educational organisations offer new
opportunities for personal and professional development in a life-long learning
course. Nevertheless, teachers’ education has been severely criticized on the
grounds of both quantity and quality (e.g. Perraton & Potashnik, 1997; Darling-
Hammond & Bransford, 2005; Thompson & Schmidt, 2007).

To solve this problem, UNESCO suggests that countries need to keep pace with
technological development and the changing competencies, reflected in the

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 3


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 1: Introduction

curriculum and teacher training. Economic advantage will accrue to a population that
acquires competencies in processing information into knowledge and applying it in
work and everyday life. These competencies are not only related to using the devices
but also working on procedures that give access to information and skilfully
transforming information into knowledge. As this is the task of the educator,
educational systems will become a national resource as important as the traditional
factors of production-land, labour, and capital. This in turn would cause educators to
become more important, their productivity and their wages should increase, but they
can also expect the nature of their jobs to change with a great deal of specialization
(UNESCO, 2002:633-640). To support this life-long learning context, companies and
institutions often use commercial Learning Management Systems (LMS) for online
teachers’ training. Commercial LMS like Centra, and Blackboard, and Open Source
software as Bodington, Dokeos or Moodle are nowadays widely used; for example,
there are more than 26,124 registered sites from 182 countries, 1,855 in the U.K.
(http://moodle.org/sites/, last access 29/05/2007), and 97 in Greece, one of them is in
the service provided by the Greek School Network (GSN).

However, e-learning international and national projects, such as UK eUniversity,


Universitas 21, or Global University Alliance, have not succeeded in meeting a
number of promises (Garrett, 2004; Oliver, 2005). Research studies have
demonstrated that e-learning has both positive and negative impacts in terms of
effectiveness and achievement of outcomes (Franklin et al., 2001; Sims et al., 2003).
One of the reasons is because e-learning has created confusion between the mere
supply of information and knowledge-building (Barbera, 2004; Whatley, 2004).

A number of projects have addressed teachers’ online training using LMS


successfully, for example, the international project ‘Tapped In’ on a voluntary basis
(Schlager & Fusco, 2004), the Australian National Quality Schooling Framework
(NQSF) (Hartnell-Young, et al, 2006), or the European projects ‘Implementing
Standards for European e-Tutor Training’ (on going, 2006-2007 1 ) and ‘E-Learning
Fundamentals’ (started in May 2007 2 ). However, not all projects were successful; an
example that failed to engage teachers in an e-learning community is the European
Minerva Project ‘Star Science’ aimed at collaboration between science teachers from
Ireland, UK, and Bulgaria (Harvey, 2003). According to Harvey, the UK failed to

1
Leonardo DaVinci project ‘Implementing Standards for European e-Tutor Training’ (ISEeTT,
http://www.etutorportal.net/). ISEeTT aims to define the core curriculum and quality standards for European e-Tutor
training in relation to different national contexts. The Greek partner is the Educational Research and Evaluation
Group, Foundation for Research Technology - Hellas (http://www.forth.gr/).
2
There was no additional information for this project other than the website: http://fecone.passionforlearning.eu/.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 4


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 1: Introduction

participate and Bulgaria’s participation was minimal. Similarly, there was no


significant evidence of socio-cultural learning in the European project ‘E-learning
Fundamentals’ in which I took part in June of 2007.

In short, e-learning outcomes seem to be unpredictable.

1.2.1 Greek Teachers’ Online Training

In Greece, teachers’ training is mainly onsite and organised by the Greek


Pedagogical Institute in collaboration with the Greek Ministry of Education and
Religious Affairs. There were efforts to facilitate online teacher training either from
universities or governmental organisations in collaboration with Higher Education
Institutes. This means that isolated courses were developed as part of PhD studies,
for example at the University of the Aegean (Hlapanis & Dimitrakopoulou, 2005) or
projects based on collaborative activities between the Greek Pedagogical Institute,
the Research Academic Computer Technology Institute, and other Greek
universities. One of these initiatives is the use of Moodle as part of the Greek School
Network (GSN) services; Moodle@GSN was built to aid Greek teachers’ online
training by developing an online community. (There is no agreement between
researchers on a single definition of community. In this study, a community is “a
group of people who consciously share a sense of belonging anchored in common
interests and enhanced by social interactions”. An online community is a community
where social interactions are facilitated by information and communication
technologies.)

However, Moodle@GSN appears not to have worked in that there has been a high
level of passive participation, that is absence of posting, for more than three years
(1077 days on the 13/10/2006 according to the log files). Although the Greek
teachers do visit and download material as apparent from the log files, there is no
evidence that the Greek teachers use the e-learning service effectively and skilfully to
transform information into knowledge. Following one of the participants in one of the
studies “absence of participation in GSN is probably because (desire, time, money
for) planning and organisation are lacking. Collaboration in design and planning by all
stakeholders to bring pedagogy and technology together is lacking as the most
valuable idea/suggestion: learner / user involvement in the design”. It also appears
that broad collaboration is now crucial for contemporary organisations.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 5


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 1: Introduction

Even though e-learning has delivered mixed results, some advantages have been
reported in using e-learning for teachers’ training (Golian, 2000): e-learning is
individualised and self-paced; there are more opportunities to access learning
resources; e-learning is based on activities and experience (active and experiential
learning) within groups and communities (collaborative learning); time and cost are
less because of the use of the electronic form of resources; and communication is
nonlinear. Nevertheless, there are several obstacles: institutional, instructional,
technical, and personal (Piotrowski & Vodanovich, 2000).

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 6


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 1: Introduction

1.3 QUALITY IN ONLINE EDUCATION

Online Education can support employees’ new competencies and training.


However, despite the huge investments in technology and e-infrastructure, and the
high levels of interest among educators, administrators and policy makers worldwide,
e-learning remains an unproven experiment. Nevertheless, the collapse of several
initiatives may not indicate the failure of the e-learning concept per se but rather a
lack of quality. The questions that arise are related to what constitutes e-learning
quality, why it is important, and whether quality assurance is feasible.

In general, quality refers to fitness for purpose, and in this context as applied to
learning (Stephenson, 2005). A survey on e-learning quality for CEDEFOP, the
European Agency for Vocational Training, showed that 61% of the 433 respondents
rated the overall e-learning quality somewhat negatively, as ‘fair’ or ‘poor (Massy,
2002:3). The European Foundation for Quality in eLearning (EFQUEL) conducted a
European survey between 15 August 2004 and 15 November 2004 (Panorama
Report, Ehlers, et al., 2005). In this survey, 5,023 people called up the questionnaire,
of whom 28 % actually completed it, and a further 7% finished the two basic sections
on quality in e-learning. According to the results, quality relates to obtaining the best
learning achievements (50%) and ‘something that is excellent in performance’ (19%).
In brief, the PANORAMA report revealed the importance of e-learning quality: the
need for critical awareness; the need for specific analytical frameworks as the
respondents although believed that they knew about quality they showed a general
lack of information; and the need for quality requirements in e-learning design. The
researchers stressed that ‘learners must play a key part in determining the quality of
e-learning services’ and insisted on the involvement of all e-learning participants in
quality design and development by 2010 (p.11).

Overall, design for socio-cultural learning appears to be connected to e-learning


quality; this is not only a Greek but also a global phenomenon. Socio-cultural
learning is related to e-learning systems and tools used to facilitate and support
socio-cultural learning principles and in particular collaborative learning. UNESCO’s
definition suggests that collaborative learning takes place:

when learners work in groups on the same task simultaneously,


thinking together over demands and tackling complexities.
Collaboration is here seen as the act of shared creation and/or

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 7


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 1: Introduction

discovery. Within the context of electronic communication,


collaborative learning can take place without members being
physically in the same location.
Technology & Learning definitions, UNESCO (n.d.)

This means that collaborative learning is related to co-creativity and has the potential
to occur online. Therefore, interventions to support social interactions, collaborative
learning and associated tools may influence e-learning quality. It also suggests that if
one of the elements is missing then e-learning quality may be impaired.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 8


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 1: Introduction

1.4 AIMS AND RESEARCH QUESTIONS

The problem this research addresses is the lack of e-learning quality as


evidenced by the Greek teachers’ passive participation in e-learning communities.
Since this appears to be a common problem it is worth investigating. Thus, the goal
of this thesis is to shed light on passive participants in e-learning communities and
propose conceptual frameworks to ensure their participation. More specifically, this
study aims to carry out the following in regard to Collaborative e-Learning
Communities:

The study translates into two sets of questions in order to study the conditions to
tackle e-learning quality. The first set of research questions will be investigated in the
literature review. These are:

Q1. In what ways can studies of Collaborative e-Learning Communities (CeLC)


be exploited in a concrete way by educational designers?
Q1.A. In what ways has collaborative learning research evolved up to the
present?
Q1.B. Are there any collaborative e-learning frameworks to support CeLC?

Q2. Is there an educational design approach that can ensure quality in specific
e-learning contexts such as the Greek teachers’ community?
Q2.A. What is design?
Q3.B. Are there any design principles for specific educational contexts?

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 9


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 1: Introduction

Q3. Are there any tools and techniques that can be used to facilitate the
formation of CeLC? If so, what are the effective characteristics and usage of
these tools?
Q3.A. Are tools and techniques for evaluating participation helpful to enable
participation?
Q3.B. Are tools for observing and analysing interactions helpful to enable
participation?

The second set of questions has an exploratory nature and will be based on the
previous results:

1. Collaborative e-Learning Communities (CeLC)


Q1.Ex1. Is there a collaborative learning scheme to identify, analyse and
evaluate CeLC?
Q1.Ex2. Are there any evaluation techniques that can be used to facilitate the
formation and maintenance of CeLC?

2. Educational Design
Q2.Ex1. In what ways can core design principles be integrated in the process
of educational design?
Q2.Ex2. In what ways can quality by design be achieved for the Greek e-
learning communities?
3. Tools to Support CeLC
Q3.Ex1. Are tools for evaluating participation helpful to enable participation?
Q3.Ex2. Are tools for structuring collaborative learning helpful to enable
participation?
Q3.Ex3. Are tools for observing and analysing interactions helpful to enable
participation in the Greek e-learning community?
Q3.Ex4. Are tools for observing reporting information on interactions helpful in
enabling participation in the Greek e-learning community?

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 10


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 1: Introduction

1.5 OVERVIEW OF THE CHAPTERS

The chapters in this thesis are developed as follows:

Table 1.5-1. Chapters Overview

TOOLS AND EVALUATION TECHNIQUES


FOR COLLABORATIVE E-LEARNING COMMUNITIES

Chapters Description
Motivation, research problem
Aims and objectives
1 Introduction
Research questions
Chapters overview
Collaborative e-learning
E-Learning communities and participation
2 Literature review Tools and evaluation techniques
Design
Conceptual framework
3 Research design Research methodologies
Understanding the research context
4 Research context Ethnotechnological inputs
Preliminary studies
5 Tools and evaluation techniques Planning and design
Implementation of previous conceptual framework
6 Main study Findings
Discussion
Summary of the Thesis
Conclusions
7 Conclusions
Recommendations
Future trends

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 11


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 1: Introduction

REFERENCES
Barbera, E. (2004). Quality in virtual education environments. British Journal of
Educational Technology, 35(1), 13-20.
Berners-Lee, T. (2007). Berners-Lee warns of changes ahead. Computing Magazine.
Retrieved 17/06/2007, from
http://www.computing.co.uk/computing/analysis/2186086/berners-lee-warns-
changes-ahead.
Blackboard (n.d.). Retrieved 15/02/2006, from http://www.blackboard.com/.
Bodington (n.d.). Retrieved 15/02/2006, from http://www.bodington.org/.
CENTRA Education & Training (n.d.). Retrieved 15/02/2006, from
http://www.centra.org.uk/.
CEDEFOP (n.d.). Cedefop: The European Centre for the Development of Vocational
Training. Retrieved 21/08/07, from http://www.cedefop.europa.eu/.
Chen, D.H.C. & Kee, H.L. (2005). A Model on Knowledge and Endogenous Growth
World Bank Policy Research Working Paper 3539. Retrieved 22/10/2006,
from http://info.worldbank.org/etools/library/latestversion.asp?135703.
Daniels J.S. (2002) “Foreword” in Information and Communication Technology in
Education – A Curriculum for Schools and Programme for Teacher
Development. Paris: UNESCO
Darling-Hammond, L. & Bransford, J. (2005). Preparing Teachers for a Changing
World: What Teachers Should Learn and be Able to Do. San Francisco, CA:
Jossey-Bass.
Delich, P. (2006). Pedagogical and interface modifications: What instructors change
after teaching online. Published doctoral dissertation, Pepperdine University,
Malibu, CA. Retrieved 10/09/2007, from
http://proquest.umi.com/pqdlink?Ver=1&Exp=09-09-
2012&FMT=7&DID=1144195631&RQT=309&attempt=1.
Dokeos (n.d.) Retrieved 15/02/2006, from http://www.dokeos.com/.
Downes, S. (2005). E-learning 2.0. eLearn Magazine. Association for Computing
Machinery, Inc. Retrieved 12/04/2006, from
http://elearnmag.org/subpage.cfm?section=articles&article=29-1
EFQUEL (n.d.). European Foundation for Quality in eLearning. Retrieved
20/08/2007, from http://www.qualityfoundation.org/.
Ehlers, U., Hildebrandt, B., Görtz, L. & Pawlowski J. (2005).Quality in European e-
learning – PANORAMA report. A study by the European Quality Observatory.
Retrieved 12/11/2006, from http://cms.eun.org/shared/data/pdf/quality_in_e-
learning_panorama_en.pdf.
ESRC (2006). Economic and Social Research council. Retrieved 13/05/2006, from
http://www.esrcsocietytoday.ac.uk/ESRCInfoCentre/opportunities/current_fun
ding_opportunities/tel_call.aspx?ComponentId=14262&SourcePageId=5964.
Franklin, S., Peat, M., Lewis, A., & Sims, R. (2001). Technology at the cutting edge:
A large scale evaluation of the effectiveness of educational resources. Paper
presented at ED-MEDIA, the 13th Annual World Conference on Educational
Multimedia, Hypermedia & Telecommunications. 25-30 June 2001, Tampere,
Finland.
Garrett, R. (2004). The real story behind the failure of the UK eUniversity. Educause
Quarterly, 27(4), 3-6.
Golian, L. M. (2000), ‘Utilizing Internet Resources by Education Professionals in the
New Millennium’, Information Technology and Libraries, 19(3): 136-143.
GSN (n.d.). The Greek School Network. http://www.sch.gr/en/.
Hartnell-Young, E. & McGuinness, K. (2006). Evaluation of an Online Community:
Australia's National Quality Schooling Framework. In Niki Lambropoulos and
Panayiotis Zaphiris (Eds) (2006). User-Centred Design of E-Learning
Communities. Hershey, USA: Idea Group.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 12


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 1: Introduction

Harvey, J. (2003). Supporting Communities in Practice: an exploration of two case


studies. Paper presented at the Conference Communities of Practice, 10th
Association for Learning Technology Conference (ALT-C), 8-10 September
2003, The University of Sheffield & Sheffield Hallem University (UK).
Hlapanis, E., & Dimitrakopoulou, A. (2004). Greek Teachers' Training via the Internet:
Presentation of the Greek Online Community KME from the University of the
Aegean. Paper presented at the 4th Conference in ICT in Education, 29
September – 3 October 2004, Athens, Greece.
Massy, J. (2002). Quality and eLearning: BizMedia. Retrieved 21/06/2003, from
http://www.elearningage.co.uk/docs/qualitysummary.pdf.
Moodle (n.d.) Modular Object-Oriented Dynamic Learning Environment. Retrieved
20/08/2007, from http://moodle.org.
Moodle@GSN (n.d.). Moodle at the Greek School Network. Retrieved 21/08/2007,
from http://e-learning.sch.gr/.
Oliver, R. (2005). Quality assurance and elearning: Blue skies and pragmatism. ALT-
J, Research in Learning Technology, 13(3), 173-187.
Peratton, H. & Potashnik, M. (1997) Teacher Education at a Distance. Education and
Technology Technical Notes, Vol. 2, No 2.
Piotrowski, C. and S. J. and Vodanovich (2000), ‘Are the Reported Barriers to
Internet-Based Instruction Warranteed?, A Synthesis of Recent Research
Education 121(1), 48-53.
Reddy, V.V. & Manjulika, S. (2002) The Changing Context of Higher Education in the
21st Century, in V.V. Reddy and S Manjulika (eds.) Towards Virtualization-
Open and Distance Learning, pp. 1-47. New Delhi: Kogan Page.
Schlager, M., & Fusco, J. (2004). Teacher professional development, technology,
and communities of practice: Are we putting the cart before the horse? In S.
Barab, R. Kling, & J. Gray (Eds.), Designing for Virtual Communities in the
Service of Learning. Cambridge, UK: Cambridge University Press. Pp. 120-
153. Retrieved 22/05/2006 , from
http://tappedin.org/tappedin/web/papers/2003/TPDBarab.pdf.
Sims, R., Dobbs, G., & Hand, T. (2001). Proactive evaluation: New perspectives for
ensuring quality in online learning applications. Paper presented at the 18th
Annual Conference of the Australasian Society for Computers in Learning in
Tertiary Education (ASCILITE), Meeting at the Crossroads. Melbourne,
Australia, December 9-12, 2001, pp. 509 – 517. Retrieved 22/11/2006, from
http://www.ascilite.org.au/conferences/melbourne01/pdf/papers/simsr.pdf.
Stephenson, J. (2005). Definitions of indicators of quality on the application of ICT to
University Teaching. Paper presented at a workshop for the European
Schoolnet in Saragossa, Spain. Retrieved 12/11/2006, from
http://cms.eun.org/shared/data/pdf/qual_onlinehe.pdf.
Thompson, A., & Schmidt, D. A. (2007). Winter 2006-2007. Journal of Computing in
Teacher Education (JCTE), 23(2).
UNESCO (n.d.) Technology & Learning definitions: Collaborative Learning. Retrieved
12/07/2004, from
http://www.unesco.org/education/educprog/lwf/doc/portfolio/definitions.htm.
UNESCO (2002). Information and Communication Technology in Education – A
Curriculum for Schools and Programme for Teacher Development. Paris:
UNESCO. Retrieved 22/10/2007, from
http://unesdoc.unesco.org/images/0012/001295/129538e.pdf.
Whatley, J. (2004). An Agent System to Support Student Teams Working Online.
Journal of Information Technology Education, 3, 53-63. Retrieved 24/10/2005,
from http://jite.org/documents/Vol3/v3p053-063-033.pdf.
WR Hambrecht & Co (2000). Corporate E-learning: exploring a new frontier.
Retrieved 14/03/2006, from http://www.astd.org/NR/rdonlyres/E2CF5659-
B67B-4D96-9D85-BFAC308D0E28/0/hambrecht.pdf.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 13


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Literature Review:

2 Design for Collaborative e-Learning Communities

Key Topics Covered in This Chapter:

• Why collaborative learning is important for today’s e-learners

• How and why educational research moved from the study of the
individual to the study of the community

• Participation in online and e-learning communities

• How design supports the user as a learner within a community

• The need for further investigation for tools and evaluation


techniques to support collaborative e-learning communities

• New propositions

Chapter 2 discusses the importance of collaborative learning to effective e-


learning. Collaborative learning research as well as current design for e-learning
suggest that the research focus has evolved from the study of the individual to
the study of the community. However, there are still questions about instructional
design as well as quality in e-learners’ interactions. Investigation of the literature
pointed to new propositions for tools and evaluation techniques to enhance
participation in collaborative e-learning communities.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 14


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

2.1 INTRODUCTION

Education as a discipline was initially anchored in Cognitive Psychology


and Pedagogy. Only recently, has learning with the use of tools introduced
design issues. This chapter attempts to shed light on the multidisciplinary nature
of e-learning, to explore its social nature and the role of tools in collaborative e-
learning communities.

2.2 COLLABORATIVE LEARNING

According to Lave (1988:1), cognition is a complex social phenomenon


that occurs within the individual’s head. It refers to intermediate variables that
describe social interactions and their relationships with the conditions that
facilitate learning. Because the intermediate variables are invisible, their
observation and study is difficult. However, it is possible to study human activity
and inference from this cognitive change. Activity in the form of discussion of
shared experience has been considered an effective means for adult learning
(e.g. Brookfield, 1990; Brown & Duguid, 2000). Nonetheless, participation in
discussions can be active, where the individual participate by posting, and
passive, where the individual does not. Consequently, without active participation
passive participation is not possible.

Other than the importance attached to socio-cultural learning, collaborative


learning as well as passive and active participation in dialogue are concepts also
related to recent approaches to adult learning. Knowles (1984) proposed the
theory of andragogy (adult learning) to complement pedagogy (child learning):
adults were responsible for their learning; there should be specific targets relative
to their backgrounds; instruction should be task oriented; and discovery should
be guided and facilitated thus the relationship between the instructor and the
learner needs to be redefined. In reality, adults not only accept but also pursue
passive learning (Rogers, 2002; Jarvis, 2004). Mezirow (2000), in his theory of
transformative learning, proposed that adults were reflective, critical, and open to
others’ opinions; thus, active participation in groups was essential. However,
external factors such as organisational and financial problems could be major
obstacles (e.g. Brookfield, 2001); additionally, adults were less flexible towards

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 15


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

change and for this reason, they did not acknowledge their passive participation
(Illeris, 2002). Nonetheless, Mezirow proposed that dialogue, the collaborative
learning pillar, was the solution.

Two terms have been used interchangeably in collaborative learning history:


cooperation and collaboration. Cooperation was the basis of sociability “acting
together, in a coordinated way at work, or in social relationships, in the pursuit of
shared goals, the enjoinment of the joint activity, or simply furthering the
relationship” (Argyle, 1991:15). (Johnson & Johnson, 1994 provide an overview
on cooperative learning.)

An attempt to propose a distinction between cooperation and collaboration was


made by Teasley and Roschelle (1993):

Collaboration is a coordinated, synchronous activity that is the result


of a continued attempt to construct and maintain a shared conception
of a problem… Cooperative work is accomplished by the division of
labour among participants, as an activity where each person is
responsible for a portion of the problem solving.
Teasley & Roschelle, 1993:235

Teasley and Roschelle provided a clear distinction anchored in the idea that
tasks are divided between participants: “each person is responsible for a portion
of the problem solving”. However, such division is not deliberately required from
the participants, although roles exist naturally as a spontaneous division of
labour. In their review, Borgers and Baranauskas (2003) advocated collaborative
learning as it had more advantages than other types of group learning. It
empowered and enabled learners to solve problems and understand subjects
more easily since discussing ideas and constructing arguments through dialogue
could shape in-depth learning. Collaboration is an interactive process that
engages two or more participants working together to achieve outcomes they
could not accomplish independently (Salmons & Wilson, 2008). UNESCO’s
definition embraces most of the aforementioned concepts; collaborative learning
occurs

when learners work in groups on the same task simultaneously,


thinking together over demands and tackling complexities.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 16


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Collaboration is here seen as the act of shared creation and/or


discovery. Within the context of electronic communication,
collaborative learning can take place without members being
physically in the same location.
Technology & Learning definitions, UNESCO (n.d.)

Thus shared creativity for new knowledge building is the ultimate collaboration
learning target that can also be expanded to online settings. Working on shared
tasks implies learners’ participation and engagement.

However, some e-learners do not actively participate but lurk 1 . According to the
Free Online Dictionary for Computing (http://foldoc.org, added on 14/06/1997),
lurking is a messaging jargon for activity of one of the "silent majority" in an
electronic forum; it is posting occasionally or not at all, but reading the group's
postings regularly. This term was not pejorative; for example, reading the
Frequently Asked Questions was recommended netiquette for beginners who
needed to learn about the history and practises of the group before posting. This
distinction of passive and active participation appeared in computer-supported
tasks; the participant who controls the mouse tended to be "executor", while the
other was likely to be the "reflector" (Blaye et al., 1991). According to Miyake
(1986:174): "The person who has more to say about the current topic takes the
task-doer's role, while the other becomes an observer, monitoring the situation.
The observer can contribute by criticising and giving topic-divergent motions,
which are not the primary roles of the task-doer." Miyake referred to active as
well as reflective learning since criticism could support new knowledge
production.

If passive and active modes are acceptable, there should be some processes
and pedagogical methodologies to create and maintain the transition between
these. Rather than the facilitator or educator providing rules for learning (Berge &
Collins, n.d.; Wegerif, et al., 1998:495), learners took responsibility for their own
learning, as, according to Mercer (1995), they had to: talk to create the context;
engage in collaborative learning activities; create common knowledge; and follow
ground rules that encourage the exchange of relevant ideas and active
participation. Initial intention is another prerequisite, important enough to form a
1
Passive participation and lurking will be used interchangeably in this thesis as a demand for changing the term
lurker was observed in the literature as well as online communities and conferences (e.g. E-mint community,
the JISC, Joint Information Systems Committee online conference discussions on ‘Innovating E-Learning’,
http://www.jisc.ac.uk/elp_conference07).

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 17


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

cooperative principle needed before interactions occur (Grice, 1975). This idea
was further developed in psychology as the principle of the “least collaborative
effort” (Clarke & Wilkes-Gibbs, 1986:26), as an active concern for the
construction of mutual understanding in order to develop a shared focus and a
shared narrative (Crook, 1994:176). In other words, it is the e-learners’ shared
goals and generated context that make collaborative learning occur; then,
collaborative e-learning continuity creates the shared narrative that fabricates its
history and builds a successful e-learning community. Prerequisites also refer to
a joint interactive space for grounding, as interactions intended to create
common ground (Clark et al, 1983), mutual understanding, knowledge, beliefs,
assumptions or presuppositions (Baker et al., 1999; Mäkitalo et al., 2001) or
“repairing” misunderstandings (Dillenbourg, 1999). Therefore grounding is built
on sociability. This means that sociability creates the initial conditions for
collaborative learning.

Overall, it appears that sociability is the basis for collaborative learning


supporting both a reflective and active mode. Shared goals and activities,
cooperation and collaboration through dialogue, spontaneous roles and division
of labour, co-creativity as exploration and discovery, and enjoinment of the joint
activity are concepts found in collaborative learning. According to Dillenbourg
and colleagues (1996), the key to supporting collaboration is to find suitable
intermediate variables to describe and support collaborative interactions and their
relationships with the conditions that facilitate collaborative learning.

2.2.1 From the Individual to the Community:


a Brief History

Dillenbourg and colleagues (1996), said that the development of an


understanding of collaborative learning began with the learner as an individual;
then it moved to group learning in a more socio-cultural mode, and finally,
expanded to the community. During the 70s and early 80s, research was focused
on the individual’s learning processes. The context of their interaction was seen
as a backdrop rather than the focus of research in its own right. When the group
became the unit of analysis, the focus shifted to the social construction of
knowledge; however, this was still on the basis of studying individuals. In terms
of empirical research, the focus was on comparative processes to establish

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 18


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

whether and under what circumstances collaborative learning was more effective
than learning alone. Because collaborative learning is inherently complex, it was
almost impossible to establish causal links between the conditions and the
effects of collaboration. Therefore, Dillenbourg and colleagues indicated the need
for new tools and methods for observing and analysing interactions to increase
understanding of the collaborative learning social mode. This development of an
understanding of learning is briefly presented next.

In Piaget’s early writings (1932), the potential productivity of peer interaction in


relation to cognitive development was related to the achievement of concrete
operational modes of thought in the early years. Egocentrism was the main
obstacle to operational thinking and requires its “decentralisation”. This is the
ability to take into account multiple points of view and multiple covarying factors
in a given situation. One of the fundamental concepts that helped collaborative
learning to evolve was the socio-cognitive conflict derived from the interaction
with other learners as a result of decentralisation. Socio-cultural researchers
were able to build on the egocentric thought as the inner dialogue, and expand it
with the outer dialogue required in socio-cultural contexts.

The second major theoretical foundation that influenced collaborative learning


was the socio-cultural approach. In brief, it advocated that knowledge acquisition
was based on the alignment of asymmetrical interactions between learners and
more capable peers (Vygotsky, 1962, 1978). The research focus was shifted to
the causal relationship between social interaction and individual cognitive change
stressing the importance of social activity within a group to promote cognitive
development. Dialogue as social speech was the medium of communication, and
inner dialogue was the medium for self-regulation. To Light and Littleton (1999),
Vygotsky’s work introduced two significant concepts, the “zone of proximal
development” (zpd) and “scaffolding” (Wood, Bruner & Ross, 1976); the
individual could reach a higher level of development with the help of a more
capable other. Socio-cultural learning attached significance to the level of
symmetry/asymmetry between the members of a group. In fact, Tharp and
Gallimore (1988) suggested that peer assistance at the same level of asymmetry
is required so that peers can help themselves. The individual often gets more of
a chance to participate actively in critical planning and decision making when
interacting with an expert tutor, more capable peers or other peers. Studies on
this relationship led to the community knowledge building approach (e.g. Lave &

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 19


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Wenger, 1991). Here, the learning process occurred within a larger physical and
social context of interactions and culturally constructed tools and meanings.
Social and individual were not different levels of study but inexorably
interconnected; this created two distinct traditions of situated cognition, one
focused on the individual and a second focused on community (Wilson & Myers,
2000). The community focus is next.

There were several community definitions; for example, Peck (1987) stated that
“If we are going to use the word [community] meaningfully we must restrict it to a
group of individuals who have learned how to communicate honestly with each
other, whose relationships go deeper than their masks of composure, and who
have developed some significant commitment to "rejoice together, mourn
together," and to "delight in each other, make others' conditions our own." The
development of communication channels, trust, support, and a sense of
belonging seemed to be significant to help a community to emerge. Lazlo and
Lazlo (1997, 2000) described the community as a group of two or more
individuals with a shared identity and a common purpose, committed to the joint
creation of meaning. The authors echoed Lave and Wenger in Communities of
Practice (CoP) (1991); CoP members shared the characteristics of joint
enterprise, mutual engagement and shared repertoire to clarify, define, and
evolve practices (Wenger, 1998). One implication of the social reproduction of
CoP was that the sustained participation of newcomers, as they become old-
timers, must involve conflict between the forces that support processes of
learning and those that work against them. This was because learning,
transformation, and change were always mutually present. Thus, CoPs were
engaged in the generative processes of producing their own future. Learning in
communities was configured through the process of becoming a full participant in
practice and being able to get involved in new activities, perform new tasks and
functions, and master new understandings.

Thus learning is the process of participating in communities. It was proposed that


this centripetal process of engagement was proposed to be legitimate for all
participants. Lave and Wenger (1991) called this learning process Legitimate
Peripheral Participation (LLP). More specifically, LPP was a decentred model
where a specialist field contained different levels en route for newcomers’
engagement and practice. The peripheral members drifted into the centre as their
interests were stirred. Thus, there were hierarchical levels of engagement
depending on many factors, both external and internal to the participants. The

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 20


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

newcomers were located at the first level, as potential contributors, and were not
voiceless or powerless but a vital part of the community. Overall, Lave and
Wenger proposed that LPP was a descriptor of engagement in social practice
that entailed learning as an integral constituent. In fact, central and peripheral
participation should not be on different levels after all (1991:35). The next table
describes CoP, the research focus, the key tenets, the research methods and
results (Table 2.2.1-1):

Table 2.2.1-1. The Socially-Shared Cognition Approach: Communities of Practice

Communities of Practice (CoP)

Research
Research Focus Key Tenets Results
Methods
-Learning is situated -Inter - -There are
-The individual, the and grounded in disciplinary different types
social and the everyday actions. research of
physical is one -The situation needs -Real settings communities.
research context. to be studied as a -Ethnographic -Conflicts may
whole. inputs allow
Learning is a -CoP characteristics: resolutions.
process of joint enterprise, -The
participating in mutual engagement, conditions that
CoP. shared repertoire facilitate
-Legitimate participation:
Research Peripheral -management
Questions Participation of the
increases in participation
What is learning engagement, process
within a complexity and -accessibility
community? responsibility. to information,
-Learner’s &
Is learning via participation in the -tools that
interactions with community lies in the enable
other individuals concept of “becoming members’
more efficient than part of the participation
learning alone? community. and learning.
-Lurkers are
What are the types legitimate participants
of social and community
engagement that members.
facilitate learning? -Tools are part of the
community’s heritage
What are the social and cultural life.
conditions that
facilitate learning?

It appears that the research focus involved the individual, the social as well as
their situated context where the individual participates in collaborative practices.
For this reason, multidisciplinary methods needed to be evolved to investigate

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 21


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

learning on an individual and social level as well as the tools and the
environment of participation. Thus, the sense of belonging appeared to be the
driving force for collaborative learning enhanced by tools.

A project focused on lurking in CoP that accommodated community


management, explored the meaning of “legitimate peripheral participants”
(McDonald, 2003) viewed as occurring in separate levels (Figure 2.2.1-1):

Figure 2.2.1-1: The Eyeball of Participation

The participants were located in four levels, based on the “numeric amount” of
posting to the community (number of messages). With a direction from the
periphery to the centre, lurkers were the participants who did not exhibit any
activity. On the second level there were the members who occasionally
contributed to the community. The participants and key contributors were located
on the third and fourth level. The “eyeball of participation” provided a structure to
better understand legitimate peripheral participation. McDonald and colleagues
stressed the fact that the active members are the ones who “add value” and fill
the gaps for all members in order to sustain the community.

Learning in communities was built on social interactions. Identity (e.g. Donath,


1996; Lee et al., 2006) and the sense of belonging (Nonnecke, 2000) appeared
to be essential concepts in community research. In fact, the latter was suggested
as the basic factor that distinguished a guest from a member in an online
community and was initially studied in relation to empathy 2 (e.g. Preece, 1999,

2
Empathy is a “complex psychological inference in which observation, memory, knowledge and
reasoning are combined to yield insights into the thoughts and feelings of others” (Ickes, 1997:2)

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 22


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

2004; Preece & Ghozati, 2001; Lambropoulos, 2005a, 2005b). Nonnecke and
Preece (2000:127) found that some lurkers felt a sense of community, especially
when the dialogue engenders “a sense of trust and care”; this was an indirect
way to become active contributors. Lambropoulos found that the members who
developed empathy became active participants (2005a). Ramachandran
suggested that empathy was a cognitive activity triggered by the mirror neurons
(2000) and appeared to be a key in human communication regardless of the
medium used.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 23


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

2.3 ONLINE & E-LEARNING COMMUNITIES

While educators and learners in classroom-based courses have


already discovered the benefits of an engaged learning approach to
education, the power of engagement in online courses is yet to be
fully realised.
Conrad & Donaldson, 2004:ix

After Rheingold’s book ‘The Virtual Community’ (1993), the idea of


simulating a community remained popular. Researchers had examined a range
of phenomena related to online and text-based discussion groups including
member contribution patterns, sustainability, and motivations to contribute (e.g.
Rheingold 1993; Constant et al, 1996; Nonnecke, 2000; Wasko & Faraj 2004;
Gulati 2006; Butler et al., 2007). An online community referred to people who
make the community where group dynamics, needs and roles shaped the
community; purposes, that is people come together for a purpose(s); and policies
that is the behaviour governed by group norms, rules and sometimes formal
policies (Preece, 2000). Furthermore, software was needed to mediate, support,
and influence or restrict community activity. A community is something more than
merely an aggregation of users using a collection of communication tools
(Typaldos; 2000) As for e-learning communities, these explicitly target learning,
e-learning leaders agreed that interaction was the key for their effectiveness (for
a review see Conrad & Donaldson, 2004).

Research on e-learning communities appeared in the mid 80s (e.g. Hiltz, 1985).
However, Barab and Duffy (2000) argued that e-learning environments were
practice fields rather than authentic Communities of Practice (CoP) because the
activities were not real; they were educational and not part of the communities’
authentic work. On the other hand, being a participant in a community was an
essential component of the educational process (Goodfellow 2003): ‘communities
of practice’ differ from ‘communities of learners’ in that the latter are reflexively
concerned with learning whereas the former are concerned with practice, of
which learning is a corollary. (p. 3). It appears that there is a difference between
online CoP and e-learning communities. Active and passive behaviour is not a
new phenomenon; an experiment on active and passive reading was conducted
by Janis and King in 1953 (cited in Hovland et al.1953). Two experimental groups
of college students worked as ‘active participants’, who delivered talks in a group

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 24


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

situation; and ‘passive controls’, who read and listened to the source material.
The results indicated that all members had the potential to become active
participants; in other words, they were located in the grey zone of potential active
participation. This grey zone was characterised by the “sleeper effect” as a
change of behaviour after a lapse of time (Hovland et al., 1953). If all passive
participants have the potential to become active this requires an effort on their
behalf.

The Greek teachers belong to an off-line CoP; however, this community is not
reflected in an online CoP. For this reason passive participation has been
identified as the key variance in this study. Thus, the Greek teachers remained
on the first level of participation, that is passive participation. This variance is
imperative to be controlled on a pedagogical and operational level. The question
that arises is, if the Greek teachers do belong in an offline CoP why is it so
important to force them in active participation in collaborative e-learning
communities?

Participation in CoP involves the use of tools; artefacts used within a cultural
practice carry a substantial portion of that practice's heritage. Thus,
understanding the technology of practice is more than learning to use the tools; it
is a way to connect with the history of practice and participate more directly in its
cultural life (Lave & Wenger, 1991:100-102). In the age of ubiquitous computing
and the social phenomenon of the Web, and with the growth of what Berners-Lee
(2007), calls collaboration and creativity, communities will continue to evolve.
Thus, the question is not why it is important to engage the Greek teachers in e-
learning, but finding ways to enable it.

2.3.1 Passive Participation in Online


Communities

Some online communities’ members act as invisible observers of


synergetic activities; they never seem to cross the threshold of observation and
remain in the periphery (e.g. Nonnecke & Preece, 1999; McDonald, 2003; Gulati,
2006). (For a review see Schultz & Beach, 2004.) Nonnecke and Preece (1999)
suggested that ‘lurking is a systematic and idiosyncratic process, with well-
developed rationales and strategies. All interviewees lurked, but not all the time,

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 25


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

and several developed a sense of community through their lurking.” In other


words, lurking is personal and situated; at a particular time an individual is
located within specific external and internal conditions that create reasons that
affect his/her online behaviour and performance. A lurker can generate the sense
of belonging but is it enough?

Posting and interactivity have come under research scrutiny early in online
research (e.g. Rafaeli, 1986; Bruckman & Resnick, 1995) where interactivity was
related to dependency among messages (Rafaeli & Sudweeks, 1997). Figures
related to interactivity and passive participation are different, depending on the
dynamics of the media, and result in specific lurking behaviours (e.g. Fish et al.,
1990; Whittaker et al., 1998; Monthienvichienchai, 2004). Passive participation
can be tracked approximately using different methods provided that the software
supports it, for example counting the views (e.g. Nonnecke & Preece, 2000) or
using proprietary tools that log lurkers’ communicative behaviour (e.g. Soroka, et
al., 2003). Carroll & Rosson (1996) referred to lurkers-to-posters ratios 100:1,
Sproull and Faraj (1997) reported an 80% of lurking, Preece from 46% to 82%
(2000) and Lambropoulos, working with a Greek teachers’ online community,
99% (2002). Lurkers in one community can actively participate in other
communities where they believe they have something important to say (Klemm,
1998).

Nonnecke, in his PhD Thesis (2000), increased understanding of online lurking


by addressing three primary questions: why lurkers lurk, what lurkers do, and
how many lurkers there were at the time of the investigation. The primary reason
for passive participation was uncertainty about the community’s goals. Most of
the reported reasons fell into the personal sphere. Preece and colleagues
reported the top five reasons for lurking (2004): not needing to post; needing to
find out more about the group before participating; thinking that they were being
helpful by not posting; not being able to make the software work; and not liking
the group dynamics or the community was a poor fit for them. Social loafing or
free riding (e.g. Kollock & Smith, 1996; Wellman & Gulia, 1999; Ling et al., 2005)
proposed to be a robust phenomenon that occurred when people work less to
achieve some goal when they thought they were working jointly with others. This
is a limitation on lurkers’ research as free riders might never have responded.
(For example, Preece and Nonnecke’s survey response rate was 2.3%.)

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 26


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Although passive participation is common and normal there is a contribution


paradox: even though passive participants may feel a sense of belonging, and
their participation is not a prerequisite; it is necessary for the life of the
community.

2.3.2 Passive Participation in e-Learning


Communities

Studies of passive participation initially targeted online communities, then


e-learning communities. In comparison to online communities, it appears that the
major reason for being a lurker is a lack of knowledge of how to take part
effectively.

Passive participation was observed for discussion forums or blogs (Williams &
Jacobs, 2004); the relationship between writing and reading behaviour (Ebner &
Holzinger, 2005); if activity influences learning efficiency (Beaudoin, 2002); or
underlying assumptions of e-learning pedagogy (Gulati, 2006). In particular,
Beaudoin (2002) found that almost 23 out of 55 (42%) of online education
master’s degree students, preferred to read other learners’ messages. A
questionnaire sent via email to 23 "low visibility or "no visibility" learners (p. 150),
showed that e-learners were: reading assignments and others’ comments;
conducted web searches; writing assignments; and spent least time on writing
comments for online discussions. Other observed behaviours were: connecting
(visiting the community); browsing (passively participating in community’s life);
attending time durations; contributing opinions; responding to specific posts; and
interacting (responding in a reciprocal manner) (Rafaeli et al., 2004).

Although passive participation is legitimate, the situation seemed to be more


complex in e-learning if passive participants were beneficiaries of other people’s
discussions who did not share their own ideas (Salmon, 2000). Reasons for
lurking were related to the individual, the community, the tools for
communication, and research methodologies (for a review see Rafaeli et al.,
2004). Beaudoin found that 40% (n=23) were not sure how to articulate their
ideas, 30% did not understand the topic, 30% did not know what to say as the
discussion drifted away from the topic, and 25% said that they were not
comfortable in presenting their ideas online. These e-learning managerial

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 27


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

reasons may be related to reasons inherent in the teaching and learning modes
(e.g. Klemm, 1998; Earley & Gibson 2002). Klemm (1998) advocated that there
were psychological and social reasons with roots in the passive conditioning and
the “entertain me” mode of mass media. Both learners and teachers had been
exposed to television and traditional classroom teaching that deprived them of
critical thinking. In particular, Klemm, working on an online conferences context,
believed that lurking restricts creative thinking and teachers foster this behaviour
with their practices. To Khine and colleagues (2003) there is trainee teachers’
inability to actively participate in online discussions; the participants were not
critical thinkers and failed to sustain interaction. E-learners have been found
reluctant to criticise each other (Hughes & Daykin, 2002:222) even if they are
“forced” to be active (e.g. Williams, 2002; Oliver & Shaw, 2003). Greeks are not
different; in fact, Buhayer (2005) referred to the Greek media as transmitting a
model of a reluctance to follow the stories through, lack of investigation and
questioning that killed conversation in Greek homes (p. 157-158).

In her PhD research, Gulati (2006) suggests that passive participation is an


informal mode of learning and an essential part of formal education. She
opposed the controlled and structured type of e-learning that considered lurking
as dysfunctional and problematic. She said that the e-tutors do not have a
problem with lurking; the problem lies in their inability to display lurkers’ learning
in order to justify their existence. Furthermore, she suggested that compulsory
contributions proved inadequate in increasing collaborative learning since it was
connected to Foucault’s (1984) discourse of normalisation through policing. In
other words, compulsory contributions tried to police and repress the outlaws, the
lurkers. The role of the learner was more concerned with what an individual
constructed as an engaging process, so ‘visible’ and ‘silent’ roles were personal
constructs, and were enacted in order to understand the world and showed how
individuals constructed their own perspectives. She concluded that there was a
need to open the concept of learning and include silent, unmeasured learning;
redefining participation in learning allowed an open understanding of adult
learning. The reasons for passive participation are summarised next (Table
2.3.2-1):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 28


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Table 2.3.2-1. Reasons for lurking in e-learning communities

REASONS FOR LURKING IN E-LEARNING COMMUNITIES


Software
Personal Social
limitations
- lack of awareness about -differences in learners’ -inadequate
informal learning, goals, processes and learning
-deep or surface views engagement approaches platforms
-different perceptions of what -access to best practices
was required -organizational obstacles
-familiarization with the -inconvenient procedures
community -security considerations
-learn about a new topic -community-knowledge
-being shy building as a public good
-information overload -integration of novices into
-lack of recognition informal work environments
learning -unite geographically
-differences on goals dispersed work units
-differences in processes
-different perceptions of what is
required from learners

Lack of awareness about informal learning, different levels of understanding and


differences in e-learning targets were the main reasons for passive participation.
This implies that e-learners were unaware of tools and techniques. Social
reasons were identified as differences in goals, processes, engagements
approaches, access to best practices and organisational obstacles. In addition,
Rafaeli and colleagues (2004) suggested that personal characteristics, such as
being shy, information overload, organizational obstacles, inconvenient
procedures and security considerations increased passive participation. This
means that appropriate community management can tackle these problems;
however, there is nothing that can be done for the organisation obstacles,
procedures or security problems. Based on the reasons for lurking, researchers
developed strategies for active participation (Table 2.3.2-2):

Table 2.3.2-2. Active Participation Strategies

STRATEGIES TO ACTIVATE PARTICIPATION

Authors Active Participation Strategies


-require participation,
-form learning teams,
-make an activity interesting,
Klemm
-do not settle for just opinions,
(1998)
-structure the activity,
-require a hand-in assignment as a deliverable derived from the
discussion,

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 29


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

-e-tutors need to know what they are looking for and involve
themselves to help make it happen
-peer grading
-set of institutional norms promoting trust,
-promoting knowledge sharing as a norm of the organization,
-employees are trusted,
Archichvili
-sharing is a moral obligation.
et al.
-building multiple face-to-face CoP
(2003)
-clear norms
-standards for sharing knowledge.
-obtain active participation
-have transparent interface,
Swan et.
-importance attached to the role of the e-tutor,
al., (2003)
-valued and dynamic discussion
Rafaeli et. -familiarity with the community
al., (2004) -develop a sense of community

Strategies such as building e-learning teams were based on specific


methodologies to develop a sense of community in a trusted common space for
information sharing. Strategies also referred to community management and
members’ roles. E-learners, e-tutors, designers and engineers and the
organisation hold responsibility for planning and supporting e-learning activities.
For example, the e-tutor may structure collaborative activities while the
organisation can incorporate standards and rewards for sharing knowledge;
designers may provide transparent and usable interfaces, and engineers
contribute technical support. Overall, it appears that the key to participation is the
shape of collaborative e-learning design supported by suitable tools and
methodologies.

Overall, this section discussed the socio-cultural and pedagogical foundations of


participation in collaborative e-learning communities. The literature referred to
what reduces active participation. Since personal and social reasons refer to
community and learning management they can be tackled by aligning new tools
and evaluation techniques with pedagogical modifications. Such tools and
techniques need to support investigation for observation, description and
analysis of what happens in collaborative e-learning communities.

The next section will investigate whether current design supports them.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 30


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

2.4 DESIGN FOR LEARNERS AS USERS &


USERS AS LEARNERS

Online Education is in its infancy and has yet to construct design models
that will address the problem of the learner as a user (Wallace, 1999; Smulders,
2003). Learning how to be a user is about understanding the technology rather
than learning to use the tools (Lave & Wenger, 1991:100-102). There is an
assumption that the educational systems are easy to use by the educators and
learners. Users in educational settings act as both users and learners and design
has to support their dual persona (Smulders, 2003). However, technologists tend
to build techno-centric systems for use by academics, this means integrating
several levels of functionality which is geared towards the e-tutors rather than
considering the e-learning participants. For example, besides the Greek teachers
and e-tutors, the Greek Schools Network (GSN) consist of developers, engineers
and decision makers. Additionally, today’s users are typically multi tasking; this is
especially so for younger users who might do homework whilst listening to Mp3s
and chatting with friends (Dede, 2005). In order to support multi tasking the
question of the interface becomes important since a usable system, one with an
easy to use interface, will enable users to concentrate on their tasks. Thus,
interface design might become pivotal and make the difference between a
system that is used and one that isn’t. Design may influence participation quality.

Many design definitions exist; for example, design underpins every form
of creation from objects such as chairs to the way we plan and execute our lives
(Dini, 2005). But to Suchman, design developed strategies and tools insensitive
to particular circumstances (1987:121) because plans actually derive after the
completion of the course. Such a post-hoc design structure challenged the
foundation of computational design as a linear process of development. It
appears that the design of new systems is always problematic.

2.4.1 Socio-Technical & User-Centred Design

Socio-technical design (STD) refers to design that is influenced by an


organisations’ social structure. In 1949, researchers from the Tavistock Institute
for Sociology in London formulated the most important principle for joint

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 31


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

optimisation of social and technical systems: if a technical system is created at


the expense of a social system, that is the organisation’s social structure, the
results obtained will be sub-optimal (cited in Mumford, 1983). According to
Mumford (1983), design should consider the social context and involve users
(socio-technical design). Faulkner (2000) suggested that systems should be
user-oriented in all aspects of their functionality by adjusting the systems to the
users and their natural environments (user-centred design).

Socio-technical design (STD) develops systems for collaborative working


environments. STD offered solutions to practical problems by fitting design to its
context. STD took the idea of designing for the user and the task a stage further
and endeavoured to design within the structure of the organisation and the way
in which it operates (Faulkner, 1998:134). Therefore acquisition of the ‘native’s
point of view’ was important and was provided by ethnography. Even though
STD provided schemes to fit design to its context, most designers were not
motivated by incorporating social structures into design (Anderson, 1997). In fact,
designers believed that it was user’s inability to cope with the system that was
the problem (Mumford, 1983). Nowadays designers acknowledge that the
development of interactive technologies relies on an appreciation of social
circumstances in which systems are used (e.g. Dourish, 2006). So, STD offered
key concepts for systems design such as participation in design, participation in
decision making, and the need for evaluation methodologies to support them.

STD considered the organisation as a whole. User-Centred Design (UCD)


provided a clear focus on users’ needs. These had to dominate the interface
design, and the interface had to dominate the design of the system. This was
achieved by a dialogue between the stakeholders and the developers and by
their involvement in the early planning stages. This is the purpose of UCD and is
what Shakel (1991) described as having the: “…capability in human functional
terms to be used easily and effectively by the specified range of users, given
specified training and user support, to fulfil the specified range of tasks, with the
specified range of environmental scenarios”. He suggested that systems need to
be used easily and effectively to support specific users; training and support on
the use of the system was central. UCD guidelines were needed to ensure
product quality even in the planning stages. Gould and Lewis (1985) proposed
four principles for useful and easy ways to create usable computer systems: (i)
early focus on users and tasks, (ii) empirical measurement for evaluation, (iii)

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 32


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

iterative design, and (iv) integrated design. When systems did not operate as
expected then user feedback should be sought and used in the redesign as
systems should conform to expectations. User-centred studies provided more
coherent frameworks and models for analysis of what is going on and what
should be done to ensure users’ satisfaction. Shackel (1991) suggested that
usability could be measured by examining learnability, effectiveness, attitude and
flexibility; Nielsen (1993) thought it was efficiency, learnability, memorability,
errors and whether the system is subjectively pleasing. The International
Organisation for Standardisation (ISO 9241-11, 1998) defined usability as a
measure of the quality of user’s experience when interacting with a system, in
terms of effectiveness, efficiency, and satisfaction. Measurements and
assessment procedures had to ensure the product met the purpose of design.

From an educational viewpoint, technology is to enable students to reach their


potential rather than technology being the goal itself. Educational designers
needed to consider not only the educational system as a whole but also the
learners’ dual persona as learners and as users, and facilitate learning without
any additional cognitive and physical struggles to use the system. Users of a
learning system needed to be free to learn the subject and not have to spend
time on learning about the system. In other words, the system must be usable
(Faulkner, 2000). Laurillard, in her interview with Neal for the E-learn Magazine
(2003) stressed this fact; she also said that there were critical issues for
technology enhanced learning and its future. More specifically, there was a need
to: adopt pedagogical perspectives; focus on user interface; build on learning
activities design; assess performance; and evaluate in the form of checking
whether the learning objectives have been met.

These recommendations have translated the user-centred design principles into


learner-centred design principles.

2.4.1.1 Towards a Learner-Centred Design

Educational design was built on the integration of the pedagogical and


technological levels, and was called Learner-Centred Design (LCD) or
Instructional Design. Whereas User-Centred Design focused on making users
more effective, LCD focused on making learners more effective. For Norman and
Spohrer (1996) LCD had three dimensions: (a) engagement as the result of

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 33


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

motivation, (b) effectiveness, and (c) viability of interventions. Engagement was


tightly associated with motivation, the provision of rapid compelling interaction
and feedback. Feedback was used to achieve effectiveness by measurement
and benchmarking. With a social perspective in mind, Squires and Preece (1999)
provided the first set of ‘learning with software’ heuristics echoing socio-technical,
user-centred, and learner-centred design.

‘Learning with software’ heuristics opened the way to learner-centred usability, or


Pedagogical Usability (PU) (e.g. Muir et al., 2003; Nokelainen, 2006). PU derived
as part of utility as software has high quality if users can perform their tasks (the
software is useful to them, Nielsen 1993). With utility in mind, Finnish
researchers (Silius et al, 2003; Nokelainen, 2006) proposed that PU should
question whether the tools, contents, interfaces, and tasks provided within the e-
learning environments that supported e-learners. For Silius and colleagues, PU
was tested with a questionnaire linked to the educational website. The
questionnaire targeted the suggested tools’ basic use and utility triangulated by
logs and data from the discussions anchored in pedagogical usability (Silius et
al., 2003a, 2003b). Muir and colleagues (2003) also worked on an e-learning PU
pyramid for educational effectiveness and practical efficiency of a course-related
website. They stressed the e-learning participants’ involvement in design,
evaluation, and decision making. (For a review of pedagogical usability attributes
see Nokelainen, 2006.)

Overall, educational systems and their integrated tools constitute the means by
which learners are permitted or restricted in the use of their learning capabilities.
Instructional design and engineering are design processes to ensure their
educational fulfilment.

2.4.2 Instructional Design & Engineering

Instructional Design (ID) is the systematic process of activities to solve an


instructional problem with the aid of technologies (Fenrich, in press). There were
different approaches in ID models (e.g. Molenda, 1987; Ryder, n.d.; Fenrich, in
press). For most instructional designers, the process fell into a framework called
ADDIE , that is analyze, design, develop, implement, and evaluate (Bichelmeyer
et al., 2005). Fenrich proposed a coherent ID model anchored in Bloom’s
taxonomy (1956) with practical guidelines targeted at instructional multimedia

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 34


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

solutions. ID begins with initialization and project planning (how the instructional
design is carried out), the design and development phase (appropriate strategies
and approaches in targeted contexts), and a quality assurance phase focused on
deployment, evaluation, and assessment. It also includes easy-to-use checklists
and presentation tips offering a comprehensive insight in ID and deployment
outside the control room. This interdisciplinary approach involved all stakeholders
in design by covering all stakeholders’ benefits defined under specific criteria:
roles, skills, characteristics, activities, commitments, and responsibilities. As with
the previous design approaches, identification of intents and planning based on
stakeholders’ goals, skills, background and characteristics provided the design
backbone. Fenrich said that his model works in ideal situations; in reality
instructional designers need to adjust the plan to given situations by integrating
evaluation results into system requirements and re-design specifications. This
means that Fenrich implicitly agreed with Suchman on the post-hoc nature of
design.

Instructional design and engineering are interrelated processes to support


planning, analysis, design, and delivery of a learning system. Instructional
Engineering (IE) integrated the concepts, the processes and the principles of ID,
software engineering and cognitive engineering (Paquette, 2004) and explicitly
referred to e-learning (Shepherd, 2001; Myrach & Knolmayer, 2005). Paquette
(2004) adopted Human-Computer Interaction (HCI) approaches directly into his
IE framework. His model described e-learning engineering in relation to:
engineering and re-engineering e-learning systems (macro design); producing
instructional materials (micro design); delivering training on networks; and
reviewing and maintaining e-learning. Paquette provided a complete knowledge
presentation system on platforms and portals and described all functions and
roles for the stakeholders called actors: the teacher-designer, the learner, the
facilitator, the manager, and the platform administrator. His MISA method was an
automated system that supported 34 main tasks and about 150 secondary tasks.
It has been developed continuously and tested successfully in different contexts
for several years.

However, it was suggested that systematic ID models had been accused of not
reflecting actual practice and not supporting new competencies for the 21st
century and post-industrial societies (Table 2.4.2-1):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 35


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Table 2.4.2-1. Instructional Design problems

INSTRUCTIONAL DESIGN PROBLEMS


Authors Description
Visscher-Voerman et al., 1999;
The ID process itself
van den Akker et al., 2004
Richey et al., 2000; Waters &
ID competencies
Gibbons, 2004
Gordon & Zemke, 2000 Being cumbersome, ineffective, inefficient
Gentry, 1994; Berger & Kam,
The nature of the task
1996
Seels & Glasgow, 1998;
Methods to inform practice
Shambaugh & Magliaro, 2005
Tripp & Bichelmeyer, 1991;
Costly to implement
Fenrich, in press
Schwier, Campbell & Kenny, Not taking of advantages of new
2006 technologies

It appears that the major problem is related to the ID process itself with regard to
its implementation and flexibility. Some other problems related to e-learning
quality and recorded in the Panorama report (Ehlers, et al., 2005) were caused
by: e-learning participants’ unfamiliarity with the design process; division between
‘academic’ and ‘corporate’ approaches; lack of awareness of the need for quality
standards; the dual identity of the learner as a user was neglected; stakeholders’
engagement in the early stages of design was neglected; and the fact that design
in artificial settings makes it vulnerable to the Hawthorne effect, so little evidence
existed as to how to use technology effectively.

These problems were relevant to the situated nature of the learning environment
as they have a “conclusion-oriented” nature instead of a “decision-oriented” one.
In addition, social interactions were partially neglected; with the Web 2.0
signpost, instructional designers and engineers foresee the need of an e-learning
2.0 stage (Downes, 2006; Karrer, 2006; Jennings, 2005). At this point in time,
educational design needs to consider the social and collaborative structure of e-
learning communities for learners’ content creation. The question that arises is
not why ID and IE suffer from so many problems, but to what extent designers
need a more flexible or a more systematic model. Depending on the
organisational target, a managemental educational change suggests a more
open model, whereas design for particular systems requires a more coherent
model. According to Leigh (1998), instructional designers need to focus either on
one aspect of learning and instruction and act as consultants or matter experts.
Since the field is becoming too broad for most designers to work with authority,

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 36


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

these options give freedom to instructional designers. To answer this question,


following Winn (1986:20), “we must have at our disposal a whole battery of
methods to deal with the different types of things we need to find out”.

Overall, it appears that systems design can be the catalyst to overcome


variances or even crises. This implies that if design is relevant and suitable to
situation it can fulfil its purpose; then there is high probability not only of
overcoming the variance but reaching a high level of quality. The rest of the
literature review explores whether socio-cultural learning principles have been
incorporated in tools to support collaborative e-learning communities.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 37


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

2.5 TOOLS FOR COLLABORATIVE


E-LEARNING COMMUNITIES

For active learning, point and click is not enough (Klemm, 1998).
Dillenbourg (2000) stressed the social aspect of e-learning as a designed
information space where learners are actors i.e. they co-construct the information
space. Kollock (1998) emphasised the need for such social computing: ‘The key
challenges the Internet community will face in the future are not simply
technological, but also sociological: the challenges of social interaction and social
organization.’. In other words, e-learning applications need to consider social
interactions between e-learning participants. So far, there are different types of e-
learning tools categorised under several criteria: temporal, as in synchronous,
asynchronous activities or both; the medium of communication as in text, audio,
visual, simulation and their combinations; the direction of activity as in broadcast,
email, messaging or interactive; web-based as offline or online; and the privacy
levels, these are public or private. The computer-based collaborative learning
tools are divided into conversation and collaboration tools. (For reviews, see
Conrad & Donaldson, 2004; Jonassen, n.d.; Kommers et al., 1992) Jonassen
said there was a chasm to consider between tools that support information
provision and groupware.

As for groupware, what supports collaborative learning is the visibility of the


collaborative learning structure, tools for observing interactions, and tools for
analysing interactions (Dillenbourg et al., 1996). Thus, design can impact
collaborative learning on the kinds of social interactions that aid learning since
aspects of software can modify the socio-dynamics between the learning
partners. Designing e-learning tools to influence collaborative e-learning has
methodological advantages. For example, such tools can give explicit control of
the process and support the type of interactions that were expected to promote it.
Lack of such tools results in lack of understanding collaborative e-learning and
vice versa. This is evident in current research by a lack of interesting papers
(Nick Rushby, editor of the British Journal of Education Technology, personal
communication via email, 06/12/2006). The question that arises is whether
collaborative e-learning tools can support the socio-dynamics of an e-learning
community.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 38


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

The quest for tools in the e-learning history goes back to the 60’s. (For a detailed
view, see Moodle http://docs.moodle.org/en/Online_Learning_History, and
wikipedia http://en.wikipedia.org/wiki/History_of_virtual_learning_environments.)
Systems to support e-learning communities were not included in associated
reviews because they were considered as a “third place”, a room for social
collaboration outside home and work (Bruckman & Resnick, 1995). They were
also excluded from the history of online communities by Ambrozek and
colleagues (2004), and had a reference of 33 words in Preece and colleagues
(2000). The following graph from Google Trends depicts the separate routes of
the two terms in the literature (Figure 2.5-1):

Figure 2.5-1. Google trend history for online community and e-learning

The separate routes meet just before 2007 suggesting a trend within e-learning
communities. Nonetheless, the social aspects of collaborative e-learning were
apparent from the early years of the educational technology research. It is
interesting to observe that the word “collaborative” exists in early designs.

The social aspects of learning was scrutinised in Computer-Supported


Collaborative Learning (CSCL). There were early systems in the 70s such as the
Programmed Logic for Automated Teaching Operations (PLATO, Figure 2.5-2)
(Tuss, 2001; Van Meer, 2003) or a Time-shared, Interactive, Computer-
Controlled Information Television (TICCIT) (Bunderson, 1973).

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 39


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Figure 2.5-2. PLATO III 3

PLATO (Figure 2.5-2) pioneered concepts such as online forums, message


boards, learning management, online testing, email, chat rooms, instant
messaging, remote screen sharing, and online games.

In his review of e-learning software developments over the last 30 years, Rumble
(2001) proposed four models of teaching and learning: the transmission, the
constructivist, the socio-cultural, and the metacognitive model. Following
Rumble’s review, current LMS can be located only on the first three levels. In
other words, higher order thinking is not supported by current LMS. For example,
Moodle, related to this research, was based on socio-constructivist theories
(http://docs.moodle.org/en/Philosophy). Today, Moodle developers have
synthesized Rumble’s first three concepts in the “social constructionist
pedagogy”:

The design and development of Moodle is guided by a particular


philosophy of learning, a way of thinking that you may see
referred to in shorthand as a "social constructionist pedagogy".
Moodle, http://docs.moodle.org/en/Philosophy Last access
20/12/2006

3
Figure retrieved from the blog http://siliconuser.com/?q=node/12 - Posted June 8th, 2007 by Joshua Coventry
(Permission to use the image in the Thesis was acquired via email on 06/12/2007.)

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 40


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Despite the fact that the Moodle developers built on sound pedagogical
approaches, Moodle does not explicitly support collaborative e-learning
communities. Overall, it appears that current learning management systems have
not evolved significantly, compared to the pace of emerging technologies of
social networks and user-generated context.

2.5.1. When the Online Community Met


E-Learning

The first e-learning system that supported e-learning communities was


MediaMoo (Bruckman & Resnick, 1995). MediaMoo was excluded from the e-
learning history because it was a shared environment and set of activities for
people with the same research interests. MediaMoo community management
was heavily based on the members’ conversation as the primary activity.
MediaMoo became publically available in Jan 1993. MediaMoo developers saw
three aspects of an e-learning community: the platform, e-learning, and
community management. MediaMoo targets were explicitly referred to all
Rumble’s stages, the transmission, the constructivist, the socio-cultural, and the
metacognitive model and thus, ensuring the quality of participation in a
community by design. Kim, in an interview for Elearningpost (February 27, 2001)
about the design that ensures a successful e-learning community, said:

To create an effective learning community, you need to develop a social


and technical infrastructure that supports the key activity that's
happening there - which can be quite different for various online
learning communities.
Kim, 2001

To Kim, success is related to both the social and textual infrastructure for
situated activity and learner-created context. It also seems that more than ten
years after MediaMoo, a design to support socio-cultural e-learning is still an
issue.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 41


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

2.5.1.1. Design Principles for E-Learning


Communities

Researchers have proposed design principles for online and e-learning


communities since the early 80s (Table 2.5.1.1-1):

Table 2.5.1.1-1. Design principles for online and e-learning communities

DESIGN PRINCIPLES FOR ONLINE & E-LEARNING COMMUNITIES


Author Description
Requirements for the possibility of cooperation:
• Arrange that individuals will meet each other again
Axelrod • They must be able to recognize each other
(1984) • They must have information about how the other has
behaved until now

Design principles of successful communities:


• Group boundaries are clearly defined
• Rules governing the use of collective goods are well
matched to local needs and conditions
• Most individuals affected by these rules can participate in
modifying the rules
• The right of community members to devise their own rules
Ostrom (1990) is respected by external authorities
• A system for monitoring members' behaviour exists; this
monitoring is undertaken by the community members
themselves
• A graduated system of sanctions is used
• Community members have access to low-cost conflict
resolution mechanisms

Principles for making virtual communities work:


• Use software that promotes good discussion
• Don't impose a length limitation on postings
• Front-load your system with talkative, diverse people
• Let the users resolve their own disputes
Godwin (1994) • Provide institutional memory
• Promote continuity
• Be host to a particular interest group
• Provide places for children
• Confront the users with a crisis
• Integrate the online environment with the "real" world

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 42


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Principles for creating a community:


• Define the purpose of the community
• Create distinct gathering places
• Create member profiles that evolve over time
• Promote effective leadership
Kim (1998) • Define a clear-yet-flexible code of conduct
• Organize and promote cyclic events
• Provide a range of roles that couple power with
responsibility
• Facilitate member-created subgroups

Guidelines for Online Communities:


Sociability Web Usability
• Purpose (clear • Navigation
definition) • Access
• People (access, • Information design
riles and • Communications software
effective
Preece (2000) • Finding people & Information
communication)
• Ensuring readability of
• Policies instructions
(registration,
governance, • Providing tools to support
trust and moderators and other role-
security) players

Design Principles for e-Learning Communities:


Pedagogical Level Operational Level
• Intention
• Information
Lambropoulos • Interactivity
(2006) • Real-time evaluation
• Visibility
• Control
• Support

It appears that most principles for online and e-learning communities are related
to community management and targeted to promote group coherence and
resolve differences as well as supporting members’ individuality. The e-learning
community guidelines for synthesizing pedagogy, community management, and
socio-technical design were suggested by the writer (Lambropoulos, 2006).
Other more informal principles to support teachers’ e-learning stressed the need
for a socio-cultural basis and came from the teachers’ community TappedIn®;
Gray and Koch (n.d.) suggested that the use of familiar settings, support of social
facilitation, and allowing flexible grouping supports e-learning communities.
Design factors for collaborative e-learning communities appear to be complex
and interrelated.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 43


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Kim (2001) suggested that there is no single answer for developing principles for
e-learning communities. However, one solution may be that the principles should
be open and agile enough so to be incorporated in any situated e-learning
interface design.

2.5.1.2. E-Learning Tools & Interface Design

Two types of interfaces were found in the literature, information provision


and groupware. Information provision interfaces dominate the field and tend to be
more data-centric and context-independent. They often deal with issues of
search and retrieval, information presentation, and visualisation. This is because
contextual information and socially relevant representations were rarely used
outside the lab (Hoadley, 1999; Faulkner, 2000). Shneiderman and Maes (1997)
suggested that learning management systems needed to overlap both types.
There are no indications that e-learning interfaces have been developed to
support both types to a great extend.

E-learning communities’ context derives from monological and dialogical


sequences so current tools that are directed to simply facilitate these processes
are mostly characterised as information-centric media. Hoadley and Enyedy
(1999; 2006) criticized such tools as computer-supported intentional learning
(Scardamalia & Bereiter, 1991), SpeakEsay (Hoadley, His & Berman, 1995), and
SenseMaker (Bell, 1998). They said that a chasm between internalisation and
externalisation in monological and dialogical sequences was reflected in the
interface. The challenge was to build tools that could help the learner to bridge
the “middle space”. Because monologue derived from dialogue as reflective
thought, there was a need to make this structure visible. Thus, visible dialogic
interactions could allow the interlocutors to give each other feedback leading to a
gradual refinement of partial meanings and construction of increasingly
sophisticated approximations of concepts. Therefore, tools need to fill out the
“middle spaces” of the continuum, that is the transitions for inclusion and
centralisation, reflections and interactions, topical and discursive coherence, and
the trade off between convergence and divergence (such as reaching a
consensus). They proposed the initiate-respond-evaluate (IRE) scheme used in
computer-supported collaborative learning argumentation and has reported both
positive and negative results (e.g. Siegel et al., 2001, Mercer and Wegerif, 1999).
Their IRE model was criticised for restricting creativity because of its limited three

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 44


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

levels scheme that did not allow critical thinking (Nassaji & Wells, 2000). (For a
review on a state of art technology for supporting collaborative learning see
Jermann et al., 2001.)

By studying such initial attempts, the problem appears to be related to a creative


interplay between collaborative learning theory and its translation into tools to
support collaborative leaning in a social and contextual basis. This was
discussed by Scardamalia and colleagues (1996) and Jonassen (n.d.).
Scardamalia and colleagues worked on knowledge databases so learners’
knowledge could be objectified, represented in an overt form, and then
evaluated, examined for gaps and inadequacies, added to, revised, and
reformulated (1996:201). Jonassen’s constructivist learning environments include
tools like listservs, electronic mail, bulletin boards, MUDs (multi-user
dimensions), and MOOs (MUDs Objected Oriented, like MediaMoo). Active and
constructive participation in learning proved to be essential but did not happen
naturally.

2.5.2. Tools to Support E-Learning


Communities

It appears that there are two kinds of tools to support human-human and
human-computer interactions: tools for observing and tools for analysing
interactions, divided in three categories, social/cognitive, cognitive/metacognitive,
and task/communicative (Figure 2.5.2-1):

Collaborative Learning
Tools
(Dillenbourg et al., 1996)

Observing Analysing
Interactions Interactions

social/cognitiv cognitive/metacognitiv task/communicativ


e e e

Figure 2.5.2-1: Categorisation of collaborative learning tools

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 45


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Observing social interactions were the first step built on a learner-generated


dialogical context for showing presence and co-presence to make the user
visible. As for analysing interactions, tools could capture the collaborative
learning cognitive and metacognitive distinction, for example exploring and
verifying solutions as well as the discrimination between task and
communication. The latter is referred to as dialogue management and
conversation models drawing from linguistics (for example, argumentation,
speech types, managing turn taking or making relevant contributions). The
following section presents existing tools that fit with the collaborative e-learning
dialogue scheme.

2.5.2.1. Tools to Structure Dialogical Sequences

The initiate-respond-evaluate (IRE) scheme (Hoadley & Enyedy, 1999)


triggered efforts on studies to support collaborative learning dialogical sequences
by predicting the forms of desirable dialogue as well as interactivity mechanisms.
Thus, it formed a basis for the design of computational models. Jonassen (n.d.)
introduced instructional activities for an interplay between learning and instruction
(Table 2.5.2.1-1):

Table 2.5.2.1-1: Learning and Instructional Activities

Learning Activities Instructional Activities


Exploration Modelling
Articulation Coaching
Reflection Scaffolding

Jonassen said that each attribute should represent an instructional activity. There
were different tools supporting collaborative learning discourse such as the
collaborative notebook (O’Neil & Gomez, 1994) or the knowledge integration
environment (Bell, Davis & Linn, 1995) based on utterances as particular kinds of
speech acts (initiate, continue, repair and acknowledge), sentence openers and
floor control (I agree, Could you explain etc) (Traum, 1994). Another approach
that will be explored in detail in the next chapter was introduced by Mercer (1995)
and Wegerif and colleagues (1998); the authors worked on a collaborative
learning model for disputational, cumulative, and exploratory talk. Baker (2000)
suggested that what was required for tools to structure dialogical sequences was
relations between theory, model and corpus (i.e. transcriptions of interactions

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 46


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

data), and then design legitimate objects. Two examples of such models
translated into collaborative e-learning tools were InterLock and Messageforum.

InterLock was the by-product of McAlister’s PhD project Academic Talk (2004)
(http://learning.north.londonmet.ac.uk/ltri/academictalk/). As with AcademicTalk,
InterLock, worked on the same attributes to support collaborative e-learning
argumentation (http://www.interloc.org/; last accessed 21/12/2006), and Mercer
and Wegerif, participated in the new project (e.g. Ravenscroft, Wegerif, Mercer &
Hartley, 2006). InterLock viewed collaborative e-learning argumentation as a
game that promoted engagement and learning (Figure 2.5.2.1-1):

Figure 2.5.2.1-1. InterLock Interface

Message openers remained one of InterLock key features


(http://www.interloc.org/about.htm). According to its developers, the message
openers promoted coherent dialogue, thinking and deep learning which aids in
reusable and adaptable learning activity and dialogue game templates. However,
InterLock: (a) was too complicated for most users; (b) restricted creativity;
learners had to think in frames because they were obliged to use the tool and this
interrupted the discussion resulting in lack of context depth in reasoning; (c) the
attributes were distributed on the interface, so the structure of the conversation
restricted visibility of dialogical sequences overview; (d) there is no indication that
it promoted group-knowledge building; and (e) Java programming cannot be
easily integrated in wide use learning management systems; however, this was
not intended by the developers.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 47


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

The next example had a more flexible nature and structured collaborative e-
learning in discussion forums. MessageForum was built by Jeong (2005) about
the same time as InterLock (2006) to support computer-supported collaborative
argumentation. Jeong suggested that online dialogical argumentation lacked
depth and was redundant. To solve these problems he created the
ForumManager, a tool to support collaborative e-learning argumentation based
on Toulmin (1958). The ForumManager was an MS Excel application for
downloading and analyzing messages (and message texts) in Blackboard
threaded discussion forums using Internet Explorer browser. This discussion
analysis tool created reports in Excel including reports on social interactions built
on social network analysis, that is visualisation of e-learners’ interactions. Table
2.5.2.1-2 describes the collaborative e-learning attributes in detail using symbols
and attributes rather than message openers:

Table 2.5.2.1-2. MessageForum Collaborative Learning Attributes

MessageForum Collaborative Learning Attributes


Symbol Description of symbol
Identifies a message posted by a student assigned to the team
+ supporting the given claim/statement
Identifies a message posted by a student assigned to the team
- opposing the given claim/statement
Identifies a message that presents one and only one argument
or reason for using or not using chats (instead of threaded
ARG# discussion forums). Number each posted argument by counting the
number of arguments already presented by your team. Sub-
arguments need not be numbered. ARG = "argument".
Identifies a reply/message that provides additional support,
EXPL
explanation, clarification, elaboration of an argument or challenge.
Identifies a reply/message that questions or challenges the
BUT merits, logic, relevancy, validity, accuracy or plausibility of a
presented argument (ARG) or challenge (BUT).
Identifies a reply/message that provides proof or evidence to
EVID
establish the validity of an argument or challenge.

The translation of the above model in a tool is depicted next (Figure 2.5.2.1-2):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 48


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Figure 2.5.2.1-2. MessageForum Attributes in a Discussion Topic

As Dillenbourg and colleagues had predicted in 1996, Jeong found that the
visibility of the structure helped learners’ reflection; more replies were elaborated
on previous ideas; there were greater gains in knowledge acquisition; there were
fewer unsupported claims, greater knowledge of argumentation processes; there
was no difference in knowledge acquisition, application of domain content, and
convergence towards consensus; and lastly, there were fewer challenges per
argument. The findings about the effects of using message constraints and
message labelling were similar to studies on other projects such as
ShadowPDForum (Jonassen & Ramirez 2005); NegotiationTool (Beers, 2004);
Future Learning Environment 3 (FLE3) (Leinonen et al., 2003); and Computer-
Supported Collaborative Learning Environment (CSILE) (Scardamalia et al.,
1994).

According to Jeong, the tool supported the visibility of the mechanisms that
facilitated three main areas: the strategic uses of message labels; the potential
group performance problems associated with personality traits; and methods and
tools used with message labelling to diagnose problems. Lastly, one of Jeong’s
recommendations for future research was to integrate tools in the forums for real-
time feedback to optimize group performance. One of the consequences of not

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 49


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

having the results in real-time is lack of situated social presence. Jeong has
presented similar results from other studies (e.g. Jeong & Davidson-Shivers,
2006; Jeong, 2008). Jeong’s tool also produced social network analysis (SNA)
graphs (sociogrammes) via Microsoft Excel (not in real time). Since SNA tools
appeared in online community research after 2000, it seems that Dillenbourg and
colleagues (1996) may have predicted another e-learning trend. The similarities
and differences between InterLock and ForumManager were (Table 2.5.2.1-3):

Table 2.5.2.1-3. Similarities and differences between MessageForum & InterLock

MessageForum VS InterLock
+ n/a Initial information
Attributes

- n/a Agree
ARG = Question
ARG = Challenge
BUT = Reason
EXPL = Reason
EVID / Maintain

The differences as regards the interface are mainly based on the different ways
in which the implementation of the main concepts have been carried out.
InterLock is a system whereas MessageForum is a tool incorporated in e-
learning systems such as Blackboard. The differences were connected to
interface usability as well as the visibility of attributes structure. Overall, it
appears that any tools to support collaborative e-learning need to aid:

• observing and analysing human-human and human-computer


interactions;
• evaluation to be close to decision making and preferably in real-time;
• flexible attributes to allow space for creativity;
• visibility of the communicative actions (interactions); and
• implementation in wide use leaning management systems.

The question that arises is whether conceptual frameworks can aid the design
process. This will be discussed next.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 50


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

2.6 PROPOSITIONS FOR DESIGN

Based on the literature review, there are three propositions for design, the
Collaborative e-Learning Episodes, the Participation Levels, and the Sense of the
e-Learning Community Index.

2.6.1 Collaborative E-Learning Episodes

The dialogical context differentiated collaborative learning dialogue from


other “simple” dialogues, providing a distinction between mere information and
knowledge acquisition. Thus, collaborative learning appears to be the answer to
the initial question on quality in e-learning as these dialogical sequences promote
learning. A collaborative e-learning model is process-based and can be identified
in a dialogical context. Such a framework was found as a “knowledge sharing
episode” (Soller et al, 2002). A knowledge sharing episode was initiated when
there was new information, and ended with new knowledge construction.

A dialogical sequence for collaborative learning can be an episode; hereafter


called a collaborative e-learning episode (CeLE 4 ). A CeLE is a communicative
discussion episode that has to have a starting point, a transition and an end point
that indicates a collaborative e-learning cycle. Based on Gumperz (1982:328-9)
for identifying the end point or point for CeLE completion, a CeLE inference is
involved in a complex series of judgments, including relational or contextual
assessments on how items of information are to be integrated into what we know
to expand our knowledge. These end points are defined by silence or mutual
agreement when an idea arises, which is new to some participants. The basic
interaction variables (attributes) can indicate collaborative learning value and
reflect pedagogical values in one CeLE. A numerical measurement was also
considered; however, all attributes function as inputs in a CeLE and are of equal
importance. In addition, the scheme needs to be flexible, not imposed and easy
to use. So the proposed process of one CeLE in accordance to its pedagogical
values is:

4
Examples of CELEs can be found on Appendix A_VII_13 and Appendix X.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 51


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

1. Information-based as one way of communication;


2. Interactional as two ways communication (Explain, Question);
3. Collaborative Learning – Exploration (Explore, Reflection); and
4. Evaluation / Feedback.

This process of progressive dialogue can be depicted as follows (Figure 2.6.1-1):

Inform Explain

Interaction Pedagogical
Value
Inform Suggestion
Explain Elaboration
Question Reflection
Evaluate Question
Explore Elaboration
Evaluate Reflection

Explore

Figure 2.6.1-1. The Collaborative E-learning Episode (CeLE)

The CeLE is proposed to indicate the collaborative e-learning levels, and as such
its value in real-time. This means that all e-learning participants can instantly be
aware of the level and the value of participation, as soon as the learner describes
the conversation act in an accurate way. Secondly, CeLE can identify the actors
and the modes of preferred participation. Lastly, CeLE, in combination with other
tools, can support the e-learning community’s socio-dynamics, that is the type of
interactions that can foster the community’s creativity. So if a CeLE can be
translated into a tool it will support observing and analysing collaborative e-
learning interactions as it:

• makes e-learners aware of active participation prerequisites;


• raises awareness of possibilities in collaborative dialogue and
argumentation;
• structures data in collaborative e-learning;
• measures the collaborative e-learning value;
• identifies e-learners’ type and thus posting value and quality;
• provides an overall view of the collaborative e-learning process; and
• facilitates proactive decision making.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 52


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

If CeLE can provide the attributes to observe and analyse interactions in


collaborative learning, then measurement, observation and analysis of
participation may be feasible. In an effort to measure the collaborative e-learning
value, Wiley (2002) suggested a qualitative classification built on a mathematical
approach. Evaluating participation in multi-thread discussion was conducted by
calculating an adjusted mean reply depth (d value) for each participant; the value
in the depth of discussion was considered on three levels:

1. 0 to 0.3 Monologue or lecture; no discussion


2. 0.3 to 1.2 Simple Q & A; chit-chat
3. 1.2 and higher Discussion, Multilogue

This is an interesting way to calculate value based on internal (monologue) or


external levels of discussion (dialogue); however, specific prerequisites and
attributes were missing. Acknowledging this limitation, Wiley’s reply depth or
thread depth will be used to calculate the level of persistence to a topic as
discussion depth.

Other than evaluating online messages as CeLE, participation is of major


importance in the search for quality in collaborative e-learning discussions. So a
proposition for measuring e-learners’ participation levels is presented next.

2.6.2 Levels of Participation

The Greek teachers’ passive participation is a normal and legitimate


behaviour and it is not a barrier to developing a sense of belonging. The literature
review increased understanding on passive and active participation but it did not
provide distinct and practical assessment. Recent approaches for measuring
participation levels are not designed to be automated (Table 2.6.2-1):

Table 2.6.2-1. Levels of participation measurement

Authors Levels of Participation


Beaudoin (2002) No Low
Taylor (2002) Lurkers Shirkers Workers
Oriogun (2006) No Low Medium High

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 53


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Beaudoin suggested a distinction between low and no visibility learners; Taylor


proposed a classification between three groups, the workers or active
participants; the lurkers, who participated occasionally but mostly in a read-only
mode; and shirkers, the ones, that followed the minimum requirements. Oriogun
considered low, medium and high levels of engagement.

To automate a practical measurement of participation, the bell curve (Herrnstein


& Murray, 1994) is proposed for calculation on the basis of the total number of
messages sent in a forum (Table 2.6.2-2):

Table 2.6.2-2. Levels of activity measurement

Levels of Activity Measurement on the overall messages

X Sense of Community
4 High 76-100%
3 Medium 26-75%
2 Low 1-25%
1 Zero

Based on this measurement, a centripetal process, the participation eye, targets


to the sense of belonging to the community (Figure 2.6.2-1):

Figure 2.6.2-1. Participation Levels in Collaborative e-Learning Communities

The above levels show the symmetry/asymmetry in participation and also depict
the grey zone between passive and active participation characterised by the
sleeper effect; the latter is the white area where the participants decide to make
the first step, make the least collaborative effort. Furthermore, there is an area
outside the taxonomy and in the middle of the participation eye that refers to the
sense of belonging to the community and does not depend on active
participation. Active participation is initiated with the very first message and has

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 54


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

three levels, low, medium and high. If these levels can be displayed in real-time
on a discussion topic level and a course level, then participants in e-learning will
be able to be proactive by managing and evaluating participation in real time.
Real time online community management and evaluation of participation
provides immediate information delivery as needed for all stakeholders. It is cost-
effective; it is iterative rather than one-off as results are concentrated on the
process; it minimizes risk as decision making depends on facts rather than
assumptions; it is a catalyst for the stakeholders to accomplish their goals faster.

2.6.3 The Sense of E-Learning Community Index

Studying the Greek teachers’ participation in collaborative e-learning


communities suggested two ways to measure success and quality in e-learning,
the collaborative e-learning episodes (CeLE), and the sense of community. At the
first stages of this research, part of collaborative learning evaluation was initially
anchored in empathy to provide a sense of belonging. Based on the framework
suggested by Levenson and Ruef (1992:234), the participants in the initial study
had to answer questions on: (a) knowing what another person was feeling when
reading the message; (b) feeling what another person was feeling when reading
the message; (c) whether they took any action; and (d) whether the tools helped.
It is important to note that incorporating these attributes in a questionnaire was
found the only way to acquire results as all information in online discussions is
textual.

However, the measurement based on empathy was found inadequate to


evaluate the sense of community; the only tangible outcome was that the
members who were found to develop empathy were the ones who were very
active in the community (Lambropoulos, 2005c). This result does not describe
the sense of belonging and its interrelated attributes in a coherent way. So a
second approach was developed, the Sense of e-Learning Community Index
(SeLCI) and empathy was one attribute of the new framework. This process will
be further elaborated in the research design in Chapter 5.

The question here is on suitable research methodologies to inform design


towards the development of tools and evaluation techniques for collaborative e-
learning communities. This will be explored in the next chapter.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 55


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

REFERENCES
Ambrozek, J., Bundesen, L., & Cothrel, J. (2004). Online Communities in
Business: Past Progress, Future Directions. Paper presented at
International Conference on Virtual Communities 2004. Retrieved
15/12/2006, from http://www.infonortics.com/vc/vc04/slides/cothrel.pdf.
Anderson, B. (1997). Work, Ethnography and System Design. In A. Kent & J. G.
Williams (Eds.), The Encyclopedia of Microcomputers (Vol. 20, pp. 159-
183). New York, NY: Marcel Dekker.
Archichvili, A., Page, V., & Wentling, T. (2003). Motivation and Barriers to
Participation in Virtual knowledge-sharing Communities of Practice.
Journal of Knowledge Management, 7(1), 64-77.
Argyle, M. (1991). Cooperation: The basis of sociability. London: Routledge.
Axelrod, R. (1984). The Evolution of Cooperation. New York, NY: Basic Books.
Baker, M. J. (2000). The Roles of Models in Artificial Intelligence and Education
Research: A Prospective View. International Journal of Artificial
Intelligence in Education, 11, 122-143.
Baker, M.J., Hansen, T., Joiner, R., & Traum, D. (1999). The role of grounding in
collaborative learning tasks. In P. Dillenbourg (Ed.), Collaborative
learning. Cognitive and computational approaches (pp. 31--63).
Amsterdam: Pergamon.
Barab, S., & Duffy, T. (2000). From Practice Fields to Communities of Practice. In
D. Jonassen & S. Land (Eds.), Theoretical Foundations of Learning
Environments (pp. 25-55). Mahwah, NJ: Lawrence Erlhbaum Associates.
Beaudoin, M. F. (2002). Learning or Lurking? Tracking the 'Invisible' Online
Student. The Internet and Higher Education, 5(2), 147-155.
Beers, P. J., Boshuizen, H. P. A., & Kirschner, P. (2004). Computer support for
knowledge construction in collaborative learning environments. Paper
presented at the AERA'04 - American Educational Research Association,
Learning in innovative learning environments Symposium, San Diego, Ca.
Bell, P. L. (1997). Using argument representations to make thinking visible for
individuals and groups. Paper presented at the Computer Support for
Collaborative Learning, Toronto, Canada. Retrieved 02/12/2005, from
http://www.oise.utoronto.ca/cscl/papers/bell.pdf
Bell, P., Davis, E. A., & Linn, M. C. (1995). The knowledge integration
environment: Theory and design. Paper presented at the CSCL'95 -
Computer Supported Collaborative Learning Conference, Bloomington,
IN. Mahwah, NJ: Lawrence Erlbaum Associates, 14-21.
Berners-Lee, T. (2007). Berners-Lee warns of changes ahead. Computing
Magazine. Retrieved 17/06/2007, from
http://www.computing.co.uk/computing/analysis/2186086/berners-lee-
warns-changes-ahead.
Berge, Z. L., & Collins, M. (n.d.). E-moderators. Retrieved 23/04/2005, from
http://emoderators.com/.
Berger, C., & Kam, R. (1996). Educational Software Design and Authoring.
Retrieved 24/05/2005, from http://www.umich.edu/~ed626/626.html.
Bichelmeyer, B. A. (2005). The ADDIE "Model" - A Metaphor for the Lack of
Clarity in the Field of IDT. IDT Record. Retrieved 25/06/2005, from
http://www.indiana.edu/idt/shortpapers/documents/aect2004.htm.
Blaye, A., Light, P. H., Joiner, R., & Sheldon, S. (1991). Joint planning and
problem solving on a computer-based task. British Journal of
Developmental Psychology, 9, 471-483.
Bloom, B. S. (1956). Taxonomy of Educational Objectives, the classification of
educational goals – Handbook I: Cognitive Domain. New York, NY:
McKay.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 56


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Borges, M. A. F., & Baranauskas, M. C. C. (2003). CollabSS: a Tool to Help the


Facilitator in Promoting Collaboration among Learners. Educational
Technology & Society, 6(1). Retrieved 20/10/2006, from
http://ifets.ieee.org/periodical/vol_1_2003/borges.html.
Brookfield, S. (2001). Repositioning Ideology Critique in a Critical Theory of Adult
Learning. Adult Education Quarterly, 52(1), 7-22.
Brookfield, S. (1995). Adult Learning: An Overview. In A. Tuinjman (Ed.),
International Encyclopaedia of Education. Oxford: Pergamon Press.
Brown, J.S. & Duguid, P. (2000). The social life of information. Boston: Harvard
Business School Press.
Bruckman, A., & Resnick, M. (1995). The MediaMOO Project: Constructionism
and Professional Community. Convergence, 1(1), 94-109. Retrieved
25/05/2004, from http://llk.media.mit.edu/papers/convergence.html.
Buhayer, C. (2005). Greece - Culture Smart!: A Quick Guide to Customs and
Etiquette. London: Random House.
Butler, B., Sproull, L., Kiesler, S., & Kraut, R. (2002). Community Effort in Online
Groups: Who Does the Work and Why? In S. P. Weisband (Ed.),
Leadership at a Distance: Research in Technologically-Supported Work.
Mahwah, NJ: Lawrence Erlbaum Associates.
Carroll, J. M., & Rosson, M. B. (1996). Deliberated evolution: Stalling the View
Matcher in design space. In T. P. Moran & J. M. Carroll (Eds.), Design
rationale: Concepts, techniques, and use (pp. 107-145). Mahwah, NJ:
Lawrence Erlbaum Associates.
Clark, H. H, & Wilkes-Gibbs, D. (1986). Referring as a collaborative process.
Cognition, 20, 1-39.
Clark, H. H., Schreuder, R., and Buttrick, S. (1983). Common ground and the
understanding of demonstrative reference. Journal of Verbal Learning
and Verbal Behavior, 22, 245- 258.
Conrad, R.-M., & Donaldson, J. A. (2004). Engaging the Online Learner. San
Francisco, CA: Jossey-Bass.
Constant, D., Sproull, L., & Kiesler, S. (1996). The kindness of strangers: On the
usefulness of weak ties for technical advice. Organization Science, 7,
119-135.
Crook, C. (1994). Computers and the Collaborative Experience of Learning
London, UK: Routledge.
Dede, C. (2005). Planning for “Neomillennial” Learning Styles: Implications for
Investments in Technology and Faculty. In J. Oblinger & D. Oblinger
(Eds.), Educating the Net Generation (pp. 226-247). Boulder, CO:
EDUCAUSE Publishers. Retrieved 05/12/2005, from
http://www.educause.edu/ir/library/pdf/pub7101o.pdf.
Dillenburg, P. (2000). Virtual Learning Environments. Paper presented at the
EUN'00 - Learning In The New Millennium: Building New Education
Strategies For Schools, Workshop on Virtual Learning Environments,
Brussels, Belgium. Retrieved 20/12/2005, from
http://tecfa.unige.ch/tecfa/publicat/dil-papers-2/Dil.7.5.18.pdf.
Dillenbourg P. (1999) What do you mean by collaborative learning?. In P.
Dillenbourg (Ed) Collaborative-learning: Cognitive and Computational
Approaches. (pp.1-19). Oxford: Elsevier.
Dillenbourg, P., Baker, M., Blaye, A., & O'Malley, C. (1996). The evolution of
research on collaborative learning. In E. Spada & P. Reiman (Eds.),
Learning in Humans and Machine: Towards an interdisciplinary learning
science. (pp. 189-211). Oxford: Elsevier.
Dini, D. (2005). GamePlay Design. Paper presented at the GDTW-2005, Game
Design and Technology Workshop, 8-9 November 2005, Liverpool, UK.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 57


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Donath, J. S. (1996). Identity and Deception in the Virtual Community. In P.


Kollock & M. Smith (Eds.), Communities in Cyberspace (pp. 29-59).
London: Routledge.
Dourish, P. (2006). Implications for Design. Paper presented at the CHI'06 - ACM
Conference on Human Factors in Computing Systems: Interact. Inform.
Inspire. 22-27 April 2006, Montreal, Canada, 541-550. Retrieved
26/01/2007, from
http://www.ics.uci.edu/~jpd/publications/2006/implications-chi2006.pdf.
Downes, S. (2006). E-learning 2.0. eLearn Magazine. Retrieved 02/01/2007,
from http://www.elearnmag.org/subpage.cfm?section=articles&article=29-
1.
Ebner, M., & Holzinger, A. (2005). Lurking: An Underestimated Human-Computer
Phenomenon. IEEE MultiMedia, 12(4), 70-75.
Earley, P. C., & Gibson, C. B. (2002). Multinational Teams: New Perspectives.
Mahwah, NJ: Lawrence Earlbaum Associates.
Ehlers, U., Hildebrandt, B., Görtz, L. & Pawlowski J. (2005).Quality in European
e-learning – PANORAMA report. A study by the European Quality
Observatory. Retrieved 12/11/2006, from
http://cms.eun.org/shared/data/pdf/quality_in_e-
learning_panorama_en.pdf.
Faulkner, X. (2000). Usability Engineering. New York, NY: Palgrave, MacMillan.
Faulkner, X. (1998). The Essence of Human-Computer Interaction. London:
Prentice-Hall.
Fenrich, P. (in press). Instructional Design. In the Collaborative Book Project 'A
Practical Guide to Online Teaching and Learning'. British Columbia
Campus, Canada. In press.
Fenrich, P. (2005). Creating Instructional Multimedia Solutions: Practical
Guidelines for the Real World. Santa Rosa, Ca, USA: Informing Science.
Fish, R. S., Kraut, R. E., & Chalfonte, B. L. (1990). The VideoWindow system in
informal communication. Paper presented at the 1990 ACM conference
on Computer-Supported Cooperative Work. October 07 – 10, los Angeles,
Ca.
Foucault, M. (1984). The politics of health in the eighteenth century. In P.
Rabinow (Ed.), The Foucault Reader (pp. 279-282). New York, NY:
Pantheon Books.
Free Online Dictionary for Computing. Retrieved 23/03/2004, from
http://foldoc.org.
Gentry, C. (1994). Introduction to instructional development. Belmont, Ca, USA:
Wadsworth Publishing Co.
Godwin, M. (1994). Nine Principles for Making Virtual Communities Work. Wired,
2.06 (June), 72-73.
Goodfellow, R. (2003) Virtual Learning Communities. A report for the National
College for School Leadership. Milton Keynes, Institute of Educational
Technology, The Open University.
Gordon, J., & Zemke, R. (2000). The attack on ISD. Training, 37(4), 42-54.
Gould, J. D., & Lewis, C. (1985). Designing for usability: Key principles and what
designers think. Communications of the ACM, 28(3), 300-311. (Reprinted
in Baecker, R. M., & Buxton, W. A. S. (Eds.) (1987), Readings in human-
computer interaction: A multidisciplinary approach, 528-539. San Mateo,
CA: Morgan Kaufmann Publishers.)
Grice, P. H. (1975). Logic and Conversation. Syntax and Semantics, 1, 41-58.
Gulati, S. (2006). Learning during online and blended courses: Unpublished
Research, PhD Thesis at the Department of Education and Lifelong
Learning, City University, London.
Gulati, S. (2004). Constructivism and emerging online learning pedagogy: a
discussion for formal to acknowledge and promote the informal. Paper

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 58


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

presented at the UACE, the Annual Conference of the Universities


Association for Continuing Education - Regional Futures: Formal and
Informal Learning Perspectives, Centre for Lifelong Learning, 5–7 April
2004, University of Glamorgan, UK. Retrieved 20/11/2006, from
http://www.leeds.ac.uk/educol/documents/00003562.htm.
Gumperz, J. J. (1982). Discourse strategies. Cambridge, MA: Cambridge
University Press.
Herrnstein, R. J., & Murray, C. (1994). The bell curve. New York, NY: Free Press.
Hiltz, S. R. (1985). Online Communities: A Case Study of the Office of the
Future. Norwood, NJ: Ablex Publishing Corp.
Hoadley, C. M. (1999). Scaffolding scientific discussion using socially relevant
representations in networked multimedia. Unpublished research, PhD
Thesis, University of California, Berkeley.
Hoadley, C. M., & Enyedy, N. (1999). Between information and collaboration:
Middle spaces in computer media for learning. Paper presented at the
CSCL’99 - Computer Supported Collaborative Learning 1999, Stanford
University, Palo Alto, CA, Palo Alto Ca. Hillsdale, NJ: Lawrence Erlbaum
Associates.
Hoadley, C., Hsi, S., & Berman, B. P. (1995). The multimedia forum kiosk and
SpeakEasy. Paper presented at the 3rd ACM international conference on
multimedia, San Francisco, CA: ACM, 363–364.
Hovland, C. I., Janis, I. L., & Kelley, H. H. (1953). Communication and
Persuasion: psychological studies of opinion change. New Haven, C.T:
Yale University Press.
Hughes, M., & Daykin, N. (2002). Towards Constructivism: Investigating
Students’ Perceptions and Learning as a Result of Using an Online
Environment. Innovations in Education and Teaching International, 39(3),
217-223.
Jarvis, P. (2004). Adult Education and Lifelong Learning: Theory and Practice.
London: RoutledgeFalmer.
Jennings, D. (2005). E-learning 2.0, whatever that is. Retrieved 02/01/2007, from
http://alchemi.co.uk/archives/ele/elearning_20_wh.html.
Jeong, A. (2008). Varying the number of required postings versus posting days:
Promoting student interaction and sustained discourse in online
discussions. British Journal of Educational Technology.
Jeong, A., & Davidson-Shivers, G. (2006). The effects of gender interaction
patterns on student participation in computer-supported collaborative
argumentation. Educational Technology, Research, and Development,
54(6), 543-568.
Jeong, A. (2005). Methods and Tools for the Computational Analysis of Group
Interaction and Argumentation in Asynchronous Online Discussions.
Paper presented at the Technology and Learning Symposium. 5-6 April
2006, New York, NY. Retrieved 10/12/2006, from http://dev22448-
01.sp01.fsu.edu/Research/Proposals/LearnTechSymposium/MethodsTool
sAnalyzingInteraction_Jeong2005.pdf.
Johnson, R. T., & Johnson, D. W. (1994). An Overview of Cooperative Learning.
In J. Thousand, A. Villa & Nevin (Eds.), Creativity and Collaborative
Learning. Baltimore: Brookes Press.
Jonassen, D., & Remidez, H. (2005). Mapping alternative discourse structures
onto computer conference. Inderscience, International Journal of
Knowledge and Learning, 1(1-2), 113-129.
Jonassen, D. H. (n.d.). Design of Constructivist Learning Environments(CLEs).
Retrieved 02/12/2005, from
http://tiger.coe.missouri.edu/~jonassen/courses/CLE/
Ickes, W. (Ed.). (1997). Empathic accuracy. New York, NY: Guilford.
Illeris, K. (2002). Three Dimensions of Learning. Leicester: NIACE.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 59


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Interlock, http://www.interloc.org/ Last access 28/08/2007.


International Organisation for Standardisation (ISO) (1998). (1998). Ergonomic
requirements for office work with visual display terminals (VDTs) -- Part
11: Guidance on usability.
Karrer, T. (2006). What is e-learning 2.0? Blog entry. Retrieved 05/09/2006, from
http://elearningtech.blogspot.com/2006/02/what-is-elearning-20.html.
Khine, S. M. (2003). Creating a Technology-Rich Constructivist Learning
Environment in a classroom management module. In S. M. Khine & D.
Fisher (Eds.), Technology-Rich Learning Environments (pp. 21-39). New
Jersey: World Scientific.
Kim, A. J. (2001). Building Successful eLearning Communities. elearningpost,
May 2001. Retrieved 15/09/2005, from
http://www.elearningpost.com/articles/archives/exclusive_interview_with_
amy_jo_kim/.
Kim, A. J. (1998). Social Scaffolding: 9 Design Principles for Online Community-
Building. Web Techniques- Jan 1998. Retrieved 26/07/2006, from
http://www.naima.com/articles/webtechniques.html.
Klemm, W. R. (1998). Eight Ways to Get Students More Engaged in Online
Conferences. The Higher Education Journal, 26(1), 62-64.
Knowles, M. (1984). Andragogy in Action: Applying Modern Principles of Adult
Learning. San Francisco: Jossey-Bass.
Kollock, P. (1998). Design Principles for Online Communities. PC Update, 15(5),
58-60.
Kollock, P., & Smith, M. (1996). Managing the Virtual Commons: Cooperation
and Conflict in Computer Communities. In S. Herring (Ed.), Computer-
Mediated Communication: Linguistic, Social, and Cross-Cultural
Perspectives (pp. 109-128). Amsterdam, the Netherlands: John
Benjamins.
Kommers, P. A. M., Jonassen, D., & Mayes, T. (1992) (Eds). Cognitive Tools for
Learning. NATO ASI Series / Computer and Systems Sciences, Vol. 81.
Berlin: Springer-Verlag.
Lambropoulos, N. (2006). User-Centred Design of Online Learning Communities.
In N. Lambropoulos & P. Zaphiris (Eds.), User-Centred Design of Online
Learning Communities (pp. 1-28). Hershey, PA, USA: Idea Publishing.
Lambropoulos, N. (2005). Online Empathy: Social Consequences for Community
Building. In S. Dasgupta (Ed.), Encyclopaedia of Virtual Communities and
Technologies (pp. 346-348). Hershey, PA: Idea Publishing.
Lambropoulos, N. (2005). Paradise lost? Primary Empathy in Online
Communities of Interest and Ways of Use. Paper presented at the 1st
Conference on Online Communities and Social Computing, in the 11th
International Conference on Human-Computer Interaction 2005, 22-27
July, Las Vegas, NV. Retrieved 05/09/2005, from
http://www.intelligenesis.eu/nikiweb/PDF/05/lambropoulosOnlineEmpathy
Vegas05Research.pdf.
Lambropoulos, N. (2005). Online Empathy: Social Consequences for Community
Building. In S. Dasgupta (Ed.), Encyclopaedia of Virtual Communities and
Technologies (pp. 346-348). Hershey, PA: Idea Publishing.
Lambropoulos, N. (2002). Project IRIS: I.C.T. Projects for the Hellenic Education
in the U.K. Presented at the Annual Conference for the Hellenic
Education in the U.K. London, 1-3 September, 2002.
Laszlo, K. C., & Laszlo, A. (2000). Learning to become: creating evolutionary
learning community through evolutionary systems design. In R. Miller
(Ed.), Creating Learning Communities. New York, NY: Solomon Press.
Laszlo, K. C., & Laszlo, A. (1997). Partners in life — Syntony at work. from
http://home.earthlink.net/aklaszlo/syntonyques

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 60


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Lave, J. (1988). Cognition in practice: Mind, mathematics and culture in everyday


life. Cambridge: Cambridge University Press.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral
participation. Cambridge: Cambridge University Press.
Lee, Y.-W., Chen, F.-C., & Jiang, H.-M. (2006). Lurking as participation: a
community perspective on lurkers' identity and negotiability. Paper
presented at the ICLS '06, 7th International Conference on Learning
sciences, Indiana University, June 27-July 1, Bloomington IN,
International Society of the Learning Sciences. Retrieved 10/06/2005,
from
http://portal.acm.org/ft_gateway.cfm?id=1150093&type=pdf&coll=GUIDE
&dl=GUIDE&CFID=21290811&CFTOKEN=83883790.
Lee, F. S. L., Vogel, D. R., & Limayem, M. (2002). Adoption of Informatics to
Support Virtual Communities. Paper presented at the HICSS 2002,
Hawaii International Conference on System Sciences, 7-10 January 2002,
Big Island, Hawaii, IEEE Computer Society. Paper No 214.
Leigh, D. (n.d.). A Brief History of Instructional Design. Retrieved 02/01/2006,
from http://www.pignc-ispi.com/articles/education/brief%20history.htm.
Leinonen, T., Kligyte, G., Toikkanen, T., Pietarila, J., & Dean, P. (2003). Learning
with Collaborative Software - A guide to Fle3. Media Lab, University of Art
and Design, Helsinki. Retrieved 12/10/2005, from
http://fle3.uiah.fi/papers/fle3_guide.pdf.
Levenson, R. W., & Ruef, A. M. (1992). Empathy: A physiological substrate.
Journal of Personality and Social Psychology, 63(2), 234-246.
Light, P. H., & Littleton, K. (1999). Social Processes in Children’s Learning.
Cambridge, UK: Cambridge University Press.
Ling, K., Beenen, G., Ludford, P., Wang, X., Chang, K., Li, X., Cosley, D.,
Frankowski, D., Terveen, L., Rashid, A. M., Resnick, P., and Kraut, R.
(2005). Using social psychology to motivate contributions to online
communities. Journal of Computer-Mediated Communication, 10(4),
article 10. Retrieved 30/04/2006, from
http://jcmc.indiana.edu/vol10/issue4/ling.html
Mäkitalo, K., Salo, P., Häkkinen, P., & Järvelä, S. (2001). Analysing the
mechanisms of common ground in collaborative web-based interaction. In
P. Dillenbourg, A. Eurelings, & K. Hakkarainen (Eds.) Euro-CSCL
European Perspectives on Computer-Supported Collaborative Learning.
First European Conference on Computer-Supported Collaborative
Learning. 22-24 March 2001, Universiteit Maastricht: Maastricht,
Netherlands, 445-453.
McDonald, J. (2003). Let’s get more positive about the term ‘lurker’. Retrieved
02/03/2004, from
http://www.cpsquare.org/edu/News/archives/LurkerProjectCoPWorkshop
SPring03a.doc.
Mercer, N. (1995). The Guided construction of knowledge: talk amongst teachers
and learners. Clevedon: Multilingual Matters.
Mercer, N., & Wegerif, R. (1999). Is 'exploratory talk' productive talk? In K.
Littleton & P. Light (Eds.), Learning with computers. Analyzing productive
interaction (pp. 79-101). London: Routledge.
Mezirow, J. (2000). Learning to Think Like an Adult: Core Concepts of
Transformation Theory. In J. M. a. Associates (Ed.), Learning as
Transformation (pp. 3-33). San Francisco: Jossey – Bass.
Miyake, N. (1986). Constructive Interaction and the Iterative Process of
Understanding. Cognitive Science, 10, 151-177.
Monthienvichienchai, R. (2004). Computer Support for Vicarious Learning
Unpublished Research, PhD Thesis at the University College London,
London.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 61


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Molenda, M. (1987). An Agenda for Research on Instructional Development


Paper presented at the AECT'87, Annual Conference of Association for
Educational Communications and Technology. 26 February, Atlanta,
USA. Retrieved 23/05/2005, from
http://www.indiana.edu/~molpage/Agenda%20Res%20on%20ID.pdf.
Moodle (n.d.) Modular Object-Oriented Dynamic Learning Environment.
Retrieved 20/08/2007, from http://moodle.org.
Muir, A., Shield, L., & Kukulska-Hulme, A. (2003). The Pyramid of Usability: A
Framework for Quality Course Websites. Paper presented at the EDEN
12th Annual Conference of the European Distance Education Network,
The Quality Dialogue: Integrating Quality Cultures in Flexible, Distance
and eLearning. 15-18 June 2003, Rhodes, Greece, p. 188-194.
Mumford, E. (1983). Designing Human Systems - The ETHICS Method.
Retrieved 19/12/2005, from http://www.enid.u-net.com/C1book1.htm.
Myrach, T., & Knolmayer, G. (2005). Modularity and Integrity in Instructional
Engineering. Paper presented at the NAISO World Congress on
Networked Learning in a Global Environment; Challenges and Solutions
for Virtual Education 2002-05, Berlin. Retrieved 14/11/2006, from
http://opess.ie.iwi.unibe.ch/download/100029-03-TM-024.pdf.
Nassaji, H., & Wells, G. (2000). What’s the use of ‘triadic dialogue’?: An
investigation of teacher-student interaction. Applied Linguistics, 21(3),
376-406.
Neal, L. (2003). Q&A With Diana Laurillard. Retrieved 22/10/05, from
http://www.elearnmag.org/subpage.cfm?section=articles&article=14-1
Nielsen, J. (1993). Usability Engineering. Boston, USA: Academic Press.
Nokelainen, P. (2006). An empirical assessment of pedagogical usability criteria
for digital learning material with elementary school students. Educational
Technology & Society, 9(2), 178-197.
Nonnecke, B. (2000). Lurking in email-based discussion lists. Unpublished
Research, PhD Thesis at London South Bank University, London.
Nonnecke, B., & Preece, J. (2000). Lurker demographics: counting the silent.
Paper presented at the CHI Conference on Human Factors in Computing
Systems, April 1-6, 2000, Hague, Netherlands. New York, NY: ACM
Press, 73 – 80. Retrieved 25/06/2004, from
http://www.ifsm.umbc.edu/~preece/paper/13%20Nonnecke&PreeceCHIp
aper_ACMnotice.PDF
Nonnecke, B., & Preece, J. (1999). Shedding light on Lurkers in Online
Communities. Paper presented at the Conference Ethnographic Studies
in Real and Virtual Environments: Inhabited Information Spaces and
Connected Communities. 24-26 January 1999, Edinburgh, UK, 123-128.
Retrieved 18/11/2004, from
http://www.ifsm.umbc.edu/~preece/Papers/SheddingLight.final.pdf.
Norman, D. A., & Spohrer, J. C. (1996). Learner-Centered Education.
Communications of the ACM, 39(4), 24-27.
Oliver, M., & Shaw, G. (2003). Asynchronous Discussion in Support of Medical
Education Journal of Asynchronous Learning Networks, 7(1), 56-67.
O'Malley, C. (1987). Understanding Explanation. Cognitive Science Research
Report Sussex, UK: University of Sussex.
O'Neil, D. K., & Gomez, L. (1994). The Collaborative Notebook: A distributed
knoowledge building environment for project enhanced learning. Paper
presented at the Conference Ed-Media ’94 - Educational Multimedia and
Hypermedia, Charlottesville, VA, 416-423.
Oriogun, P. K. (2006). Content Analysis of Online Transcripts: Measuring Quality
of Interaction, Participation and Cognitive Engagement within CMC
Groups by Cleaning of Transcripts. International Journal of Instructional

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 62


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Technology and Distance Learning (March 2006). Retrieved 05/06/2005,


from http://www.itdl.org/Journal/Mar_06/article03.htm.
Ostrom, E. (1990). Governing the Commons: The Evolution of Institutions for
Collective Action. New York, NY: Cambridge University Press.
Paquette, G. (2003). Instructional Engineering in Networked Environments
Canada: John Wiley & Sons.
Peck, M. S. (1987). The Different Drum: Community making and peace. New
York: Touchstone.
Piaget, J. (1932). The Moral Judgement of the Child. New York, NY: Harcourt,
Brace Jovanovich.
Preece, J. (2004). Etiquette, empathy and trust in communities of practice:
Stepping-stones to social capital. Journal of Universal Computer Science,
10, 294-302.
Preece, J., Nonnecke, B., & Andrews, D. (2004). The top 5 reasons for lurking:
Improving community experiences for everyone. Computers in Human
Behavior, 2(1).
Preece, J., & Ghozati, K. (2001). Observations and Explorations of Empathy
Online. In R. R. Rice & J. E. Katz (Eds.), The Internet and Health
Communication: Experience and Expectations (pp. 237-260). Thousand
Oaks: Sage Publications Inc.
Preece, J. (2000). Online Communities: Designing Usability, Supporting
Sociability. Chichester, UK: John Wiley & Sons.
Preece, J. (1999). Empathic communities: Balancing emotional and factual
communication. Interacting with Computers, The Interdisciplinary Journal
of Human-Computer Interaction, 12(1), 63-77.
Rafaeli, S., Ravid, G., & Soroka, V. (2004). De-lurking in virtual communities: a
social communication network approach to measuring the effects of social
and cultural capital. Paper presented at HICSS'04, 37th Hawaii
International Conference on System Sciences. 5-8 January 2004, Big
Island, Hawaii, IEEE Computer Society. Retrieved 18/11/2006, from
http://csdl2.computer.org/comp/proceedings/hicss/2004/2056/07/2056702
03.pdf.
Rafaeli, S., Sudweeks, F., & McLaughlin, M. (1998). Network & Netplay: Virtual
Groups on the Internet. Cambridge, Mass: MIT Press.
Rafaeli, S., & Sudweeks, F. (1997). Networked Interactivity. Journal of Computer-
Mediated Communication, 2(4). Retrieved 15/03/2004, from
http://jcmc.indiana.edu/vol2/issue4/rafaeli.sudweeks.html.
Rafaeli, S. (1986). The electronic bulletin board: A computer driven mass
medium. Computers and the Social Sciences, 2(3), 123-136.
Ramachandran, V. S. R. (2000). Mirror Neurons and imitation learning as the
driving force behind "the great leap forward" in human evolution. Edge,
69. retrieved 09/03/2006, from
http://www.edge.org/3rd_culture/ramachandran06/ramachandran06_inde
x.html.
Ravenscroft, A., Wegerif, R., Mercer, N., & Hartley, J. R. (2006). Reclaiming
thinking: dialectics and design for digital learning. Paper presented at the
British Journal of Educational Psychology Current Trends Conference,
Nottingham Trent University, UK.
Rheingold, H. (1993). The Virtual Community. Reading: Addison-Wesley.
Richey, R. C., Fields, D. C., & M. Foxon, M. (Eds.). (2000). Instructional design
competencies: The standards (3rd ed.). Syracuse, NY: ERIC
Clearinghouse on Information and Technology and the International
Board of Standards for Training, Performance and Instruction.
Rogers, A. (2003). What is the difference? a new critique of adult learning and
teaching. Leicester: NIACE.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 63


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Rogoff, B. (1991). Social interaction as apprenticeship in thinking: Guidance and


participation in spatial planning. In L. B. Resnick, J. M. Levine & S. D.
Teasley (Eds.), Perspectives on socially shared cognition (pp. 349-383).
Washington, DC: American Psychological Association.
Rumble, G. (2001). Re-inventing distance education, 1971-2001. International
Journal of Lifelong Education, 20(1/2), 31-43.
Ryder, M. Instructional Design (an overview). Retrieved 25/03/2005, from
http://carbon.cudenver.edu/~mryder/itc_data/idmodels.html
Salmon, G. (2000). E-moderating: the key to teaching and learning online.
London: Kogan Page.
Salmons, J., & Wilson, L. (Eds.). (2008). Handbook of Research on Electronic
Collaboration and Organizational Synergy. Hershey, PA, USA: IGI Global
Publications.
Schwier, R. A., Campbell, K., & Kenny, R. (2006). Instructional Designers’
Perceptions of Their Agency: Tales of Change and Community. In M. J.
Keppell (Ed.), Instructional design: Case studies in communities of
practice. . Hershey, PA: Idea Group.
Shackel, B. (1991). Usability – Context, framework, definition, design and
evaluation. In B. Shackel & S. J. Richardson (Eds.), Human Factors for
Informatics Usability (pp. 21–37). New York, NY: Cambridge University
Press.
Scardamalia, M., Bereiter, C., & Lamon, M. (1994). The Computer-Supported
Intentional Learning Environment (CSILE) Project: Trying to bring the
classroom into World 3. In K. McGilly (Ed.), Classroom lessons:
Integrating cognitive theory and classroom practice (pp. 201–228).
Cambridge MA: MIT Press.
Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building
communities. The Journal of the Learning Sciences, 3(3), 265-283.
Schultz, N., & Beach, B. (2004). From Lurkers to Posters (Report): Australian
Flexible Learning Framework. Retrieved 23/03/2005, from
http://flexiblelearning.net.au/resources/lurkerstoposters.pdf.
Seels, B., & Glasgow, Z. (1998). Making instructional design decisions (2nd ed.).
Upper Saddle River, NJ: Merrill Prentice Hall.
Shambaugh, N., & Magliaro, S. G. (2005). Instructional design: A systematic
approach for reflective practice. Boston, MA: Allyn & Bacon.
Shepherd, C. (2001). Engineering e-learning. Retrieved 06/11/2006, from
http://www.fastrak-consulting.co.uk/tactix/features/engineer.htm.
Shneiderman, B., & Maes, P. (1997). Direct Manipulation vs Interface Agents,
excerpts from the debates at IUI 97 and CHI 97 Interactions, 4(6), 42-61.
Siegel, M., Derry, S. J., Steinkuehler, K. A., Kim, J. B., & Seymour, J. (2001).
What and how preservice teachers learn: Designing a course that fosters
development of useful theoretical knowledge and the assessment
methods to capture it. Paper presented at the AERA'01 - Annual Meeting
of the American Educational Research Association. April 10-14, Seattle,
WA.
Smulders, D. (2002). Designing for Learners, Designing for Users. eLearn
Magazine. Retrieved 05/12/2005, from
http://elearnmag.org/subpage.cfm?section=best_practices&article=11-1.
Silius, K., Tervakari, A.M., & Pohjolainen, S. (2003). A Multidisciplinary Tool for
the Evaluation of Usability, Pedagogical Usability, Accessibility and
Informational Quality of Web-based Courses. Paper presented at PEG’03,
11th International PEG Conference: Powerful ICT for Teaching and
Learning. 28 June - 1 July 2003 , St. Petersburg, Russia. CD-Rom.
Silius, K., & Tervakari, A.-M. (2003, 8-9 May 2003). An Evaluation of the
Usefulness of Web-based Learning Environments.The Evaluation Tool
into the Portal of Finnish Virtual University. Paper presented at the

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 64


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

International Conference on Network Universities and E-Learning,


Valencia, Spain. Retrieved 23/04/2006, from
http://matriisi.ee.tut.fi/arvo/liitteet/usefulness_of_web.pdf.
Soller, A., Mones, A. M., Jermann, P., & Muehlenbrock, M. (2005). From
Mirroring to Guiding: A Review of State of the Art Technology for
Supporting Collaborative Learning. International Journal of Artificial
Intelligence in Education, 15, 261-290.
Soller, A., Wiebe, J., & Lesgold, A. (2002). A Machine Learning Approach to
Assessing Knowledge Sharing During Collaborative Learning Activities.
Paper presented at the CSCL'02 - Computer-Support for Collaborative
Learning, 7-11 January 2002, Boulder, CO. Retrieved 19/03/2004, from
http://www.cs.pitt.edu/~wiebe/pubs/papers/sollerwiebelesgold02.doc.
Soroka, V., Jacovi, M., & Ur, S. (2003). We can see you: a study of communities'
invisible people through reachout. In M. Huysman, E. Wenger & V. Wulf
(Eds.), Communities and technologies (pp. 65 - 79). Deventer, The
Netherlands: Kluwer, B.V.
Sproull, L., & Faraj, S. (1997). Atheism, sex, and databases: The net as a social
technology. In S. Kiesler (Ed.), Culture of the Internet (pp. 35–51).
Mahwah, NJ: Lawrence Erlbaum Associates.
Squires, D., & Preece, J. (1999). Predicting Quality in Educational Software:
Evaluating for learning, usability and the synergy between them.
Interacting with Computers, 11(5), 467-483.
Suchman, L. A. (1987). Plans and Situated Action: The Problem of Human-
Machine Communication New York: Cambridge University Press.
Swan, K., Holmes, A., Vargas, J. D., Jennings, S., Meier, E., & Rubenfeld, L.
(2002). Situated professional development and technology integration:
the CATIE mentoring program. Journal of Technology and Teacher
Education, 10(2), 169-190.
Teasley, S. D., & Roschelle, J. (1993). Constructing a Joint Problem Space: The
Computer as a Tool for Sharing Knowledge. In S. P. Lajoie & S. J. Derry
(Eds.), Computers as Cognitive Tools. New Jersey, NJ, USA: Lawrence
Erlbaum Associates, Inc. Publishers.
Tharp, R., & Gallimore., R. (1988). Rousing Minds to Life: Teaching, Learning,
and Schooling in Social Context. New York: Cambridge University Press.
Toulmin, S. (1958). The Uses of Argument. Cambridge, MA: Cambridge
University Press.
Traum, D. R. (1994). A Computational Theory of Grounding in Natural Language
Conversation. Unpublished research, PhD Thesis. University of
Rochester, Rochester, NY.
Tripp, S., & Bichelmeyer, B. (1990). Rapid prototyping: An alternative
instructional design strategy. Educational Technology Research &
Development, 38(1), 31-44.
Typaldos, C. (2000). Community Standards. Interview with Katharine
Mieszkowski, Fast Company, 38, 369. Retrieved 02/03/2004, from
http://www.fastcompany.com/online/38/ideazone.html.
UNESCO (n.d.) Technology & Learning definitions: Collaborative Learning.
Retrieved 12/07/2004, from
http://www.unesco.org/education/educprog/lwf/doc/portfolio/definitions.ht
m.
Van Meer, E. (2003). PLATO: From Computer-Based Education to Corporate
Social Responsibility. Iterations: An Interdisciplinary Journal of Software
History. Retrieved 02/12/2005, from
http://www.cbi.umn.edu/iterations/vanmeer.html.
Van Der Akker, J., Branch, R. M., Gustafson, K., Nieveen, N., & Plomp, T. (Eds.).
(2004). Design approaches and tools in education and training. Boston:
Kluwer Academic Publishers.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 65


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 2: Literature Review

Visscher-Voerman, J. I. A. (1999). Design Approaches in Training and Education:


A reconstructive study. University of Twente, Enschede, the Netherlands.
Vygotsky, L. S. (1962). Thought and Language. Cambridge Mass: M.I.T. Press.
Published originally in Russian in 1934.
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological
processes. In M. Cole, V. John-Steiner, S. Scribner & E. Souberman
(Eds.). Cambridge, MA, USA: Harvard University Press.
Wallace, R. (1999). Learners as Users, Users as Learners: What’s the
Difference? Retrieved 24/05/2005, from
http://www.msu.edu/~mccrory/pubs/HCIC_LCDboaster.htm
Wasko, M. M., S. Faraj, S., & Teigland, R. (2004). Collective Action and
Knowledge Contribution in Electronic Networks of Practice. Journal of the
Association of Information Systems, 5(11), Article 15.
Waters, S. H., & Gibbons, A. S. (2004). Design languages, notation systems, and
instructional technology: A case study. Educational Technology Research
and Development, 52(2), 57-68.
Wellman, B., & Gulia, M. (1999). Net-Surfers Don’t Ride Alone: Virtual
Communities as Communities. In B. Wellman (Ed.), Networks in the
Global Village: Life in Contemporary Communities. Boulder, CO:
Westview Press.
Wegerif, R., Mercer, N., & Dawes, L. (1998). Software Design to Support
Discussion in the Primary Classroom. Journal of Computer Assisted
Learning, 14(3), 199-211.
Whittaker, S., L. Terveen, W. Hill & L. Cherny (1998). The dynamics of mass
interaction. Paper presented at the CSCW 98, Computer-Supported
Collaborative Work. Seattle, WA, 1998, ACM Press, New York, NY.
Press, pp. 257-264.
Wiley, D. (2002). A Proposed Measure of Discussion Activity in Threaded
Discussion Spaces v0.9. Retrieved 09/11/2005, from
http://wiley.ed.usu.edu/docs/discussion09.pdf.
Williams, J. B., & Jacobs, J. (2004). Exploring the use of blogs as learning
spaces in the higher education sector. Australasian Journal of
Educational Technology, 20(2), 232-247.
Williams, C. (2002). Learning On-line: a review of recent literature in a rapidly
expanding field. Journal of Further and Higher Education, 26(3), 263-272.
Wilson, B. G., & Myers, K. M. (2000). Situated Cognition in Theoretical and
Practical Context. In D. Jonassen & S. Land (Eds.), Theoretical
Foundations of Learning Environments. (pp. 57-88). Mahwah NJ:
Lawrence Erlbaum Associates.
Winn, W. (1986). Emerging trends in educational technology research. Paper
presented at AECT'86, Annual conference of the Association for
Educational Communications and Technology. 16-21 January, Las
Vegas, NV.
Wood, D., Bruner, J., & Ross, G. (1976). The role of tutoring in problem solving.
Journal of child psychology and psychiatry, 17, 89-100.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 66


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

The Research Design:

3 Research Methods for Investigating Collaborative E-Learning


Communities

Key Topics Covered in This Chapter:

• E-Learning research advantages and limitations

• The need for a systematic e-research design to examine the research


context

• Why and how ethnotechnology can support investigation in


collaborative e-learning communities

• The dual nature of human-human and human-computer interaction


methodologies

• The research coordination and design

Chapter 3 introduces the research design. A review of research methods to support


design and e-learning shows advantages and limitations caused by their
multidisciplinary nature. Ethnotechnology was found to be a suitable approach to
examine the research context as well as to analyse human-human and human-
computer interactions in e-learning environments. The questionnaire design, time
series design and research constraints are presented.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 67


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

3.1. INTRODUCTION

Rapid technological changes influence communication, collaboration,


information and knowledge management for e-learning. Within the context of these
new challenges, research design needs to explore human-human and human-
computer interactions which have learning as their purpose, that is Human-Computer
Interaction Education.

Sasse, referring to the different aspects of the HCI nature, says that “the lack of a
single agreed research strategy discipline leaves an individual researcher planning a
specific research undertaking out on a limb” (n.d.). This multifaceted nature is
reflected in the lack of a single agreed strategy; however, this disadvantage can be
tackled by triangulation. Nonetheless, there is a risk; some times an amalgamation of
different research methods fails because of the disengagement between methods
and their underlying methodology (Boehner et al., 2007).

This chapter builds on situated e-research. The first section discusses issues for e-
learning research and ethics. Then, it employs an aspect of ethnography, that is
ethnotechnology, to examine the research context. Ethnotechnology is used
alongside human-computer and human-human interaction analysis to shed more
light on collaborative e-learning communities by triangulating sides of space and
time. Time-series design is suggested as a means of coordinating e-research.

3.1.1. E-Learning Research

E-Research employs web server analysis, application logs, data mining,


participant research, virtual ethnography, and Internet use. Because of its virtual and
interdisciplinary nature, e-research challenges research methods, e-researcher’s
skills, and ethics (Anderson & Kanuka, 2003). According to the British Psychological
Society (http://www.bps.org.uk/, n.d.), the ethical principles for conducting research
with human participants include general principles that may differ in e-research.
Anderson & Kanuka said that creating and maintaining respectful relationships with
e-participants is underpinned by three principles: voluntary informed consent;
privacy, confidentiality and anonymity; and recognising the elements of e-research
risk. E-researchers need to balance the conflicts which derive from the
implementation of these principles. For example, a hacker hacked the server where

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 68


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

the final study was hosted and wiped out all database tables; on a second occasion,
at the researcher’s request, the system operators collected data including
demographic details for GSN’s own use and also for research purposes.

On the other hand, Robinson and colleagues (2007) suggest that a combination of
methods is needed, but in order to avoid any form of control over the participants,
data can be collected as it occurs naturally. To Robinson and colleagues, working on
ethnographically-informed empirical studies, it is legitimate to consider such data in
order to understand practice in its own terms. This approach may resolve the
problem in research created by the Hawthorn effect. However, in reality “there are no
clearly defined criteria for appropriate ethical behaviour for all researchers or all
research activity” (Anderson & Kanuka, 2003:58).

In order to address the ethical issues, this study was approved by three different
organisations: the London South Bank University Ethics Committee, the Greek
Pedagogical Institute, and the Innovations Office, part of the Greek Ministry of
Education and Religious Affairs. Prior to the transfer of this research to LSBU there
had been no approval sought for the two initial studies not presented in this thesis.
Nonetheless, these were conducted under all ethical considerations and principles
and participants’ consent was obtained for all studies. The consent was obtained by
the participants’ signature on a pro-forma which emphasised that they could withdraw
from the study at any time without providing any explanation. The participants were
allowed to keep a copy; in addition, the right to withdraw was stressed in the user
policy (netiquette) available online as well as sent to the participants prior to the
study.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 69


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

3.2. RESEARCH DESIGN

The proposed research design is (Table 3.2-1):

Table 3.2-1. Research Design

RESEARCH DESIGN
Target Use
Research Methodologies Implications
Coordination Evaluation
for Design
1 HCI-HHI Ethnotechnology √ √ √
2 Qualitative Methodologies
HHI A Message analysis √
3 HHI Social network analysis √
4 Quantitative Methodologies
HCI-HHI A Questionnaires √ √
HCI B Log Analysis √ √
5 n/a Time-series design √

The research design targets methodological triangulation, that is the application and
combination of several research methodologies of the same phenomenon to
corroborate one set of findings with another; the hope is that two or more sets of
findings will converge on a single proposition (for a review of triangulation, see
Massey, 1999). Initially, ethnotechnology can increase understanding of the research
context for human-human and human-computer interactions (HHI & HCI).

In addition, time-series design coordinates the research activities on a temporal basis


for baselines and interventions. These approaches will be analysed in this chapter as
regards their scope, principles, and research instruments as well as their advantages
and disadvantages for this particular study.

3.2.1. Examining the Research Context

For the purpose of this study, the examination of research design should
scrutinise the situation as regards participation in collaborative e-learning
communities and determine implications for designing successful collaborative e-
learning communities. Ethnographic research was found to represent a long tradition
of studying social processes in real life situations, used to uncover the knowledge,
ideas, beliefs, values, and purposes of systems use, and thus informing the
designers about the setting. It has been exploited in computing research, especially

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 70


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

after Suchman (1987) discussed the profound mismatch between the generic models
of work on which IT systems are built at the time and the actual nature of work in
which they are used.

Ethnography is located in the field of naturalistic research (qualitative), distinguishing


it from positivist research (quantitative). It involves the ethnographer immersing
herself in people’s daily life for an extended period of time, watching what happens,
listening to what is said, asking questions, and collecting whatever data is available.
It is an ongoing attempt to place specific encounters, events, and understandings into
a more meaningful context. It is not simply the production of new information or
research data, but rather the way in which such data are transformed into a written or
visual form (Tedlock, 2000:455).

The advantages of ethnography in the investigation of tools and evaluation


techniques for collaborative e-learning communities are related to the rich description
of the research context and the exploratory stance that relies on the researcher’s
own ability to provide rich descriptions of data and understand the meaning of social
situations. However, the ethnographer’s role towards a rich fieldwork is also a
disadvantage. Many ethnographers advocate the need for a trained ethnographer in
order to go beyond simple descriptions (fieldwork), to a genuine analytic one (scenic
fieldwork) (McGarry, 2005:67; Ian Sommerville, Professor of Software Engineering at
Lancaster University; personal communication via email, 01/05/2007). This has
created two approaches in the use of ethnography in systems design, one that
suggests there should be two groups of experts, the engineers and ethnographers,
and the second that sees the engineer conducting the fieldwork. After stressing the
importance attached to the use of ethnography for fieldwork analysis in design, two
examples from these traditions will be presented.

3.2.1.1. Ethnography

Ethnography was suggested as the preferred approach to study lurkers and


thus, participation in online communities (Nonnecke, 2000). One way to utilise
ethnography in design is by ethnography’s scenic fieldwork, that is descriptive and
historical accounts for understanding social settings. These provide the explanatory
frame and the narrative explaining of “why” and “how” these implications one arrived
at. This is a demanding process, so some systems designers employed
ethnographers to provide description of the context needed for design. However,

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 71


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

there was a problem related to finding common ground and communication channels
for both the fieldworker and the engineer. Despite the efforts to ease their differences
(e.g. Hughes et al., 1997), and after almost 10 years of research, Hughes (1999)
concluded that there are major differences between ethnography and systems
design; they are two different mentalities.

A more recent approach considers that an engineer is able to conduct the fieldwork
herself. Although this approach has created questions on the engineer’s
epistemological validity there were successful implementations (Furnis, 2004; Sharp
et al., 2006). Studying teamwork in their own agile software development groups,
Sharp and colleagues (e.g. 2005, 2006) adopted ethnographic perspectives in order
to put design to work in particular contexts, adopted and adapted by people in the
course of practice, contradicting the definition of the “user” as a passive recipient of
technology. Thus, design is the active process of incorporation and co-evolution of
technologies, practices, and settings.

Assumptions on the use of ethnography for the ethnographer’s role as well as the
locus of e-research were revisited. If the designer/researcher is located within their
domain of study, in this case her own community of practice, can immerse herself
easily since the prolonged engagement and immersion in the context required in
ethnography already exists. Thus, she can progressively reconfigure new ways to
understand ‘users’ and ‘user contexts’ by understanding work practices within her
own culture and society as inspirations and foundations for design activity in order to
support new ways of working. These were the assumptions that were used in this
study.

3.2.1.2. Ethnotechnology: the Virtual Ethnography

Ethnography employed in real environments has taken advantage of


conversations, written materials, and observation. In fact, Button (2000) suggested
that it is important to consider how relevant people do what they do, that is the
‘interactional what' of their activities. This is the explication of members' prior
knowledge: what people have to know to do work, and how that knowledge is
deployed in the ordering and organisation of work. He proposed that this
consideration provides the key to understanding the contribution of sociology to
engineering and design.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 72


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

Ethnotechnology, has been only recently used to shape a strategy in computing and
e-learning research as in collaborative e-learning (Guribye & Wasson, 2002; Guribye,
2005) and online Communities of Practice (CoP) (Paiva, 2005; Talamo, 2005;
Talamo & Ligorio, 2002). It is a specific field for studying the impact of technology
raised from the observation of a mismatch among the users’ way of tools’
implementation and the functions for which the developers had planned them
(Grossen et al., 2006). Ethnotechnology has been recently employed for developing,
animating, and analysing online CoP in the ITCOLE EU project (Talamo, 2005;
Talamo & Ligorio, 2002). The project aimed at the creation of software tools,
pedagogical best-practices, and testing, based on cycles of software development.
Talamo and Ligorio combined ethnotechnology and discourse analysis in
synchronous and asynchronous communication within the ITCOLE CSCL system,
built to support collaborative e-learning communities. Following Talamo,
ethnotechnology can be supported by textual analysis and can rely on
ethnotechnologist’s empathic understanding of her context, abandoning her
preconceptions, and treating everything as ‘strange’. Full participation is feasible
when she is fully engaged as when they belong to a CoP, however, her research role
is partly covert.

The research issues are similar to ethnography, such as the role of the researcher
and lack of generalisability since ethnographic approaches are always in principle
incomplete (Hine, 2005).

3.2.1.3. Ethnotechnological Methods

The GSN Deputy Director, Michael Paraskevas, said that the Greek teachers
were not using the tools provided. His observation brings forward one of the primary
aims of ethnotechnology, studying the impact of technology raised from the
observation of a mismatch among the users’ way of tools’ implementation and the
functions for which the developers had planned them. Conducting scenic fieldwork
within the Greek Community of Practice (CoP) appears to have both a real and
virtual locus. Participant observation in activities such as seminars and conferences,
talking to colleagues, and conducting interviews with individuals from the Greek
educational authorities can reveal some aspects of the problem of participation in
collaborative e-learning communities. As for the online environment, focus groups as
well as the quantitative human-computer (logging) and qualitative human-human

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 73


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

interaction analysis (messages analysis) can shed more light in understanding the
Greek e-learners. Thus ethnotechnology can be utilised in e-learning design.

Focus groups, widely used in Human-Computer Interaction (e.g. Faulkner, 2000), are
compatible with ethnography and thus ethnotechnology. Suter (2000) worked on the
dilemma of their use in ethnography and provided three reasons they should be
seriously considered as an alternative way for generating data: (a) access to
participants' interaction on topics that are either difficult to observe or rare in
occurrence; (b) a focus group improves ethnographic practice by providing another
option for generating appropriate data; and (c) the method raises important questions
of relevance between focus groups and participant observation. Provided that the
participants in a focus group discuss the topic of their specialisation and interest, a
focus group can provide insights in hidden aspects of design with data that could not
be acquired otherwise. So in this study, one focus group is used to reveal best
practices and tools for activating the passive participants and a second to evaluate
the tools and evaluation techniques before their implementation.

Ethnotechnological research conducted in e-learning environments can acquire


information about the users by tracing them. A log, also referred to as web-log,
server log or log-file, is usually in the form of a text file and is used to track users’
interactions with the computer system they are using. Examples of what information
can be collected include (Laghos & Zaphiris, 2006): when people visited a site; the
areas they navigated; the length of the visit; frequency of visits; patterns of
navigation; where they are connected from; or details of the computer they are using.
A typical logging operation consists of collecting messages and then performing
analysis, frequently by counting and/or sorting using various criteria (Nonnecke,
2000:21-22). The advantages are attractive (Nonnecke, 2000; Preece et al., 2002;
Laghos & Zaphiris, 2006): log files can be automated; are time stamped; are useful
for finding usage/activity patterns; can study users’ behaviour; provide opportunities
of quantitative analysis and further research. The disadvantages include the need for
power tools for large amounts of data; permission to conduct the research which is a
bureaucratic process in governmental organisations; privacy issues are raised; and
there may be inadequacy in answering questions such as “why” and “how”. In this
study, logging can offer information on the Greek teachers’ passive participation in
Moodle@GSN. Such information can be translated to:

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 74


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

a. the number of passive and active participants;


b. passive participation levels measured by time spent on the system,
c. the depth of discussion threads to measure interactivity; and
d. tool usage.

Logging can also offer opportunities for correlations in order to identify practices that
work better than others.

However, there are some disadvantages. Considering everything as data suggests a


data overload and thus, there should be analytical frameworks in place to facilitate
transitioning from ethnographic inputs to designing concepts (Jones, 2006). Existing
analytical frameworks build on ethnographic inputs to provide designers with a series
of tools and techniques for understanding social settings and organizing their
observations to derive models for design (e.g. Contextual Inquiry, Beyer & Holtzblatt,
1999; DiCoT, Furniss, 2004). Another problem is the danger of amateurisation of
ethnography in practice because fieldwork itself is only the first step; the analytical
treatment of that data determines whether it is fieldwork or scenic fieldwork
(McGarry, 2005:68). Reservation exists on the Greek teachers’ passive participation
in Moodle@GSN; if there is no active participation, there is no narrative, that is zero
history. But e-learners do leave traces that appear in the log files. The key variance
of passive participation may not be the cause and this can be detected.

So ethnotechnology can be incorporated in e-learning engineering building on scenic


fieldwork, implications for design, and ethnographic inputs.

ETHNOTECHNOLOGY

Thematic Analysis Social Network Analysis


Messages Interactions
(qualitative data) (quantitative data)

Questionnaires Logging
Open & closed questions Log files
(qualitative & quantitative data) (quantitative data)

Fieldwork
observation, interviews
(qualitative data)

Figure 3.2.1.3-1 Ethnotechnology and methods

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 75


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

3.2.2. In the Search of Quality: Human-Human


Interaction Analysis

Human-human interaction analysis employs thematic analysis and Social


Network Analysis so as to improve understanding of e-learning interactions quality.

3.2.2.1. Posts: Thematic Analysis

Analysis of the posted messages could provide information about active


participation and the quality of the Greek teachers’ communication and learning.
Theory and prior-research driven thematic analysis for collaborative learning (see
MA dissertation, 2002) was used for code development. This is a process for
encoding qualitative information in order to relate the data to prior ideas (Boyatzis,
1998: 99-127; Braun & Clarke, 2006). Coding needs to be ‘usable’ from other
researches for a high inter-rater reliability level; simple to understand and remember;
relevant to the context; allow quantitative analysis; and use of a wide variety of
information. Codes had been developed in a continuum following a cyclic process of
sampling, developing themes and codes and validating and using the code scheme.
Themes and clusters for developing protocols were grouped based on a
developmental scale on cognitive complexity (Boyatzis, 1998:143). Furthermore,
statistical comparison could determine valid differences by reducing large amounts of
text into numerical data so as to be analysed statistically. Consequently, thematic
analysis has been used in posts analysis searching for Collaborative e-Learning
Episodes (CeLEs) as well as in open questions in the final questionnaire.

In this study, CeLE analytical framework was found to neglect significant information
related to social interactions; these were the social cues such as words at the
beginning and end of a message (e.g. greetings and sign offs) and emoticons. Thus
two more analytical frameworks were considered in order to increase understanding
of collaborative e-learning; Henri’s (1992) five dimensions for message analysis, and
Fahy and colleagues’ Transcript Analysis Tool (TAT) (2001, 2005). Her model
provided an initial framework for coding CMC discussions; however, it lacked
detailed criteria for systematic and robust classification of electronic discourse
(Howell-Richardson & Mellar, 1996). In addition, it has been criticised for poor
theoretical support and being strictly a teacher-centred instructional paradigm
(Gunawardena & Lowe, 1997). Despite these limitations, there is an advantage in

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 76


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

Henri’s proposition, which is the social cues coding for analysing social messages:
self-introduction, expression of feeling, greeting or emoticons for describing emotions
(Henri, 1991:126). After Henri, more researchers have attempted to analyze such
social effects of conferencing exchange (e.g. Rice & Love, 1987; Walther, 1996) or
sociability of online communities (Preece, 2000).

As a result, one CeLE was the unit of coding as the most basic meaningful segment.
The attributes of a CeLE were initially based on conceptual and empirical work;
however, inter-rater reliability had to be reached to ensure correct coding. Therefore,
an analysis of 25% of the data was conducted by an independent researcher in
Atlas-ti™ based on the CeLE codes without any prior discussion. This revealed an
initial reliability of 3% on the data which was clearly unacceptable. The classification
indicators were rewritten in a clearer and simpler way and a further 10% of the data
was coded giving a reliability of 50%. Then, the themes and clusters were discussed
and analysed with the independent researcher giving a reliability of 90% on a further
25% of the data. A second independent researcher verified the process with a
reliability of 93% on a 25% sample. Reliability in the 90%+ range was operationally
considered as being acceptable.

The schema has been also tested by 2 colleagues using messages from different
units of analysis from SKYPE and chats (Lambropoulos et al., 2008). It appeared that
social cues can be related to building social interactions prior to collaborative e-
learning and can be included in coding sub-units within initiation (Table 3.2.2.1-1):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 77


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

Table 3.2.2.1-1. Collaborative e-Learning Episodes Codes Matrix

Collaborative e-Learning Episodes


LEVELS OF ABSTRACTION
Level 3 Level 2 Level 1
CeLE Elements Analytical Corroboration Definitions Indicators for classification
Information, statement, definition,
Initiations, additions or superficial amendments,
0 Initiation & Social cues emoticons, abbreviations, lexical items,
repetitions, uncritical information, social cues, etc.
quoting, images, audio etc
Recommendation, question, bullet points, I
Question, proposition, instruction, opinion, history
1 Question - Information think, I believe, instruction, I know, have
of something, , etc
worked, I prefer.
because, this is why, thus, therefore,
Explanation and self-explanations, requirements,
2 Explanation example, further explanation, help, nice
examples, summaries, etc
behaviour & suggestion.
It is very interesting, refer-to-a-name,
3 Agreement Agreement, confirmation, corroboration, etc.
same, Yes, I agree & you are right.
Disagreement, difference, discrepancy, flaming,
3a Disagreement but, however, on the contrary & different.
etc.
alternative, I have an idea, something else,
Hypothesis, comparison, example, argument,
what about, what do you mean, I tried
4 Exploration resource interdependence, critical information,
if, might, could, would, should, think &
competition of ideas, reasoning, argument, etc.
suggestion.
best, it is important, comparison, easiest,
5 Evaluation Comparison, assessment, best practice, etc.
worst, unfortunately & having no meaning.
New ideas - solution, summary, overall, we agreed &
6 Strategy, plan, method, plan, procedure etc..
Co-construction finally.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 78


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

Enhancing social cues required a specific community management framework. A


focus group provided a scheme for best community development practices and tools
to promote active participation.

Quantifying qualitative data was essential in thematic analysis. Other than counting
the words in a post as an indicator of text richness, two relevant concepts were found
in Fahy and colleagues’ research on their Transcript Analysis Tool (TAT) (2001,
2005), density and intensity. Density was the ratio of the actual number of
connections observed, to the total potential number of possible connections: Density
= 2a/N(N-1), where "a" was the number of observed interactions between
participants, and "N" was the total number of participants. The researchers stressed
that the measure of density is sensitive to the size of the network, so larger groups
will be likely to exhibit lower density ratios than will smaller groups.

Intensity in this study referred to the levels of participation and persistence where
persistence is the level to which participants pursue topics. Persistence was related
to interactivity and discussion depth operationalised by measuring the number of
levels of communication in a particular discussion thread from the first posting to the
last (depth of discussion threads).

Other than the posts’ quantitative and qualitative examination, investigations of social
interactions can shed more light in the research context and can triangulate the
findings with ethnotechnological fieldwork.

3.2.2.2. Interactions: Social Network Analysis

Social Network Analysis (SNA) was used to understand e-learning related to


the Greek teachers’ sense of belonging to the collaborative e-learning community.
SNA has been used to visualize communication and relationships between people
and/or groups through diagrams by depicting social relationships between a set of
actors (Baroudi, et al, 1986). The most widely used SNA attributes are nodes (the
actors of study), relations (the strands between actors), and centrality (central or
isolated person). SNA focuses on complete (or group) and ego networks; however,
only group analysis on cohesion and centrality was found suitable for this study. In
addition, several tools were considered for SNA as well as their integration in
discussion forums as to support co-presence.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 79


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

Cohesion: Network density for group thickness, reciprocity, cliques, and structural
equivalence were used to measure the level of cohesion. Network density is the
proportion of possible links in network that actually exist; it was evaluated by the
adjacency connection reports. Sent-Received (S-R) number of messages is related
to participants’ reciprocity. More specifically, reciprocity is the number of ties that are
involved in reciprocal relations relative to the total number of actual ties (Hanneman
& Riddle, 2005). A clique is a set of actors with each being connected to each other
in smaller groups. Structural equivalence and in particular the CONCOR technique
(CONvergence of iterated CORrelations, White, Boorman, Breiger, 1976; Breiger et
al., 1975), describes the actors that have similar relations to others in the network
with dendrogrammes. So the degree to which two nodes are structurally equivalent
can be evaluated by measuring the degree to which their columns are identical:

Centrality: Group centrality (Everett, 2005; Freeman, 1979) refers to the distribution
of power between the community members and is measured by centrality, closeness
and betweenness. In this study it referred to the total number of Sent-Received
Messages (direct links), out-degree (replies made) and in-degree (received
messages) centrality. Group closeness is defined by the normalised inverse sum of
distances from the group to a node outside the group (Everett, 2005:61); in this study
closeness was related to reciprocal distances. Betweeness is the number of indirect
links in which the actor is required as an intermediary; this characterise the mediator
as the controller of the information flow in a network.

SNA Tools: SNA has been used for offline (e.g. Breiger, 2004; Bender-deMoll &
McFarland, n.d.), off-line and online (e.g. Wellman, 2001), and online educational
contexts (e.g. Daniel, 2007; Laghos & Zaphiris, in press). Presenting relationships
and perceiving solutions derived from visualisation can assist annotations,
consultancy or revision since data visualisation proved to be important for user locus
of awareness, control and initiative (Shneiderman, 2000). However, even though
SNA tools can depict the activity in online human-human interactions, the use of
these tools in educational contexts is offline. Offline SNA tools need the researcher to
input the data either based on observation (e.g. Petropoulou, 2006; Laghos &
Zaphiris, in press) or extract the data in a file such as excel files (Jeong, 2005).
These data are inserted to SNA software for visualisations graphs, however,
providing an out-of-date evaluation and decision making.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 80


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

SNA tools could support e-learning discussions in real time. Kreijns and colleagues
(2002) have suggested that if a group awareness widget existed, a software tool
providing the learner group awareness about the others in the task and in the non-
task context can enhance groups’ sociability. Additionally, social presence and co-
presence can enhance the sense of community by providing a picture of the
community (Beer et al., 2005). Social presence was the degree by which a person
was perceived as real in an online conversation (Meyer, 2002:59). Real-time SNA
has started to support e-research; for example, Microsoft uses SNA tools to visualise
users’ clicks on web pages (Milic-Frayling, 2007). To date, some open source offline
SNA software used in discussion research are SoNIA (Stanford University; Bender-
deMoll & McFarland, 2006), Pajek (University of Ljubljana, Slovenia; Nooy et al.,
2005), UCINET (Analytictech, http://www.analytictech.com/download_products.htm)
and JUNG (Java Universal Network/Graph Framework, http://jung.sourceforge.net/).
JUNG was found to be compatible to Moodle and was integrated in Moodle for the
purposes of this study. Other than the SNA online use, an SNA software is essential
for data analysis. After studying comparisons between several SNA desktop
applications (Laghos, 2007; Huisman & van Duijin, 2005) UCINET (Borgatti et al.,
2002) was found most suitable for this study. The main reasons were its usability and
the support provided (Everett and Borgatti; personal communication via email,
September 2007).

Overall, fieldwork and thematic analysis provide the data for qualitative analysis that
can be quantified for statistical analysis; logging and Social Network Analysis provide
data for quantitative analysis and social networks visualisation. Open and closed
questions in a questionnaire could be used for triangulation so as to reveal whether
two or more sets of findings could converge in a single proposition.

3.2.3. Questionnaire Design

A questionnaire is a self-reporting technique whereby participants fill in the


answers to questions themselves; its purpose is to elicit facts about the respondents,
their behaviour and their beliefs/attitudes (Nielsen, 1993). According to Nielsen, there
are three types of questions: open-ended, closed, and scaled. Open-ended questions
give freedom to the participants to respond, closed, where the participants have to
choose from several choices, and scales where the respondents must answer on a
pre-determined scale. They have been used for: online communities (Andrews et al.,
2003); evaluation of the sense of community (e.g. Brook & Oliver, 2006; Daniel,

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 81


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

2007); revealing lurkers’ opinions (e.g. Nonnecke, 2000; Gulati, 2006); or evaluation
of evaluation tool’s effectiveness (e.g. Silius et al, 2003).

Advantages derived from methodology were: easy access to participants, especially


via emails or web-based questionnaires; use of qualitative or quantitative questions
or a mix; there can show differences over time. The disadvantages were related to:
an idealised version of information; lack of honest and accurate responses; very low
response rates from passive participants and less than 20% response from active
participants; and difficulty in having an immediate follow up (Mason, 1999). However,
these disadvantages can be eliminated if used in conjunction with other
methodologies. The questionnaires stages and objectives are (Zaharias, 2004;
Kirakowski and Corbett, 1990) (Figure 3.2.3-1):

Figure 3.2.3-1. Questionnaire Design Methodology

Three questionnaires were given to the participants, at the beginning, in the middle
and in the end of study. The first questionnaire was used to acquire information on
demographics, conditions of working and learning on the internet and initial

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 82


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

knowledge on collaboration. The second questionnaire was used for e-tutoring


purposes and it is not part of this study. The third questionnaire targeted to
triangulation via the evaluation of: community evolution, the sense of belonging,
empathy, trust, and knowledge acquisition. All questionnaires were given to 3-5
respondents in three phases in both English and Greek languages prior to the study.

Lastly, the final section presents the e-research design needed for the e-research
coordination.

3.2.4. E-Research Coordination: Time-Short Series


Design

Time-based coordination was used to capture the development and to


triangulate sides of space and time of the unit of analysis. The use of quasi
experimental time-short-series design was found a suitable approach to set a
timeframe (Shadish et al., 2002). Quasi-experimental design is not preferable to
other research design such as experimental design, but it is used when an
experiment is not feasible, most of the time due to the group selection criteria: the
groups were chosen and assigned out of convenience rather than randomization.
Most importantly, controlled environments are impossible where participants have
access from remote and distributed locations. Also, there is little loss of status or
prestige in doing a quasi-experiment instead of a true experiment.

Time settings refer to two main sets, defining the baseline(s), and time series.
Baseline refers to the observation of behaviour prior to any treatment designed to
alter behaviour. As such, the treatment effect is demonstrated by a discontinuity in
the pattern of pre-treatment and post-treatment responses. The groups which are
going to be used in this study are inactive. The latter suggests a solid baseline for
treatments and effects related to causal inference, not affected by threats like history,
natural development and maturity for studies mostly observed in children’s research.
In time-short-series design aggregation and causal inference are not necessarily
affected if a detailed amount of data could be collected. There are three dimensions
to be investigated in order to examine the nature of intervention: (a) the form of the
effect (the level, slop, variance and cyclicity); (b) its permanence (continuous or
discontinuous) and (c) its immediacy (immediate or delayed).

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 83


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

Two baselines are initially suggested, before treatments, community management


and use of new tools, and after treatments. The time series design is (Table 3.2.4-1):

Table 3.2.4-1. Observations, interventions and evaluation

Observation Course Day INTERVENTION & EVALUATION


Community Management Tools
0 O0 Baseline 1 - days √ n/a
1 O1 1 √ n/a
2 O2 3 √ n/a
3 O3 7 √ n/a
4 O4 14 √ n/a
5 O5 21 √ n/a
6 O6 Baseline 2 √ √
7 O7 28 √ √
8 O8 End of the course √ √

The initial baseline describes the research context before any intervention. Then
community management and tools (treatments) support the e-learning community.
Limitations and strengths of time-short-series design in this study are (Sanson-
Fisher, 2004) (Table 3.2.4-2):

Table 3.2.4-2. Limitations and strengths in Time-Short Series

Time-short-series
Limitations Strengths
• Process-based framework creates close examination
• Fewer study units of both units and interventions as well as causal
limits generalisability inference
• Measures must be • All units could get intervention if it is effective
suitable for repeated • New theories can be created
use
• Flexibility related to individuals, small and large units
• Depends upon • More intervention research and knowledge
successful, temporal • Can examine each intervention component
relationship between
• Interdisciplinary Research exists
intervention and
• Clear research design and data analysis
measure
• Consisted with decision making processes

Time-short-series strengths add value to research, and limitations are considered as


similar to any other research; thus, provision of measures can ensure the validity and
reliability. Overall, time series design seems to have the potential to provide the
needed time-based coordination of measurements to provide implications for design
and e-learning evaluation.

The methodological approaches and their attributes are (Table 3.2.4-3):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 84


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

Table 3.2.4-3. Research Methodology

ETHNOTECHNOLOGY DESCRIPTION

1. Participation in projects, conferences, seminars,


online discussions of Greek teachers’ associations
and schools in Greece
1 Fieldwork 2. Documentation, manuals and presentations by the
Greek educational authorities
3. Personal opinions, conversations, and emails
4. Richness of text
1. Demographics
2 Pre-Questionnaire 2. Conditions of working and learning over the Internet
3. Initial knowledge on collaboration
1. Frequency of visits
2. Number of active and passive participants
3 Logging
3. Active participation levels
4. Passive participation levels
1. Collaborative e-learning episodes analysis
4 Thematic Analysis 2. Sense of e-Learning Community Index
• Number of Collaborative e-learning episodes
Social Network 2. Group Centrality
5 Analysis • Centrality
1. Sense of e-Learning Community Index
• Community evolution
• Sense of belonging to the e-learning community
• Empathy
6 Post-Questionnaire • Trust: knowledge exchange, help and support
• Collaborative e-learning quality: participants’
opinions
2. Usage and usability of collaborative tools
3. Professional development
1. Examination of the data, missing data, normality.
2. Descriptive Statistics: crosstabulation, frequencies,
arithmetic means, standard deviations, and
7 Statistical Analysis exploration
3. Correlation: Pearson correlation coefficient (r).
4. Inferential statistics: statistical significance, p-value
(Cronbach's alpha, α), and null hypothesis.
1. Form of the effect: the level, slop, variance and
cyclicity
8 Intervention Analysis
2. Its permanence (continuous or discontinuous)
3. Its immediacy (immediate or delayed)

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 85


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

3.2.5. Research Constraints

There were several constraints for the study.

First, a number of constraints occurred because the research was conducted


completely over the Internet (e-research): data overload; time and knowledge
required to access, filter, sort, and combine the quantity of information produced, and
endless opportunities.

Secondly, additional constraints reflect the nature and circumstances under which
the research is carried out including the time frame, having one individual performing
all research functions and simultaneously being an active participant in the Greek
educational community, use of open source tools, not being an expert ethnographer,
and having limited programming skills.

Thirdly, there is also a need to acknowledge the Greek teachers’ passive


participation; the ratio on questionnaire response is anticipated to be less than 20%.
This is not only a phenomenon observed in the Greek teachers’ reality; Cuthell
(2005) reported limited activity in a 5 months project on MirandaNet, a British
teachers’ network similar to GSN. Another limitation is related to the target
population, which is not representative of the total population of the Greek teachers
but naturally occurred in the Computer-Mediated-Communication space.

Lastly, even though there are recent studies on participation in online communities,
there is no similar research for comparison in the Greek and international research
community.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 86


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

REFERENCES
Anderson, T., & Kanuka, H. (2003). E-research: Methods, strategies, and issues.
Boston, MA: Allyn & Bacon.
Andrews, D., Nonnecke, B., & Preece, J. (2003). Electronic survey methodology: A
case study in reaching hard to involve Internet Users. International Journal of
Human-Computer Interaction, 16(2), 185-210.
Baroudi, J. J., Olson, M. H., & Ives, B. (1986). An Empirical Study of the Impact of
User Involvement on System Usage and Information Satisfaction.
Communications of the ACM, 29(3), 232-238.
Beer, M. D., Slack, F., & Armitt, G. (2005). Collaboration and Teamwork: Immersion
and Presence in an Online Learning Environment. Information Systems
Frontiers, 7(1), 27-37.
Bender-deMoll, S., & McFarland, D. A. (2006). The Art and Science of Dynamic
Network Visualization. Journal of Social Structure, 7(2). Retrieved
15/12/2006, http://www.stanford.edu/group/sonia/papers/DNV_JOSS.pdf.
Bender-deMoll, S., & McFarland, D. (n.d.). Interaction, Time, and Motion: Animating
Social Networks with SoNIA. Report, Stanford University.
Beyer, H., & Holtzblatt, K. (1999). Contextual design. Interactions, 6(1), 32-42.
Retrieved 12/12/2005, from
http://www.csc.ncsu.edu/faculty/anton/csc/rep/beyer.pdf.
Boehner, K., Vertesi, J., Sengers, P., & Dourish, P. (2007). How HCI Interprets the
Probes. Paper presented at the CHI’07 - SIGCHI Conference on Human
factors in computing systems. 28 April - 3 May, 2007, San Jose, CA, 1077-
1086. Retrieved 10/07/2007, from
http://cemcom.infosci.cornell.edu/uploads/pubs/Boehner_Vertesi_Sengers_D
ourish_Probes.pdf.
Borgatti, S. P., Everett, M. G., & Freeman, L. C. (2002). Ucinet for Windows:
Software for Social Network Analysis. Harvard, MA: Analytic Technologies.
Boyatzis, R. E. (1998). Transforming Qualitative Information: Thematic Analysis and
Code Development. Thousand Oaks, CA: Sage Publications.
Braun, V., & Clarke, V. (2006). Using Thematic Analysis in Psychology. Qualitative
Research in Psychology, 3, 77-101.
Breiger, R. L., Boorman, S. A., & Arabie, P. (1975). An algorithm for clusteirng
relational data with applications to social network analysis and
comparison with multidimensional scaling. Journal of Mathematical
Psychology, 12, 328-382.
Breiger, R. L. (2004). The Analysis of Social Networks. In M. Hardy & A. Bryman
(Eds.), Handbook of Data Analysis (pp. 505-526). London: Sage Publications.
Brook, C., & Oliver, R. (2006). Exploring the influence of instructor actions on
community development in online settings. In N. Lambropoulos & P. Zaphiris
(Eds.), User-Centered Design of Online Learning Communities (pp. 341-364).
Hershey: Idea Group.
Button, G. (2000). The ethnographic tradition and design, Design Studies, 21, 319-
332.
Cerulo, K. A. (1997). Reframing Social Concepts for a Brave New (Virtual) World.
Sociological Inquiry, 67(1), 48-58.
Cuthell, J. P. (2005). What does it take to be active? Teacher participation in online
communities. International Journal of Web Based Communities (IJWBC),
1(3).
Daniel, B. K. M. (2007). A Bayesian Belief Network Computational Model of Social
Capital in Virtual Communities. Unpublished Research, PhD Thesis,
University of Saskatchewan, Canada. Retrieved 01/08/2007, from

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 87


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

http://library2.usask.ca/theses/available/etd-07132007-
141903/unrestricted/ben_m.pdf.
Dourish, P. (2006). Implications for Design. Paper presented at the CHI'06 - ACM
Conference on Human Factors in Computing Systems: Interact. Inform.
Inspire. 22-27 April 2006, Montreal, Canada, 541-550. Retrieved 26/01/2007,
from http://www.ics.uci.edu/~jpd/publications/2006/implications-chi2006.pdf.
Dourish, P., & Button, G. (1998). On "Technomethodology": Foundational
Relationships between Ethnomethodology and System Design. Human-
Computer Interaction, 13(4), 395-432.
Everett, M. (2005). Extending Centrality. In P. J. Carrington, J. Scott & S.
Wasserman (Eds.), Models and Methods in Social Network Analysis (pp. 57-
76). New York, NY: Cambridge University Press.
Fahy, P., & Ally, M. (2005). Student Learning Style and Asynchronous Computer-
Mediated Conferencing (CMC) Interaction. American Journal of Distance
Education, 19(1), 5-22. Retrieved 12/12/2005,
http://www.leaonline.com/doi/abs/10.1207/s15389286ajde1901_2.
Fahy, P. J., Crawford, G., & Ally, M. (2001). Patterns of Interaction in a Computer
Conference Transcript. International Review of Research in Open and
Distance Learning, 2(1). Retrieved 12/12/2005, from
http://www.irrodl.org/content/v2.1/fahy.html.
Faulkner, X. (2000). Usability Engineering. New York, NY: Palgrave, MacMillan.
Freeman, L. C. (1979). Centrality in Social Networks: Conceptual Clarification. Social
Networks, 1, 215-239.
Furniss, D. (2004). Codifying Distributed Cognition: A Case Study of Emergency
Medical Dispatch. MSc Dissertation at UCL Interaction Centre, London.
Retrieved 10/05/2006, from
http://www.uclic.ucl.ac.uk/people/d.furniss/Documents/DomProjectFinal.pdf.
Grossen, M., Pochin, L. O., Resnick, L. B., Säljö, R., Pontecorvo, C., & Burge, B.
(2006). Interactional perspectives on the use of the computer and on the
technological development of a new tool : The case of word processing.
NATO ASI series. Series F : computer and system sciences (NATO ASI ser.,
Ser. F : comput. syst. sci.) ISSN 0258-1248 CODEN NASFEG North
Atlantic Treaty Organization Advanced Science Institutes series.
Gulati, S. (2006). Learning during online and blended courses: Unpublished
Research, PhD Thesis at the Department of Education and Lifelong Learning,
City University, London.
Gunawardena, C., Lowe, C., & Anderson, T. (1997). Analysis of a Global Online
Debate and the Development of an Interaction Analysis Model for Examining
Social Construction of Knowledge in Computer Conferencing. Journal of
Educational Computing Research, 17(4), 397-431.
Guribye, F. (2005). Infrastructures for learning: Ethnographic inquiries into the social
and technical conditions of education and training. Unpublished Research,
PhD Thesis at the Department of Information Science and Media Studies,
University of Bergen, Norway.
Guribye, F., & Wasson, B. (2002). The Ethnography of Distributed Collaborative
Learning. Paper presented at the CSCL'02 - Computer-Supported
Collaborative Learning. 7-11 January, Boulder, CO. retrieved 09/05/2006,
http://newmedia.colorado.edu/cscl/148.pdf.
Hanneman, R. A., & Riddle, M. (2005). Introduction to social network
methods. Riverside, CA: University of California, Riverside.
Henri, F. (1992). Computer Conferencing and Content Analysis. In A. R. Kaye (Ed.),
Collaborative learning through computer conferencing: The Najaden Papers
(pp. 117-136). Berlin: Springer-Verlag.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 88


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

Hine, C. (2005). Virtual methods and the sociology of Cyber-Social-Scientific


knowledge. In C. Hine (Ed.), Virtual Methods: Issues in Social Research on
the Internet (pp. 1-13). Oxford: Berg.
Howell-Richardson, C., & Mellar, H. (1996). A methodology for the analysis of
patterns of participation within computer mediated communication courses.
Instructional Science, 24, 47-69.
Hughes, J. A. (1999). The general acceptance of ethnography in system design.
Paper presented at the Conference Ethnography in systems design, An
Industrial Approach to Work Analysis and Software Design. 29 April, 1999,
The Conference Centre, Lancaster University, UK.
Hughes, J. A., O'Brien, J., Rodden, T., Rouncefield, M., & Blythin, S. (1997).
Designing with ethnography: a presentation framework for design. Paper
presented at the Symposium on Designing Interactive Systems, In the
Proceedings of the conference on Designing interactive systems: processes,
practices, methods, and techniques. August 18-20 1997, Amsterdam, The
Netherlands. New York, NY: ACM Press 1997, pp. 147 – 158.
Huisman, M., & Van Duijn, M. A. J. (2005). Software for social network analysis. In P.
J. Carrington, J. Scott & S. Wasserman (Eds.), Models and Methods in Social
Network Analysis (pp. 270-316). New York, NY: Cambridge University Press.
Jeong, A. (2005). Methods and Tools for the Computational Analysis of Group
Interaction and Argumentation in Asynchronous Online Discussions. Paper
presented at the Technology and Learning Symposium. 5-6 April 2006, New
York, NY. Retrieved 10/12/2006, from http://dev22448-
01.sp01.fsu.edu/Research/Proposals/LearnTechSymposium/MethodsToolsAn
alyzingInteraction_Jeong2005.pdf.
Jones, R. (2006). Experience Models: Where Ethnography and Design Meet.
Ethnographic Praxis in Industry Conference Proceedings, 1, 82-93. Retrieved
2/03/2007, from
http://www.anthrosource.net/doi/pdfplus/10.1525/epic.2006.2006.1.82.
Kirakowski, J. and Corbett, M. (1990). Effective Methodology for the Study of HCI.
Elsevier. Retrieved 5/10/2005, from http://sumi.ucc.ie/sumipapp.html.
Kreijns, K., Kirschner, P. A., & Jochems, W. (2002). The Sociability of Computer-
Supported Collaborative Learning Environments. Educational Technology &
Society, 5(1).
Laghos, A. (2007). Assessing the Evolution of Social Networks in e-Learning.
Unpublished doctoral dissertation at Center for Human-Computer Interaction
Design, City University, London.
Laghos, A., & Zaphiris, P. (In press). Social Network Analysis of Self-Taught e-
Learning Communities. International Journal of Knowledge and Learning.
Laghos, A., & Zaphiris, P. (2007). Investigating Student Roles in Online Student-
Centered Learning. Paper presented at the ICEM'07 - International Council
for Educational Media Conference. 20-22 September 2007, Intercollege,
Nicosia, Cyprus.
Laghos, A., & Zaphiris, P. (2006). Sociology of Student-Centred e-Learning
Communities: A Network Analysis. Paper presented at the e-Society 2006
Conference. 13–16 July 2006, Dublin, Ireland. Retrieved 22/01/2007, from
http://www.soi.city.ac.uk/~zaphiri/Papers/eSociety.pdf.
Lambropoulos, N., Kampylis, P., Papadimitriou, S., Vivitsou M., Gkikas, A.
Minaoglou, N. & Konetas, D. (2008). Hybrid Synergy for Virtual Knowledge
Working. In Janet Salmons & Lynn Wilson (Eds) (2008), Handbook of
Research on Electronic Collaboration and Organizational Synergy, Hershey,
PA, USA: IGI Global Publications.
Lambropoulos, N. (2002). Collaborative and Individualistic Learning in a Situated,
Computer - Supported Co-operative Environment for Multimedia
Constructions. MA dissertation at the Institute of Education, University of
London.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 89


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

Mason, R. D. (1999). IET’s Masters in Open and Distance Education: What have we
learned? CITE Report No 248. Institute of Educational Technology (IET), The
Open University, Milton Keynes, UK: The Open University. Retrieved
15/12/2005, from http://iet.open.ac.uk/pp/r.d.mason/downloads/maeval.pdf.
Massey, A. (1999). Methodological Triangulation, Or How To Get Lost Without Being
Found Out. Studies in Educational Ethnography, 2, Explorations in
Methodology, Stamford, JAI Press, 183-197. Retrieved 01/06/2006, from
http://www.voicewisdom.co.uk/htm/triangulation1.htm.
Meyer, K. A. (2002). Quality in distance education: Focus on on-line learning. San
Francisco, CA: Jossey-Bass.
McGarry, B. (2005). Things to Think With: Understanding Interactions with Artefacts
in Engineering Design. Unpublished Research, PhD thesis at the Department
of ITEE, School of Information Technology and Electrical Engineering,
University of Queensland. Retrieved 14/05/2006, from
http://eprint.uq.edu.au/archive/00003422/01/McGarry%20PhD%20Thesis%20
-%20Things%20to%20Think%20With.pdf.
Milic-Frayling, N. (2007, 1 May 2007). Internet - Where Information and
Communication Technologies Meet. Paper presented at the Women in
Computing Research: the Hopper Colloquium. 1 May 2007 London Hopper
2007, British Computer Society, London.
Moodle@GSN (n.d.). Moodle at the Greek School Network, http://e-learning.sch.gr.
Nielsen, J. (1993). Usability Engineering. Boston, USA: Academic Press.
Nonnecke, B. (2000). Lurking in email-based discussion lists. Unpublished Research,
PhD Thesis at London South Bank University, London.
Nooy, W. d., Mrvar, A., & Batagelj, V. (2005). Exploratory social network analysis
with Pajek. Cambridge: Cambridge University Press.
Paiva, V. L. M. d. O. e. (2005). A pesquisa sobre interação e aprendizagem de
línguas mediadas pelo computador (A research on computer-mediated
interactions in language learning). Calidoscópio, São Leopoldo, 3(1), 5-12.
(Translated by Margarida Romero.) Retrieved 22/05/2006, from
http://www.veramenezes.com/cmc.htm.
Petropoulou, O., Lazakidou, G., & Retalis, S. (2006). Πώς να Αξιολογήσουμε τη
Διαδραστικότητα σ’ ένα Τεχνολογικά Υποστηριζόμενο Περιβάλλον Μάθησης;
How can we evaluate interactivity in a technology enhanced learning
environemnt? Paper presented at the 3ο Πανελλήνιο Συνέδριο Ε.Ε.Ε.Π. -
Δ.Τ.Π.Ε. "Εκπαίδευση & Νέες Τεχνολογίες". 3rd Panhellenic Conference
EEEP-DTPE "Education and the New Technologies", 23-46. 30 September -
1 October 2006, Korydallos, Athens, Greece.
Preece, J., Rogers, Y., & Sharp, H. (2002). Interaction Design: Beyond Human-
Computer Interaction. New York, NY: John Wiley & Sons, Inc.
Preece, J. (2000). Online Communities: Designing Usability, Supporting Sociability.
Chichester, UK: John Wiley & Sons.
Rice, R. E., & Love, G. (1987). Electronic emotion: Socio-emotional content in a
computer-mediated communication network. Communication Research,
14(1), 85-108.
Robinson, H., Segal, J., & Sharp, H. (2007). Ethnographically-informed Empirical
Studies of Software Practice. Information and Software Technology. In press.
Sanson-Fisher, R. (2004). Multiple Baseline Designs. Paper presented at the
Workshop on Research Designs for Complex, Multi-level Health Interventions
and Programs. 4 -5 May, 2004, Bethesda, MD. Retrieved 11/01/05, from
http://obssr.od.nih.gov/Conf_Wkshp/Complex%20Interventions%20Workshop
/Sanson-Fisher.ppt.
Sasse, A. (1997). Eliciting and Describing Users’ Models of Computer Systems.
Unpublished Research, PhD Thesis, University College London, London, UK.
Retrieved 15/02/2006, from
http://www.cs.ucl.ac.uk/staff/a.sasse/thesis/Frontpage.html.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 90


Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 3: Research Design

Shadish, W., Cook, T., & Campbell, D. (2002). Experimental. & Quasi-Experimental
Designs for Generalized Causal Inference. Boston, MA: Houghton Mifflin.
Sharp, H., Robinson, H., Segal, J., & Furniss, D. (2006). The Role of Story Cards and
the Wall in XP teams: a distributed cognition perspective. Paper presented at
the Agile 2006. 23-28 July 2006, Minneapolis, Minnesota, USA. Retrieved
12/05/2006, from
http://www.dcs.qmul.ac.uk/research/imc/hum/pubs/SharpEtAl-
CardsAndWallFinalVersion1%20PREPRINT.pdf.
Sharp, H., & Robinson, H. (2005). Some social factors of software engineering: the
maverick, community and technical practices. Paper presented at the
HSSE'05 - 2005 Workshop on Human and Social factors of Software
Engineering. 16 May 2005, St. Louis, Missouri. Retrieved 12/05/2006, from
http://delivery.acm.org/10.1145/1090000/1083117/p11-
sharp.pdf?key1=1083117&key2=8932278711&coll=&dl=&CFID=15151515&C
FTOKEN=6184618.
Shneiderman, B. (2000). Creating Creativity: User Interfaces for Supporting
Innovation. ACM Transactions on Computer-Human Interaction, 7(1), 114–
138.
Silius, K., & Tervakari, A.-M. (2003, 8-9 May 2003). An Evaluation of the Usefulness
of Web-based Learning Environments.The Evaluation Tool into the Portal of
Finnish Virtual University. Paper presented at the International Conference on
Network Universities and E-Learning, Valencia, Spain. Retrieved 23/04/2006,
from http://matriisi.ee.tut.fi/arvo/liitteet/usefulness_of_web.pdf.
Suchman, L. A. (1987). Plans and Situated Action: The Problem of Human-Machine
Communication New York: Cambridge University Press.
Suter, E. A. (2000). Focus Groups in Ethnography of Communication: Expanding
Topics of Inquiry Beyond Participant Observation The Qualitative Report, 5(1
& 2).
Talamo, A. (2005). Developing, Animating and Analysing Virtual Communities of
Practices. Paper presented at the Workshop on Learning Communities in the
era of Ubiquitous Computing. 13 June 2005, Milan, Italy. Retrieved
21/05/2006, from
http://www.idi.ntnu.no/~divitini/ubilearn2005/Final/talamo_ubilearn.pdf.
Talamo, A., & Ligorio, M. B. (2002). Ethnomethods as resources for developing
CVEs in the ITCOLE project. Paper presented at the CSCL'02 Conference. 7-
11 January, Boulder, Colorado, USA. Retrieved 21/12/2005, from
http://newmedia.colorado.edu/cscl/167.htm.
Tedlock, B. (2000). Ethnography and ethnographic representation. In N. K. Denzin &
Y. S. Lincoln (Eds.), Handbook of qualitative research (2nd ed) (pp. 455-486).
Thousand Oaks, NJ: Sage.
Walther, J. B. (1996). Computer-mediated communication: Impersonal, Interpersonal
and Hyperpersonal Interaction. Communication Research, 23, 3-43.
Wellman, B. (2001). Physical Place and Cyber-Place: Changing Portals and the Rise
of Networked Individualism. International Journal for Urban and Regional
Research, 25(2), 227-252.
White, H. C., Boorman, S. A., & Breiger, R. L. (1976). Social structure from
multiple networks: I. Blockmodels of roles and positions. American
Journal of Sociology, 87, 517-547.
Zaharias, P. (2004). A Usability Evaluation Method for E-Learning Courses.
Unpublished Doctoral Dissertation at Athens University of Economics and
Business.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 91


Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

The Research Context:

4 Ethnotechnological inputs and preliminary studies

Key Topics Covered in This Chapter:

• Understanding the research context

• The Greek teachers’ background and characteristics

• The intentions of the Greek educational authorities

• The Greek School Network and the e-learning environment provided


to the Greek teachers

Chapter 4 reveals aspects of the research context with the aid of ethnotechnology.
This is connected to the Greek educational authorities’ intentions, targets, and tools
provided for the Greek teachers’ professional training and development. One of the
tools was Moodle@GSN, the e-learning platform supported by the Greek School
Network. The chapter also sheds light to the Greek teachers’ needs, background,
and characteristics as well as their overall activity at Moodle@GSN.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 92


Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

4.1 INTRODUCTION

The main objectives of this chapter are:

• understanding the context from the Greek teachers’ viewpoint;


• finding best practices for communication and social interaction; and
• implications for design.

Because some parts of the jigsaw were not found in Greece, observant participation
in other groups was considered necessary. Thus the following projects were also
used to provide information:

1. the European projects “E-Tutor” (June 2006; http://www.etutorportal.net/) and


“e-learning fundamentals” (June 2007- http://fecone.passionforlearning.eu/);
2. the Greek English language teachers (PEKADE; http://pekade.gr);
3. the Greek association for the valorisation of ICT in education (EEEP;
http://www.eeep.gr);
4. the Greek teachers with a special interest in music (EEMAPE;
http://www.primarymusic.gr/);
5. the primary teachers’ training programme conducted by the Greek
Pedagogical Institute within the Greek schools (September 2006).; and
6. participation in several conferences with publications in collaboration with
other Greek teachers.

Access was provided by the researcher’s colleagues and there were discussions with
the organisations’ directors as well as members.

4.1.1. The Native’s Point of View, background and


characteristics

The participants in e-learning at Moodle@GSN were the directors, the


engineers as technical support, the e-tutors, the e-learners, and the Greek
authorities. The directors and the engineers were not Greek teachers. The e-tutors
had a twofold role, the moderator and the e-tutor. There were no Greek teachers’
online community managers at the time of the research and I was a member of the
Online Community Managers Association, called E-mint (http://emint.org/). E-mint is

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 93


Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

widely known for members’ genuine interest and creative inputs in the field of online
communities. Among other results, thematic analysis revealed a template for writing
online messages, and a community management framework (Appendix I).

I also participated in the Moodle-based Leonardo DaVinci project ‘e-Tutor’


(http://pegasus.clab.edc.uoc.gr/ite/) and the Greek English language teachers’ forum
(www.pekade.gr/forum/). The aims were to identify:

• the Greek teachers’ ability of working online;


• soft skills in online environments;
• the ability to incorporate learning derived from discussion forums for
educational tasks and working conditions; and
• the potential for professional development via e-learning.

The results revealed that the Greek teachers had difficulties in integrating new
practices learned from online seminars in their teaching and learning practice. The
main reasons were the lack of training, lack of opportunities, lack of soft skills, and
absence of professional guidance, help and support.

Lastly, I acquired the opinions of the ‘Greek Primary Teachers’ Association for the
Valorization of ICT in Education’ (EEEP, http://www.eeep.gr; formed in December
2003). Fourteen out of 61 participants (22,95%) responded to a questionnaire in July
and August 2004. The factors that appeared to influence members’ participation,
were: (a) organisational, related to educational authorities; (b) school-based; (c)
personal: age, gender, training or absence of training on the use of computers, years
of teaching, previous experience and familiarization with ICT, writing and typing skills,
and personal characteristics; and (d) “the real world”: for example, expensive rates
for the internet access in Greece, the Internet connection from the schools,
participants’ spare time and time available for training. (The results of these studies
have been published in Lambropoulos 2005a, 2005b, 2005c.)

The overall findings suggested changes on a social cognitive/learning, and technical


level. The Greek teachers were not familiar with the new technologies: soft skills for
online collaboration were almost absent, and communication was fair. Professional
help and support was not available as regards e-learning, and lastly, Moodle
modules were not facilitating the communication gaps. One interesting observation
was the importance of timing regarding 4 spots in a community life-span:

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 94


Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

1. Registration – enrolment and initial contact with the community,


2. One to two days after the registration,
3. One week after the registration – towards full participation, and
4. Maintenance of the community

These stages appear to be important; for example, initiations and interventions need
to be applied after the first week in order to allow time for the e-learners to familiarise
with the system and the community. The next section investigates the context of
educational authorities and GSN.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 95


Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

4.2 IDENTIFYING INTENTIONS


Intentions determine the stakeholders’ goals and acknowledge the real need
and leads towards adaptation of the right attitude to accept and incorporate change.
In the case of the Greek School Network (GSN), passive participation may be the
real need or a circumstance serving as visible evidence of a different need. GSN
initially attempted to provide an online environment for Greek teachers’ training and
professional development but the legislation and pedagogical frameworks were not
there to support teachers’ involvement. The Greek Ministry of Education and
Religious Affairs introduced GSN as the educational intranet to provide certified
telematic services for all schools and teachers in Greece. Moodle was used for the e-
learning service to host and distribute digitized e-lessons, for teachers with different
specialisations working in primary, secondary state schools, and adult centres
(Figure 4.2-1):

Figure 4.2-1. Organisation of the Education System in Greece 2003/04


(Euridice, 2003:6)

The Moodle@GSN teachers are located within the black arrow area; they teach
different subjects and have different roles in the Greek education system. GSN asked
e-tutors and teachers to participate on a voluntary basis.

E-Learning at GSN aimed to support Greek teachers’ life-long learning; however,


there were problems. For example, they could not acquire a valid certificate because
associated legislation was absent. More particularly, legislation (Law#3328,
01/04/2005) has prohibited online certification and accreditation provided by
educational organisations, unless this was part of their constitution so e-learning was
not considered as part of the national policy. Evidence refers to the first distance
education students registered at the British Open University in 1970

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 96


Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

http://www.open.ac.uk/about/ou/p3.shtml. The first students registered at the Greek


Open University in 1998 (http://www.eap.gr/general/history.htm). Even today, there is
difficulty apply for a job in a governmental organisation if the degree is acquired via
distance learning (personal communication via email with Efthimios Kalaitzidis, MSc
graduate with special needs from Manchester University, UK; 08/12/2006). Unclear
criteria to provide rewards have been reported as a blocking factor for e-learning
quality (E-Quality project, 2006).

The Greek teachers talked about absence in governmental planning on pedagogy,


technology, and legislation. The right attitude, shared vision, and legislation to adopt
change were absent. This was an observation in discussions with the teachers in
schools, seminars, conferences, and emailing lists. Evidence to support this
observation came from a recent PhD on investigating Greek teachers’ needs and
training design; Tsetsilas (2006) reported:

• teachers’ training needs at school level were concerned to the social relations
and collaborative activities within the school environment;
• teachers needed support and training in collaboration, practical application of
innovations as part of scientific and professional training;
• there was lack of self-esteem and self-confidence derived from insufficient
training (Tsetsilas refers to “insufficient cognitive equipment”, 2006:6);
• the basic characteristics of the Greek educational system and the official
educational policy was centralism, restrictions in school autonomy, insufficient
basic and ineffective professional training; and
• certain parameters in the school environment required training.

Tsetsilas found that teachers currently exhibit insufficient professional skills on a


social and collaborative, cognitive/learning, and technical level. Lack of new
capabilities may result in inability to communicate their subjects with the students,
and lack of communication and collaboration within their environment. Tsetsilas
proposed that an in-service training program could be planned and organised
according to the training needs and suggestions of the Greek teachers themselves.
Alekos Alavanos, a Greek MP, representative of the Communist Party, reported this
problem to the Greek Parliament (24/11/2006 -
http://www.syriza.gr/modules/news/article.php?storyid=777; last access 05/06/2007).
In his speech after the Greek teachers’ 6 weeks strike, he said that everyone
involved in Greek education realises the need for change. It is interesting to see that

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 97


Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

the Greek teachers’ demonstrations continued every Wednesday until this section
was written (18/02/2007). Therefore a risk analysis was needed (Appendix II).

Overall, it appears that there were discrepancies between the stakeholders’ goals,
common ground and right attitude are not established. However, there is space for
improvement on a social, cognitive, and technical level.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 98


Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

4.3 PLANNING
Ethnotechnology can capture relevant information for planning in e-learning
engineering. The first section described the Greek teachers’ context; this section will
refer to the e-learning environment.

4.3.1. MOODLE@GSN: Moodle at the Greek


School Network

The Greek School Network (GSN) is the educational intranet similar to the
British JANET (Vivitsou et al., 2008). With regards to the e-learning service,
evaluation of various open-source learning management systems led its developers
to the open source package Moodle (Modular Object Oriented Dynamic Learning
Environment; http://www.moodle.org). The GSN e-learning platform is called
Moodle@GSN (http://e-learning.sch.gr). The Greek teachers played the roles of both
the course creators and students on a voluntary basis. Moodle@GSN is an
autonomous, self-organised service for teachers’ distance education training (Figure
4.3.1-1):

Moodle@GSN

Figure 4.3.1-1. Moodle@GSN research context

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 99


Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

4.3.1.1 Users and Activity in Moodle@GSN

Several observations and studies were conducted at Moodle@GSN as part of


time-series design. Two studies appeared to be significant for its history. The first
study was an overall quantitative evaluation until the 01/11/2006 (Vivitsou,
Lambropoulos, Konetas, Paraskevas & Grigoropoulos, 2008). Then, a baseline study
drew the line before and after any implementation in GSN based on descriptive and
discourse analysis.

Moodle@GSN started in 2003 with 2 online courses. Nine more courses were
created in 2004, 2005 and 26 in 2006, in a total number of 46 courses. The baseline
was the 1st of November 2006; on this date, there were 1,910 users registered online
whereas 64 users were registered in the 14 courses that were under construction. In
a total number of 4,350 members, there are 853 registered active participants, and
25 e-tutors. This means that 19% of the total number of registered users exhibited
some kind of activity (e.g. visiting the online courses, posting in chats and
discussions) in Moodle@GSN and 3% were the e-tutors. Due to the fact that
participation was on a voluntary basis, the offered courses had an open nature and
were divided in two major categories, the structured and well managed and the
courses that were designed for a special interest (e.g. Yoga, bird watching). With the
subject as the criterion, the categorization of the online course (Table 4.3.1.1-1):

Table 4.3.1.1-1. Courses categories and number of e-learners


General Topic # Courses #e-learners
Familiarization with Moodle 3 16
General Interest 6 98
Greek Schools Network Training 3 59
Greek Language 1 57
Pedagogical Approaches 2 124
IT Training 10 1167
Multimedia 2 202
Open Office 4 124
Courses Under Construction 15 63
Total 46 1910

It appears that there were 1,910 e-learners in 46 Moodle@GSN courses. More


specifically the categorization was as follows (Figure 4.3.1.1-1):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 100
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

Online Courses in GSN Familiarization w ith Moodle

General Interest
1400
1200 Greek Schools Netw ork
1000 Training
Learners

800 Greek Language

600
Pedagogical Approaches
400
200 IT Training
0
1 2 3 4 Multimedia

Category
Open Office

Graph 4.3.1.1-1. Online course categories in Moodle@GSN

It appears that in 46 completed and courses under construction there were 1,910
Greek teachers having a preference in IT training, multimedia, as well as the use of
pedagogical approaches. From the overall 206 messages sent in these courses, 8
discussion threads started from the e-tutors and 18 from the e-learners themselves,
resulting in 132 messages from the e-tutors and 74 from the e-learners There were
94 new topics of discussion launched by 34 e-tutors. In 22 courses there were chats,
having 1,000 messages sent by 138 participants. In addition, 40 quizzes were
activated in 19 courses used by 106 users in 197 efforts to solve them. Lastly, there
were 67 assignments in 18 courses from 47 users in a total of 94 submissions.

The 1st of November 2006 (baseline) was the starting point of interventions. There
were two interventions, collaborative e-learning and the use of tools based on the
proposed underlying conceptual frameworks. Part of the collaborative e-learning
intervention was the introduction of community management by the investigator. It
consisted of material providing advice on how to better use the platform including
emoticons. It also had material intended to improve the participants’ soft skills for
effective communication and about collaborative e-learning principles (Appendix VI).
The second intervention was to provide access to the new tools.The following table
describes the type and the number of messages. The ratio of single messages
26.6%, in a total number of 206 messages, 16.5% were the messages with a reply,
and 56.7% were the overall replies. In addition, it appears that there is counter
analogy to the number of messages and the number of replies, for example 19

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 101
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

messages had 1 one reply, whereas 1 message produces 27 replies (Graph 4.3.1.1-
2):

Number of messages and replies

30

25

20

Number of Messages
15
Number of replies

10

0
1 2 3 4 5 6 7 8 9 10

Graph 4.3.1.1-2. Comparison between number of messages and replies

This message was initiated to test the e-mint community managers’ suggestions;
they had to talk about their experiences and the Greek teachers responded
enthusiastically. The intervention stopped in order not to interfere with the study.
From the total number of 32 courses, two courses produced the aforementioned
messages. One course was built before the intervention, and the second after.

The messages in the first course ‘Use of ICT in Religious Education (RE)’ on the
discussion topic about the use of Power Point presentation were limited as regards
the richness of the text. The messages were sent during the period 26/03/2006 -
29/03/2006. There were 28 words in 6 threads. Even though the discussion is limited
there is evidence of collaborative learning. The participants (all different individuals)
built on each others’ threads and agreed on the importance of collaboration as
follows:

1 Participant A: Η παρουσίαση είναι τρόπος παράδοσης αλλά και εξέτασης


The presentation (ppt) is a way of delivery and examination.
2 Participant B: συμφωνω
I agree
3 Participant C: Και κάνει το μάθημα πιο ενδιαφέρον (χρήση οπτικοακουστικού
υλικού).
And makes the class more interesting (use of audiovisual material)
4 Participant D: Και δεν κουράζει τους μαθητές.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 102
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

And is not tiring for the students


5 Participant E: Ενδιαφέρουσα κ΄συλλογική συνεργασία
Interesting & collaborative cooperation
6 Participant F: ΠΡΟΠΑΝΤΩΝ ΣΥΝΕΡΓΑΣΙΑ
MOST OF ALL COLLABORATION

Participant F stressed the importance of collaboration using capital letters. In the


second discussion of the same first course (22/03/2006 - 09/05/2006) there were 13
messages with no more than 2 sentences in each thread. It appears that the
participants were exploring the environment; they were reluctant to reply since they
did not know each other (Participant F: ‘Βάλε φωτογραφία για να δεις βοήθεια. Έτσι
δεν έχεις ελπίδες......’ Add a photo to find help. Otherwise you don’t have a chance…
(26/03/2006)’. This message triangulates the previous observations on interventions
as regards the provision of professional help and the need for collaboration.

Next, the participants were invited to the second course on project management, built
from the researcher with the help of two colleagues, Marianna Vivitsou and Dimitris
Konetas. The participants were given educational material (samples exist in
Appendix VI) and the e-tutors followed the online community management framework
(Appendix I). The course “Project Method” was designed to respond to the Greek
educators’ need for training on educational project management, institutionalised by
law (Greek Government Gazette 303 & 304/13-3-2003).

The number of visits and the number of messages sent were (Table 4.3.1.1-2):

Table 4.3.1.1-2. e-Learners’ posts and views in the 6 active courses

E-LEARNERS’ POSTS AND VIEWS


Online Courses Forums Discussions Users Posts Views
1 Initial Page GSN News GSN community 4576 7 653
2 ICT in R.E. The Use of Ppt. Ppt use 15 5 9
3 Project Method e-tutors discussion Intro 50 4 5
4 Project Method Initial problems What we want 50 6 18
5 ICT in R.E. ICT in RE ICT in RE 15 7 10
6 Project Method Projects4discussion Hi everyone 50 5 13
Total 4576 34 708

It appears that 4,576 users posted 34 messages with 708 views. The course under
construction for this study appeared to have initiated a significant number of posts. A
comparison demonstrates the rapid increase of the messages in the course “Project
Method” after the initiation of the community management scheme (Table 4.3.1.1-3):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 103
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

Table 4.3.1.1-3. e-Learners’ replies and dates

E-LEARNERS’ POSTS AND DATES


Online
Forums Discussions Replies Dates
Courses
Initial
1 Page GSN News GSN community 27 10/03/2004 - 23/02/2005
ICT in
2 R.E. The Use of Ppt. Ppt use 5 26/03/2006 - 29/03/2006
Project
3 Method e-tutors discussion Intro 7 11/06/2006 - 19/07/2006
Project
4 Method Initial problems What we want 9 04/06/2006 - 06/10/2006
ICT in
5 R.E. ICT in RE ICT in RE 12 22/03/2006 - 09/05/2006
Project
6 Method Projects4discussion Hi everyone 14 17/10/2006 - 01/11/2006
Total 74 10/03/2004 - 01/11/2006

The e-learners, all different individuals, produced 14 threads with more than 2,000
words in total with the Project Method course among them with 30 replies between
04/06/2006 and 01/11/2006 presenting the following activity in the first week (Graph
4.3.1.1-3):

Activity in the Online Course

13, 10% 8, 6%

Number of Topics
28, 22%
Number of new
discussions
Number of sent
messages
Number of active
participants

77, 62%

Graph 4.3.1.1-3. Activity in the online course

It appears that the there was initial significant activity after the baseline and the first
intervention; there were 8 topics, 28 new discussions, 77 sent messages and 13
active participants. From the total number of 45 participants, 71% were passive and
29% active participants with an average of 6 messages per poster. Overall, it

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 104
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

appears that Moodle@GSN has potential to initiate an active e-learning community. It


is evident that:

• Professional development anchored in e-learning is one of both GSN and the


Greek teachers’ targets.
• The Greek educational authorities, responsible for planning and support, were
absent on a pedagogical and legislation level.
• The main areas of training were suggested to be IT training, the use of
multimedia, and current pedagogical approaches.
• The Greek teachers lacked appropriate e-learning training.
• The Greek teachers themselves attached importance to the key variances of
collaboration and social interactions evident in their discussions in an explicit
and implicit way.

Considerations on a socio-cultural, pedagogical and technological level were directed


to supporting social interactions and collaboration in order to activate participation in
collaborative e-learning.

The interventions in this research were the support of collaborative e-learning and its
social aspects in particular as well as the introduction of a set of new tools to support
passive and active participation levels, interactions, and the collaborative e-learning
as such. Thus, the studies were as follows (Table 4.3.1.1-4):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 105
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

Table 4.3.1.1-4. E-Learning Engineering for Moodle@GSN

E-Learning Engineering for Moodle@GSN


Date Sample Environment Intervention Activity
Ethnotechnology
Researcher’s
1/07/2004 EEEP mailing
1 Mailing list n/a Participation in the
-30/08/2004 list
list - Questionnaire
E-mint online Researcher’s
14/04/2004
2 community Mailing list n/a Participation in the
-30/06/2004
managers list - Discussion
Greek
-Learning Online course:
21/03/2006 - education Moodle@GSN
3 Management Introduction to
07/04/2006 mailing lists, & E-mmersion
-Tools Web Design
EFQUEL list
Online course:
01/11/2006 - Learning Educational Project
4 Moodle@GSN Moodle@GSN
30/11/2006 Management Management with
Collaboration Tools
Greek Moodle
06/03/2007 Recommendations
5 developers E-mmersion Tools
-13/03/2007 for new tools
lists
Online course:
-Learning
01/03/2007 PSD e-mail Moodle@GSN Educational Project
6 Management
-31/03/2007 database & E-mmersion Management with
-Tools
Collaboration Tools

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 106
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 4: Research Context

REFERENCES
Dourish, P., & Anderson, K. (2006). Collective Information Practice: Exploring Privacy
and Security as Social and Cultural Phenomena. Human-Computer
Interaction, 21(3), 319-342.
Fernandes, C., & Montalvo, A. (2006). E-Quality Synthesis Report: Experience-based
Quality in European ODL.
Golafshani, N. (2003). Understanding Reliability and Validity in Qualitative Research.
The Qualitative Report, 8(4), 597-607.
Lambropoulos, N. (2005). EEEP Online Community of Practice: The First to Boldly
Go in Greece. "Open Education", The Journal for Open and Distance
Education and Educational Technology, 2, 29-56.
Lambropoulos, N. (2005). Online Empathy: Social Consequences for Community
Building. In S. Dasgupta (Ed.), Encyclopaedia of Virtual Communities and
Technologies (pp. 346-348). Hershey, PA: Idea Publishing.
Lambropoulos, N. (2005). Paradise lost? Primary Empathy in Online Communities of
Interest and Ways of Use. Paper presented at the 1st Conference on Online
Communities and Social Computing, in the 11th International Conference on
Human-Computer Interaction 2005, 22-27 July, Las Vegas, NV. Retrieved
05/09/2005, from
http://www.intelligenesis.eu/nikiweb/PDF/05/lambropoulosOnlineEmpathyVeg
as05Research.pdf.
Shackel, B. (1991). Usability – Context, framework, definition, design and evaluation.
In B. Shackel & S. J. Richardson (Eds.), Human Factors for Informatics
Usability (pp. 21–37). New York, NY: Cambridge University Press.
Tsetsilas, I. (2006). Reporting teachers' training needs and design training schemes:
The case study from a primary school using D.I.O.N. model. Καταγραφή
επιµορφωτικών αναγκών των εκπαιδευτικών και σχεδιασµός προγραµµάτων
επιµόρφωσης: Η µελέτη περίπτωσης µίας σχολικής µονάδας της
Πρωτοβάθµιας Εκπαίδευσης µε το µοντέλο D.I.O.N. . Unpublished research at
the Open University, Patras, Greece.
Vivitsou, M., Lambropoulos, N., Konetas, D., Paraskevas, M., & Grigoropoulos, E.
(2008). The Project Method e-course: the use of tools towards the evolution
of the Greek teachers’ online community. International Journal of Continuing
Engineering Education and Lifelong Learning (IJCEELL), 18(1), 26-39.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 107
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

Tools & Evaluation Techniques

5 Supporting Collaborative E-Learning Communities

Key Topics Covered in This Chapter:

• Presentation of the first models for tools and evaluation techniques

• Initial evaluation of tools and evaluation techniques

• Applying guidelines and heuristics from feedback for design

• Implications for e-research design

• Iterative design and e-research design

Chapter 5 presents the initial design and evaluation related to the conceptual
frameworks, the tools prototypes, the evaluation techniques, and the research
design. Three studies are presented, one with international participation, one with the
Greek teachers, and one focus group with Greek teachers who are also Moodle
developers. Guidelines and heuristics obtained from feedback on the proposed
concepts and tools.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 108
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

5.1 INTRODUCTION

Six tools 1 were initially considered for integration in Moodle, and 3 tools were
found which were either not working for the Greek context or not suitable to be
discussed here; these tools were:

1. A chatbot, that initiated ‘safe’ conversation for passive participants


(Lambropoulos, 2007), was built in ASCII (American Standard Code for
Information Interchange); ASCII was found difficult to be represented in Greek
characters.
2. A keyword search facility that was not used. According to one of the
participants, it was a great idea but current systems are not built to support
such tools.
3. A real-time pedagogical usability questionnaire that was not particularly used.

The collaborative e-learning tools to be discussed here are:


1. Social network analysis tools
2. Participation levels evaluation graphs
3. Tools to structure collaborative e-learning

The following sections refer to the design process of these tools.

1
The tools were developed by Intelligenesis, a British company specialising in market research, social
networking and e-learning.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 109
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

5.2 COLLABORATIVE E-LEARNING


TOOLS

This section will present the iterative design process from creating the first
designs to the actual re-design and implementation. Four projects (blocks) were
developed, two of them will be discussed here (Table 5.2.-1):

Table 5.2.-1. Iterative Design Blocks

ITERATIVE DESIGN
Blocks Sample Environment Intervention Activity
Greek
-Learning Online course:
21/03/2006 education Moodle@GSN
1 Management Introduction to Web
-07/04/2006 mailing lists, & E-mmersion
Tools Design
EFQUEL list
Online course:
01/11/2006 Learning Educational Project
2 Moodle@GSN Moodle@GSN
-30/11/2006 Management Management with
Collaboration Tools
Greek Moodle
06/03/2007 Recommendations
3 developers E-mmersion -Tools
-13/03/2007 for new tools
lists
Online course:
-Learning
01/03/2007 PSD e-mail Moodle@GSN Educational Project
4 Management
-31/03/2007 database & E-mmersion Management with
-Tools
Collaboration Tools

Block 1 and 3 will be discussed in this chapter. Block 2 tested the community
management as part of collaborative learning framework, and will only be used for
comparison. Block 4 will be discussed in the next chapter as the main study.

5.2.1. Design for initial design:


Create prototype testing by the e-learners
Simplicity in design helps learners to concentrate on their objectives (Boy,
2007; Alty, 1997). This means that one of the targets in e-learning design should be
simplicity in order to maintain e-learners’ cognitive stability. Targeting the sense of
community development and socio-cultural learning, the tools were designed to
support presence and co-presence awareness, participation evaluation, and
collaborative e-learning.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 110
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

5.2.1.1. Presence & Co-Presence Awareness:


Visualisations Interactions Tools

The literature revealed that observing and analysing social interactions can
increase collaborative learning understanding. JUNG Social Network Analysis (SNA)
open source software was found to be suitable to acquire real-time results. Social
interactions can be depicted in SNA Nodes and Centrality windows as Visualisation
Interactions tools (VIT) open as Java applets using algorithms (Figure 5.2.1.1-1):

(a) (b)
Figure 5.2.1.1-1. Visualisation Interaction Tools (VIT) Nodes and Centrality

Each node represents a unique user/learner and the number of messages is


indicated as numbers as well as on the interaction lines; the more the messages, the
thicker the interaction lines. The Java applet uses JUNG library to create a sparse
graph, where users are represented as vertices and their relations as edges. The
process is as follows (Figure 5.2.1.1-2):

Figure 5.2.1.1-2. Visualisation Interactions Tools (VIT) production line

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 111
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

Moodle was patched to create a link which passes the forum ID to a Java applet. The
applet executed PHP script that fetched all required information from the Moodle
database creating an XML file, with all calculations made such as user relations and
message count. Then XML passed to the JAVA applet and the JAVA applet built a
graph based on information generated from the XML file.

The hosting company upgraded PHP4 to PHP5 and constantly reconfigured PHP5 and
Apache; JPGraph, the graphing software creator, was not updated, whereas GD library
was not activated. GD creates PNG, JPEG and GIF images and is commonly used to
generate charts on the fly. Not being activated after the upgrade resulted in major
problems with the use of these tools; the images did not display correctly and were
represented by an X. Participant D1 expressed his frustration: ‘It keeps driving me
bananas all the time. It really gives me the willies when that peculiar X appears in the top
lefthand corner’.

Despite the problems, e-tutors and e-learners would be able to observe the
sociability of the human network.

5.2.1.2. Participation Awareness:


Participation Evaluation Tools

Participation awareness is related to presence and co-presence awareness


measured by course participation and discussion participation graphs. Participation
Evaluation Tools (PET) are two graphs showing the participation levels for the overall
course activity and the e-learner’s activity in one discussion (Figure 5.2.1.2-1):

Figure 5.2.1.2-1. Course and individual participation levels graph

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 112
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

The graphs were built as follows (Figure 5.2.1.2-2):

Figure 5.2.1.2-2. Participation evaluation graphs production line

Moodle was patched in order to auto-generate links with the course ID that pointed to
PHP scripts in order to enquire users’ database accessing the specified course. Then
it gathered all statistical information from databases, parsed information and passed
it to JPGraph library in order to perform an output on the screen in real-time.

5.2.1.3. Structuring Collaborative E-Learning:


MessageTag

The tool to structure collaborative e-learning followed Collaborative e-


Learning Episode (CeLE). The initial tagging was inform, explain, explore, evaluate,
and other, was structured onto a drop down menu that was designed and located
after the posting text box. This was because posters may not recognise the type of
the reply unless they finished their message. It also contradicts the existing models
and tools; drop down menus and message tagging or openers have been designed
as an initial requirement (Figure 5.2.1.3-1).

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 113
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

Figure 5.2.1.3-1. Initial Design for Message Tagging

The script for such a tool is very simple, written in PHP and HTML and can be
incorporated in any Post page. Moodle was patched to allow all messages stored in
the database to be tagged. In addition to the tool, SQL queries could be performed to
get different statistical information based on message types or complex statistics
where types are only a part of a query (for example keyword and message type that
was already implemented but not used). The use of icons was considered instead of
text as used in the “vicarious learner” project (Lee et al., 1999). However, it was
thought that there will be excessive iconic information on the interface and, since the
e-learners actually read the title and the actual message when they post, the use of
text would not interrupt their reading but can be part of it.

Overall, it is anticipated that the tools can broaden and illuminate the space between
human-human interactions so as to open space for reflection for the e-learners. From
a research viewpoint, the tools could expand our understanding on the socio-
cognitive aspects of collaborative e-learning.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 114
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

5.2.2. The E-mmersion Block


The first block for iterative design to acquire participants’ opinions was
conducted from 21/03/2006 to 07/04/2006. The aims were:

The call for participation was sent initially to registered users in the web design
course at Moodle@GSN; only 5 out of 128 registered users responded to the call
(28/01/2006). Help was sought and found from the European Foundation for Quality
in eLearning (EFQUEL) (http://www.qualityfoundation.org/). The call was also sent to
the most popular Greek educational mailing lists, EEEP and PEKADE as well as
EFQUEL, and was announced on EFQUEL front page.

The online course in the experimental environment was supplementary to the web
design course in Moodle@GSN. From the 68 initial subscribers 12 did not enrol. The
remaining 56 participants are shown in the following graph (Graph 5.2.2-1):

Participating Countries

1, 1, 1, 1,
2%2% 2%2%
1,1,2%
2%
Greece
1,
1, 1%2%
2, 2% UK
Hungary
Germany
India
13, 19% Portugal
Spain
Sweden
43, 64%
Switzerland
USA
Turkey

Graph 5.2.2-1. Participating countries

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 115
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

The majority of the participants were from Greece (n=43, 64%) and the UK (n=13,
19%). The participants were divided in two groups and worked in two research pools,
the web design course for the Greeks and a discussion to evaluate the tools for all.
Also, there was a ‘Living Room’ for all participants. The participants were free to
communicate in both languages, English and Greek, however, English was proposed
in the tools evaluation discussion.

Methodology: A questionnaire was distributed and the collaborative e-learning


analytical framework was used for messages analysis. The suggested research
methodology proposed messages analysis, logging, and questionnaire statistical
analysis. The demographics, messages quantitative analysis and a collaborative e-
learning episode example from the web design pool are presented in Appendix VII.
Four CeLEs were identified in the Web Design pool with the Greek teachers. The
total code network view was depicted in ATLAS.ti (highlights added by the author):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 116
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

Figure 5.2.2-1 Total Codes Network in the Web Design pool

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 117
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

Figure 5.2.2-2 shows the code network for the 4 CeLEs in the Web Design pool. The
codes were: initiation, explanation, agreement, conflict, exploration, evaluation, new
ideas for co-construction, and other. After the analysis, it appeared that social cues
can be a separate attribute for the CeLE framework. Additionally, disagreement was
collapsed with agreement as they are opposite without being separate.

The next section presents the results from the second pool relevant to this chapter,
the tools evaluation pool.

5.2.2.1. Tools Evaluation Pool

In the tools evaluation pool there were 7 low, 2 medium and 1 high activity
user and a total of 33 messages were sent. (The high activity user A1 was the
Swedish Moodle developer.) (Figure 5.2.2.1-1):

Figure 5.2.2.1-1. Participants and number of messages (ATLAS.ti)

In addition to the pattern for writing online messages from the E-mint study, a second

pattern was identified and used (Appendix A_I_2).

Visualisation Interactions Tools (VITs): There were two graphs providing

information about the locus of participants within one discussion in a forum, based on

Social Network Analysis and the levels of participation. The two tools were located on

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 118
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

the top of discussion (Figure 5.2.2.1-2) and were VIT Nodes (Figure 5.2.2.1-3) and

VIT Centrality (Figure 5.2.2.1-4):

Figure 5.2.2.1-2. Location of VIT on the discussion forum

Figure 5.2.2.1-3. VIT Nodes

Figure 5.2.2.1-4. VIT Centrality

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 119
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

In the discussion in English language, A1 was initially sceptical on using algorithms


and Java compared to PHP applications. He was not familiar with sociogrammes but
found them interesting and mostly useful by moderators: ‘It would be interesting to
see who is active and in what role and what contributions people make. I think it is
the teacher's job to e.g. monitor/moderate group work and see to it that the
assignments are distributed evenly and that credit is given fairly… [resulting in] better
learning.’. M1 as an online tutor she ‘would like to know which students are active
and participate in the e-lessons and which not. I would use it in me lessons!!’. M1
liked VIT as ‘vit nodes also work with me and i agree they can be very useful for
class participation statistics (quality measurement)’. However, she could not
understand the concept of centrality and had to read the instructions. Also, AM, GF,
M2, and D1 expressed the same problem. E1 expressed her unfamiliarity with the
tools as well: ‘I'm not familiar with what they stand for, so I'm afraid I can't be of much
help here. They seem nice, though!’.

A1 suggested that learners ‘can lurk intensively and actively and learn more than

those visibly active’, so she proposed passive participation to be visible on VIT. The
participants had to face new concepts and work with new tools without previous

experience. Participant M1 suggested: ‘…i (at least tried to) familiarised with new

concepts (quality, e-/web/online learning etc)’; ‘through this experience I realised

(once again) that there are some many things left for me to ‘learn’ in the LMS

management area and the evaluation of technology-based learning. Also: Some

concepts need to be defined (e.g. what are ‘best learning achievements?”), some
applications / tools to be improved (e.g. philippa’s repertoire of responses), some to

be further explored (e.g. VIT tools, activity graphs, message tags) and the moodle

environment itself needs more elaboration (e.g. listening activities?)’.

Participation Evaluation Graphs: Two types of graphs were created associated with
the online course and the individual e-learner describing the e-learning community
progress (Graphs 5.2.2.1-1 to 5.2.2.1-3):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 120
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

Graph 5.2.2.1-1. 31/03/2006 – 5/28 Graph 5.2.2.1-2. 04/04/2006 – 10/28


e-learners e-learners

Graph 5.2.2.1-3. 08/04/2006 – 10/28 e-learners

As viewed in the previous graphs, only two members appeared in high and medium
activity during the first days (31/03/2006). Thus the moderator sent an email
suggesting that all participants need to contribute to the discussion. This intervention
appeared successful; 5 more participants appeared in the low participation graph
5.2.2.1-2 (04/04/2006).

M1 said that such a tool is not helpful: ‘for a newbie like me passive participation very
often proves to be very constructive’ (M1). E1 found difficulty in interpreting them. A1
said: ‘they show that e.g. my activity level in the Induction course is medium, that
there are 5 other users on low level and 1 user, probably admin with high level.’ To
A1 the kind and the quality of the activity is missing ‘or whether anything, and if so

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 121
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

what, may have been learnt.’, so log files provide more valuable information. E1 said:
‘they are more useful to teachers than fellow students.’. M1, talking about the sense
of community, said: ‘how can you, from in this way graphically presented statistics,
can conclude that someone is 'immersed'.’.

From an e-tutor’s and researcher’s viewpoint, the graphs were effective. Similar tools
exist in other learning management systems such as Blackboard, where bar graphs
depict e-learners’ overall activity rather than distinguishing passive and active
participation levels. There were some issues due to systems’ instability; the first
column of participation was unstable; passive participants were not shown on the
graphs (participation column should have shown 28 participants in Graphs 6.2.2.1-1-
3). In addition, it was not possible to retrieve their log files. Other problems were
related to more than one login names registered for each user exhibiting multiple
appearances, indicating the need for an initial account confirmation to ensure
accurate and reliable results.

Collaborative e-learning tool MessageTag: The participants were encouraged to


use a drop-down menu for tagging their messages 2 (Figure 5.2.2.1-5):

Figure 5.2.2.1-5. MessageTag

The tool was used more in the Evaluation Pool than the Web Design pool as follows
(Table 5.2.2.1-1) (only the participants’ messages were counted):

2
At this point in the study, disagreement was coded under evaluation. Later it appeared as on option in
conjunction with agreement

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 122
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

Table 5.2.2.1-1. Use of MessageTag

USE OF MESSAGETAG
Overall CeLE Tagged
Pools # Percentage
replies Attributes messages
Inform 9
Living Question 2
Room 35 17/35 48.5
Explore 4
Explain 2
Inform 4
Web Question 8
Design 73 Explore 5 20/73 27.3
Explain
3
Inform 12
Question 6
Tools
47 Explore 2 26/47 55.3
Evaluation
Explain 4
Evaluate 2
Inform 25 39.6
Question 16 25.3
Total 155 Explore 11 63/155 17.4 40.6
Explain 7 11.1
Evaluate 2 3.1

It appears that MessageTag was mostly used in the tools evaluation pool with 26 out
of 47 messages tagged (55.3%) and less in the Greek teachers’ only pool with 20 out
of 73 messages tagged (27.3%). In the introductory forum (Living Room) 17 out of 35
messages were tagged (48.5%). Because the tool was integrated in discussion, the
log files are the same with the forum and discussion logs. Based on the tagged
messages, the inform tag was used more than the other tags (39.6%) and the
evaluate tag less (3.1%); 16 (25.3) messages were tagged as questions, 11 (17.4) as
explorations, 7 (11.1%) as explanations, and 2 (3.1) as evaluations.

A1 found this tool interesting whereas E1 said that she never saw a tool like this
before so she did not know the purpose it served or how it can be used (‘for statistics
maybe?’). She wondered whether ‘users read messages according to their tagging
(from a long list of messages). A help file next to "message tag" would be useful.
Perhaps the text of each tag could be accompanied by an icon.’. AM and M1 agreed
with E1. M1 said that ‘each tag provides a thematic categorisation of the reply.
personally i read all the replies to the topic i'm interested in no matter what the tag is,
perhaps because what matters is the content and not the title. it might prove more

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 123
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

useful for younger learners' induction to the system though’. G1 agreed with M1. A1
added that the tool would have worked better if it could show the very structure of the
discussion as not many people are capable ‘of seeing such structures by themselves,
this meaning that they are likely to miss, or misunderstand the crucial points. All
learners are not good at 'deep' learning.’. He suggested that this tool could be
involved into becoming an instrument ‘for learners to enhance their literacy i.e.
becoming better e-readers/listeners and e-writers/speakers, then that would certainly
be a 'huge step for humanity'’. A1 recommended some additional tags for
consideration:

• add/develop
• ask for additional information
• ask for clarifications
• confirm
• approve
• agree
• disagree
• offer conclusions
• point to resources

Overall, it appears that some issues influenced MessageTag functionality: the


instructions were not helpful; it was the first time the participants used such tools;
there were severe technical problems; and the participants thought that the tools
were more helpful to e-tutors. This may result in the increase of low and medium
active participation levels as in the previous example. Nonetheless, implications for
design, discussed in the next section, can improve tools’ usability and usefulness.

Lastly, it was evident that the use of MessageTag attributes was descending as the
level of reflection and higher order thinking was ascending. This finding stresses the
importance attached to the difference between information provision and knowledge
acquisition and that the tool can make this difference visible. In this way the e-
learners participants can moderate their own messages and use the tool to facilitate
their critical thinking skills development; as such, the e-tutors can support this
process. This will have an impact on the e-learning quality.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 124
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

5.2.2.2. Application of guidelines and heuristics from


feedback in design

Based on participants’ recommendations and the researcher’s experience


four changes were made on Visualisation Interaction Tools (VITs), participation
graphs, and MessageTag.

Visualisation Interactions Tools (VIT): Following A1’s recommendation the lurkers


were added in the VIT centrality that provided more space on the interface. A
disadvantage is that the Java applet restricts login names’ clickability (Figure 5.2.2.2-
1):

Figure 5.2.2.2-1. Lurkers overall view in VIT Centrality (right)

Participation Evaluation Tools: The two graphs for overall course participation
levels and discussion participation levels had to depict all participation levels for all
participants so there were three changes (Figure 5.2.2.2-2):

• overall participation column was corrected;


• because e-tutors’ messages change the results, these messages were not
counted in the graphs. They can be counted if the e-tutor wanted to appear as
one of the participants so the messages were calculated again; and
• date and time were added.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 125
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

Figure 5.2.2.2-2. Redesign of participation graphs

Participation Avatars: Based on A1’s proposition, a new tool thought to depict


participation levels and an overall view. After discussing this proposition with the
graphic designer MR and supervisors, the avatar appeared to be a good solution.
Based on the notion that the avatar is an Indian concept, and thus indicates different
energy levels in the human body (Isaackson, 2003), the avatar was designed as
follows:

• Zero participation (grey)

• Low participation (red, grey)

• Medium participation (red, blue, grey)

• High participation (red, blue, yellow)

The avatars were integrated on the threaded discussion view. This means that the
participants can compare collaborative e-learning structures and participation levels
in one glance. Lastly, the log files will be the same as the discussion forum log files.

MessageTag: First, two more values were added on MessageTag, Agree and
Summarise. A1 suggested adding conflict but the decision was made; conflict was a
subcategory used for coding only along with disagreement. So the new collaborative
e-learning episode analytical framework was redesigned (Figure 5.2.2.2-3):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 126
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

Figure 5.2.2.2-3 CeLE MessageTag tool

Following A1’s recommendation for overviews, a messages overall view in the


threaded form was adopted. In this way the e-learners would be able to see the
internal structure of the message.

5.2.2.3. Implications for Research Design

The findings indicated implications for design and research design:

1. Questionnaire: The e-learners did not use the real-time online questionnaire
(N=9/31, 29%); several changes were made so the questionnaire could be
sent via email:

• Three sections were elaborated and added:


o the initial sense of community identification was based on empathy;
then, a Sense of E-Learning Community Index was adapted;
o the learning section was expanded; and
o Pedagogical Usability targeted usability and utility in detail.

• An absence of collaboration with the Greek educational authorities on the


implementation of the tools, led to the addition of an extra question
targeted to Greek teachers’ communication and collaboration with the
Greek authorities.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 127
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

2. Changes on coding: Following changes on MessageTag, the two new codes


“agree” and “summarise” were added in the collaborative e-learning analytical
framework.

3. Changes on the Collaborative E-Learning Episode (CeLE): Two structures


appeared to occur in a CeLE, a horizontal and vertical (see figure ..). The
horizontal structure referred to interaction between the participants and thus,
a horizontal and interactive direction of learning. The vertical structure exists
within the message of the participants and is related to her reflective and
evaluation processes, and thus, a vertical and individualised direction of
learning.

A Sense of e-Learning Community Index (SeLCI) was further investigated (Rovai,


2002; Preece & Maloney-Krichmar, 2003; Wright, 2004; Daniel, 2007). Rovai (2002)
worked on the role of spirit, trust, interaction, and commonality of expectations and
goals towards the development of the sense of community in e-learning. He also
suggested that there are additional elements in e-learning such as social presence,
social equality, small group activities, group facilitation, the teaching and learning
style as well as the community size. Preece & Maloney-Krichmar provided a
framework to support sociability and usability. For online communication they
stressed the need for common ground, sense of social presence, and empathy and
trust. To support the group dynamics online they suggested the use of social
networks and reciprocity, roles, rituals, norms and policies. Wright (2004) emphasised
the dynamics between elements in a Sense of Community Index and suggested that
there should be different indexes if the study has an educational purpose. Daniel
(2007), working on social capital suggested the importance of shared experiences,
interaction, common ground, trust, awareness, social protocols, and the use of social
networks. Anchored in the aforementioned recommendations as well as the findings
the Sense of e-Learning Community Index (SeLCI) was developed for community
evolution, sense of belonging, empathy, trust, intensity, collaborative e-learning
quality, social network analysis for global cohesion and centrality.

Thus, the research design was adapted as follows (adaptations in grey) (Table
5.2.2.3-1):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 128
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

Table 5.2.2.3-1. Research Design

ETHNOTECHNOLOGY DESCRIPTION
1. Participation in projects, conferences, seminars,
online discussions of Greek teachers’ associations
and schools in Greece
1 Scenic Fieldwork 2. Documentation, manuals and presentations by the
Greek educational authorities
3. Personal opinions, conversations, and emails
4. Richness of text
1. Demographics
2 Pre-Questionnaire 2. Conditions of working and learning over the Internet
3. Initial knowledge on collaboration
1. Frequency of visits
2. Number of active and passive participants
3. Active participation levels
4. Passive participation levels
5. Sense of e-Learning Community Index
3 Logging • Persistence: depth of discussion threads
• Density: formula 2aN(N-1)
• Reciprocity: number of messages sent and
received
• Intensity: levels of participation
1. Collaborative e-learning episodes analysis
4 Thematic Analysis 2. Sense of e-Learning Community Index
• Number of Collaborative e-learning episodes
The Sense of e-Learning Community Index
1. Global Cohesion
• Density
• Reciprocity
Social Network • Cliques
5 Analysis • Structural equivalence
2. Global Centrality
• Centrality
• Closeness
• Betweenness
3. Local real-time nodes & centrality
1. The Sense of e-Learning Community Index
• Community evolution
• Sense of belonging to the e-learning community
• Empathy
6 Post-Questionnaire • Trust: knowledge exchange, help and support
• Collaborative e-learning quality: participants’
opinions
2. Usage and usability of collaborative tools
3. Professional development
1. Examination of the data, missing data, normality.
2. Descriptive Statistics: crosstabulation, frequencies,
arithmetic means, standard deviations, and
7 Statistical Analysis exploration
3. Correlation: Pearson correlation coefficient (r).
4. Inferential statistics: statistical significance, p-value
(Cronbach's alpha, α), and null hypothesis.
1. Form of the effect: the level, slop, variance and
cyclicity
8 Intervention Analysis
2. Its permanence (continuous or discontinuous)
3. Its immediacy (immediate or delayed)

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 129
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

Overall, PHP applications cannot easily work with algorithms and Java. This perhaps
indicates the need for new programming techniques to support systems
interoperability. Since such programming has not been used yet in available e-
learning management systems a new hosting company was needed to meet most of
the requirements. Furthermore, if such tools and evaluation techniques do not exist, it
is difficult for the participants to observe they are missing.

With the provided time and resources in this study only redesign of the tools was
feasible. The redesign was tested and developed further with the help of a focus
group of Greek teachers Moodle developers.

5.2.3 Greek teachers Moodle developers

Three individuals, being both Greek teachers and Moodle developers,


participated in a focus group targeting tools redesign before the Greek teachers’
renewed participation. The discussion was from 06/03/2007 to 13/03/2007. Three of
them returned the questionnaire so the demographics are as follows (Table 5.2.3-1):

Table 5.2.3-1. Demographics for the three Greek Teachers / Moodle Developers
Demographics
Use of
Working
computer Use of Use of
Age Gender Occupation Experience
/ software Internet Moodle
(years)
(years)
1 30-40 Male Teacher 1-5 6-10 All day 1-5 years
2 40-65 Male Instructor 11-20 6-10 All day 1-5 years
Once -
Doctor /
3 40-65 Male 11-20 1-5 twice a 1-5 years
instructor
day

All three were male, two between 40-65 years old and one between 30-40 years old
and none of them had participated in any of the previous blocks or studies. P1 was a
teacher, P2 an instructor, and P3 a doctor, all working, training adults using Moodle
for 1-5 years. P1 had 1-5 years of professional experience, he was using educational
software for 6-10 years, and he had been connected on the internet all day. P2 and
P3 had the same working experience, 11-20 years; P2 was using the Internet all day
and P3 once or twice a day. None of them said he knew any tools and evaluation
techniques to support participation whereas all of them thought participation
necessary. As for the reasons, P1 said that ‘self-efficiency is important in e-learning’,

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 130
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

P2 because ‘feeling part of the community’ is important, whereas P3 replied with a


question mark (‘?’).

The discussion took place in the experimental environment with 11 discussion topics
initiated by the researcher with a brief description of each tool. Thematic analysis
was used on the 53 replies to categorise participants’ opinions on the tools; in
addition, the new collaborative e-learning scheme was tested for use and usefulness.
The discussions topics were: an introduction, new ideas, tools, and suggestions on
Greek names. Only discussions on the new ideas and tools are presented here. The
comments about the tools on the provided questionnaire were (Table 5.3.1-2):

Table 5.2.3-2. Questionnaire open questions

Open Questions Results


Participation Tools Message Tag VIT Nodes VIT Centrality Other
Show the quantity but
not the quality of the
Structures Not needed, Useful and
P1 messages. The avatar
information but not sure interesting
was simpler and
equally good.
P2 More information More tools
Clickable
P3
images

Overall, the respondents said the following:

• E-learner’s participation graph was not calculated correctly.


• Understanding the usefulness of the MessageTag and VIT was not easy
without reading the manual;
o Simpler names were needed (P3).
o On VIT Centrality, the list with the lurkers’ login names was distractive
and perhaps it should be removed. As for the centrality circles, they
were quite understandable; it may be a good idea to use this image as
the VIT Centrality icon (P3).
• Graphics:
o Multicoloured avatars are not clear as regards their meaning (P3: ‘Η
πολυχρωμία ...είναι λίγο δυσνόητη’). The use of one colour for each

participation level may work better ( ) (P3).


o The more information existed on an icon, the less the participation,
especially for the people who are not familiar with such environments
(P3). In other words, an image should explain its own use (P2).

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 131
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

o The results should be available on the interface without the


user/learner having to click on them (P3).
• They would like more tools.

P1, P2 and P3 also answered the second session of the questionnaire based on a
Likert scale 1-5 and focused on efficiency and effectiveness, satisfaction (ISO DIS
9241-11, 1994), enjoyability, learnability, and overall evaluation for imagination and
satisfaction (Zaharias, 2004; Silius et al., 2003) (Table 5.2.3-3):

Table 5.2.3-3. Tools Pedagogical Usability Scores

PEDAGOGICAL USABILITY
1 Efficiency & Effectiveness μ = 3.9
Participation
Message Tag VIT Nodes VIT Centrality
Tools
μ 4.3 3.3 3.6 4.3
2 Satisfaction = 4
Participation
Message Tag VIT Nodes VIT Centrality
Tools
μ 4.3 3.6 3.6 4.6
3 Enjoyment = 4.5
Participation
Message Tag VIT Nodes VIT Centrality
Tools
μ 4.6 4.3 4 5
4 Learnability = 4
Participation
Message Tag VIT Nodes VIT Centrality
Tools
μ 4.3 4.3 4 3.6
5 Imaginative μ = 4.6
6 Satisfaction μ = 4.3
Overall Score μ = 4.2

The tools scored 4.2 out of 5 as regards pedagogical usability. This means that the
participants were overall satisfied with the tools but made additional
recommendations as seen previously. Lastly, there were some final propositions. P2
suggested that the interface should provide two views; an overall users/learners’ view
so the moderator can support the learners’ individuality; and a user/learner’s view for
self-organisation and self-learning. He also suggested the group format for e-learners
flexibility. In this way, a user/learner who lost his interest and lurked can be activated
in a different team. Finally, P3 indicated the problem spotted in the literature,
structuring or not structuring information: ‘I think I understood!!!no need for explaining
the use of tags. But is it necessary to make use of tags? Isn't it easy to understand
the nature of the writer's message?’

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 132
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

However, this measured was found inadequate to accurately measure utility and
usability for new tools. The Collaborative e-Learning Episodes (CeLE) was successful
and 7 CeLEs were identified.

5.2.3.1 Application of guidelines and heuristics from


feedback in design

When this discussion finished the Greek teachers were in the middle of the
online course in Moodle@GSN (main study). Meanwhile work was carried out
collaboratively with another programmer to update the tools and she actively
participated in the course as an e-tutor. The considerations on design and research
design were:

• The participants did not immediately understand the usefulness of the tools.
This may partly be because they were not familiar with such tools, and partly
because of design.
• The names of the tools came from social network analysis references; the
participants did not provide any alternative names.
• The lurkers’ login names on VIT Centrality were not removed because there
was no other way to make them visible on the tool.
• Providing all information on the interface was found to be of poor usability
because of the information overload on the interface so no additional
information was provided.
• The use of one colour for each participation level did not show the
user/learner’s potential. However, this was a good idea and the high

participation graph was changed to yellow ( ). The proposed red colour was
found inappropriate due to alert signalling of red and usability purposes.
• The icons for the VIT were changed to icons acquired from a discussion.
• Two views of the tools, as well as having the tools in a group format could not
be implemented in a study such as this one. It is a bigger project and requires
team working as well as time and funding.
• The pedagogical usability questionnaire was elaborated in more detail.
• A combination of participation levels and CeLE from a bird’s eye view can
provide comparable information (Figure 5.2.3.1-1):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 133
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

Figure 5.2.3.1-1. Discussion on tools’ Greek names

Following the graph, it appears feasible to compare collaborative e-learning values


and levels of participation. Due to limited space, this will be analysed further in the
next chapter.

Lastly, since the utility and usability attributes were found inadequate, they were
transformed; new attributes were anchored in Zaharias usability questionnaire,
focused on 5 out of his 13 suggested attributes (Zaharias, 2006:198-199) as well as
Preece and Maloney-Krichmar’s framework on sociability and usability for online
learning communities (2003). Further correlation analysis suggested the
interactivity/engagement and motivation to learn to be combined under ‘motivation to
participate’. Moreover,

• relative items were grouped under one attribute;


• the aim was to support a utility & usability overview of the tools related to
participation in collaborative e-learning communities;
• the questionnaire was tested twice with 2 groups of different participants with
3 members in each group to ensure that the items were adapted and included
appropriately in the questionnaire.

Considering users’ unfamiliarity with new tools the utility & usability questionnaire
items were developed as follows (Figure 5.2.3.1-1):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 134
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

Figure 5.2.3.1-2. Pedagogical Usability Attributes

Overall, all participants agreed that the tools, and in particular the Visualisation
Interactions Tools (VIT) were useful and important, especially for the moderator,
because she can: direct and control online discussions; discover interlocutors’
weaknesses and strengths; activate the lurkers with specific questions; aid in team
building by “bounding” the team; and record the discussion. They also said that self-
efficiency is important in e-learning and feeling part of the community is what
participation is about, addressing learning on an individual and a social level.

From a researcher’s viewpoint, the numeric assessment provided by the tools and
evaluation techniques was successful. Despite the discrepancies with the graphics
and difficulty in understanding the meaning of the tools, the Greek teachers Moodle
developers were satisfied with the tools, they enjoyed using them, and asked for
more.

The last part of this research is the main study discussed in the next chapter.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 135
Tools & Evaluation Techniques for Collaborative E-Learning Communities Ch. 5: Tools & Evaluation Techniques

REFERENCES
Alty, J. L. (1997). Multimedia. In A. B. Tucker (Ed.), The Computer Science and
Engineering Handbook (pp. 1551-1570). Boca Raton, FL: CRC Press.
Boy, G. A. (2007). Perceived Complexity and Cognitive Stability in Human-Centered
Design. Paper presented at the HCII'07 - HCI International 2007 Conference,
22 – 27 July 2007, Beijing, China.
Daniel, B. K. M. (2007). A Bayesian Belief Network Computational Model of Social
Capital in Virtual Communities. Unpublished Research, PhD Thesis,
University of Saskatchewan, Canada. Retrieved 01/08/2007, from
http://library2.usask.ca/theses/available/etd-07132007-
141903/unrestricted/ben_m.pdf.
Isaacson, C. (2003). The Way of Yoga. London: Element Books.
International Standards Organization. 1994. Ergonomicrequirements for offi ce work
with visual display terminals.Part 11: Guidance on usability (ISO DIS 9241-
11). London: International Standards Organization
Lambropoulos, N. (in press). Preparing the On-line Learners: Information Provision
and Intention Setting by Chatbots. In P. Shank (Ed.), Online Learning Idea
Book. Hershey, PA: Idea Publishing.
Lee, J., McKendree, J., Stenning, K., Cox, R., Dineen, F., & Mayes, T. (1999).
Vicarious Learning from Educational Dialogue. Paper presented at the
CSCL'99 - Computer Supported Co-operative Learning, 11 - 12 December,
1999, Stanford University. Retrieved 12/12/2005, from
http://portal.acm.org/ft_gateway.cfm?id=1150283&type=pdf&coll=portal&dl=A
CM&CFID=15151515&CFTOKEN=6184618.
Preece, J., & Maloney-Krichmar, D. (2003). Online Communities. In J. Jacko & A.
Sears (Eds.), Handbook of Human-Computer Interaction (pp. 596-620).
Mahwah: NJ: Lawrence Erlbaum Associates Inc. Publishers.
Silius, K., Tervakari, A.M., & Pohjolainen, S. (2003). A Multidisciplinary Tool for the
Evaluation of Usability, Pedagogical Usability, Accessibility and Informational
Quality of Web-based Courses. Paper presented at PEG’03, 11th
International PEG Conference: Powerful ICT for Teaching and Learning. 28
June - 1 July 2003 , St. Petersburg, Russia. CD-Rom.
Wegerif, R. (in press). Dialogic, Educational and Technology: Expanding the Space
of Learning. New York: Springer-Verlag.
Wright, S. P. (2004). Exploring Psychological Sense of Community in Living-Learning
Programs. Unpublished research, PhD Thesis, University of Maryland.
Zaharias, P. (2004). A Usability Evaluation Method for E-Learning Courses.
Unpublished Doctoral Dissertation at Athens University of Economics and
Business

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 136
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Main Study:

6 The quasi experimental study on Moodle@GSN and the


research pool

Key Topics Covered in This Chapter:

• How initial difficulties were resolved in the main study

• Demographics and initial knowledge or lack of it

• Tracing active and passive participation

• The collaborative e-learning episodes in the service of e-learning


quality

• Unfolding the Greek teachers’ sense of community

• How pedagogical usability and usability informed human-computer


interactions

• How and why the intervention was successful

• Comparison with other contemporary studies

Chapter 6 presents the preparations, interventions, findings, and discussion of the


main study. There is an initial examination of the Greek teachers’ human-human
interactions for working and learning online. Then the levels of participation reveal
their passive and active presence in the course in comparison to the experimental
environment. The comparison continues with the quest for quality in their discussion
under the lens of the analytical framework of collaborative e-learning episodes. The
study also investigates whether a collaborative e-learning community exists using the
sense of e-learning community index. In addition, pedagogical usability and utility
values provide information about the Greek teachers’ human-computer interactions.
Finally, the intervention analysis and comparison with current research are presented
in this chapter.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 137
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

6.1 INTRODUCTION

The aim of this study is to address the Greek teachers’ passive participation
in the e-learning environment provided by the Greek School Network
(Moodle@GSN). Because the Moodle@GSN technicians were reluctant to
implement the new tools, these tools were incorporated in an experimental Moodle,
version 1.4.5 as with Moodle@GSN. The online course was on the ‘Use of New
Technologies for Educational Project Management’, started on 01/03/2007 and
finished on 31/03/2007. I participated as the 5th e-tutor (Figure 6.1.1-1):

Figure 6.1-1. The e-tutors in the online course

A link in Moodle@GSN in the last week led the participants to the research pool.
Thus comparison was feasible between the two contexts, Moodle@GSN and the
experimental environment. The e-research methodologies used to aid e-learning
engineering were centred on ethnotechnology: fieldwork, logging, thematic and social
network analysis, and questionnaires. More specifically, the next section in this
chapter describes demographics, conditions of working and learning over the
Internet, and examples of Collaborative e-Learning Episodes. Then, it explores the
attributes of the Sense of e-Learning Community Index: community evolution, sense
of belonging, empathy, intensity, trust, e-learning quality; in addition, global and local
social network analysis will aid in investigating group cohesion and the importance of
social networking and collaborative e-learning. The evaluation of the new
collaborative tools is conducted by utility and usability evaluation. Lastly, intervention
analysis examines the overall success and compares the findings with other
contemporary research.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 138
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

6.1.1. Initial activities

The initial email call for participation in the online course attracted insufficient
response and another call was issued. To improve communication flexibility, the
participants were given gmail accounts. The course had to be expanded by one more
week than initially planned; this resulted in having the experimental session the week
before Easter 2007. It had 4 sections, project management, blog, wiki, and
videoconference. The participants had to study the educational resources, create
their own context using the tool and evaluate its use. The evaluation of the
videoconference was conducted in the research pool after modifications to the
interface based on previous research carried out before hand into the effectiveness
of the system. Because interface and pedagogical modifications (Delich, 2006) as
well as methods and tasks (Draper, 1993) are interlinked, a decision was made to
change from an instructional (e-tutoring) to a student-centred )collaborative e-
learning) approach.

Two questionnaires were sent: one at the beginning of the course to gather
information about demographics and some background and one at the end with
questions related to the exploratory research questions. One hundred and seventeen
(117) participants returned the first and 59 the third questionnaire. Only 40
questionnaires from the participants who returned the first and third questionnaire
and appeared on both Moodle@GSN and the research pool were accepted for
analysis (Table 6.1.1-1):

Table 6.1.1-1. The questionnaires selection process

Accepted / Rejected Questionnaires

Number of Questionnaires Value Justification


Returning Q1, Q2, Q3; participation in
Total 59
Moodle@GSN and the research pool
Accepted 34 Participation in all previous activities
Not participating in either Q1, Q3 or
Rejected 19
Moodle@GSN and the research pool
Accepted for analysis 40 (67.7%)

Forty questionnaires were selected for analysis using the above criteria and were
given a unique number. Considering the total absence of activity and the ratio
referred to in the literature (20% lurkers’ response), the response ratio compared to
the initial expression of interest (117) was satisfactory (34.1%).

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 139
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Even though 2.03% of the overall responses in the questionnaires items were
missing, this was a low percentage and did not impose any serious problems with the
subsequent analysis (Tabachnick & Fidell, 1996). When participants had failed to
supply a response this was treated as a no response rather than using the mean.
Correlation analysis was used to suggest the highly correlated factors in order to
integrate them in one variable. These results, the small initial sample in the trials and
previous absence of variables justification in the literature suggested that the
questionnaire items could remain the same. Hierarchical Clustering Explorer 3.0 ∗
(HCE) (Seo, 2005) was used for the examination of the data (Figure 6.1.1-2):

Figure 6.1.1-2. Normality overview for tools pedagogical usability and utility in HCE

The responses from the questionnaires were also screened for normality skewness
and kurtosis in HCE (e.g. Hair et al., 1998) to determine whether the variables are
“normal” enough to be analysed; skewness and kurtosis suggested that original
variables can to be used. (Tests of significance for skewness and kurtosis test the
obtained value against a null hypothesis of zero for a normal distribution.)


HCE is a visualization tool for interactive exploration of multidimensional datasets to help users explore
and understand multidimensional datasets by maximizing the human perceptual skills
(http://www.cs.umd.edu/hcil/hce/).

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 140
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

6.2 DESCRIPTION OF THE RESEARCH


CONTEXT

From the initially registered 177 teachers, 49 never fully enrolled. The participants
were asked to create their profiles immediately after their registration. It was hoped
that this would create a co-presence feeling by getting to know each other before
their attempts for active participation. Of the remaining 128 participants, 36 created
illegible profiles due to Greek language encoding, and 95 participants produced
usable profiles. Most of the participants with Moodle profiles were primary school
teachers (n=36, 36%), and IT secondary school teachers (n=22, 28%). Based on the
initial questionnaires (N=40) there were 9 female (n=9, 22.5%) and 31 male (n=31,
77.5%) Greek teachers. The age range in the closed questions was created
anchored in the way teachers are hired and previous studies (e.g. Hlapanis &
Dimitrakopoulou, 2004), and was successfully tested in the preliminary studies.

6.2.1. Who are the Greek Teachers?

The analysis showed 1 participant between 20-30 years old (n=1, 3%), 14
between 30-40 years old (n=14, 36%), and 24 more than 40 years old (n=24, 61%) (1
missing). They participated from different parts of Greece, 16 from Athens (Figure
6.2.1-1):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 141
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Figure 6.2.1-1. Participants’ locations in Greece

This selection process assigned a representative sample of the teachers with a


special interest in the use of ICT in education in Greece (Figure 7.2.1-1; originally
retrieved 03/09/2007, from http://www.lib.utexas.edu/maps/europe/greece_div96.jpg):

6.2.2. Conditions of Working and Learning Online

The participants’ years in education, as well as their experience in computers


and Learning Management Systems (LMS) were (see data in Appendix VIII):

ICT in Education: Most of the participants have worked in the Greek education
system for more than 6 years (N=40). There were 2 new teachers (n=2, 5%), 10
teachers working for 6-10 years (n=10, 25%), 18 working for 11-20 years (n=18,
45%) and 9 more than 20 years (n=9, 22.5%). The participants have used computers
in education: 3 for 1-5 years (N=3, 7.5%), 14 for 6-10 years (n=14, 35%), 15 for 11-
20 years (n=15, 37.5%), and 7 for more than 20 years (n=7, 17.5%). However, their
familiarity with Learning Management Systems (LMS) is relatively low; the majority

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 142
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

have not used LMS at all (N=22, 55%) and 3 for 1 to 6 months (n=3, 7.5%). Only 13
participant have been using LMS more than one year (n=13, 32.5%) whereas one
participants said that she was using LMS for more than 6 years (i.e. less than 6 years
ago). This response may not be reliable since Moodle was only made available to the
public in November 2001 (Moodle e-learning history, retrieved 17/07/2007, from
http://docs.moodle.org/en/Online_Learning_History). The correlations between these
factors are (Chart 6.2.2-1):

Correlations between Time in Education,


Use of Computers and LMS

100
Number of Participants

22
18 Years in employment
13 14 15
10 10 9 Years using computers
7 Years using LMS

3
2 2

1 1
0 years Months 1-5 years 6-10 11-20 20+ years
years years
Time Scale

Graph 6.2.2-1. Correlations between time in education,


use of computers, and Learning Management Systems (LMS)

The previous graph presents that the use of LMS starts in early employment. This
result may be related to the short time LMS in education has been available; for
example Moodle@GSN was available to the public in August 2002. The use of
computers is in parallel with the years of employment. As for training on Information
and Communication Technologies (ICT) in education, most of the participants did not
attend courses in higher education but outside the formal Greek education system
(n=29, 72.5%). The vast majority did not have any training in LMS (n=26, 65%); 8
learned to use Moodle in an ICT course (n=8, 20%), and 3 within the official Greek
education system in graduate or postgraduate courses (n=3, 7.5%); one of the
respondents had 6-10 years and 2 had 11-20 years in education. As they were not
recent graduates, this means that their course was part of a life-long learning course
whereas LMS were not part of the curriculum. Lack of training and launching online
courses as initiatives on an individual basis has been reported as an e-learning
quality blocking factor (Fernandes & Montalvo, 2006).

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 143
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Familiarisation with the social aspects of the Internet: The majority of the
participants used the Internet on a daily basis: 13 are on-line all day (n=13, 32.5%),
23 once or twice a day (n=23, 57.5%) and 2 every 2 or 3 days (n=2, 5%). Before the
study, the respondents seemed to think that the use and importance of profiles and
discussion forums are significant in e-learning; a small ratio responded on the Likert
scale as low and very low although the ratio of not answering these questions was
relatively high (n=8-13, 22–32.5%). Most of the respondents considered the use and
importance of profiles and forums is of average importance (n=4-9, 10-22.5%),
relatively high or very high: 14 participants have used profiles before (n=14, 35%)
whereas 18 think they are important in e-learning (n=18, 45%); 21 participants have
used forums (n=21, 52.5%) and the vast majority believe in their importance in e-
learning (n=26, 65%). Despite the fact that the participants use the Internet, and
because the Internet is mostly based on textual communication, it appears that they
might not know ways to represent themselves and talk in online discussions.

Overall, the Greek teachers use the Internet regularly and participate in discussion
forums. On the significance of co-presence and e-learning management, they think
that profiles and discussions forums are of average importance. Their opinion was
checked in the final questionnaire and appeared to explicitly correlate communication
and the creation of profiles. Lastly, formal pedagogical training on ICT and LMS is
lacking. These results were similar to my observations and Tsetsilas’ findings (2006)
on Greek teachers’ lack of soft skills, lack of opportunities and professional training
incorporating current pedagogical approaches. In addition, Hartley (2007) suggested
the need for improving British teachers’ social and learning skills, self-instruction, and
their presentation on the web from a life long learning perspective. These results are
also reflected to the next findings.

6.2.3. Previous Knowledge of Collaborative


E-Learning Techniques & Participation

The Greek teachers replied to the question on collaborative e-learning


techniques and participation as follows (Table 6.2.3-1):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 144
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Table 6.2.3-1. Greek teachers’ knowledge and attitudes


on collaboration and participation

Collaboration & Participation

Yes No N/A
Freq. % Freq. % Freq. %
Do you know...
1 collaborative ways for e-learning? 11 27.5 20 50.0 9 22.5
Do you think that…
2 your participation is useful? 26 65.0 0 0.0 14 35.0
3 active participation is necessary? 31 77.5 0 0.0 9 22.5

The previous table presents that only 11 participants knew some e-learning
collaborative techniques (n=11, 27.5%); however, most of them either said they did
not know or did not answer (n=29, 72.5%). As for their participation, most of them
said that their participation is useful (n=26, 65%); a significant number (n=14, 35%)
did not answer the question. The majority replied that participation in e-learning is
necessary (n=31, 77.5%), none denied that participation is not necessary, and 9
participants did not answer the question (n=9, 22.5%). The N/A response on
collaboration and participation as well as on the use and importance of profiles and
forums is relatively high compared to the response on questions about personal
details (n=1-3, 2.5-7.5%).

Overall, despite the fact that the Greek teachers did not know any particular
collaborative e-learning techniques, they believed that their own and other e-learners’
active participation is of great importance. This implies a gap between knowing the
importance of participation but not being able to act upon it because of lack of know
how. This result is similar to other results (e.g. Beaudoin, 2002; Gulati, 2006) and this
lack of knowledge of how to work and learn together is inherent in the teaching and
learning modes for teachers’ training found in the literature review. This was found to
be the major difference between the online communities and the e-learning
communities, as the e-learning participants have a specific purpose: to learn. For this
reason, the e-learners were given explicit guidelines on: netiquette, the use of
emoticons, how to write online messages based on results from this project’s
preliminary studies; and the structure of the collaborative e-learning episode (see
Appendix X).

On the question about collaboration with the Greek educational authorities everyone
agreed on the absence of communication channels (Graph 6.2.3-1):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 145
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

The Greek educational authorities' communciation


with participants

35 34
33
84%
30
82%

25

20
Pedagogical Institute (N=40)
Ministry of Education (N=40)
15

10

5 5
5 13% 13% 2 5%
3%1
0
Some times Very few times Never

Graph 6.2.3-1. Communication with the educational authorities

The communication channels with the educational authorities are not open as most
participants have never been contacted by the Pedagogical Institute or the Ministry of
Education (average 83%). Some participants (average 16.5%) were approached a
few times by either authorities.

Overall, only two young Greek teachers took part in this study; despite my efforts it
was impossible to find reports on the workforce so no assumptions can be derived
from this result. A reason for this lack of consideration of the age of employment
comes from Katsaros and Karageorgiou on their study on the absence of the Greeks
primary teachers training for the environment in their area between 2001-2006
(2006); the age range had three parameters: up to 40 years old (22 participants), 40-
50 (47 participants) and over 50 years old (21 participants).

The majority use the Internet regularly and have an active online life. However, basic
knowledge of the Internet as a tool for pedagogical activities, how to represent
themselves online as well as techniques to collaborate are all lacking. In addition, the
little communication with the Greek educational authorities implies lack of
participation in any major changes introduced in the Greek education system as
regards the use of ICT in education.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 146
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

6.3 TRACING PARTICIPATION

Logging for e-learning environments provides a description of e-learners’


activities. Since particular modules to facilitate logging in Moodle do not exist, the
mdl_log file was extracted to create an Excel compatible Comma Separated Value
(CSV) file format. So logging will present a quantitative view on the frequency of
visits, as well as the number of active and passive participants. (My messages as the
observer participant in the study were not counted unless indicated otherwise. These
were general comments in Moodle@GSN and initiations to discussions in the
experimental environment.)

6.3.1 Frequency of Visits

This study targets participation in collaborative e-learning communities;


therefore, only the messages with at least one reply discussion depth were
considered appropriate for analysis. These messages were 175 from Moodle@GSN
(N=616) and 80 from the research pool (N=98); most other messages were either
announcements or members’ introductions without seeking a response from other
learners.

Views VS Posts: The activity in the two environments was (Table 6.3.1-1.):

Table 6.3.1-1. Moodle@GSN forums and users view log files

MOODLE@GSN LOGS (01/03-31/03/2007 – 31 days)


Average Percent
Type of Activity Activity
per day on total views
1 User view 20,799 671 47.4
2 View individual discussion 12,193 393 27.7
3 View individual forum 7,666 247 17.4
4 View all users 1,597 51 3.6
5 View all forums 1,555 50 3.5
6 Add forum/post 175 6 0.4
Total 43,985 1,418 100

Almost double the clicks were related to viewing other e-learner’s profiles and then
discussions and forums (45.1%) whereas ‘forum add forum/post was limited’ (N=175,
0.4%). As stated before, this was deliberately encouraged so that the e-learners
could know more about each other before participation started (Table 6.3.1-2):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 147
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Table 6.3.1-2. Forums and users view log files in the research pool

RESEARCH POOL LOGS (25/03-31/04/2007 – 7 days)


Average Percent on
Type of Activity Activity
per day total views
1 Forum view discussion 799 114 55.7
2 Forum view forum 427 61 29.2
3 View forums 86 12 5.8
4 Forum add forum/post 80 11 5.5
5 User view 39 6 2.8
6 User view all 30 4 2.0
Total 1,461 208 100.0

The logs related to forums (view discussion, view forum, and view forums) in the
research pool were of high priority (1,312 logs, 90.7%) whereas the users view was
limited (69 logs, 4.8%). This also means that the social reasons for passive
participation were almost eliminated at the beginning of the course.

Next, the total number of posts, and posts for analysis appeared as follows (the total
number of posts including my posts is in the parenthesis) (Table 6.3.1-3):

Table 6.3.1-3. Total number of posts

NUMBER OF POSTS
Percentage
Frequency Average per day
(N=714)
Moodle@GSN
1 (31 days) 616(850) 86.3 19.8
Message for Analysis 175 24.5 5.6
2 Research pool
98 13.7 14
(7 days)
Message for Analysis 80 11.2 11.4
Total 714 23.0 (31 days)
Total for Analysis 255 35.7 8.2 (31 days)

The previous table presents that the overall average posting per day was 19.8
messages for Moodle@GSN and 13.6 for the research pool. From the overall 616
posts in Moodle@GSN, 175 (28.4%) were suitable for analysis, whereas from the 95
messages in the research pool, 80 (84.2%) were analysed. A comparison is
presented (Graph 6.3.1-1):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 148
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Comparison: Number of Messages & Messages for Analysis

200
175
180
Number of Messages 160
140
120
Moodle@GSN
100
80 Experimental Environment
80
60
40 24.5
20 11.2 11.4
5.6
0
Analysed Messages Percentage on total Average per Day
number of messages (N=31)
(N=714)

Graph 6.3.1-1. Comparison between sent messages and messages for analysis

The number of selected messages for analysis is higher in Moodle@GSN: there


were 175 messages (24.5%) in Moodle@GSN and 80 (11.2%) in the research pool
with an average of 5.6 messages in Moodle@GSN and 11.4 messages in the
research pool posted per day. Active participation in Moodle@GSN was richer as
regards the discussion depth. However, the proposed tools and evaluation
techniques are suggested to be catalysts for active participation and collaborative e-
learning. It appears that about half of the messages were posted in the experimental
environment in a quarter of the time compared to Moodle@GSN.

Temporal View: A temporal overview of all activities can provide an in-depth view
(Table 6.3.1-4):

Table 6.3.1-4. Temporal overview of all activities

Observation Course Day INTERVENTION & EVALUATION


Community Management Tools
0 O0 Baseline 1 Moodle@GSN n/a
On date Until date
1 O1 01/03/2007 3,821 6,808 n/a
2 O2 03/03/2007 2,088 10,987 n/a
3 O3 07/03/2007 2,427 20,990 n/a
4 O4 14/03/2007 1,243 30,007 n/a
5 O5 22/03/2007 1,553 42,312 n/a
6 Baseline 2 On date Until date
7 O6 26/03/2007 365 44,771 501 501
8 O7 30/03/2007 505 46,793 387 3,796
9 O8 31/03/2007 176 46,969 364 4,160
10 Baseline 3

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 149
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

The previous table presents that the number of all activities in Moodle@GSN is
decreased as the course progresses. From almost 4,000 activities on the first day of
the course, there were 176 activities in the end of the course. Posted messages
appear as follows (Table 6.3.1-5):

Table 6.3.1-5. Temporal overview of posted messages (add post/forum)

Observation Course Day INTERVENTION & EVALUATION


Community Management Tools
0 O0 Baseline 1 Moodle@GSN n/a
On date Until date
1 O1 01/03/2007 38 76 n/a
2 O2 03/03/2007 33 152 n/a
3 O3 07/03/2007 32 313 n/a
4 O4 14/03/2007 11 416 n/a
5 O5 22/03/2007 11 567 n/a
6 Baseline 2 On date Until date
7 O6 26/03/2007 2 585 16 16
8 O7 30/03/2007 6 601 9 69
9 O8 31/03/2007 5 606 29 98
10 Baseline 3

Similar to activities, the posted messages decreased towards the end of the course
with some more messages in the research pool. There were 38 messages on the first
day of the course and 5 messages on the last day of the course in Moodle@GSN. As
the course continued in the research pool, there were 16 messages on the first day
and 29 messages on the last day of the course. A comparison between activity and
posted messages reveals the following (Graph 6.3.1-2):

Overall activity VS posting

10,000
3,821
2,088 2,427
1,243 1,553
1,000 866 892
540
Logs

100
38 33 32 34
18 15
10 11 11
Activity
1 Posted messages
01/03/07
03/03/07
05/03/07
07/03/07
09/03/07
11/03/07
13/03/07
15/03/07
17/03/07
19/03/07
21/03/07
23/03/07
25/03/07
27/03/07
29/03/07
31/03/07

Observation dates

Graph 6.3.1-2. Logs of overall activity VS posting

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 150
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Activity was significantly decreasing whereas posting was slightly rising in the
research pool. In fact, the posts on the last day (34 messages) were almost as many
as on the first day of the course (38 messages). Because the messages were related
to each topic and were not on social networking (e.g. saying goodbye or staying in
touch after the course), it might indicate that the e-learners had acquired enough
knowledge to finish the course with a good rate of quality posts. Another explanation
may be the Hawthorn effect.

Overall, online community management was initially focused on enhancing individual


actions (one way of communication) and social relationships (interactions) in order to
increase the feeling of presence. Then it was focused on enhancing participation
quality. This seemed to work; the logs revealed more activity on viewing the profiles
during the first days of the course that were diminished in the end whereas posting
was relatively stable and the messages quality was increasing. Lastly, the next
section examines participation.

6.3.2 On Participation

This section discusses active and passive participation. The calculation of


active participants was conducted by counting the members who posted at least one
message.

6.3.2.1. Active & Passive Participants

An overview of participants’ active and passive participation shows (Table


6.3.2.1-1):

Table 6.3.2.1-1. Number of active and passive participants

NUMBER OF ACTIVE & PASSIVE PARTICIPANTS


Participants Percentage
Total
Active Passive Active Passive
Moodle@GSN 59 36 95 62.1 37.9
Participants in both 34 6 40 85.0 15.0
Research pool 26 14 40 65.0 35.0

From the 59 active participants in Moodle@GSN, 34 of them participated in the


research pool. Overall, there were 26 active participants in the research pool; as for

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 151
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

the participants in both environments there were about 20% (N=14) more active in
Moodle@GSN. The next sections will try to shed more light in understanding
differences in active and passive participation as mere calculation may not provide a
coherent view.

6.3.2.2. Active Participation Levels

There were two phases in determining active participation levels as the initial
calculation was found not to be practical on a large scale.

Active Participation Levels I: The levels of active participation were calculated


according to the proposed model on low, medium and high participation, based on
the total number of sent messages (Table 6.3.2.2-1):

Table 6.3.2.2-1. Active participation levels (Initial proposition)

A. LEVELS OF ACTIVE PARTICIPATION (TOTAL NUMBER OF POSTS)


Active Participation Levels (%)
Percentage
(posts)
Total
Posts Low Medium High
(1-25%) (26-75%) (76-100%) Low Medium High
(1-154) (155-462) (463-616)
Moodle@GSN 616 616 100
(1-83) (84-250) (251-333) Low Medium High
Participants 333 333 100
(1-24) (24-71) (72-95) Low Medium High
Research pool 98 98 100

The previous table presents that the proposed model calculated on the total number
of sent messages was not functional on a large scale as all participants appeared to
be on the low activity level.

Active participation Levels II: A second attempt was made to calculate the
messages based on the highest respondent’s posts; as previously, the calculation
was anchored in the bell curve to determine low, medium and high participation
levels (Table 6.3.2.2-2):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 152
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Table 6.3.2.2-2. Active participation levels (Second proposition)

B. ACTIVE PARTICIPATION LEVELS (HIGHEST POSTER)


Active Participation Levels
Percentage
(% & highest participant’s posts)
Highest
Poster Low Medium High
(1-25%) (26-75%) (76-100%) Low Medium High
Moodle@GSN
(1-38) (39-76) (77-152)
54(P58) 152(P52)
32
P52:152 1 1 94.0 3.0 3.0
participants
participant participant
Low Medium High
Highest
(1-25%) (26-75%) (76-100%) Low Medium High
Research Poster (1-3) (4-9) (10-12)
pool
16 5 5
P50: 12 62.0 19.0 19.0
participants participants participants

The second attempt provided better practical results, There were 32 low, 1 medium
and 1 active participants in Moodle@GSN, and 16 low, 5 medium and 5 high
participants in the research pool. The highest poster sent 152 messages in
Moodle@GSN (P52), and 12 posts in the research pool (P50). Both medium (P58)
and high (P52) participants were e-tutors in Moodle@GSN. However, P58 posted 1
message and P52 did not post any messages in the research pool. With 12 posts, e-
learner P50 was the highest poster in the research pool, whereas he was a low
activity poster in Moodle@GSN sending 14 messages. P50 posted almost the same
number of messages in 31 days at Moodle@GSN and in 6 days at the research pool.
This means that P50’s posting activity was accelerated from low to high activity in a
week.

Overall, it appears that Moodle@GSN required significantly more e-tutoring than the
research pool and this was reflected in e-tutors and e-learners’ behaviour. Since
there was no e-tutoring in the research pool, active participation was e-learners
directed. Once more it appears that the initial ‘social’ kick on the participants and
instructional modifications to adjust e-learning behaviour to the new collaborative
tools worked. Lastly, the e-learner-generated text provided the context to some
participants to express themselves.

6.3.2.3. Passive Participation Levels

Calculation of passive participation was conducted on four levels (zero, low,


medium and high) based on the days of viewing; zero level is when a participant only
subscribed to the 2 environments and returned the 2 questionnaires; high passive

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 153
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

participation is when a participant appears to visit the course but never crossed the
threshold of participation. The course lasted 31 days; this means an average of 10
viewing days for each level. Tracking the exact hours and minutes spent on the
system was found to be extremely difficult. (There is a need to note that the graph for
the activity overview in Moodle was misleading, so all participants’ logs needed to be
checked and verified from the individual logs.) (Table 6.3.2.3-1):

Table 6.3.2.3-1. Passive Participation Levels

PASSIVE PARTICIPATION LEVELS (VIEWS IN DAYS)


Zero Low Medium High
Total
(0 days) (1-10 days) (11-20 days) (-31 days)
Moodle@GSN P51: 0 P47: 3 days
(31 days, P38: 7 days
6 participants, P19: 3 days
15%) P1: 4 days
P7: 8 days
Total 1 5 0 0 6
Percentage 17 83 0 0 100
Zero Low Medium High
Total
(0 days) (1-2 days) (3-4 days) (5-7 days)
P51: 1 day P21: 7 days P56: 6 days
Research P59: 2 days P40: 7 days
pool P16: 1 day P49: 4 days
(7 days, P43: 3 days P19: 3 days
9 participants, P29: 2 days
22.5%) P38: 2 days
P1: 2 days
Total 7 4 1 12
Percentage 0 59 33 8 100

There were 6 passive participants in Moodle@GSN (15%) in 31 days and 9


participants in the research pool in 7 days. P51 was the only participant who
remained in zero participation. All 5 of the Moodle@GSN e-learners remained on the
low level of passive participation (83%) whereas 7 (59%) were on low, 4 (33%) on
medium, and 1 on high passive participation. Overall, most participants were located
in low passive participation, not exceeding the 10 viewing days. There were different
passive participants in the two environments; P1, P19, P38, and P51 remained
passive in both environments (10%); 7 participants were active in Moodle@GSN
(17.5%); and 2 in the research pool (5%), these were the activated lurkers, this
means 33.3% of the passive participants. An overview of the activated passive
participants was as follows (Table 6.3.2.3-2):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 154
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Table 6.3.2.3-2. Posts from passive participants

PASSIVE PARTICIPANTS’ POSTING BEHAVIOUR


To: Moodle@GSN To: Research pool

Participants Posts Participants Posts


From: P47: Low 1
Moodle@GSN P7: High 1
P59: Low 1
P21: High 2
P43: Medium 2
From: P29: Low 3
Research pool P16: Low 3
P40: High 6
P49: Medium 11
P56: High 25

There were 2 Moodle@GSN low and high passive participants who posted 1
message each in the research pool. There were 3 low, 2 medium and 3 high passive
participants who posted in Moodle@GSN but not in the research pool. The tendency
of passive and active participation can be depicted in Graph 6.3.2.3-1:

Graph 6.3.2.3-1. Active & Passive Participation locus from the same participants

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 155
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

The previous graph presents that low, medium and high level passive participants
remained in the same level of active participation. In other words, a person who
observes a lot of participation is highly likely to be an active participant in a different
forum provided favourable circumstances to the e-learner allow the change of
behaviour. It also depicts the idiosyncratic character of the individual as found in
Nonnecke and Preece (1999), and stresses the importance of investigating the space
between passive and active participation if more participation in the e-learning
communities is desirable. This gap was found as the sleeper effect in the literature
review (Chapter 2.2.1).

Because the result of the increased passive participation in the research pool was
unexpected, I contacted the passive participants and asked them why they did not
participate. It appears that there were different reasons. P1 complained that time was
limited for him (personal communication via email 27/03/2007). P38 said: “I had
extreme difficulty with communication, I did dedicate many hours to build my profile
and in the end I didn’t make it. I couldn’t participate in any of the videoconferences
due to the Internet connection. Once I was asked to prepare a text and I wrote about
the water. I did receive the newsletters from time to time explaining exactly what I
had to do; however, I couldn’t follow either because I couldn’t find the specific link or,
when I could find the link, I couldn’t get into the course and I was wondering in the
classes.” (Personal communication via email, 01/08/2007). Looking into P38’s first
questionnaire, it appears that she was using computers for 5 years; she was not
trained, and never worked with LMS. O1 quit because of GSN technical problems,
and he got scared of the environment and the immediate responses of co-learners;
he said he needed to follow at own pace. Although O2 also quit, he sent a message
addressed to me in Youtube and said the following: “Professional work, kindness,
positive thinking was beyond any expectations. I was proud to be a Greek teacher;
however, I was angry because the Greek educational community does not take
advantage of its talents and knowledge of technology in many fields.”.

P51 visited the research pool for one day and never looked at Moodle@GSN. P1,
P19, and P51 reported that they did not have any spare time. P49, an active
participant in Moodle@GSN, appeared to have serious login problems in the
research pool and the log showed continuous failed logins, I contacted her and tried
to solve the technical problem (28/03/2007). The online course was extended by one
week, the week of participation in the research pool was the week before Easter; this
meant that participants were preparing for Easter or as in the case of P16 were in

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 156
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Europe on Comenius projects. These participants were not in fact passive. When the
participants were asked about the worst thing in the course, most said lack of time
(N=40, 12, 30%), technical problems (10, 25%), and being unable to participate as
much as they wanted (4, 10%). Overall, the main reasons were lack of knowledge
(both personal and technical) as to how to collaborate online (Table 2.3.2.2-1), and
perhaps this prevented the participants from engaging in the way they wanted.

This section discussed active and passive participation; the overall participation was
decreasing and increased slightly at the end of the course. Active and passive levels
of participation provided a technique for coherent and accurate measurement that
can be used by the e-tutors to support the e-learners. Furthermore, the messages for
analysis in the research pool were found to have more replies than the ones in
Moodle@GSN. One way or another, numeric analysis found inadequate for in-depth
messages analysis and as a means to determine their quality. This was however
feasible with the Collaborative e-Learning Episodes analysed in the next section.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 157
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

6.4 THE COLLABORATIVE E-LEARNING


EPISODES

Messages analysis was used to determine the number and quality of the
Collaborative e-Learning Episodes (from now on CeLE) and aimed to:

• distinguish mere provision of information and collaborative e-learning


as one measure of e-learning quality;
• find correlations between passive - active participation and
collaborative e-learning;
• provide some evaluation towards the pedagogical usability and use of
the new collaborative tools; and
• evaluate the CeLE updated version.

The proposed CeLE analytical framework considers 4 variables, 3 quantitative and 1


qualitative. The quantitative variables are: richness of text, discussions depth, and
density. The CeLE structure as such is the qualitative variable. All messages from
both Moodle@GSN and the research pool were analysed. All images and Urls were
considered in the analysis and counted as one item. (My messages were not counted
unless indicated otherwise.)

6.4.1. Quantitative Variables

Comparison of the messages quantitative analysis in the two environments


(see data in Appendix XI) led to the following results (Graph 6.4.1-1):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 158
In
t ro

Ph.D. Thesis
du
cti
Ti on
m (0
So et 1-
cia ab
le 21

1000
10000

1
10
100
ln (0 /0
et 1 3/
wo – 20
rk 28 07
ing /0 )
M (0 3/
oo 1 20
-2 07
dl
e 4/
03 )
Pr (0/2
Pr ob 1 -0
00
oje lem 8/
ct
sA s (0
03 7)
/2
rc 1- 00
hi 0 8/ 7)
ve 05
( /2
B l 01- 00
og 07 3)
s( /0
01 3 /
To -3 200
Bl o 1 /0 7
og ls 3/ )
& (0 2

Centre for Interactive Systems Engineering LSBU


HT 3- 00
12 7)
M /0
Pr L 3/
ob (0 20
lem 5-1 07
2/ )
s( 03
De 12 /2
si g -2 00
3/ 7)
Pr n 0 3
ac (1
7- / 20
tic 18 07
ali
ty / 03 )
Te Gr (1 /2
00
ch o u 3 -1 8 7)
ni c ps /0
3/
Subjects & Dates

al (1 2
Pr 8-
ob 24 0 0 7
lem /03 )
s /2

Page 159
G r (1 00
Pr ou 6-2 7)
oje p 2
Tools and Evaluation Techniques for Collaborative E-Learning Communities

ct s /
id (1 0 3 /
ea 8- 0
s 23 7)
(0 /0
1- 3/
Ot 07
h e 0 6/
0 )
r( 3/
0 2
VC 1- 0
in N e 18/ 07)
0
E- ws 3/
lea
rn ( 2
20
07
8-
in
g 3 0/ )
(2 03
8/
03 /0
7)
-0
1/
Chapter 6: Main Study

04
/0
Correlations between Messages for Analysis, Richness of Text, & Discussion Depth

7)
Richness of Text

Graph 6.4.1-1. Comparison between messages for analysis, richness of text, and discussion depth
Messages for Analysis

Discussion Depth (E-Tutors)


Discussion Depth (Participants)
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Graph 6.4.1-1 provides a comparison between messages for analysis, richness of


text as well as the discussion depth related to both e-learners and all participants (e-
learners and e-tutors). Richness of text and discussion depth involved counting the
messages and the words included. All these variables were found to be interrelated;
in other words, when the text was rich, then there were more messages for analysis
this means having more than one replies and the higher the discussion depth.

Density was calculated with Fahy and colleagues’ density formula, 2a/N(N-1)
(a=interactions, N=number of participants). Messages density was almost double in
Moodle@GSN (0.19) compared with the research pool (0.1). However, Fahy’s
formula does not consider interaction time, and is highly sensitive to the size of the
group. Quantitative variables can give a specific picture of the research context;
however, there is a need to investigate its quality.

6.4.2. Qualitative Variables

E-learning quality in this study was measured by the Collaborative e-Learning


Episode (CeLE) analytical framework. A CeLE is an argumentation cycle with starting
points (social cues, information and questions), middle (explanations, explorations,
agreements, disagreements, and evaluations), and ending points (summaries, social
cues, and silence). A productive CeLE occurs when the participants introduce a new
idea. There were 255 messages suitable for analysis, 175 in Moodle@GSN and 80 in
the research pool. These messages were categorized and inserted in Atlas-ti™ in
order to find the messages that indicated at least one CeLE. The analysis suggested
thirteen CeLEs (Table 6.4.2-1):

Table 6.4.2-1: Collaborative E-Learning Episodes Overview

COLLABORATIVE E-LEARNING EPISODES


CeLEs in Moodle@GSN
CeLEs
Section Forum # Threads
# Thread # Words
A Introduction 1 Introduction 58 I #18/ 10(3)-PX2 601(8)
Total 5 2
II #25/ 8(1)-PX1 623(11)
III #29/ 7(1)–P18 133(6)
C Blogs 1 Blogs 61 IV #46/ 5(1)-P2 276(93)
V #50/ 18(2)-P52 1,782(82)
VI #54/ 6(1)-P15 526(9)
Total 3 1

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 160
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

CeLEs in the Research pool


1 News 8 VII #4/ 3-P13 725
VIII #3/ 18-R 2,213(9)
IX #4/ 10-P37 882
VC in E-
G VC in E- X #6/ 8-P50 941
Learning 2 9
learning XI #7/ 22-R 3,990(86)
XII #8/ 5-R 474(27)
XIII #9/ 6-R 153(18)
Total 2

It appears that in regard to Moodle@GSN, 1 CeLE was found in the introduction


forum, and 5 in the blogs discussion. Similarly, there was 1 CeLE in the news forum
and 6 in the videoconferencing discussion in the research pool. I initiated 4
discussions in the research pool and none in Moodle@GSN. P52 was also an e-
tutor; this means that the richest in words CeLEs were initiated by e-tutors.

CeLEs overview in the two environments is presented next in relation to individual


posters and duration in days (researcher’s posts in parentheses) (Table 6.4.2-2):

Table 6.4.2-2. Collaborative e-Learning Episodes selected for analysis


FINAL CeLEs
Moodle@GSN
# Individual Duration
CeLE # Dates # Words
Posters (days)
I 4(1) 01-05/03/2007 5 601(8)
II 6(1) 06-06/03/2007 1 623(11)
III 5(1) 05-05/03/2007 1 133(6)
IV 5(1) 03-03/03/2007 1 276(93)
18-22/11/2006 &
V 9(1) 11 1,782(82)
25/02-02/03/2007
VI 4(1) 28-28/02/2006 1 526(9)
Total 3,941(209)
Research pool
VII 3 27-29/03/2007 3 725
VIII 10(1) 21-31/03/2007 10 2,213(9)
IX 7 29-31/03/2007 3 882
X 6 27-29/03/2007 3 941
XI 17(1) 26-29/03/2007 4 3,990(86)
XII 5(1) 26-29/03/2007 4 474(27)
XIII 4(1) 24-28/03/2007 5 153(18)
Total 9,378(140)

CeLEs in the research pool were more than twice as rich as in Moodle@GSN, with
3,941 words (30%) in Moodle@GSN and 9,378 words (70%) in the research pool. It
appears that the richest CeLEs last on average 4 days and in 2 occasions the richest
CeLEs were open 10 and 11 days rather than the average of 4 days. They become

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 161
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

richer as the course is unfolding related to the number of days and the number of
words. For example, the CeLE-I lasted 5 days and had 601 words; and more CeLEs
appeared towards the end of the course. It is interesting to note that there is a gap
after the first week; the CeLEs stopped on 06/03/2007 and appeared again on
18/03/2007. The CeLEs’ temporal overview is presented next (Table 6.4.2-3):

Table 6.4.2-3. Collaborative e-Learning Episodes temporal overview

Observation Course Day INTERVENTION & EVALUATION


CeLE in Moodle@GSN Tools & CeLEs
0 O0 Baseline 1
1 O1 01/03/2007 n/a
2 O2 03/03/2007 n/a
3 O3 07/03/2007 I, II, III, IV n/a
4 O4 14/03/2007 n/a
5 O5 21/03/2007 V & VI n/a
6 O6 Baseline 2
7 O7 28/03/2007 VII, X, XI, XII, XIII
8 O8 31/03/2007 VIII, IX

Most of the Collaborative e-Learning Episodes in Moodle@GSN were technical


enquiries and most of the time the discussions were developed in small chunks. For
example CeLE III had three chunks of smaller enquiries that led to answers to
questions unfolding within the CeLE. The first four CeLEs were initiated in the first
week and then a 2 weeks gap appears. These first CeLEs were on getting to know
each other as well as technical problems; the last ones were of more quality. For
now, an overall comparison on duration in days, number of words, threads and
individual posters can also provide a posting timeline (Graph 6.4.2-1):

CeLEs: Comparison between Duration (days) and number of


10000
Words, Threads & Indivudual Posters
3,904
1,700 2,204
Number of words

1000 725 882 941


593 612 517 447
183 135
100 127

18 18 22
16
10 10 11 10
9 10
8 7 8 6 7 8
6 6
5
4 5 4 5
4 4 5
4 5
3 3 3 3 3
1 1 1 1 1
CeLE-I CeLE-II CeLE-III CeLE-IV CeLE-V CeLE-VI CeLE-VII CeLE-VIII CeLE-IX CeLE-X CeLE-XI CeLE-XII CeLE-XIII

Duration 5 1 1 1 11 1 3 10 3 3 4 4 5
Words 593 612 127 183 1,700 517 725 2,204 882 941 3,904 447 135
Threads 10 8 7 5 18 6 3 18 10 8 22 5 6
Posters 4 5 4 4 8 3 3 9 7 6 16 4 3

Graph 6.4.2-1. CeLEs factors’ comparison graph

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 162
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

The previous graph presents that there are interrelations between the factors in a
successful CeLE. Since a CeLE is related to co-creativity and new knowledge
construction, idea generation is found to be related to all factors. In other words,
when duration is in days, number of words, threads and individual posters reach a
peak, and idea generation reaches a peak.

However, the aim of collaborative e-learning is e-learners’ generated text; for this
reason the same analysis is presented with and without e-tutors’ contributions (Table
6.4.2-4):

Table 6.4.2-4. CeLEs: e-learners and e-tutors’ contributions


FINAL CeLEs
Moodle@GSN
# E-learners – Duration # Words
CeLE # E-tutors
Unique # posts (days) E-learners E-tutors
1 I 2 4 5 396 205
2 II 3 6 1 455 168
3 III 2 5 1 93 40
4IV 2 5 1 192 84
5 V 5 9 11 726 1,056
6VI 2 4 1 384 142
Total 2,246 1,695
Percentage 53% 47%
Research pool
7 VII 3 3 3 725 0
8 VIII 10 10 10 2,213 0
9 IX 5 7 3 715 167
10 X 5 6 3 905 36
11 XI 15 17 4 3,204 786
12 XII 5 5 4 474 0
13 XIII 4 4 5 153 0
Total 8,389 989
Percentage 89% 11%

It is evident that the e-learners were more active in the research pool. In
Moodle@GSN there were 2,246 words from the e-learners (53%) and 1,695 (47%)
from the e-tutors. As for the research pool, there were 8,389 words from the e-
learners (89%) and 989 (11%) words from the e-tutors. This difference is significant
as illustrated in the next graph (Graph 6.4.2-1):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 163
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Number of words posted in CeLEs


by E-learners & E-tutors
89%
9000 8389
8000

Number of Words 7000


6000
5000 E-learners
4000 E-tutors
57%
3000 2246 43%
11%
2000 1395
989
1000
0
1 2
Moodle@GSN Experimental Environment

Graph 6.4.2-2. Comparison for number of words posted by


e-learners and e-tutors

The previous graph presents that the e-learners’ ratio of words in posting was
increased by 32% in the research pool CeLEs. To conclude the section on the
qualitative variable, 2 CeLEs were selected, from Moodle@GSN and the research
pool. The two environments were viewed as complementary so selection was based
on the most interesting and representative examples, one from each environment,
CeLE-III from Moodle@GSN and CeLE-IX from the research pool. CeLE-II and
CeLE-IX found to be similar as regards the number of words as well as the number of
e-learners and posters. In addition, CeLE-VIII seems to be interesting for
investigation since there were 10 messages sent by 10 different individuals, all being
low active participants. The last CeLE detailed analysis is presented in Appendix X
(A_X_1-4).

In summary, information exchange in CeLE III had a linear structure which unfolded
as a problem solving activity on a technical problem. On the other hand, CeLE IX did
not have a linear structure and referred to the use of specific e-learning tools in
building a project. In CeLE IX there were more agreements, disagreements,
arguments, and exploration of each other’s ideas. These were expressed both as
monologues as well as threaded dialogue. These processes can be seen as
knowledge internalisation and externalisation through monologue and dialogue which
promoted participants’ critical and creative thinking (Appendices A_X_5 and A_X_7).

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 164
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

As there was only one participant in a medium activity level and seven participants
on a low activity level, the appearance of these structures with the aid of previous
knowledge on collaborative e-learning and even the limited use of MessageTag may
be reasons for tackling passive participation (e.g. Klemm, 1998; Khine et al., 2003;
Jeong & Davidson-Shivers, 2006). The use of tools to reveal these structures even
on a limited level, can be indicated by the fact that previous knowledge existed in
Moodle@GSN; however, the CeLEs were not as rich, varied and consistent as in the
research pool. Knowing, viewing, and carefully using the collaborative e-learning
technique can give confidence to and encourage e-learners in their participation. It
appeared that there were transitions between internalisation and externalisation of
learning and this means that design should explicitly support dialogical sequences by
broadening and deepening this space.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 165
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

6.5 THE SENSE OF E-LEARNING


COMMUNITY INDEX

The Sense of e-Learning Community Index (SeLCI) aims to provide in-depth


insights to human-human interactions within e-learning communities. The SeLCI
attributes are:

1. Community evolution: initial opinions on the community evolution, shared


interests and values, knowing about the community, new members’
contribution, and the collaborative tools;
2. Sense of belonging to the e-learning community;
3. Empathy as a representation of what co-learners know and feel;
4. Trust: knowledge exchange, help and support; and
5. Intensity: levels of passive and active participation, and persistence;
6. Collaborative e-learning quality: participants’ opinions, and the number of
collaborative e-learning episodes.
7. Social Network Analysis
o Global cohesion: density, reciprocity, cliques, and structural
equivalence;
o Global centrality: centrality degree, closeness and betweenness; and
o Local Real-Time Nodes and Centrality: tools results.

Quantitative, thematic, and social network analysis were employed within the
ethnotechnological framework to evaluate the SeLCI. As before, the questionnaires
from 40 participants (N=40) were analysed. The data for quantitative analysis were
inserted in the Statistical Package for the Social Sciences (SPSS 11.5) and Microsoft
Excel. Thematic analysis was conducted for the open questions and the collaborative
e-learning episodes using Atlas-ti™. Finally, social network analysis was aided by
UCINET (Borgatti et al., 2002).

The next sections will explore the SeLCI starting with the community evolution.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 166
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

6.5.1 Community Evolution

The first SeLCI attribute is related to understanding the community evolution,


shared interests and values, knowing about the community, new members’
contribution, and the community roles.

Evolution: Almost all participants (n=38, 95%) thought that the community was
developing; 2 responses were N/A (not applicable) so there was not an opposing
opinion. As regards the time needed for community evolution, most of them (n=17,
42.5%) said it needs some time to develop: 2 weeks, 13 (32.5%) 1 week, 10 about 4-
5 days (10%) and the rest 6 (15%) 1-3 days. If collaborative e-learning started
developing after the first week, this means 1 to 2 weeks is necessary for developing
a community and collaborative e-learning where the sleeper effect is considered to
occur. In other words, the first week the participants explore and familiarise
themselves with the system and the other learners and make decisions upon passive
or active participation.

Participants were asked to suggest elements that showed community evolution on an


open question (Graph 6.5.1-1):

Increasing interactivity / help


Community Evolution
Communication outside and
Number of responses (N=62) after the course
Common interests / goals
0 2 4 6 8 10 12 14
Online communication
19% 12 Affective elements
13% 8
Quick familiarisation
Communtiy evolution elements

11% 7
10% 6 Increasing number of
6 participants
Communication – general
6% 4
4 Collaborative atmosphere
5% 3
E-learners’ participation in
1 2 planning
3% 2 Sense of belonging
2
Personal messages /
2 experiences / first name
1 New colleagues
2% 1
Photos
1
1 Visiting others’ web pages
1
Knew nobody initially

Participation quality / quantity

Graph 6.5.1-1. Community evolution elements

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 167
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

The previous graph presents that the most important community evolution elements
were the increasing participation based on mutual help (n=12, 19%) as well as
increasing communication outside and after the course (e.g. phone, SKYPE, blog,
Facebook; n=8, 13%); then shared goals and interests (n=7, 11%) communication
during the course (n=6, 10%); affective elements such as trust and support
(n=6,10%); the quick familiarisation with the e-learning environment (n=4, 6%), the
increasing number of participants (n=4, 6%); the level of communicative activities
(n=3, 5%); and from 2 responses (3%) the collaborative atmosphere, participation in
course planning as adjustment to particular circumstances, the sense of belonging
and expressions of familiarisation e.g. exchange of personal messages, experiences,
and address of e-learners by their first name (n=2, 3%). Other elements (1 response,
2%) were on meeting new colleagues, the number of photos in profiles, visiting each
other’s personal web pages and adding each other’s links, some people did not know
anyone initially, and the quantity and quality of participation.

When they were asked whether they would continue their collaboration outside the
course, 24 said yes (60%), nobody said no; however, there were 16 N/A and missing
responses (40%). Willingness to keep the community going was also evident on their
demand for a blog, a Facebook group be notified if a new course will start. However,
because this is not organised by GSN and the online course was part of this
research, so this request was not feasible.

The increasing interactivity and participation indicated that the community was an
evolving organic entity. This means that community evolution can be moderated by
supporting communication and participation in collaborative activities by helping e-
learners becoming passive and then active participants. For example, the use of
profiles and the initial social interactions to enhance trust and empathy supported e-
learners’ quick familiarisation with each other as well as with the LMS. Having
common targets and discovering similar interests creates a tendency known as self-
disclosure reciprocity as the participants exchange personal information and
experience (Wallace, 1999). The use of other communications means (e.g. phone,
SKYPE, blogs, and Facebook) other than the tools inside the e-learning system has
been found crucial for the community maintenance (Boase et al, 2006), and indicates
a healthy practice in this study. This may also suggest that different people have
different desires when it comes to communication modes.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 168
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Shared Interests & Values: Shared interest was found to be the third community
evolution element. Moreover, most participants replied to the closed question that
they had same interests (n=35, 87.5%), 2 denied (5%) and 2 were N/A (5%). Most
also said they had shared values (n=32, 80%), although 1 denied (2.5%) and 7 were
N/A (17.5%). Following the literature (e.g. Preece, 2000), shared values and interests
is the most important indicator to identify whether a community existed or not. In this
study these results were more than 80% which means that there was a strong
common ground from the participants’ viewpoint, an essential element for
collaborative e-learning.

Knowing the community: The participants said the following about the community
(Table 6.5.1-1):

Table 6.5.1-1. Knowing the community


1 2 3 4 5
THE COMMUNITY: N/A
Very
Did you... Neither
small

1 know the netiquette? 2 5 1 2.5 6 15 17 42.5 14 35


know the kind of
2 11 27.5 4 10 12 30 13 32.5
community?
like working
3 2 5 5 12.5 17 42.5 16 40
together?
express yourself
4 5 12.5 5 12.5 15 37.5 14 35 1 2.5
freely?
5 participate actively? 6 15 7 17.5 8 20 13 32.5 6 15
think that the LMS
6 helped the 5 12.5 14 35 21 52.5
community?
think that the new
7 tools helped the 5 12.5 13 32.5 20 50 2 5
community?

The respondents liked working together (n=33, 82.5%) and were expressing
themselves freely (n=29, 72.5). The LMS provided the platform for the community to
exist (n=35, 87.5) and the new tools significantly helped in this process (n=33,
82.5%). Most of the participants thought that they knew about the netiquette (n=31,
77.5%) as well as the community (n=25, 62.5%). On freedom of expression the
results were lower (n=11, 27.5%) as well as active participation (n=7, 17.5%). Also,
half of the participants knew nobody before the course and the other half knew a few;
28 (70%) said they developed online relationships, 10 (25%) did not (2 were N/A).

Although the majority of the participants did not have experience in e-learning
communities and online collaboration, they were positive on collaboration and the

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 169
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

only thing that they were not happy about was not to be able to participate as much
as they wanted. Using tools in e-learning was important to them; knowing what the
community was about and developing relationships was a strong determinant in
SeLCI.

New members’ contributions: The results from the closed question on new
members’ contribution were in favour of new members: 34 (85%) supported their
contribution, 6 (15%) were N/A and nobody opposed. Thematic analysis on the e
open question also revealed the following (Graph 6.5.1-2):

Graph 6.5.1-2. New members’ contributions

The respondents believed that the new members can regenerate the community by
bringing new ideas (n=16, 24%) to share with the older members (n=9, 14%) via
active interaction and participation in the community (n=6, 9%). The most important
element is their enthusiasm (n=5, 8%). The participants believe that heterogeneous
groups function better (n=4, 6%) as they bring up questions (n=4, 6%). Other
responses refer to the need for training (n=3, 5%) and to keep the community going
(n=3, 5%); from two responses each (n=2, 3%) are freedom of expression, bringing
new perspectives and abilities into the community, provide feedback for older
members, whereas willingness to learn and collaborate is essential as well as the
use of new technologies. Shared interest, exploration with propositions and criticism,
quality and life-long learning are the last elements on new members’ contribution.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 170
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Moreover, P7 and P9 made interesting suggestions. P7 said that new members


‘…contribute their own knowledge that most experienced members can take
advantage of depending on e-learning targets. Based on their questions, more
experienced members can elaborate and analyse the coming data to transform the e-
learning context and future actions as interaction that can only bring positive results.’.
To P9, ‘old members can identify the new knowledge coming from newbies and help
in its transfer in positive ways’.

Additionally, two issues were revealed with regard to the new members in an e-
learning community. Despite the evidence on the importance of newcomers bringing
new ideas to regenerate the community, there was only one young participant in the
study; however, there were older newcomers. This means that the Greek teachers
even as newcomers have not recently graduated so there is a gap in their
professional training especially when there is a demand for faster training cycles in
recent years. In other words, the lack of training in the use of ICT in Education and
knowledge of collaborative techniques may be due to absence of teachers’ life-long
learning courses in Greece (Tsetsilas 2006). In addition, it is evident that new ideas
are of major importance to keep the community going; this is another collaborative e-
learning beneficial factor to community evolution.

Roles: The participants said that there were roles assigned to the members.
According to the closed question, 23 (57.5%) thought there were roles, 6 (15%) there
were not, and 11 (27.5%) thought there were no specific roles. However, thematic
analysis on the open question revealed the following (6.5.1-3):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 171
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Graph 6.5.1-3. Roles in the e-learning community

Participants’ responses were on community management, e-learning, and


participation. The moderator’s role was found important (n=7, 18%) along with
motivators (n=5, 14%) and leaders (n=4, 11%). There were also teachers (n=5,
14%), students (n=6, 17%), and technical support (n=2, 6%); active participants (n=2,
6%) and observers (n=1, 3%) were also included. Based on the assigned role of the
moderator and motivator, it appears that organisation of the community and e-
learning were of equal importance. As for the extent of provision of help that can
define roles between community members, 21 (52.5%) said that was neither great
nor small, 13 (32.5%) very great, 4 great (10%) and 2 (5%) to a small extent.

Overall, community management, e-learning and participation were themes for roles
which could be assigned. Equality in participation was also indicated by the fact that
e-tutoring and moderating was evident in Moodle@GSN and did not exist in the
research pool. In other words, the assignment of roles was not found to be a very
strong element in community evolution and collaborative e-learning.

6.5.2 Sense of Belonging

On the question of bonding or togetherness 16 respondents said it was strong


(47.5%); 18 said it was neither strong nor weak (45%); 1 said it was not strong

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 172
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

(2.5%); and 2 were N/A (5%). In the question on the factors that kept the community
together they said (N=40) (Graph 6.5.2-1):

Graph 6.5.2-1. Participants’ opinions on e-learning community elements

More than half of the participants (n=22) said that sharing the same goals and
interests was the main reason for holding the e-learning community together (n=22,
30%); then curiosity for new knowledge (n=12, 16%); and equally e-tutors (n=5, 7%)
and collaborative tools (n=5, 7%). More responses were: desire for success and
communication (n=4, 5%), willingness to participation, enthusiasm, participation as
such and effective learning (n=3, 4%), affective elements such as mutual help and
trust, the subject, collaborative learning for projects development and problem
solving (n=2, 3%). Other suggestions (n=1, 1%) were the immediate success and
feedback as well as reliability.

Furthermore, P11 believed that something that shows the sense of community is
active participation despite the technical problems; to P35 there was an ‘incredible
increase of active participation’; to P22, there was ‘willingness to collaborate beyond
and after the online course’, to P39 there was a ‘desire to keep the community going’,
and to P38 and P47 there was mutual help, understanding, and trust. P14 said that
concerning the use of profiles that the ‘photos created a climate of familiarisation with
each other and helped in developing a sense of belonging’.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 173
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Overall, it seems that the same factors that defined the sense of belonging are the
factors that contribute to the e-learning community evolution: shared interests and
values. Also, new knowledge is the collaborative e-learning outcome. The mediators
in this process are the e-tutors and the tools.

It is interesting to see an overview of the importance the participants applied to the


observed clusters grouped in three themes (Graph 6.5.2-2):

Correlations between Codes and Resposnes

100

48

21
Codes
10 10
Responses
6
5

1 1
Community Management e-Learning (35%) Technology (6%)
(59%)
(Percentage on codes)

Graph 6.5.2-2. Comparison of themes and community elements

Community management is considered the most important factor in this study (59%),
then e-learning (35%), and third the tools (6%). In other words, community
management facilitates community evolution and thus allows e-learning to take
place. Learning is an important element of community evolution and vice versa,
meditated by artefacts; in this case e-learning tools (Lave & Wenger 1991).

6.5.3 Empathy

Table 6.5.3-1. Empathy factors

EMPATHY FACTORS
Know what Other Feel what Other was
was feeling when feeling when reading Action
EXTEND reading a message a message
Number Percent Number Percent Number Percent
Very Small 1 2.5 2 5 11 27.5
Small 8 20 6 15 7 17.5
Neither 17 42.5 15 37.5 9 22.5
Great 13 32.5 15 37.5 11 27.5

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 174
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Very Great 0 0 2 5 2 5
N/A 1 2.5 0 0 0 0
SD 0.9 0.9 1.3
Total 40 100 40 100 40 100

On the question of knowing what someone else was feeling (42.5%) replied they
more or less knew what the other person was feeling when reading a message and
13 (32.5%) they knew to a great extent; 8 (20%) said they knew to a small extent, 1
very small (2.5%) and one response (2.5%) was N/A. On “feeling what the Other was
feeling”, the responses were equally distributed between neither great or small and
great extent (n=15, 37.5%); 6 (15%) said they could feel other’s feeling to a small
extent, and equally 2 (5%) were on the very small and very great scale. As for
whether they took any action, 11 respondents (27.5) equally said to a very small and
great extent; 7 (17.5) to a small extent and 2 (5%) took action to a very great extent.

It appears that the results reach a peak in “knowing what the other person was
feeling”, they are more distributed on “feeling what the other person was feeling”, and
they reach a down peak in “action taking”. A scatter plot provided a more detailed
view (Graph 6.5.3-1):

Graph 6.5.3-1. Scatter plot for empathy

Convergence of the three empathy parameters when reading a poster’s message,


these are “knowing what Other was feeling”, “feeling what the Other was feeling” and
“action taking” seem to be on both a small extent and a great extent. Plotting
suggested a linear relationship between the three variables, so significance
evaluation was conducted with Pearson's Correlation Coefficient (Table 6.5.3-2):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 175
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Table 6.5.3-2. Correlations for Empathy factors

Correlations for Empathy Factors


Feel what Other
Feel Other
was feeling
when reading Action
when reading a
a message
message
Know what Pearson Correlation 1 .489(**) .347(*)
Other was
feeling when
reading a Sig. (2-tailed) . .001 .028
message
Feel what Other Pearson Correlation .489(**) 1 .527(**)
was feeling
when reading a Sig. (2-tailed) .001 . .000
message
Pearson Correlation .347(*) .527(**) 1
Action
Sig. (2-tailed) .028 .000 .
N 40 40 40
** Correlation is significant at the 0.01 level (2-tailed).
* Correlation is significant at the 0.05 level (2-tailed).

The strength of association between the three variables is positive but it does not
suggest a strong linear relationship. A moderate association (r=0.527) is between
“Feel what Other was feeling when reading a message” and “Action” with correlation
coefficient p≤0.01; “Feel Other” and “Action” association was the weakest (r=0.347)
with correlation coefficient p<0.05. Cronbach's alpha, as a coefficient of reliability and
consistency between the variables found α=0.698, which is on the limit of acceptance
in social research (0.70). Although empathy has been related to gaze and body
language (e.g. Lanzetta and Englis, 1989) it seems that it can occur online and it is
influenced by the properties of different communication media (Preece, 2004). It was
not possible to extract explicit relationships between the results of this study and
others as the frameworks for investigating online empathy were different. In this
study, the respondents appeared to know what the other person was feeling when
posting more than actually feel the poster. Nonetheless, in regard to action taking,
people may need time to actively participate in activities (the sleeper effect) before
consciously decide to work with others (the least collaborative effort).

There is a need to note that empathy has been related to the mirror neurons; this
means that the e-learners in their profiles provided information to the mirror neurons
to build representations of the other learners and their actions (Goleman, 2007).
However, investigation of such correlations was not part of this study.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 176
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

6.5.4 Trust

The participants responded to statements on trust (Table 6.5.4-1):

Table 6.5.4-1. Trust levels

TRUST
Statements Yes Percent No Percent N/A Percent
I can trust most of the
1 37 92.5 1 2.5 2 5
participants.
I have to be very careful as some
2 37 92.5 3 7.5
take advantage of others.
3 Most are trying to help. 38 95 2 5
4 Nobody can trust anyone else. 38 95 2 5

The contradictory closed questions on trust suggest high level of participants’


reliability; the majority (over 90%) responded that they could trust the other learners
as most were trying to help. So trust was evident to a significant extent. More
specifically (Table 6.5.4-2):

Table 6.5.4-2. Trust development towards individuals


1 2 3 4 5
TRUST Very Very
Small Neither Great N/A
small great
Individuals who had
1 a similar 11 27.5 16 40 12 30 1 2.5
specialisation
Individuals who had
2 12 30 16 40 12 30
similar writing skills
3 E-tutors 1 2.5 9 22.5 30 75
Experienced
4 9 22.5 16 40 15 37.5
individuals
You can’t trust
5 32 80 6 15 1 2.5 1 2.5
anyone

It appears that most of the participants could trust other e-learners with a similar
specialisation to a great (n=16, 40%) and very great extent (n=12, 30%) whereas 11
(27.5%) could trust them more or less; as for individuals who had similar writing
skills, the results indicate trust to a great (n=16, 40%) and very great extent (n=12,
30%) whereas 12 (30%) could trust them more or less. The results on trusting the e-
tutors were the most positive with 39 (97.5%) to trust them on a great or very great
extend and 1 neither a small or a great extent (2.5%). As for trusting the experienced
individuals the results were: 16 (40%) to a great extend, 15 to a very great extent
(37.5%) and 9 (22.5) neither small or great extent. The question to ensure that the

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 177
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

participants did not respond randomly revealed that the majority (n=32, 80%) could
trust other people.

Lastly, all but 1 participants reported that the level of trust had risen (n=39, 97.5%).

Almost all participants felt that they could trust the e-tutors and their co-learners.
Trust has been reported to be related to different forms of awareness, such as
personal information, presence on the community, demographic backgrounds,
capabilities and skills in performing specific tasks (Daniel, 2007:124). Trust is also
related to the mental models people develop when they first meet as well as the
content of their conversation and tend to develop very quickly (Norman, 1988).
However, these suggestions were not explicitly investigated in this study; here, trust
was linked to reciprocity and levels of participation to allow participants to work freely
together, evident in the freedom of expression to a great extent (n=29, 72.5%).

6.5.5 Intensity

Intensity refers to the participation levels and persistence; the latter is the
level to which participants pursue topics.

Participation Levels: Active and passive participation levels were found to vary in
the two environments. Seeing these levels as a process from low passive to high
active provides a different overview for intensity (Graph 6.5.5-1):

Passive & Active Participation Process

35
32
30
Number of participants

25

20
A: Moodle@GSN
16 B: Research Pool
15

10
7
5 5 5 5
4
1 1 1 1
0
Zero Low Passive Medium High Passive Low Active Medium High Active
Participation Passive Participation Participation Active Participation
Participation Participation

Graph 6.5.5-1. Passive & Active Participation Process

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 178
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Line A corresponds to Moodle@GSN and is interrupted; the participants were located


in zero and low passive participation, high low active participation and one e-learner
in medium and active participation. As described in the previous sections, this may
be because medium and high activity users were e-tutors whereas there were no e-
tutors in the research pool. Line B corresponds to the research pool and seems more
broadly distributed showing higher activity than Line A. In other words, from an
activity viewpoint, intensity in the research pool seemed to be in greater balance.
Lastly, it is evident that participation increased from passive to active levels on a
social and temporal basis.

Persistence: The intensity element of persistence is the level to which participants


pursue topics in order to evaluate the emergence of a clear focus. Persistence was
located on two levels, discussion topic and thread. (Note that the e-tutors used the
‘split forum’ facility to facilitate the flow of the conversation and intervened in the
persistence ratio in a positive manner.) Both the number of initiations and replies with
more than two messages were calculated (Table 6.5.5-2):

Table 6.5.5-2. Persistence in Moodle@GSN

PERSISTENCE IN MOODLE@GSN
Depth of persistence
Section Forums
Forums Messages
Introduction (01- 21/03/2007) 43/49 66/231
Timetable (01 – 28/03/2007) 7/7 127/127
A Introduction Social networking (01-24/03/2007) 1/1 30/30
Moodle (01-08/03/2007) 2/2 7/7
Problems (01-08/05/2003) 6/22 117/127
Total 5 59/81 347/522
Project
B Projects Archive (01-07/03/2007) 1/1 6/6
Management
Total 1 1/1 6/6
Blogs (01-31/03/2007) 42/43 187/198
C Blogs Tools (03-12/03/2007) 3/3 9/10
Blog & HTML (05-12/03/2007) 1/1 3/3
Total 3 46/47 199/211
Problems (12-23/03/2007) 9/9 37/37
Design (17-18/03/2007) 0/2 0/6
D Wikis
Practicality (13-18/03/2007) 2/2 10/18
Groups (18-24/03/2007) 6/7 31/31
Total 4 17/20 78/92
Technical Problems (16-22/03/07) 1 12/12
E Videoconferencing
Groups (18-23/03/07) 1 16/16
Total 2 2/2 28/28
F Internet Cafe Project ideas (01-06/03/2007) 1/1 3/3

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 179
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Other (01-18/03/2007) 2/2 6/6


Total 2 3/3 9/9
Overall 16 128/154 667/868
Overall Percentage 83.1 76.8
Average Percentage 80

The previous table presents that 59 out of 81 forums and 347 out of 522 messages
were following the subject of discussion in the introductory section; in the project all
messages were on topic these are 1 out of 1 forums and 6 out of six messages; in
blogs, persistence was strong as 46 out of 47 forums and 199 out of 211 messages
were on topic; in wikis the persistence depth was 17 out of 20 in forums and 78 out of
92 for messages; as in videoconferencing persistence was 100%, 2 in 2 forums and
28 out of 28 messages; the same rate appeared in the internet café with 3 in 3
forums and 9 out of 9 messages. The forums overall percentage was 83.1% with 128
out of 154 forums persistence depth which meant that the forums were relevant to
the forum topics. Slightly lower was the messages overall percentage (76.8%) with
667 out of 868 messages to follow the discussion topic. The overall persistence
depth in Moodle@GSN was 80%.

More persistence was observed in messages with 2-3 replies. As for the rest of the
messages, there were a small number of them irrelevant to the topic, sometimes
coming from the same e-learner; for example, 5 out of 6 irrelevant messages in the
‘Problems’ forum came from P48; they were events announcements. (Note that there
were no members’ announcements area in the online course.) Both forums on ‘wiki
design’ were irrelevant; this perhaps was because of the title forum: ‘What are the
main characteristics in wiki design? the goal to match, to serve the audience, both or
something else?’ As for levels of persistence in replies, the persistence depth was
found to be relevant to the thread depth; in other words, the more the replies the
more the probability of lack of persistence. Some times there was a reason for
shifting the discussion focus; for example, in Wiki topic, the focus was shifted in the
practicality discussion when one of the members found to have posted in another
person’s wiki without realising it (Reply 5): “Hi O3, I am P50; without doing it
intentionally, you created a webpage post in my wiki…”. Two similar changes of
focus occurred in the blog/blogs discussion when a problem on the suggested url
appeared and 5 messages were on finding the correct url. Lastly, a large number of
forums indicated a small number of replies so the level of persistence was maximum
for these forums. Overall, shift of focus was justified.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 180
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Similar results on persistence appear in the research pool (Table 6.5.5-3):

Table 6.5.5-3. Persistence in the research pool

PERSISTENCE IN MOODLE@GSN
Depth of persistence
Section Forums
Forums Replies
News (28-30/03/07) 2/2 5/5
A Introduction
VC in E-learning (28/03-01/04/07) 8/8 63/70
Total 2 10/10 68/75
Percentage 100 90.6
Average 95.3

The forums had 100% persistence; with 10 out of 10 messages following the topic.
As for the messages, there were 68 out of 75 messages exhibiting persistence, with
90.6% percentage. The overall persistence in the research pool was 95.3%. In depth
analysis revealed that the discussion on re-using videos in a project in the
videoconferencing forum produced 21 replies; however, after the 13th reply the focus
changed to how the Ministry of Education supports the teachers on the projects and
what happens when the project finishes. Then the focus shifted to the Greek
education system and teachers’ training, and the last two messages partly brought
the focus back to the use of tools in projects.

More intensity found in the research pool: the process of participation was more
stable and coherent, and there was an overall persistence of 95.3% comparing to
80% in Moodle@GSN even though the number of messages was low.

Intensity appeared to be higher in the research pool: the participation levels were
more coherent and stable in the research pool having all levels from low passive to
high active participation; and persistence was found 90.6%. It appears that initial
online dialogical argumentation lacked depth and was redundant as participants
failed to sustain interaction, also found to be evident in the research by Khine and
colleagues (2003); as the participants gained knowledge and experience of
collaborative techniques based on information provision and observation of the e-
tutors, their behaviour changed in the research pool. It is interesting to note that for 3
years there were 19 messages with 1 one reply, whereas 1 message produces 27
replies (Chapter 4.3.1.2). Intensity also provides evidence for establishing common
ground. This is done by checking whether a conversation partner has heard and
correctly understood what is being said (Preece & Maloney-Krichmar, 2003). Lastly, it

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 181
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

is evident that intensity can depict any imbalance or equilibrium between passive and
active participation and a level of persistence as a SeLCI determinant.

6.5.6 E-Learning Quality

The Collaborative e-Learning Episodes analysis provided a qualitative view in


online discussions, aiming to assess quality in e-learning interactions. There were 13
Collaborative e-Learning Episodes, 6 in Moodle@GSN and 7 in the research pool.
However, it was considered important to acquire the e-learners’ viewpoint: whether or
not the e-learners asked for help and who helped them the most; if the e-learners
learned to collaborate; how they worked together; and to determine how the tools
aided their learning.

The respondents said that they asked help from the e-tutors (n=37, 92.5%) and less
from their co-learners (n=3, 7.5%). The majority replied that they actually learned
ways for collaborative learning (n=37, 92.5%), 1 did not (2.5%), and 2 N/A (5%).
More specifically, they suggested ways to achieve collaborative e-learning (N=41;
one participant provided more than one suggestion) (Graph 6.5.6-1):

Collaborative e-Learning
Number of responses (N=41)
0 2 4 6 8 10 12

Information & knowledge


exchange
Elements for community evolution

37% 11
Collaborative activities
27% 8

17% 5 Dialogue development

7% 2 Mobility of ideas
1
1
Learn to communicate
3% 1

1 Vicarious learning

1 New skills acquisition

Number of created projects

Graph 6.5.6-1. Elements that show community evolution: e-learning

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 182
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Information and knowledge exchange were the most important factors for
collaborative e-learning (37%); then collaborative activities (27%), dialogue
development (17%), and mobility of ideas (7%). One suggestion each (3%) referred
to learning to communicate, vicarious learning, new skills acquisition, and the number
of created projects. Only 5 responses were on tools, 4 on the new collaborative tools
(80%) and 1 on profiles (20%). It appears that mere information was essential to
trigger collaboration but it was the collaborative techniques and the use of tools that
transformed information to collaborative e-learning by supporting different learning
styles. This result indicates that progressive discourse can be the outcome of
increasing participation in collaborative e-learning communities.

The next graph shows more specific opinions with regard to whom they learned from
(Graph 6.5.6-3):

Whom did you learn from? (N=35)

3, 6%
3, 6%

E-tutors
Otherlearners
More experienced learners
17, 32% 30, 56% Own work

Graph 6.5.6-3. The e-learning facilitators

The previous graph presents that the participants learnt from the e-tutors (n=30,
56%), other learners (n=17, 32%) and equally more experienced learners (n=3, 6%)
and on their own (n=3, 6%). It appears that passive participation and vicarious
learning was one of the learning styles; P21 also said that she was ‘watching how
other people were working’. This result is in agreement with the centrality scores and
the responses on passive participants.

Overall, thematic analysis pointed at three themes, community management, e-


learning and technology (Graph 6.5.6-2):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 183
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Collaborative e-Learning

100

62
41

17
Codes
10
8 Responses

1
Community Management e-Learning (30%) Technology (7%)
(63%) (Percentage on codes)

Graph 6.5.6-2. Correlations between codes on collaborative e-learning quality

The importance attached to the factors on collaborative e-learning quality is in favour


of community management, then e-learning and finally, technology is the last one. It
is interesting to see that this attached importance is even higher for collaborative e-
learning than e-learning community (see Graph 6.5.2-2): the percentage is increased
as regards community management (63% from 59%), decreases for e-learning (30%
from 35%), and slightly increases for technology (7% from 6%). These results stress
the importance of the social aspect of collaborative e-learning.

Overall, quality in collaborative e-learning can be a Sense of e-Learning Community


Index (SeLCI) attribute as it provides a clear indication of quality in textual online
interaction by the number of collaborative e-learning episodes and participants’
viewpoint on their learning. Three distinct learning styles were revealed, instructional,
collaborative and vicarious learning. As before, community management was the
most important job for the e-tutors, then e-learning activities and lastly, the
technology. Some results can be triangulated by social network analysis in order to
provide a different viewpoint towards SeLCI. These will be discussed in the next
section.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 184
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

6.5.7 Global Social Network Analysis

Social Network Analysis (SNA) can describe Greek teachers’ interactions as


well as triangulating previous findings. This is feasible because the relationships
based on text and words only have limited capacity to represent the social network;
in other words, photos provide a different view of a group of people than the script of
what they said. SNA focuses on global (found also as complete or group) and ego
networks. Global cohesion and centrality were investigated using UCINET (Borgatti
et al., 2002); cohesion can represent the interactions’ weight (density), participants’
preferences (reciprocity), any small groups (cliques), and similar behaviour (structural
equivalence); centrality can depict interaction direction (in-out degree centrality),
speed (closeness), and control (betweenness).

Two adjacency matrices were produced one 64*64 for GSN (Figure 6.5.5-1) that
included the main participants and others who had been interacting but not on a
regular basis and did not fit the criteria of participants’ selection. Also another 41*41
matrix was produced for the research pool that included my interactions (Valued links
and passive participants zeros are depicted; all other zeros were eliminated to make
the matrix more legible) (Figure 6.5.7-1):

Figure 6.5.7-1. GSN adjacency matrix in UCINET

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 185
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Several problems occurred in producing the matrices: one was the discrepancy
between the system reply and the actual address of reply and the other was the
participation of two e-tutors and myself. First, the function ‘Reply’ in Moodle was
referred to the previous message whereas the actual reply could be addressed to
one or more e-learners sometimes higher up the forum. So in this study, where
apparent, the input in SNA was the actual addressed reply as conceived from the
message rather than as depicted on the system; this indicated a limitation in the
forum application in Moodle. For example, someone may send a post later in the
week by replying to the last message; however, the message may have been
addressed to somebody else. Messages that were replied to everyone as when the
participants were introducing themselves were linked to themselves on the adjacency
matrix. Then, two new sets of adjacency matrices were analysed, one with the
researcher and the 2 e-tutors as the most active participants in GSN, and one without
the 3 actors, these were two 38*38 matrices. This decision was made because this
study focuses more on collaborative e-learning for e-learners-generated text rather
than the e-tutors-generated-text.

6.5.7.1. Global Cohesion

The level of global cohesion was measured by assessing network density,


reciprocity, cliques, and structural equivalence. Density is the proportion of possible
links in network as it is the ratio of the number of links present in the network, to the
maximum possible links. Density was evaluated by the adjacency connection reports
in UCINET (Table 6.5.7.1-1):

Table 6.5.7.1-1. Group Network Cohesion: Density & Reciprocity

GROUP NETWORK COHESION: DENSITY & RECIPROCITY


GSN Research Pool
All E-learners All E-learners
Total nodes 698 122 81 73
Density (matrix average) 1.0872 0.0256 0.0470 0.0418
Standard deviation 2.0167 0.1819 0.2376 0.2226
Reciprocity (Hybrid) 0.3618 0.2222 0.1852 0.2174

E-learners’ density is rather low, 0.0256 in Moodle@GSN, however stable in the


research pool (0.0418). This means that 2.6% in Moodle@GSN and 4.7% in the
research pool of all possible links were present; however, there was an increase in

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 186
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

density. The participants actually recognised their limited participation; they said they
were not as active as they wanted to be. In addition, the 2 highest posters in
Moodle@GSN influenced the groups’ density level; this means that the actual
increase in participation was almost doubled (0.0418 - 0.0256= 0.0162). This was
also evident in the collaborative e-learning episodes text richness, as it was doubled
in the research pool (see Table 6.4.2-2).

Reciprocity in SNA is the number of ties that are involved in reciprocal relations
relative to the total number of actual ties (Hanneman & Riddle, 2005). Reciprocity
appears higher within the e-learners (Graph 6.5.7.1-1):

(a) (b)
Graph 6.5.7.1-1. Reciprocal ties in GSN (a) & the research pool (b)

There were 4 reciprocal ties in GSN (28.6%) and 10 (71.4%) in the research pool.
However, due to the e-tutors role in GSN, it appears more as an evolutionary
process.

The increase of reciprocal ties is another indication of evolution in discussion from


monological to dialogical sequences between two participants. Strong and weak
reciprocal ties can also define strong or weak relationships within an e-learning
community. Therefore, reciprocal ties can maintain a strong social network; thus,
they are important for knowledge exchange and community knowledge building as it
means members’ constant by give and take within a community (Preece, 2004).

Reciprocity is also related to the social exchange theory (Blau, 1964); it posits that
individuals engage in social interaction based on expectations or the benefits active
participants can get from active participation, for example some sort of personal gain

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 187
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

or status. However, reciprocity can also be triggered by intangible returns in the


forms of intrinsic satisfaction and self-actualisation (Äkkinen, 2005). Based on
participants’ opinions for feeling guilty because of inadequate participation in the
course as well as their initial voluntary involvement in the GSN courses, it appears
that their target was learning, and thus, only implicitly related to the social exchange
theory. This also means that social loafing was not evident in this study as they
rather had shared interests and values.

A clique is a subgroup, a set of actors with each being connected to each other as a
maximal complete subgraph of three or more nodes (members) adjacent to each
other and there are no other nodes in the network that are also adjacent to all of the
members of the clique (Laghos, 2007). Cliques may overlap, that is a forum member
(node) can be a member of more than one clique (Bock & Husain, 1950). The results
presented in the following table are cumulative and refer to cliques created by 3, 4, 5
and 6 participants (Table 6.5.7.1-2):

Table 6.5.7.1-2. Cliques

CLIQUES
Moodle@GSN Research Pool
Minimum set size
of participants All E-learners All E-learners

6 2 0 0 0
5 33 0 0 0
4 58 0 2 1
3 68 4 15 12

Most cliques were created by 3 participants in both environments. The e-tutors


dominated the cliques gathering up to 6 participants. The cliques were developed
without any intervention by any of the participants, e-tutors or myself. It is interesting
to note that the top scorers had inter-clique connections. When the cliques increase,
the social network remains active and thriving, especially if e-learners interact with
other e-learners who did not appear in a clique before; these are the activated
lurkers. In other words, the absence of cliques could have indicated a lack of
clustering that would have reflected the prevalence of weak ties.

As most the participants did not know each other before the study and were more
skilled in the research environment, the cliques were the glue for forums. However,
what fostered the cliques was not investigated in this study.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 188
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Structural equivalence describes the actors who have similar patterns of relations
to others in the network and exhibit similar communication behaviour. It presents a
different clustering view within a human network. Equivalence is important for
generalizations about social behavior and social structure; actors must not be
thought about as unique persons, but as examples of categories (sets of actors) who
are in some way, "equivalent” (Hanneman, 2001).

Two actors (nodes) are said to be structurally equivalent if they have identical ties
with themselves, each other and all other vertices (de Nooy et al., 2005). It is
computed by the Euclidean distance of tie-value from and to all other nodes (Lorrain
& White, 1971). The CONCOR technique (CONvergence of iterated CORrelations;
White et al., 1976) uses dendrogrammes (tree-diagrammes) for hierarchical
clustering whereas other techniques use algorithms to calculate network members’
individual behaviour (e.g. Everett & Borgatti, 1993). The CONCOR technique
calculates Pearson’s correlation coefficient between columns and depicts whether
two nodes are structurally equivalent if the corresponding rows and columns of the
adjacency matrix are identical. So the degree to which two nodes are structurally
equivalent can be evaluated by measuring the degree to which their columns are
identical. CONCOR is a divisive top-down clustering technique; it begins with one
group and then divides it up so the dendrogramme looks like an inverted tree. This
structure is calculated and thus artificial, resulting in failing to identify observed
clusters.

Interpretation of dendrogrammes for network clustering is as follows: the labels of the


actors are given on the left in UCINET; the network positions appear as lines; the
numbers at the top are the clustering levels, indicating the number of clusters at the
level of sharing at least 3 ties; the column in the middle is the row number in the
UCINET matrix for the network. (Dividing clusters of 3 or less individuals is not
preferable as correlations get very unstable) (Graph 6.5.7.1-2):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 189
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Graph 6.5.7.1-2. Structural equivalence dendrogrammes in GSN (all)

If each line represents a participant, the CONCOR dendrogramme reveals 7 splits on


the first level and 2 splits on a second level with four actors participating in all groups.
This means that 7 and out of them 3 actors had exhibited similar behaviour (Graph
6.5.7.1-3):

Graph 6.5.7.1-3. Structural equivalence dendrogrammes in the research pool (all)

In the research pool 3 participants were active in 2 second level groups and 5 first
level groups. The next CONCOR dendrogramme reveals 7 first and 4 second level
participants (Graph 6.5.7.1-4):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 190
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Graph 6.5.7.1-4. Structural equivalence dendrogramme in GSN (e-learners)

Here, the ties are less than the one with the researcher and the 2 high participation
e-tutors, however, the overall structure of the groups remain the same. Lastly, the
next dendrogramme refers to the e-learners in the research pool (Graph 6.5.7.1-5):

Graph 6.5.7.1-5. Structural equivalence dendrogramme


in the research pool (e-learners)

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 191
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

This graph reveals 5 first and 2 second level multi-actor positions with one solo-actor
position.

In conclusion, if grouping actors with equivalent behaviour, the results from the
previous dendrogrammes appear as follows (Table 6.5.7.1-3):

Table 6.5.7.1-3. Equivalent e-learners

EQUIVALENT E-LEARNERS
Moodle@GSN Research Pool

All E-learners All E-learners

1st level 7 7 5 5
2nd level 3 4 3 2

Structural equivalence seems to be steady in the two e-learning environments. There


were 7 and 5 participants with first level equivalence in the two research pools. There
was one more e-learner with second level equivalence if e-tutors were excluded (4-
3=1) whereas it was the opposite situation in the research pool with one less e-
learner (3-2=1). In other words, more e-learners were imitating e-tutors’ and other e-
learners’ behaviour, and thus passive behaviour was decreasing. This was in
accordance to participants’ comments on watching what the e-tutors were doing and
learning vicariously. This means that observation had a positive effect in replicating
behaviour active participation especially if the participants did not have previous
knowledge of working and learning online.

Overall, the social network analysis attributes of the Sense of e-Learning Community
Index (SeLCI) indicated that the interactions’ weight (density) was doubled in the
research pool; participants’ preferences (reciprocity) were also significantly
increased; more similar behaviours (structural equivalence) were observed in
Moodle@GSN rather than the research pool; and there were some small groups
(cliques) that remained almost the same throughout the study.

6.5.7.2. Global Centrality

Global centrality investigates the communication nodes between the


members of a network and is characterised by direction and strength and refers to

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 192
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

out-degree centrality (replies made), in-degree centrality (received messages) as well


as group in and out closeness and betweenness.

Centrality is measured by the portion of nodes that are adjacent to each node, the
sum of each row in the adjacency matrix representing the network. The centrality
scores were (Table 6.5.7.2-1):

Table 6.5.7.2-1. Group Centrality

GROUP CENTRALITY SCORES


GSN Research Pool
All E-learners All E-learners
Network Centralization (Out-Degree) 13.852% 10.826% 8.606% 9.372%
Mean 8.328 0.974 1.667 1.590
Standard deviation 20.321 2.154 2.316 2.250
Network Centralization (In-Degree) 10.751% 5.425% 6.940% 6.671%
Mean 8.328 0.974 1.667 1.590
Standard deviation 13.999 1.459 2.089 1.983
In-closeness (Network in-Centralization) 73.97% 32.68% 49.49% 49.83%
Mean 20.581 1.447 5.885 5.565
Standard deviation 10.762 1.859 5.147 4.919
Out-closeness (Network out-Centralization) 78.30% 55.04% 42.74% 42.53%
Mean 20.581 1.447 5.885 5.565
Standard deviation 11.387 2.532 5.088 4.626
32.7I (R) 2(P32) 16.2(P50) 16.2 (P50)
Betweenness (top 3 participants) 18.7(P52) 1.7 (P24) 11 (P18) 11.8 (P18)
5(P24) 0.9(P37) 6.7 (P24) 7.3 (P24)
Mean 1.254 0.173 1.751 1.950
Standard deviation 4.655 0.431 3.235 3.436
*Data matrix dichotomized for closeness, such that Xij > 0 was recoded to 1

Degree centrality refers to a directed network (where the direction of the


communication is important); the in-degree centrality is the portion of nodes that are
adjacent to each node, and out-degree centrality is the portion of nodes that are
adjacent from each node (Freeman, 1979). Even though Borgatti (2005:70) suggests
that Freeman’s centrality has been misapplied, it is widely used because there are no
other suggestions for coherent results on global centrality. The nodes with the
highest degree scores are the ones which are more central (powerful) in the network.
Degree centrality was preferred rather than eigenvector centrality as a measure of
immediate influence - the ability to influence others directly or in one time period
(Borgatti, 2005:70). This was because the subject of investigation was the levels of
activity and thus the centrality and peripherality degree of the members.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 193
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

In this study, the e-tutors’ role was apparent in Moodle@GSN. For the e-tutors the
scores were: out-degree=13.852% (SD=20.321), in-degree=73.97% (SD=13.99). For
the e-learners the scores were very limited: out-degree=10.826% (SD=2.154), in-
degree=5.425 (SD=1.459). The e-learners had more stable scores in the research
pool; with the e-tutors the scores were: out-degree=8.606% (SD=2.316), in-
degree=49.49% (SD=2.089); without the e-tutors the scores were: out-
degree=9.372% (SD=2250), in-degree=6.671% (SD=1.983).

The out- and in- degree centrality appear to have great differences depicted mostly in
the standard deviation rather than centrality itself. Standard deviation dropped in the
e-learners centrality scores in the research pool which means that the differences
between the e-learners were diminishing. There was an increase in in-degree
centrality and a decrease in out-degree centrality. This means that the e-tutors had
the power in the information flow in GSN and the e-learners in the research pool in a
more distributed manner. In a more in depth analysis, the top 10 centrality scorers in
out- and in- degree centrality (Table 6.5.7.2-2) were:

Table 6.5.7.2-2. Top 10 Scorers in Out-Degree Centrality

OUT-DEGREE CENTRALITY: TOP 10


GSN Research Pool
All E-learners All E-learners
Out- Out- Out- Out-
Participant Participant Participant Participant
Degree Degree Degree Degree
1 P52 120 P32 13 P50 12 P50 12
2 R 108 P18 4 P22 6 P18 6
3 P58 47 P48 3 P18 6 P22 5
4 P18 25 P37 2 P14 4 P48 4
5 P37 20 P24 2 P48 4 P37 3
6 P32 19 P9 2 P37 4 P24 3
7 O2 13 P13 1 P6 4 P14 3
8 P56 10 P2 1 P2 3 P6 3
9 O4 9 P50 1 P12 3 P12 3
10 O9 9 P6 1 P13 3 P13 2

If the 3 e-tutors (P52, R, and P58) are omitted, the next 3 active participants appear
first in out-degree centrality (sent messages) in GSN. P18 and P37 from GSN appear
to be among the top scorers in the research pool. The top scorers for in-degree
centrality were (Table 6.5.7.2-3):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 194
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Table 6.5.7.2-3. Top 10 Scorers in In-Degree Centrality

IN-DEGREE CENTRALITY: TOP 10


GSN Research Pool
All E-learners All E-learners
In- In- In- In-
Participant Participant Participant Participant
Degree Degree Degree Degree
1 R 95 P9 7 P50 10 P50 9
2 P52 52 P24 4 P18 5 P18 5
3 P37 31 P13 4 P37 5 P37 5
4 P9 25 P18 3 P24 5 P24 5
5 P18 19 P37 3 P6 4 P6 4
6 O2 19 P2 2 P13 4 P13 4
7 P56 19 P6 2 P22 3 P22 3
8 O9 18 P32 1 P12 3 P12 3
9 P58 17 P48 1 P2 3 P2 3
10 P15 15 P3 1 P33 3 P33 3

As for the received messages (in-degree centrality), the same participants appear to
have incoming messages posted specifically for them; this is more apparent in the
research pool where exactly the same e-learners were in the top 10 as there were
central students controlling in-coming and out-coming connections. In addition, the
increase of in-degree centrality indicates that they were more communicative and
received more messages; the simultaneous slight decrease in the out-degree
messages indicates that they lost some of their power and this power was distributed
to the other e-learners. In other words, the responses were originated from a group of
members that was larger than the group that received the messages. In a way, it
indicated a movement from a powerful group of e-tutors to a group that was working
more and more collaboratively to increase their learning; the active participants
became more democratic in their communication instead of maintaining their status.

Closeness: Nodes with low closeness scores have short distances from the others.
In other words, a node has high closeness centrality if it has very short
communication paths to the others. ‘In-closeness centrality’ is measured as a
function of the minimum geodesic distance from all other nodes to the selected node;
while ‘out-closeness centrality’ is measured as a function of the minimum geodesic
distance linking that node to the other nodes. While degree centrality measures use
only direct and local connectivity information, closeness centrality measures also use
indirect connectivity information (Braha & Bar-Yam, 2004:25). An example for its use
is the following (Borgatti, 2005:59): organizations with low closeness in an R&D

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 195
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

technology-sharing network can develop products sooner than others; a virus can
sooner infect the members with low closeness.

As with degree centrality, with regard to in- and out- closeness, there were major
differences between the e-tutors and e-learners in the two environments. This was
indicated by the closeness score, the means and standard deviation. For example,
including the e-tutors in Moodle@GSN the in-closeness score was 73.97%
(SD=10.762) and the out-closeness score 78.30% (SD=11.387); without the e-tutors
in Moodle@GSN, the in-closeness score was 32.68% (SD=1.859) and the out-
closeness score 55.04% (SD=2.532). In the research pool the results are more
equally distributed and distance of communication was reduced: with the e-tutors the
in-closeness score was 49.49% (SD=5.147) and the out-closeness score 42.74%
(SD=5.088); without the e-tutors the in-closeness score was 49.83% (SD=4.919) and
the out-closeness score 42.53% (SD=4.626).

Overall, ‘in-closeness centrality’ represents the speed of interaction from all other
nodes to the selected node; so in Moodle@GSN this speed was high when the e-
tutors were included; the standard deviation was high as well. ‘Out-closeness
centrality’ represents the speed of interaction from one node to the other nodes; this
score and standard deviation was also high in Moodle@GSN. These scores
indicated that the e-tutors were controlling the speed of information flow. However,
the scores without the e-tutors and in the research pool were more equally distributed
and the standard deviation was very low, indicating a more stable interaction speed
between the e-learners. A reason may be that e-tutors were connected more times
than the e-learners and this affected their interaction speed.

Betweennes measures the node’s prominence according to its position in the


network as an intermediary measuring the volume of traffic moving from each node
to every other node that would pass through a given node. Some active participants
act as “brokers” or “gatekeepers” between groups of nodes, therefore playing an
important role in the network. As for betweenness, e-learners stayed significantly
behind me (32.71) and the e-tutor 52 (18.7) in Moodle@GSN; P24 scored 5 and
others 2 (P32), 1.7 (P24) and 0.9 (P37). As before, this might be affected by how
long they were on-line. The scores remained on the same level in the research pool
with and without the e-tutors: 16.2 from P52, 11 and 11.8 from P18, and 6.7 and 7.3
(P24). It was evident that the highest values came from the e-tutors in Moodle@GSN
and the e-learners in the research pool. There was a significant difference between

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 196
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

the information gatekeepers as regulators of the information flow in the network


within the two environments despite the fact that the participants did not see any
particular roles or, to a small extent, e-tutors and e-learners. The participants with the
lowest betweenness values could be considered outsiders in the conversation, or
with no mediation power (Willging, 2005:51). These can be located in the low active
participation level; they can be classified as low active participants or, in SNA,
isolates. Lastly, there was no active participant who could control the information flow
in both environments.

Overall, the centralisation indexes were: for interaction direction (in- and out-degree
centrality) the in-degree centrality increase and the out-degree centrality decrease
indicated that the e-tutors had the power in the information flow in Moodle@GSN and
the e-learners in the research pool in a more distributed manner. This was also
evident in the interaction speed (closeness) and control (betweenness) that showed
that the e-tutors hold the network power in Moodle@GSN whereas this power was
quite evenly distributed in the research pool and the discussions have not been
monopolised. Moreover, there was no single participant who ranked high in all the
centrality measures in both environments including the e-tutors and myself.

Other than global Social Network Analysis (SNA), local SNA describes the human
and information network in a particular situation. The Visualisation Interaction Tools
Nodes and Centrality (VIT Nodes and VIT Centrality) supported local SNA nodes and
centrality in real time.

6.5.7.3. Local Nodes and Centrality in Real-Time

Just as a photo and a recording give a different ‘picture’ of two people


discussing something, so too, two SNA tools running in real time aimed to provide a
different viewpoint of the discussion and triangulate the events of active participation
and collaborative e-learning. Visualisation Interactions Nodes (VIT Nodes, Figure
6.5.7.3-1) and Centrality (VIT Centrality, Figure 6.5.7.3-2) were integrated in Moodle
in the research pool. A different abstract representation was given with regard to
interaction density (weight), reciprocity (preferences) as well as in- and out-degree
centrality (direction). Closeness as the interaction speed was represented on both
graphs as the geodesic distances indicate the temporal distance between the
messages. Information control (betweenness) could also be observed. (Note that the

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 197
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

participants were not given specific information on the exact use of these tools as the
tools should indicate their own use (usability).)

In VIT Nodes the individuals are represented as circles (nodes), the direction of the
messages is indicated by an arrow and the number represents the number of
messages (Graph 6.5.7.3-1):

Graph 6.5.7.3-1: VIT Nodes in CeLE IX

P37 was the information broker in this CeLE. The reciprocal tie with O2 was an
argument. She also responded to her own message a couple of hours later after the
argument with O2. Most participants were replying to P37 and two of them talked to
each other. It is interesting that this CeLE was developed by different individuals with
only two interlocutors exchanging 2 messages. In other words, the discussion was a
collaborative activity between 7 individuals. VIT Centrality provided a different
viewpoint (Graph 6.5.7.3-2):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 198
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Graph 6.5.7.3-2. VIT Centrality in CeLE IX

In VIT centrality P37 is clearly located in the middle of the e-learning social network.
VIT centrality also indicates the response time space related to geodesic distances
between the participants. As a central connector and information broker she moved
the knowledge around leading to a new proposition by taking into account her co-
learners responses even though they appeared as low activity e-learners (i.e. only
O2 was an e-tutor).

Overall, the Visualisation Interaction Tools Nodes and Centrality provided


opportunities to the e-learning participants to observe their personal styles and
performance within the e-learning community as well as observe small groups
created within their discussions. Being self-aware corresponds to self-organised
learning and development. Information organisation roles can be unofficially assigned
to e-tutors to support the collaborative e-learning development.

To sum up, it appears that after the initial knowledge acquisition and information
exchange, the collaborative techniques and tools helped participants to learn from
the e-tutors, their co-learners and on their own. As they did not have any previous
knowledge of collaborative learning and techniques they acquired this knowledge for
community knowledge building. This means that e-tutors have a complex job that
incorporates moderating as well as e-tutoring (Salmon, 2000). It was also suggested
that knowledge awareness plays a major part in the creation of opportunities for

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 199
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

efficient and effective collaborative opportunities (Ogata & Yano, 2000) which leads
to the fact that different learning styles are supplementary to each other in e-learning
environments.

Lastly, with regard to collaborative e-learning quality, the participants learned from
the e-tutors (instructional learning), the other e-learners (collaborative learning) as
well as on their own how to work together and how to use the new tools and new
technologies (vicarious and self-organised learning). Almost the same number of
collaborative e-learning episodes, 6 in Moodle@GSN and 7 in the research pool, add
to the evidence for the e-learning quality. CeLE was found to build on progressive
discourse and fill the middle space between internalisation and externalisation in the
form of monological to dialogical sequences. This was evident not only within one
CeLE but in the participation process as monological postings stand alone without
open clues for dialogue as in information provision. In a way, e-learners were
progressively adopting a two-way communication. This means that there were more
clues as opportunities for critical engagement in dialogue in the research pool caused
by two events, the initial need for familiarisation with collaborative e-learning in the
early stages of the online course and the sleeper effect, and the use of MessageTag
that revealed the collaborative e-learning structure. Moreover, correcting
communication gaps between the collaborative learning discussion stages is feasible
for the e-learning participants.

The effectiveness of the Sense of e-Learning Community Index will be discussed in


the interventions section at the end of this chapter.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 200
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

6.6 PEDAGOGICAL USABILITY

The participants’ responses in the closed question revealed that the new tools
supported the e-learning community development (n=33, 82.5%). The tools were: (a)
Participation graphs and avatars; (b) MessageTag; (c) Visualisation Interaction Tool for
Nodes (VIT Nodes), and (d) Visualisation Interaction Tool for Centrality (VIT Centrality).
The overall results are presented in Appendix XII.

6.6.1 Pedagogical Usability

The participants responded to 15 closed questions on a Likert scale 1-5. In


sum, the pedagogical usability values were (Table 6.6.1-1):

Table 6.6.1-1. Pedagogical usability values

PEDAGOGICAL USABILITY VALUES


1 Instructions 3.6
2 Frequency of use 2.4
3 Alignment with educational goals 3.5
4 Support collaborative e-learning 3.5
5 Learnability 2.7
6 Accessibility 3.7
7 Originality 3.4
8 Motivation to participate 3.5
9 Information overload 2.8 (5.0-2.2)
10 Tool failure 3.1 (5.0-1.9)
11 Functionality 3.4
12 Graphics 3.4
13 Attractiveness 3.0
14 Fast response 3.6
15 Overall satisfaction 3.4
Overall 3.2
Values 9 & 10 were inverted for reasons of compatibility and clarity

The overall score for pedagogical usability was found to be just higher than the
average of 1 to 5 Likert scale (3.2). Most of the pedagogical usability scores for the
new collaborative tools were satisfactory. However, originality (2.4) and learnability
were low (2.7). The law learnability rating may explain why the tools were
infrequently used (2.4). This result was similar to the preliminary studies although
there the tools were being used by the developers who perceived the tools
differently. Nonetheless, according to the participants, the use of the new tools was
the third best thing in the project after collaboration and feeling a sense of
contributing to something great (Appendix A_XII_4).

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 201
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

The messages were tagged with MessageTag (Table 6.6.1-2):

Table 6.6.1-2. Use of MessageTag in the Research pool

USE OF MESSAGETAG IN THE RESEARCH POOL


Forums CeLE Attributes # use Tagged messages Percentage
Inform 0
Question 0
Explain 0
Explore 0
News 0/9 0
Agree 0
Evaluate 0
Summarise 0
Other 0
Inform 7
Question 3
Explain 3
Explore 2
VC in E-learning 20/70 28.6
Agree 2
Evaluate 2
Summarise 0
Other 1
Total 20/79 25.3

The tool was not used in the introductory news section and the participants started
using it in the main discussion. The results were very similar to the e-mmersion
block; 20 out of 79 messages were tagged with an average 27.3% use. Comparing
the results with the developers’ results (55.3%) a great difference appears to be
between the way average users’ utilisation of the new tools and the more advanced
users. Involving users with different levels of experience or filtering of users based on
their profile for basic and advanced user testing can provide different results;
advanced users might know uncommon ways to overcome problems on the
interface, get familiarised quickly or be keen to experiment with new tools.

Interestingly, P6 sent a message to the forum saying that the new tools are invisible
and lost in the interface. He said that most users will not notice them and he gave
examples on MessageTag and Visualization Interactions Tools nodes and centrality:
“it is not easy to actually see the links that lead to them because the MessageTag is
at the end of the message and the images for the VIT are invisible”. In addition, his
message indicated frustration concerning their use which is in accordance to the low
learnability score: “At last! I managed to find a way with the VIT nodes and

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 202
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

centrality…” (“Επιτέλους! Έβγαλα μια άκρη με τα VIT nodes & centrality.”). Lastly, he
expressed his enthusiasm about the tools and suggested that their use will be
determined by time.

P6’s messages raises questions on the decisions made about the images selected
after the developers’ recommendations. A developer needs to listen to users’
opinions; however, her decision making is based on her own expertise and usability
tests in usability laboratories, especially when the developers’ sample is relatively
small. (There were 3 Greek teachers who were Moodle developers participating in
tools’ testing.) It also appears that new tools need more than one week of use and
evaluation. However, the use of visualisation interactions tools was not apparent
without the aid of the literature; this is in contradiction to usability rules as the tools
should be easy to use. As tools to represent social networks have now reached the
average user (e.g. different social network applications on Facebook,
http://www.facebook.com), their utilisation will be more obvious and the users will be
more familiar with their practicality.

Overall, in comparison to the previous usability evaluation in the two preliminary


studies with e-learners and developers, the actual e-learners scored significantly
less; for example, all previous scores were much higher than the average (>4), and
no differences appeared between the participation and collaborative e-learning tools,
and VIT. This also means that the ‘real’ users are more reluctant to use the tools.
The participation graphs and avatars as well as the MessageTag achieved more
scores than the Visualisation Interactions Tools (VIT) in all pedagogical usability
metrics. The alignment with the educational goals and support for collaborative e-
learning was more than the average. In other words, the relevance of the tool to the
particular task had a positive effect. From an HCI viewpoint, as Draper (1993)
suggested, this was the relationship between the task and the method. He also said
that usability measurement of one task cannot be applied in exactly the same way for
all users. In addition, because the design process is technical and social, new
products development requires cycles of communication between the organisational
stakeholders (Safayeni, et al., 2008).

This section discussed results from the questionnaires, the tools, and the
participants’ comments. The next section refers to the most noticeable correlations
and crosstabulations between the pedagogical usability and utility factors as
appeared in the questionnaires.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 203
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

6.6.2 Correlations & Crosstabulations

Two ways were found suitable to measure and analyse inter-correlations


between the participants’ responses in the final questionnaire, in SPSS (Appendix
A_XII_1-2) and in Hierarchical Clustering Explorer (HCE) (Appendix A_XII_3). SPSS
was used to find all correlations as well as the ones related to specific issues. HCE
provided an overview of correlations focused on the most important positive and
negative correlations.

In more detail, the matrix with the data was exported from SPSS to Excel and then
inserted in the HCE to distil them. In HCE, N/A were viewed as missing data, and all
data were initially filtered with SD≤1. Twenty out of 40 rows remained and then, in
order to increase similarities and reduce differences between the data, the data were

normalized row by row in a scale 0.0 to 1 using the equation . Then Pearson’s r
correlation coefficient was used to identify 1,770 correlations variables with lower
score -0.747 and highest score 0.981 (scores<1). The information visualisation
provided a simultaneous correlations overview, scatter plotting, and in depth analysis
(Figure 6.6.2-1):

Figure 6.6.2-1. Correlations analysis in HCE 3.0

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 204
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

HCE provided a correlations overview from the questionnaires (bottom left), scatter
plotting (bottom right), and in-depth analysis of the results. The latter means that
clicking on the participants (top left) located them on the scatter plot; additionally
there was a visual summary (top right).

Not all 1,770 variables are presented here; the variables with r>0.7 (0.977<r>0.703)
and r>-0.600 (-0.600>r<-0.747). As in the previous findings, it appeared that
correlations exist between the two groups of tools the collaborative e-learning and
participation tools (Participation graphs and avatars, and MessageTag) as well as the
Visualisation Interactions Tools Nodes and Centrality (VIT Nodes, VIT Centrality)
used for social network analysis. This can be explained under the rubric of
functionality; the two groups of tools supported different practices in discussions, the
first to structure and analyse collaborative e-learning, and the latter to support
visibility in social networks. In other words, the practicality of the first was more
explicit than the latter.

The positive correlations between participation graphs and avatars, and MessageTag
were: failure (r=0.944); accessibility (r=0.948); educational goals (r=0.883); fast
response (r=0.751); motivation to participate (r=0.71); educational goals and
functionality in graphs and avatars alone (r=0.782); and satisfaction and educational
goals in graphs and avatars alone (r=0.712). The results show that there were
problems concerning tools failure and accessibility; however, there were in
accordance to educational goals, motivated participation, and the e-learners
expressed their satisfaction. This preference was also evident in the average use
(Likert scale 1-5): for graphs and avatars 2.7 (SD.=1.3); and for MessageTag 2.7
(SD=1.2).

The positive correlations between the VIT were: where Nodes failed, Centrality failed
(r=0.977); where Nodes functioned, Centrality functioned (r=0.961); satisfaction
(r=0.955) and motivation to participate was related (r=0.930). Originality was found to
be an issue only for VIT Nodes, related to: educational goals (r=0.81); functionality
(r=0.771); graphics (r=0.752); satisfaction (r=0.721); and support Collaborative e-
Learning (CeL) (r=0.712). In Nodes alone functionality was also found relevant to
satisfaction (r=0.792); motivation to participate and support CeL (r=0.788);
satisfaction and support CeL (r=0.748); and satisfaction and graphics in visual design
(r=0.710). In Centrality alone correlations were found between: satisfaction and
functionality (r=0.801); satisfaction and support CeL (r=0.784); satisfaction and

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 205
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

graphics in visual design (r=0.777); accessibility and instructions (r=0.763);


functionality and graphics (r=0.729); functionality and attractiveness (r=0.722); and
graphics and attractiveness in visual design (r=0.718). More meaningful correlations
between Nodes and Centrality were on: instructions (r=0.961); and graphics and
attractiveness (r=0.718).

Less strong negative correlations appear only for the VIT. For VIT Nodes the results
were: satisfaction was negatively related to failure (r=-0.0609); motivation to
information overload (r=-0.632); information overload and attractiveness (r=-0.6776);
learnability and attractiveness (r=-0.712); and accessibility and learnability (r=-0.729).
For VIT Centrality the results were: information overload and motivation for Centrality
(-0.628); satisfaction and information overload (r=-0.629); information overload and
attractiveness (r=-0.747). It appears that information overload was the main and
learnability the second pedagogical usability problem that affected attractiveness,
motivation to participate and the overall satisfaction. VIT were not used as much as
graphs, avatars and MessageTag: VIT (Likert scale 1-5): VIT Nodes 2.1 (SD=1) and
VIT centrality 2 (SD=1).

The difference in the standard deviation on the use of tools showed that some e-
learners used the tools more frequently than others. After crosstabulating the first
and final questionnaires, it seemed that the e-learners who did not have any
experience of LMS used the new tools more than the experienced e-learners (Table
6.6.2-1):

Table 6.6.2-1. Time using LMS * Frequency of Use

CROSSTABULATIONS: TIME USING LMS & THE FREQUENCY OF USE

Tools
Time Using LMS VIT VIT
Graphs/avatars MessageTag
Nodes Centrality Mean
# # # #
1 1-3 2 2 2 2 2
2 Months 4-6 1 1 1 1 1
Missing 3 3 3 3 3
3
Total 40 40 40 40 40
4 0 22 22 22 22 22
5 1-5 13 13 13 13 13
6 Years 6-10 1 1 1 1 1
Missing 3 3 3 3 3
7
N/A 1 1 1 1 1
Total 40 40 40 40 40

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 206
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

The participants gave similar results in regard to the tools. Similarly, the e-learners
who did not have any training on LMS used the new tools more than the trained e-
learners (Table 6.6.2-2):

Table 6.6.2-2. Training on LMS * Frequency of Use

CROSSTABULATIONS: TRAINING ON LMS & THE FREQUENCY OF USE

Tools
Train LMS Graphs VIT VIT
MessageTag
avatars Nodes Centrality
# χ2 # χ2 # χ2 # χ2
1 Yes 11 11 11 11
2 No 26 26 26 26
3 Other 1 1 1 1
4 Missing 2 2 2 2
Total 40 40 40 40

As before, the responses are similar; the participants with no training at all used the
tools more than the others, less the ones attended an ICT course and even less the
ones who had proper training within universities. This may mean that the more
experienced the e-learners the more reluctant they are in the use of new tools.
Another explanation may be that the users who were familiar with Moodle hardly
noticed the new tools in the research pool and acted by habit.

Pearson’s correlations were also calculated for the two tables. The variables were
checked for being ordinal or categorical, no cell to have expected values less than
one, and no more than 20% of the cells have expected values less than five (Muij,
2003). In addition, because in the two rows and two columns (two-by-two tables)
case of the second table, chi-square becomes unreliable, Yates’ correction of
continuity was performed instead. Although there are arguments in the research
community against its use (e.g. Haviland, 2007), Yates’ correction of continuity is
automatically calculated in SPSS statistical output to prevent overestimation of
statistical significance for small data.

Overall, it appears that the graphs and avatars as well as MessageTag received
higher scores than VIT Nodes and Centrality. It seems that the first were found more
practical and relevant to the specific practice. In particular, the visibility of active
participation levels structure in the graphs and avatars may have helped e-learners’
reflection on their participation. Their effectiveness is a device for information

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 207
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

retrieval as the visual objects (graphs and avatars) provide users with the elements
to explore and reflect. E-learning tools need to address such participation trajectories
(Suthers, 2006). Accordingly, the visibility of the collaborative learning structure in
MessageTag seemed to help e-learners’ reflection and increased challenges per
argument. However, if compared to the results in the previous studies, for
participants lacking certain skills, training and experience in association to using new
tools may have been a disadvantage. Nonetheless, opening interaction spaces in
participation, collaborative e-learning, and social networks seemed to be essential.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 208
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

6.7 INTERVENTION ANALYSIS

The intervention analysis can be depicted as follows (Table 6.7-1):

Table 6.7-1. Intervention Analysis

Target Intervention Analysis

A. Form of the effect


Level Adequate
Slope √
Variance -
Participation in the Greek Cyclicity Adequate
teachers’ collaborative e- B. Permanence
learning community Continuous √
Discontinuous -
C. Immediacy
Immediate √
Delayed -

The target in this study was to investigate participation in the Greek teachers’ e-
learning community and provide tools and evaluation techniques to support their
development. In the intervention analysis the form of the effect, as in level, slope, and
cyclicity, was successful. The treatment was immediate and its permanence was
continuous to this point in time. More specifically the overall discussion on the
suggested tools and evaluation techniques is as follows:

E-learning engineering was found to require five corresponding changes:

• a fundamental change in perspective;


• a commitment to situated approaches;
• the coherence of action as an emergent property of moment-by-moment
interactions between the actors and between the actors and their
environment;
• unification of design, evaluation, and use under the rubric of community
and real-time research; and
• building interventions in ‘time-boxes’ to achieve coherency of action.

Thus, the suggested model for e-learning engineering is (Table 6.7-2):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 209
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Table 6.7-2. E-learning engineering

E-LEARNING ENGINEERING
Design Stages Ethnotechnology
1. Prerequisites: real need & right R
A INTENT attitude
2. Goals: stakeholders’ goals, Scenic E
requirements & variances fieldwork
A
3. Documentary review L
4. Background studies: description of
stakeholders background & -
characteristics Ethnographic T
5. Interdisciplinary research inputs
• Evaluation for the targeted I
B PLANNING
pedagogical approach (intentional M
variance)
E
• Evaluation for the targeted
technical problem (operational
variance)
E
• Pedagogical Usability guidelines
V
6. Design for initial design
Implications A
• Apply guidelines from for design
feedback for design L
ITERATIVE 7. Create one prototype for user-learner
C U
testing
DESIGN
• Evaluation to acquire A
feedback for redesign T
8. Redesign and evaluate with user-
learner to redesign I

EVALUATION O
D 9. Implementation in situ Scenic
& USE 10. Evaluation & feedback fieldwork N

Initially, design was used as a problem solving activity in which the problem was broken
down into a number of sub-problems, solved independently and reintegrated to produce
a design solution. In order to do this, the value of unarticulated expertise and tacit
knowledge was acknowledged. Design was found to be a reflective process having a
post hoc nature (Suchman, 1987) and evolved over time integrating design, use and
evaluation (Bannon, 1994). Such approaches have been found in the history of
engineering design as nowadays design requires flexibility between various ways of
thinking (McGarry, 2005).

Collaborative e-Learning Episodes found to be a successful analytical framework


in regards to e-learners’ use as well as in their in-depth analysis. This structure
allowed e-learners to externalise their thoughts in order to reflect upon them. Their
attributes, initiation, explanation, exploration, evaluation, co-construction, were
visualised in MessageTag as Inform, Question, Explain, Explore, Agree, Evaluate,

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 210
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Summarise, and Other. Translating a theoretical framework into a tool was found
difficult and it was not easy to create a tool that could be used without instructions.
However, iterative design made the tool more accessible to the e-learners and it
became the most successful tool in this study.

Active and passive participation levels in collaborative e-learning were found to be


related to social interactions and thus grouping as the participants’ interactions were
more and more intimate. The interactions depended on the learning styles and the
personal learning goals as active and passive participation were found to have an
idiosyncratic character. E-learners’ interpersonal and intrapersonal skills were found
to control the dynamics of the collaborative e-learning networks. In other words,
learning emerges from passive and active participation in collaborative e-learning
suggesting intentional rather than reactive behaviour. The tools based on the
participation were successful; graphs and avatars were found to be in accordance to
their purpose, however, missing their log files because of their failure without having
time to repair them prevented triangulation from the results.

Visualisation Interactions Tools were VIT Nodes and VIT Centrality. They depicted
the local networks in a particular discussion and were found to be less important for
collaborative e-learning. As they were built on SNA, the participants were unfamiliar
with them and it was difficult to interpret their use and incorporate it in their practice.
Another explanation may be the absence of a direct match between the task and the
method (Draper, 1993); this means that SNA methods were not explicitly connected
to the Greek teachers’ educational practices and thus, their working methods.

The Moodle discussion application was found to be rigid; in fact some participants
complained about the simplicity of the forum when they got back to Moodle@GSN.
The tools aimed at reducing misleading information and interruptions by triangulating
the otherwise separated information context and e-learners’ network; representation
of the progressive dialogue and relationships between the messages can provide a
more coherent view of the research context. This also means that e-learning
participants can manipulate information, knowledge, and interactions as interrelated
areas of one situated context in order to achieve their goals.

The Sense of e-Learning Community Index (SeLCI) provided a different viewpoint


to describe the e-learning community and processes in depth (only the e-learners’
results were considered in social network analysis) (Table 6.7-3):

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 211
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Table 6.7-3. The Sense of e-Learning Community Index Checklist

THE SENSE OF E-LEARNING COMMUNITY INDEX

Attributes Moodle@GSN Research Pool


1 Community evolution √
2 Sense of belonging √
Intensity
3 Levels of participation Interrupted Coherent
Persistence 80% 95.3%
4 Empathy √
Social Network Analysis
Group Cohesion
Density 0.0256 0.0418
0.2174 –
Reciprocity – reciprocal ties 0.2222 – 4(28%)
10(71.4%)
12/3 actors
Cliques 4/3 actors
1/4 actors
- 5 1st level groups
- 3 second level groups
- 2 2nd level groups
Structural equivalence - 4 participants with identical
5 1 solo-actor
behaviour
positions.
Group Centrality
In-Degree 5.425% 6.671%
Out-Degree 10.826% 9.372%
In-closeness 32.68% 49.83%
Out-closeness 55.04% 42.53%
2(P32) 16.2 (P50)
Betweenness 1.7 (P24) 11.8 (P18)
0.9(P37) 7.3 (P24)
6 Trust √
Collaborative e-learning
quality
7 Participants’ opinions √
Number of collaborative e-
6 7
learning episodes.
Time 31 days 6 days
Overall √

SeLCI was found to be successful in describing, evaluating and triangulating the


research context. Sharing their interests and values, the participants responded in a
positive manner about community evolution, sense of belonging, empathy and trust.
Intensity was above the average level and more coherent in the research pool.
Establishing sociability and common ground was the first level towards a progressive
course in collaborative e-learning.

It is also interesting to note that the sense of community was evident in the participants’
communication outside the course. For example, the e-tutors other than myself met for
the first time in a conference in Athens (7 October 2007). When I asked them via

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 212
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

electronic communication whether they could describe the feeling they had they said that
it was like being old schoolmates. In addition, the respondents’ comments in Other on
the final questionnaire were in accordance to the checklist. For example, P14 said he
was looking forward to login; familiarisation with the other participants and the interface
was immediate even though it was the first time he was participating in an online course.
He didn’t expect to have the same strange feeling when something like a live conference
is over. There was an interesting observation from P35 about activities and
communication: “Many people were collaborating at the same time I saw that I can
change things with my participation both as a student and as a facilitator. The formal
educational authorities do not actually know what this is about; it is a programme in
parallel with the official one (παρα-πρόγραμμα). Hope new technologies will change
things”. The participants experienced innovative collaborative e-learning tools and
techniques and stressed the importance of incorporating them in the Greek teachers’
training cycles.

Comparison with recent studies: In the recent research, Daniel in his PhD Thesis
(2007), tested 11 dimensions to sense of community within 15 undergraduate students
using an e-learning environment. There were differences in the number, age, profession
and experience of the participants; additionally, as he used a questionnaire and message
analysis to obtain his results there were several differences and similarities (p.111-126).
For example, he measured reciprocity by assessing participants’ frequency of sharing
class related resources; he related trust to the levels of awareness and co-presence. He
also measured learning as knowledge awareness depending on information about other
e-learners’ activities, that was what individual knew (competence awareness) and what
they could do (capability awareness) (p.126-127). (Note that Daniel did not always
describe his findings using numeric representation.) The similarities were:

• Participants continued their networking outside and beyond the time frame of
the class (p.111).
• The feeling of belonging to the community was high (p.113).
• There was a high ratio of shared goals and values (p.116).
• On participation and social protocols 67% of the participants said they were
aware of them.
• Peer support was evident and reinforced members’ sense of belonging
(p.120).
• There were high rates of social networking.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 213
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

• Most participants did not know each other before the course (56% said they
knew nobody, p.124-125).
• E-learners’ knowledge awareness bred trust to the community. And
• People discovered they had similar problems and interests.

In another study on online social identity, knowledge construction and passive


participation for nurses participating in an online course, Gulati (2006) found that there
were different forms of learning and different types of e-learners depending on how
sociable they were. (Note that there are no page numbers as the thesis was not on a
PDF format and there were differences on the page numbers in the chapters.) Her
investigation was mainly qualitative and the similarities to this study were:

• The participants enjoyed learning together.


• They felt part of the group.
• They exhibited desire to be connected.
• They shared same interests and responsibilities.
• Online communication skills were crucial to enable participation.
• Time was limited.
• There were access problems.

To finalise, none of the participants knew the specific aims and objectives of this
project when they replied on the open question about the best things they
experienced in the course; these were: their participation in collaborative activities
(11 responses); feeling of belonging to something greater (7 responses); having
common interests and targets (6 responses); and the use of new technologies (9
responses). (For all results see A_XII_4). P2 said that she needs ‘to look for people
more, since there are others who look for the same thing and can be useful’. (Να
ερευνώ από δω και πέρα περισσότερο γιατι υπάρχουν τελικά και άλλοι που ψάχνουν
το ίδιο και μπορούν να φανούν χρήσιμοι πάντα.). P47 expressed her ‘satisfaction that
there are many colleagues interested in the further development of educational
practices’. (...ικανοποίηση ότι υπάρχουν αρκετοι συνάδελφοι που ενδιαφέρονται για
την βελτίωση εκπαιδευτικών πρακτικών.) P45 said that ‘the still waters start to move
in the area’. (...Πιστεύω ότι σιγά - σιγά αρχίζουν να κινούνται τα λιμνάζοντα ύδατα
στο χώρο.)

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 214
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

REFERENCES
Äkkinen, M. (2005). Conceptual Foundations of Online Communities (Working Paper
W-387). Helsinki: Helsinki School of Economics. Retrieved 14/02/2006, from
http://helecon3.hkkk.fi/pdf/wp/w387.pdf.
Bannon, L. J. (1994). Use, Design & Evaluation - Steps towards an Integration. In D.
Shapiro, M. Tauber & R. Traunmueller (Eds.), The Design of Computer-
Supported Cooperative Work and Groupware Systems (Vol. 12). Amsterdam,
The Netherlands: North - Holland.
Beaudoin, M. F. (2002). Learning or Lurking? Tracking the 'Invisible' Online Student.
The Internet and Higher Education, 5(2), 147-155.
Blau, P. (1964). Exchange And Power In Social Life. New York, NY: John Wiley &
Sons.
Boase, J., Horrigan, J., Wellman, B., & Rainie, L. (2006). The strength of internet ties.
Washington, DC: Pew Internet & American Life Project. Retrieved
09/05/2006, from http://www.pewinternet.org/pdfs/PIP_Internet_ties.pdf.
Bock, R. D., & Husain, S. Z. (1950). An adaptation of Holzinger's B-coefficients for
the analysis of Sciometric data. Sociometry, 13, 146-153.
Borgatti, S. P. (2005). Centrality and Network Flow. Social Networks, 27(1), 55-71.
Retrieved 23/11/2006, from
http://www.analytictech.com/borgatti/papers/centflow.pdf.
Borgatti, S. P., Everett, M. G., & Freeman, L. C. (2002). Ucinet for Windows:
Software for Social Network Analysis. Harvard, MA: Analytic Technologies.
Breiger, R. L., Boorman, S, A., & Arabie, P. (1975). An algorithm for clustering
relational data with applications to social network analysis and comparison
with multidimensional scaling. Journal of Mathematical Psychology, 12, 328-
382.
Braha, D., & Bar-Yam, Y. (2004). Information flow structure in large-scale product
development organizational networks. Journal of Information Technology, 19,
244–253.
Daniel, B. K. M. (2007). A Bayesian Belief Network Computational Model of Social
Capital in Virtual Communities. Unpublished Research, PhD Thesis,
University of Saskatchewan, Canada. Retrieved 01/08/2007, from
http://library2.usask.ca/theses/available/etd-07132007-
141903/unrestricted/ben_m.pdf.
De Nooy, W., Mrvar, A., & Batagelj, V. (2005). Exploratory Social Network Analysis
with Pajek. Retrieved 12/02/2006, from http://vlado.fmf.uni-
lj.si/pub/networks/data/esna/default.htm.
Delich, P. (2006). Pedagogical and interface modifications: What instructors change
after teaching online. Published doctoral dissertation, Pepperdine University,
Malibu, CA. Retrieved 10/09/2007, from
http://proquest.umi.com/pqdlink?Ver=1&Exp=09-09-
2012&FMT=7&DID=1144195631&RQT=309&attempt=1.
Draper, S. W. (1993). The notion of task in HCI. Paper presented at the Interchi'93, in
the Adjunct proceedings. Amsterdam, The Netherlands, April 24 – 29, 1993.
Retrieved 02/04/2006, from http://www.psy.gla.ac.uk/~steve/task.html.
Enyedy, N., & Hoadley, C. M. (2006). From dialogue to monologue and back: Middle
spaces in computer-mediated learning. International Journal of Computer
Supported Collaborative Learning, 1(4), 413-439.
Fernandes, C., & Montalvo, A. (2006). E-Quality Synthesis Report: Experience-based
Quality in European ODL.
Freeman, L. C. (1979). Centrality in Social Networks: Conceptual Clarification. Social
Networks, 1, 215-239.
Goleman, D. (2007). Social intelligence: The New Science of Human Relationships.
New York: Bantam Book.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 215
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Gulati, S. (2006). Learning during online and blended courses: Unpublished


Research, PhD Thesis at the Department of Education and Lifelong Learning,
City University, London.
Hair, J. F., Anderson, R. E., Tatham, R. L. T., and Black, W. C. (1998). Multivariate
data analysis, Prentice-Hall Inc., Upper Saddle river, New Jersey.
Hanneman, R. A., & Riddle, M. (2005). Introduction to social network methods.
Riverside, CA: University of California, Riverside.
Hannemann, R. A. (2001). Introduction to Social Network Methods. Retrieved
14/02/2006 from
http://faculty.ucr.edu/%7Ehanneman/SOC157/TEXT/TextIndex.htm.
Hartley, J. (2007). Teaching, learning and new technology: a review for teachers.
British Journal of Educational Technology, 38(1), 42-62.
Haviland, M.G. (2007). Yates's correction for continuity and the analysis of 2 × 2
contingency tables. Statistics in Medicine, 9(4), 363 - 367.
Hlapanis, E., & Dimitrakopoulou, A. (2004). Greek Teachers' Training via the Internet:
Presentation of the Greek Online Community KME from the University of the
Aegean. Paper presented at the 4th Conference in ICT in Education, 729-
736. 29 September – 3 October 2004, Athens, Greece.
Jeong, A., & Davidson-Shivers, G. (2006). The effects of gender interaction patterns
on student participation in computer-supported collaborative argumentation.
Educational Technology, Research, and Development, 54(6), 543-568.
Katsaros, M., & Karageorgiou, E. (2006). The primary school teachers absence from
their environmental education training in five years, 2001-2006 in the 4th
Sector of Fthiotida Prefecture- Investigating the reasons. Αποχή των
εκπαιδευτικών της 4ης Περιφέρειας Πρωτοβάθμιας Εκπαίδευσης του Ν.
Φθιώτιδας από την εφαρμογή προγραμμάτων Π.Ε. κατά την πενταετία 2001-
2006 - Διερεύνηση αιτιών. Paper presented at the 2nd National Conference
on Environmental Education. Athens, 15-17 December 2006. Retrieved
24/04/2007, from http://kpe-kastor.kas.sch.gr/kpe/yliko/sppe2/oral/PDFs/327-
334_oral.pdf.
Klemm, W. R. (1998). Eight Ways to Get Students More Engaged in Online
Conferences. The Higher Education Journal, 26(1), 62-64.
Khine, S. M. (2003). Creating a Technology-Rich Constructivist Learning
Environment in a classroom management module. In S. M. Khine & D. Fisher
(Eds.), Technology-Rich Learning Environments (pp. 21-39). New Jersey:
World Scientific.
Lambropoulos, N. (2005). EEEP Online Community of Practice: The First to Boldly
Go in Greece. "Open Education", The Journal for Open and Distance
Education and Educational Technology, 2, 29-56.
Lanzetta, J. T., & Englis, B. G. (1989). Expectations of cooperation and competition
and their effects on observers' vicarious emotional responses. Journal of
Personality and Social Psychology, 56, 543-554.
Lave, J., & Wenger, E. (1991). Situated Learning : Legitimate Peripheral
Participation. Cambridge [England] ; New York: Cambridge University Press.
Lorrain, F. P., & White, H. C. (1971). Structural Equivalence of individuals in social
networks. Journal of Mathematical Sociology, 1, 49-80.
McGarry, B. (2005). Things to Think With: Understanding Interactions with Artefacts
in Engineering Design. Unpublished Research, PhD thesis at the Department
of ITEE, School of Information Technology and Electrical Engineering,
University of Queensland. Retrieved 14/05/2006, from
http://eprint.uq.edu.au/archive/00003422/01/McGarry%20PhD%20Thesis%20
-%20Things%20to%20Think%20With.pdf.
Muij, D. (2003). Doing Quantitative Research in Education with SPSS. London: Sage.
Norman, D. A. (1988). The psychology of everyday things. New York, NY, USA:
Basic Books Inc.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 216
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Preece, J. (2004). Etiquette, empathy and trust in communities of practice: Stepping-


stones to social capital. Journal of Universal Computer Science, 10, 294-302.
Preece, J., & Maloney-Krichmar, D. (2003). Online Communities. In J. Jacko & A.
Sears (Eds.), Handbook of Human-Computer Interaction (pp. 596-620).
Mahwah: NJ: Lawrence Erlbaum Associates Inc. Publishers.
Safayeni, F., Duimering, P. R., Zheng, K., Derbentseva, N., Poile, C., & Ran, B.
(2008). Requirements Engineering in New Product Development: How
Effective are the Socio-Technical Interactions? Communications of ACM,
51(3), 77-82.
Salmon, G. (2000). E-moderating: the key to teaching and learning online. London:
Kogan Page.
Seo, J. (2005). Hierarchical Clustering Explorer 3.0. Human-Computer Interaction
Lab, University of Maryland, College Park. Available at
http://www.cs.umd.edu/hcil/hce/.
Suchman, L. A. (1987). Plans and Situated Action: The Problem of Human-Machine
Communication New York: Cambridge University Press.
Suthers, D. D. (2006). Technology affordances for intersubjective meaning making: A
research agenda for CSCL. International Journal of Computer-Supported
Collaborative Learning, 1(3), 315–337.
Tabachnick, B. G., and Fidell, L. S. (1996). Using Multivariate Statistics. Harper
Collins College.
Tsetsilas, I. (2006). Reporting teachers' training needs and design training schemes:
The case study from a primary school using D.I.O.N. model. Καταγραφή
επιµορφωτικών αναγκών των εκπαιδευτικών και σχεδιασµός προγραµµάτων
επιµόρφωσης: Η µελέτη περίπτωσης µίας σχολικής µονάδας της
Πρωτοβάθµιας Εκπαίδευσης µε το µοντέλο D.I.O.N.. PhD dissertation at the
Greek Open University, Patras, Greece.
Wallace, P. (1999). The Psychology of the Internet. Cambridge: Cambridge
University Press.
White, H. C., Boorman, S. A., & Breiger, R. L. (1976). Social structure from multiple
networks: I. Blockmodels of roles and positions. American Journal of
Sociology, 87, 517-547.
Willging, P. A. (2005). Using Social Network Analysis Techniques to Examine Online
Interactions. US-China Education Review, 2(9) (Serial No.10), 46-56.
Yates, F (1934). "Contingency table involving small numbers and the χ2 test". Journal
of the Royal Statistical Society (Supplement) 1, 217-235.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 217
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

Conclusions:

7 Summary of the thesis, contributions and recommendations

Key Topics covered in this chapter:

• How the thesis reached this point

• Exploring the contributions to existing knowledge

• How this study is limited to this particular context in time and e-


learning space

• Recommendations to the participants in the e-learning design

• Suggestions for future directions

• The conclusion derived from this study

Chapter 7 provides a summary of the thesis and explores its contributions to


knowledge. In addition, it presents the main blocking factors for the Greek teachers’
passive participation, the implications from the findings, the limitations, the
conclusions as well as some future research directions.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 218
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

7.1 INTRODUCTION

The aim of this chapter is to examine the scope of the findings, to put them into
the context, scrutinize the strengths and limitations of the work and suggest
implications for findings and directions for the future investigations might take.

7.2 SCOPE OF FINDINGS

In this section the scope of the research and the overall findings are described
in relation to the aim and objectives. This study aimed to carry out the following with
regard to Collaborative e-Learning Communities:

Thus, this study set out to answer a number of exploratory questions; the conclusions
are presented next.

Collaborative learning research is multi-disciplinary. Cognition is a complex social


phenomenon based on the individual’s re-arrangements of knowledge influenced by
social interactions. Social interactions provide the common ground for discussion that
occurs on two levels: internalisation of knowledge as reflection from passive
participation, and externalisation of knowledge as active learning from active
participation. These two levels are found to be brought together in collaborative
learning as a socially shared cognition problem solving and evaluation technique.
This process was reflected in the literature review in a progressive way, from the
study of the individual to the study of groups and communities. This exact process is
also reflected in the structure of the current educational systems that fail to follow the

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 219
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

last step towards community knowledge building and thus, resulting in the e-learning
passive mode. In other words, available learning management systems do not
support users’ in co-creating the e-learners-generated context.

There have not been many attempts to support collaborative e-learning from a
community viewpoint; finding suitable variables to describe and support increasing
participation was part of the problem. Additionally, design was found to be
inadequate to carry the social interactions; it is legitimate for the participant to be in
any participation level. As a result, all Greek teachers never crossed the threshold of
passive participation for 3 years because of personal, social, learning, institutional or
technical obstacles. In other words, if the e-learners were adequately supported by
tools and evaluation techniques then passive participation would drop to the
minimum.

No coherent frameworks were found to support increasing participation in


collaborative e-learning communities.

For the past 50 years two main trends have been observed in education, the socio-
cultural focus and the use of technology. However, these two trends have evolved
almost separately. Socio-technical design and user-centred design were planning
approaches aiming to acknowledge that the development of interactive technologies
increasingly relies on an appreciation of the social circumstances in which systems
are used. Design is planning; educational or instructional design is the systematic
processing of activities to solve an instructional problem with the aid of technologies.
However, educational design and in particular e-learning design neglected the dual
and situated persona of the learner; she acts as both a user and a learner. In
addition, the e-learning systems were found to be information-based mainly
supporting monologue instead of being communication-based towards dialogue. For
this reason they fail to support e-learners’ transition between internalisation to
externalisation and becoming active participants. Thus, mere provision of information
points to poor e-learning quality. So, if educational design could understand the
technology of collaborative practice, e-learning quality could be improved.

The first signpost was the radical view that design is planning with a post hoc nature.
This means that situated design needs to follow the evolution of its context, in this
case, the collaborative e-learning community. The second signpost was the
development of a social and technical infrastructure that supports the key activity, in

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 220
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

this study, collaborative e-learning. The third signpost was the real-time setting of the
key activity and its evaluation. The last signpost was the involvement of e-learning
participants in the design. These guidelines suggested ethnotechnology as the
methodology to inform design on a social, learning, and technical level.

No tools and techniques were found to evaluate the process of increasing


participation in collaborative e-learning communities as progressive discourse.
Several projects investigated the use of progressive dialogue to support collaborative
leaning. However, none of the projects targeted to the creation and development of
an active collaborative e-learning community.

Therefore the answers to the exploratory questions led to thesis contributions.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 221
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

7.3 CONTRIBUTIONS

The second set of questions suggested specific schemes to tackle passive


participation and poor e-learning quality. These were the key contributions of the
thesis. Nevertheless, there were also secondary contributions.

7.3.1 Key Contributions

The Collaborative e-Learning Episode (CeLE) was proposed as a coherent


and cyclical analytical framework to identify, analyse, and evaluate contributions in
Collaborative e-Learning Communities (CeLC). The main attributes were found to be:
inform, question, explain, explore, agree, evaluate, summarise, and other. The CeLE
was found to support Greek teachers’ progressive discourse, reflection, critical
thinking, and co-creativity.

The Sense of e-Learning Community Index (SeLCI) was developed and


successfully tested based on collaborative e-learning community sociodynamics and
collaborative e-learning. SeLCI consists of indicators to understand how a community
functions as well as being determinants for its success. These were: community
evolution; sense of belonging; empathy; trust; intensity characterised by e-learners’
levels of participation and persistence on posting; collaborative e-learning quality
measured by the number of CeLEs and participants’ comments on their learning; and
social network analysis based on: global cohesion anchored in density, reciprocity,
cliques and structural equivalence, global centrality derived from in- and out-degree
centrality and closeness; and local nodes and centrality in real time. These
determinants were analysed on a temporal basis towards the evolution of the
collaborative e-learning community.

In addition, a coherent and measurable framework was developed upon the


increasing participation process: zero, low, medium, and high passive participation,
and low, medium, and high active participation. Lastly, the transition between passive
and active participation was called the sleeper effect.

The post hoc structure for E-Learning Engineering was revealed integrating design,
evaluation, and use, this involves all e-learning participants in design, and is informed
by ethnotechnology and real-time evaluation tools.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 222
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

Quality by design in e-learning appeared to be related to a complex web of factors.


Some of these factors have been investigated and determined in each of the
previous exploratory questions. A conscious effort to tackle some of them was based
on the suggested tools and evaluation techniques found to have solved the initial
problem of passive participation and e-leaning quality.

The participation graphs and avatars that depicted active participation were found
to have supported increasing participation in collaborative e-learning. This was due to
the fact that they were built on the technology of participants’ real practice.

MessageTag was the tool built on the CeLE analytical framework. As with the
participation tools, it successfully supported e-learners because of its relevance to
the key activity, that was collaborative e-learning.

The Visualisation Interactions Tools Nodes and Centrality were found to be less
supportive in the e-learner’s endeavour and had the most problems from a
pedagogical usability viewpoint. However, it appears that their use aided the
participation of the Greek teachers in their e-learning community.

7.3.2 Secondary Contributions

The secondary contributions refer to e-research methods and pedagogical usability.

1. Ethnotechnology was the exploitation of ethnography in design aiming at


better understanding of the context under investigation. Although it is
unadvisable for Ethnotechnology to be conducted by one person being
ethnographer, designer, e-tutor and researcher it was unavoidable in this
study. The use of ethnotechnology provided insights by:
a. Facilitating investigation on relations within a situated context for
i. The collaboration and interaction between the Greek
teachers
ii. The role of external factors such as lack of legislation and
slow Internet connections
iii. The management of human-human interactions
b. Facilitating investigation on human-computer interaction for
i. The use of tools by users as learners

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 223
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

ii. The use of tools by learners as users


c. Supporting feedback for design on
i. The translation of tasks into tools
ii. The emergence of alternative options and perspectives
d. Increasing understanding for tasks and associated practices for
i. Investigating e-learning participants’ roles, needs, targets,
and visions
ii. Transferring of knowledge within the organisation
iii. Rising levels of expectations and enthusiasm
iv. Supporting community’s continuity and Greek researchers’
professional learning
v. Advancing best practices in researcher’s own community
2. Triangulating qualitative and quantitative data with social network analysis
increased:
a. The validity and reliability in the study
b. The context for diversity of perspectives
c. The emergence of alternative perspectives
d. The derivation of new concepts

3. The measurement of pedagogical usability of new tools as related to:


a. The dual persona of the learner as a learner and a user
b. The design for new tools
c. The evaluation by distinct groups of users, teachers, and
developers
d. The researcher’s self-reflection

These contributions need to be considered within the limitations that follow.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 224
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

7.4 THESIS LIMITATIONS

There were several research constraints: the nature of e-research; the


circumstances under this study was carried out including time frame and
unwillingness of the Greek educational authorities to cooperate; the sample was not
representative but rather based on participants’ special interest in Information and
Communications Technology in education; and limited research in collaborative e-
learning communities. In addition, the Hawthorn effect was evident; one participant
explicitly said that she wanted to contribute to this project.

From a design viewpoint, the tools were not implemented in the real e-learning
environment for the Greek teachers future use. In addition, I had too many roles and
limited programming skills resulting in not being able to tackle a problem on the log
files for three of the tools, on time. Also, there was no usability testing of the tools in a
usability laboratory.

Lastly, the suggested tools and evaluation techniques need to be further tested and
developed in different contexts to ensure their validity and reliability.

7.5 CONCLUSIONS

This final section presents the overall blocking factors responsible for the
Greek teachers’ passive participation, the implications from the findings, and future
research directions.

The problem of e-learning quality was found to be related to the Greek teachers’
passive participation. A network of institutional, instructional, technical, and personal
obstacles acted as blocking factors on a micro and macro level:

• Diversity on the concept of e-learning quality in current research


• Traditional low value of collaborative e-learning reflected on lack of
soft skills and training in online collaboration
• Lack of national policies and coordinated activities on behalf of
educational authorities
• Lack of a variety of e-learning evaluation tools and techniques

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 225
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

• E-learning in Greece is still an unofficial practice based on isolated


initiatives
• Lack of time for the Greek teachers to be engaged in life-long learning
• Lack of technical support
• Lack of perceiving the direction in education in the 21st century.

In fact, these blocking factors have been reported on a European level in the recent
E-quality report (Fernandes & Montalvo, 2006) and were also identified in my
participant observation in 2 European e-learning projects. Thus, the problem of e-
learning quality in the Greek School Network was not different or more intensive than
in other European countries. This also means that although this study was designed
to be strictly situated, the recommendations for researchers, e-learning engineers, e-
learning practitioners, and the Greek educational authorities may have an impact on
e-learning quality on a broader level (see recommendations in Appendix XIII).

Implications from the findings

The findings in this study demonstrate the significance of social, learning, and
technical aspects of e-learning. Two interventions were made, collaborative e-
learning and the introduction of new tools; the first was related to community
management and collaborative e-learning, the tools were built to facilitate them. The
conceptual frameworks of the participation levels, the Sense of e-Learning
Community Index (SeLCI), the Collaborative e-Learning Episode (CeLE) analytical
framework, and the associated tools aimed to bridge the social and learning gap.
Brief description of the findings, their implications and recent related studies will be
discussed next.

Within the context of the study, the participants’ prior knowledge and ability to interact
was enhanced (Yang, 2002); there was also lack of governmental planning for e-
learning pedagogy, technology, and legislation. The implications from the first finding
suggest the need for training the Greek teachers in current pedagogical collaborative
approaches and new technologies; improvement of soft skills and communication
techniques; development of shared vision and goals; access to professional help and
support. These implications point to the problem of e-readiness for e-learning (e.g.
Kaminsky & Currie, 2008). The failure for e-readiness is evident in the great
discrepancy between different ranks in Greece (The Economist Intelligence Unit &
IMB Corporation, 2003:16); on a scale from 1 to 10, 8.80 was the government rank,

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 226
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

5.87 the industry rank, and 5.66 the society rank. Thus, the implications is that
organisations and business need to understand and correctly judge real situations,
trace solutions as processes, and educate and support everyone involved in e-
learning.

From a community management viewpoint, time and activities management was


anchored in the passive and active participation levels, as well as the “sleeper effect”,
the gap between the previous two levels. Increasing participation from passive to
active fostered interaction flow and continuity and thus, these levels confirmed the
concept of learning as participation (Lave & Wenger, 1991; Kanes & Lerman, 2008).
An implication is that in such favourable circumstances, passive participants are
highly likely to get engaged. These implications stress the impact of e-learners’
control in their own process of engagement as learners are usually not
knowledgeable enough to make effective decisions (Dron, 2007).

Community development was built on shared background, experiences, goals, and


visions. This was to enhance interactivity, as a key component in assessing the
effectiveness of e-learning (e.g. Zhao et al., 2005; Thurmond, 2003). Implications
from this study suggest that interactivity has to be increased prior to e-learning
activities and facilitated by reciprocity, empathy and trust. Sharing personal
information and experiences through the profiles and developing a sense of
identification and co-presence between the e-learners can foster community building.
Observing other e-learners’ behaviour and develop reciprocity empathy and trust are
interpersonal and intrapersonal skills with an impact on participants’ motivation,
involvement, and learning. They are related to the participants’ idiosyncratic
character and influence participation. In fact, Squires (1999) suggested the need to
design for freedom and flexibility so that educational software can adapt to their
idiosyncratic needs and styles.

Tutors-learners and learners-learners interactions were investigated using social


network analysis on a macro level across the network and on a micro level within
small groups. The interaction speed was high when the e-tutors were involved,
indicating control of the speed of the information flow. An implication is to delegate
control in interactions to e-learners in order to stabilise the interaction speed. This is
because the differences between the participants are minimised. In social networks,
information flow is relevant to issues of productivity, innovation and the classification
of useful ideas (Wu et al., 2004). Controlling and delegating interactions can be

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 227
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

connected to controlling or delegating e-learning (Dron, 2007:63). Decentralising the


power in a network increases the possibilities for co-creativity and innovation within
an e-learning community. In addition, since interaction density was directly linked to
text richness, richness is a measure of the interactive learning process (Stahl, 2002).
Lastly, as the e-tutors created groups with similar behaviour around them, an
implication refers to simulating (cloning) e-tutoring via vicarious learning. The e-tutors
should adopt different learning and interaction levels and styles based on e-learners’
idiosyncratic character aiming for all learners to achieve their goals. Therefore,
technologies need to be able to adapt to individuals’ changing needs and situations.

Overall, social awareness and the development of the sense of belonging to a


community were only recently related to social intelligence. The latter has been
accepted as an important soft skill (Goleman, 2007) even though it was mentioned in
Thorndike (1920) as knowledge to manage social situations and ‘act wisely in human
relations’. An implication is linked to the collaborative learning strong socio-cultural
nature, thus social intelligence can be a significant factor for successful e-learning
communities. For example, social emotions require self-consciousness (Goleman,
2007:131) that can be with the creation of profiles, avatars and other tools aiming to
enhance presence and co-presence. Another implication is related to the new model
for distributed leadership implemented as public consultancy from governmental
organisations and business; anyone can participate in decision making by sharing,
voting and discussing ideas (e.g. Dell, http://www.dellideastorm.com).

The development of collaborative learning networks and shared practice has been
found to be themes for continuing professional development (e.g. Pickering et al.,
2007). First, communication and problem-solving skills were improved by the e-
learners’ interactions. They were evident in the more complex discussions at the end
of the course comparing to monologues as redundant messages at the beginning of
the course. The analytical framework of collaborative e-learning episodes (CeLE) and
the associated tool MessageTag managed to bridge methods and tasks and provide
simple to use and reliable assessment of the e-learning quality. An implication is that
increasing reciprocity can facilitate the transition from monologues to dialogues as
there are increasing clues as opportunities for critical engagement in dialogue. A
second implication is related to the e-tutors’ ability to guide the e-learners into the
journey of critical thinking and knowledge co-construction and then leave them on
their own capabilities.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 228
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

Overall, there is a need to re-think e-learning processes and methodologies and


make an effort to engage participants in the learning process (Rodgers, 2007).
Educators and designers need to re-visit teaching and learning strategies and their
relationship to the level of e-learners’ engagement taking under consideration the
strong idiosyncratic character of e-learners’ interaction and engagement. The direct
implication leads to the different learning and interaction styles needed in e-learning
(Cooze & Barbour, 2007).

The use of new tools was found to support collaborative e-learning to an adequate
extent. The implications are related to the need for numerical and graphical
evaluation tools that demonstrate their purpose of use. Such tools can increase e-
learners’ focus, intensity, and persistence in social and learning interactions. E-tutors
can direct and control online discussions, discover e-learners’ weaknesses and
strengths, activate the lurkers with specific questions, aid in team building by
“bonding” the team, and record the discussion. From a design viewpoint, several
groups need to evaluate learning technologies and applications anchored in their
different personas such a novice and expert users (Faulkner, 2000) as e-learners as
well as developers and e-tutors. A pedagogical usability evaluation framework can be
used to evaluate usability and utility of new tools. Designers need to seriously
consider the social aspects of learning, formative and summative evaluation, and
facilitate e-learners increase control as in self-emerged collaborative learning (Dron,
2007; Anderson, 2007). Such tools need to function on multiple levels supporting
interaction for individuals, small groups and networks or as Anderson (2008)
suggested, employ social software. This implication depicts the limitations of current
interactive applications; in the recent Communication of the ACM journal, Hendler
and colleagues stress that ‘today’s applications are very early social machines,
limited by the fact that they are largely isolated from another’ (2008:65).

The interventions built the types of communities of practice discussed by Kanes and
Lerman (2008). This is because the social and learning aspect was anchored in
legitimate peripheral participation as in Community of Practice Type 1 (CPT1).
However, it created an opposing dynamic in the Greek educational system by
suggesting that the community needs to move forward and acquire the new
competencies and technologies for the 21st century. This creates a conflict within the
educational system as with Community of Practice Type 2 (CPT2) (Kanes & Lerman,
2008): ‘CPT2 is built around tension, conflict and discontinuity of practice and

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 229
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

production’. An implication is to consider conflict as changing behaviour that can


facilitate the community’s movement to another level.

A final implication for research was connected to triangulating the data via
quantitative, qualitative and social network analysis; 3 different research perspectives
can open possibilities beyond initial propositions and hypotheses. Overall, there is
room for research on usable, useful and reliable tools and evaluation techniques to
assess different levels and types of participation and critical thinking in
collaborative e-learning.

Future research directions

Some future directions are proposed on research, e-learning readiness for


organisations, e-learning, e-learning communities, and associated design.

• Research
o Multidisciplinary research on the social, learning, and technical
aspects of e-learning activity.
o Ethnotechnology as a methodology to advance community members’
own practices.
• E-learning readiness
o Institutions and organisations assessment for e-readiness on a
technical and pedagogical level.
o Soft skills as part of teachers’ training and professional development.
o Resistance to change as a longitude study of e-learning communities.
• E-Learning
o E-learning management as a social as well as a learning process.
o The process of e-learning and the ways different learning styles
influence it.
o The e-learning styles in relation to social intelligence.
o The impact of the relationships between the Collaborative e-Learning
Episode’s attributes.
o Design for each of the Collaborative e-Learning Episode’s attributes.
o Design for learning objects and their re-contextualisation based on the
different critical thinking levels in Collaborative e-Learning Episodes.
o The relationships between personalised and collaborative e-learning
and their impact to e-learning quality.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 230
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

• E-Learning Community
o Community design, management, evaluation and development, and
the attributes that most facilitate e-learning to occur.
o The impact of the relationships between the Collaborative e-Learning
Episode’s attributes.
o Design for each of the Sense of e-Learning Community Index’s
attributes.
o Evaluation of participation and its impact on students’ performance.
o The impact of interactions in e-learning across a learning network,
within groups and on an individual level.
o The role of decision making and leadership in e-learning communities.
• Design
o Multimodal interaction and interoperability between educational
technologies.
o Design to support personalised and community e-learning
environments.
o Design to support community management and evaluation.
o Tools based on numeric and graphical representation for formative
and summative evaluation.
o The role of pedagogical usability in facilitating e-learning.
o Specific pedagogical frameworks and design on associated tools.

The discovery of the mirror neurons as the neurons that enable the representation of
other humans in the brain, social intelligence, current e-learning research, the new
communication technologies and social software, as well as this study, indicate that
collaboration is part of the human nature. Building on collaborative e-learning
communities can ensure e-learning quality; collaboration is written into our DNA.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 231
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

REFERENCES

Anderson, T. (2007, June). Book Review – Control and constraint in E-learning:


Choosing when to choose. International Review of Research in Open Distance
Learning, 8(2). 1-3.
Anderson, T. (2008). Social Software to Support Distance Education Learners. In
Anderson T. (Ed.)(2008), The Theory and Practice of Online Learning.
Athabasca University: AU Press, 221-241. Available at
http://cde.athabascau.ca/online_book/. Last access 21/07/08.
Cooze, M. & Barbour, M. (2007). Learning Styles: A Focus upon E-Learning
Practices and their Implications for Successful Instructional Design. Journal of
Applied Educational Technology, 4(1). Available at:
http://www.eduquery.com/jaet/JAET4-1_Cooze.pdf. Last access 20/01/2008.
Dron, J. (2007). Designing the undesignable: Social software and control.
Educational Technology & Society, 10(3), 60-71.
Faulkner, X. (2000). Usability Engineering. New York, NY: Palgrave, MacMillan.
Fernandes, C., & Montalvo, A. (2006). E-Quality Synthesis Report: Experience-based
Quality in European ODL. CD-Rom.
Goleman, D. (2007). Social intelligence: The New Science of Human Relationships.
New York: Bantam Book.
Hendler, J., Shadbolt, N., Hall, W., Berners-Lee, T. & Weitzner, D. (2008). Web
science: An Interdisciplinary Approach to Understanding the Web.
Communications of the ACM, 51(7), 60-69).
Kaminski, J. and Currie, S. (2008). Planning Your Online Course. In Hirtz, S &
Harper, D. (2008), Education for a Digital World: Advice, Guidelines, and
Effective Practice from Around the Globe. Vancouver, British Columbia, Canada:
BCcampus and Commonwealth of Learning, 191-211. Available at
http://www.col.org/colweb/site/pid/5312. Last access 21/07/08.
Kanes, C. and S. Lerman (2008). Analysing Concepts Of Community Of Practice.
New Directions for Situated Cognition in Mathematics Education, 45, (Series:
Mathematics Education Library): 301-326.
Lave, J. and E. Wenger (1991). Situated Learning: Legitimate Peripheral
Participation. New York, NY: Cambridge University Press.
Pickering, J., Daly, C. & Pachler, N. (2007). New Designs for Teachers’ Professional
Learning. Bedford Way Papers. London: Institute of Education, UoL.
Rodgers, T. (2007). Measuring Value Added in Higher Education: a Proposed
Methodology for Developing a Performance Indicator Based on Economic Value
Added to Graduates. Education Economics, 15(1), 55-74.
Squires, D. (1999). Educational software and learning: Subversive use and volatile
design. Available at
http://csdl2.computer.org/comp/proceedings/hicss/199/0001/01/00011079.pdf
Last access 15/08/07.
Stahl G. (2002). Contributions to a theoretical framework for CSCL. In G. Stahl (Ed.)
(2002) Computer support for collaborative learning: foundations for a CSCL
community, (Cscl 2002 Proceedings), Mahwah, NJ: Lawrence Erlbaum
Associates.
The Economist Intelligence Unit and IMB Corporation (2003). The 2003 E-learning
Readiness Rankings. While Paper, Seattle: IBM.
Thorndike, E. L. (1920). Intelligence and its use. Harper's Magazine, 140, 227–235.
Thurmond, V. (2003). Examination of Interaction Variables as Predictors of Students'
Satisfaction and Willingness to Enroll in Future Web-Based Courses while
Controlling for Student Characteristics. Doctoral Dissertation, University of
Kansas, US.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 232
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 7: Conclusions

Yang, Y. S. (2002). A case study for promoting collaboration on online project-based


learning. Proceedings of the World Conference on Educational Multimedia,
Hypermedia & Telecommunications, USA, 2107-2112.
Wu, F., Huberman, B.A., Adamic, L.A. & Tyler J.R. (2004). Information Flow in
Social Groups. Physica A, 337, 327-335. Available at
http://www.hpl.hp.com/research/idl/papers/flow/flow.pdf Last access 15/03/2008.
Zhao, Y., Lei, J., Yan, B., Lai, C., & Tan, H. S. (2005). What makes the difference? A
practical analysis of research on the effectiveness of distance education.
Teachers College Record, 107, 1836-1884.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 233
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

This is the English version of the Appendixes.


In some occasions, only samples of the data are presented.

APPENDIX I: ONLINE COMMUNITY MANAGEMENT

A_I_1. Moderator’s Responsibilities - Interactivity Management

1. Information.
− Informative first page: Ensure that the community’s first page provides all
needed information to encourage and inspire the students to engage.
− Introduction of themes in collaboration with the students and the Students
Administration Office.
2. Registration system: Facilitate the registration process if necessary.
3. Welcome note: A welcome note inspires and encourages the students to
participate as well as giving additional information on technical and management
issues (e.g. software description, inappropriate behaviour etc.)
4. Profiles: The students need to be encouraged to construct their profiles. Profiles
provide a feeling of co-presence and enhance the sense of belonging.
5. Induction and training: An initial meeting for using Moodle might bring issues of
usability of the system and suggest the problems students have on using the system
on site (if any). Additionally, information will be provided on ways for writing, replying
and form an online message.
6. Subgroups: Based on students’ research interests as well as hobbies sub-groups
will create initial locus of interactions.
7. Initial one-way communication: Introduction of the students (research interests,
hobbies, personal information that would like to share, experiences etc) as well as
tools (e.g. votes, polls, surveys, newsletters) can break the ice and give the
necessary information to move to two ways of communication and productive
interactions.
8. First message: The first of each student might define later behaviour. An initial
authentic reply and warm welcoming will encourage students to continue
communication as well as shy students to send a message.
9. Discussion highlights as newsletters: A monthly newsletter that will provide a
summary of the discussions and any additional information. Newsletters inform
members for news, activities, make members aware of the previous issues and
develop a felling of belonging to a community.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 234
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

10. Motivation of silent participants: Moderatos need to ‘kick’ the students to


contribute to the community.
11. Monitoring and control: Subjects and discussions moderation could enhance or
prevent specific issues to be brought on the surface.
12. Use of the expert: Students might need more formal or expert advice in addition
to confidentiality that a community cannot provide.
13. Help and Support: The students need to feel that there is always someone there
on a 24/7 basis.
14. Give the members ownership of their learning and learning outcomes (work
behind the scene).

A_I_2. Suggestions for Writing Online Messages

Twenty five (25) messages (N=47, 53,1%) appeared to have a pattern: an initial
introduction as a response to the selected message, an extensive explanation and
justification of their point was made, an example was making suggestions very clear
and lastly, a greeting or an interesting quote used to ‘sign’ the message. A detailed
description is following:

1. Introduction, usually with an agreement with a previous message;


2. arguments and points of view;
3. an example to support the previous suggestions
4. stress of interesting points, more suggestions; and
5. signing out.

A second suggestion from Participant A1 in the e-mmersion study is as follows:

‘The lines in text on paper should,


for the sake of readability not
exceed 55-60 characters. On the
screen lines should probably be
even shorter.

Then you get longer texts. According


to Jacob Nielsen Internet user only
skim webpages for headlines and
marked keywords and they do it
very fast. IMO this makes online
communication and collaboration on
sophisticated issues more or less
impossible.’

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 235
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

APPENDIX II: RISK MANAGEMENT

The British Institute of Risk Management (IRM), the Association of Insurance and
Risk Managers (AIRMIC) and the National Forum for Risk Management in the Public
Sector (ALARM) (2002) have published a standard risk management procedure. Risk
can be defined as the combination of the probability of an event and its
consequences (ISO/IEC Guide 73). Risk assessment found to be important for this
study after the initial findings on the absence of cooperation with the Greek
educational authorities as well as the absence of the Greek teachers’ participation in
collaborative activities. Risk management increases the probability of success and
also reduces the probability of failure and uncertainty. Even though it is usually a
continuous and developing process, it was only used for the specific purpose of
completing the course under the desirable circumstances. In other words, a shift to a
different research context was not desirable.

Risk description was based on risk analysis, identification, description, and


estimation as well as risk reporting and decision making.

Risk Description

• Absence of cooperation between the Greek


educational authorities
1. Name of risk
• Absence of participation in collaborative e-learning
for 3 years
• The Greek educational authorities may have
provided a document to facilitate the conduction of the
research, however, it cannot ensure that the stakeholders
can take actions to support this study.
2. Scope of risk
• The Greek teachers’ absence of participation in
collaborative e-learning activities does not mean that they
will not cooperate in the study; fieldwork suggested that if
they were given the adequate help and support they will.
• Operational
3. Nature of risk
• Knowledge management
• The computers engineers that provide the technical
support do not have any additional benefits if they
implement the proposed tools other than some publications
in their CV. In addition, their contracts have not been
renewed which indicates a negative motivation to conduct
4. Stakeholders extra work.
• The Greek teachers can acquire knowledge on the
pedagogical use of new technologies. Two certificates can
be provided by London South Bank University and the
Greek School Network, used for their professional
development.
• Situated implementation in a real environment is the
5. Quantification of
ultimate goal for this study. However, it seems that this is
Risk
not possible to be done under the specific circumstances. In

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 236
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

addition, it is not certain that finding a different environment


can solve this problem.
• Collaborative e-learning is the pedagogical goal of
this study and it is highly likely to achieve it. The advantage
of having an inactive community for 3 years is the increase
of reliability and validity of the study. In addition, this study
has a national Greek character and this is another
advantage for the study as well as for me as being a Greek
teacher.
• Not implementing the tools in a real and situated
environment can turn the study to quasi experimental.
6. Risk Tolerance
• Not participating in collaborative e-learning will be a
failure of the suggested propositions in total.
• There is nothing that can be done to change the
situation for the computer engineers as this project is not
7. Risk treatment and
part of their everyday work.
control mechanisms
• The Greek teachers can be taught and supported
towards collaborative e-learning.
• An alternative environment linked to the e-learning
environment at the Greek School Network can be a middle
solution. The study conducted in a research pool cannot not
be completely experimental if linked to the real one; another
8. Potential action for
is the Internet nature. (Studies on the internet are highly
improvement
unlikely to be completely controlled.)
• Theory and fieldwork suggested that if appropriate
conditions are in place then absence of participation can be
eliminated.

• A server is needed to host the research pool


• Initial work on instructional learning is acquired by
9. Strategy and Policy
producing documents on the use of the e-learning
Development
environment as well as collaborative e-learning. Continuous
help and support can reduce participants’ uncertainty.

Based on the risk description, it appears that in the first case of implementing the
tools in the real e-learning environment developed for the Greek teachers the risk is
very high. Despite the fact that the Deputy Director of the Greek School Network
Technical Support said that the tools will be implemented, the risk estimation is 95%.
(A high risk is usually more than 25%). This is because cooperation with the Greek
educational authorities found to be 0% on more than one occasion. However, in the
second occasion of supporting Greek teachers in e-learning the risk is very low, less
that 2%. This is because the chance of enabling participation is considered to be
more than 75%.

The above description and risk estimation suggest that a shift to the study from a real
implementation to a quasi experimental one cannot completely alter the results and
significantly affect reliability and validity. In other words, the Hawthorn effect is
always a threat in research either in real or experimental environments. Having a

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 237
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

linked research pool can be the implemented measure to tackle the first risk;
provision of adequate help and support based on initial instructional learning and
then shift to collaborative e-learning can be the risk treatment for the second risk of
absence of participation.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 238
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

APPENDIX III: INITIAL QUESTIONNAIRE (SAMPLE)

A_III_1. The usability section in the Initial Questionnaire

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 239
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

APPENDIX IV: QUESTIONNAIRE MAIN STUDY I (SAMPLE)

A_IV_1. Demographic data

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 240
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

APPENDIX V: QUESTIONNAIRE MAIN STUDY II

Dear colleague user,

Your contribution to this research is more valuable. The information you will provide
will provide a process for the Greek teachers’ active participation in the online
community and will reduce the time of engagement. In addition, your opinions and
propositions will present existing e-learning problems. The information you will
provide are strictly confidential and will only be used for research purposes. The
results from this study will be available to you after its completion (September 2007).

Despite the fact that the questionnaire is rather long, there was an effort to be easy to
answer, with accuracy and speed in 30 minutes. It is consisted of the following 5
sections:

Α. The e-learning community


Β. Participation in the e-learning community
C. Learning in the e-learning community
D. New tools’ use and usability
Ε. Professional development

The questionnaires should be sent to Niki Lambropoulos until the 15th of April 2007
the latest, at the email nikilambropoulos@gmail.com

You can use this text box to write your name and address in Greece in order to send
you the certificates of participation from London South Bank University and the
Greek School Network. They will be sent after all questionnaires are sent, late April
2007 the latest.

Thank you very much for your participation.

Best regards,

Niki Lambropoulos

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 241
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

FINAL QUESTIONNAIRE
You are invited to dedicate some of your precious time to fill in the following questionnaire. You can
fill it in the way you prefer as there are not right and wrong answers –only different pinions-; therefore
some questions are not very clear. The questionnaire was based on Word Processor form function; you
simply click in the grey areas and select one of the answers (unless stated otherwise) on a scale from 1
(low) to 5 (very much) or simply write. The selection Don’t know / Don’t answer is referred as N/A.

Α. THE E-LEARNING COMMUNITY

1). How many people did you know in person before the online course?

(a) Nobody (b) A few (c) Almost everyone (d) N/A

2). Do you think there was community development in time on the online
course?

(a) Yes (b) No (c) N/A

2ai. If you think community development was evident, can you describe some
elements that prove it?

2bi. How did the discussion forums help/restrict the development of a sense of
belonging to the e-learning community?

2ci.Was it comfortable or uncomfortable to talk to an online environment?

3). Did you develop any relationships, either friendly or professional with
the other co-learners?

(a) Yes (b) No (c) N/A

3ai. If yes, do you think they will continue outside the online course?

(a) Yes (b) No (c) N/A

4). In your own opinion, what was the element that hold the community
members together, if any?

5). In your own opinion, most of the participants shared their values.

(a) Yes (b) No (c) N/A

6). In your own opinion, most of the participants had common interests
and goals.
(a) Yes (b) No (c) N/A

7). How much do you think that the members were helping each other in
the online course?

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 242
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

(a) Very (b) Not much (c) A little (d) Almost no help

(e) N/A

8). How long (in days) do you think it took for the community to emerge
– develop a sense of working together?

9). Was the sense of being together strong?

(a) Yes (b) No (c) neither yes or no (d) N/A

10). Do you think that... 1 2 3 4 5


...you knew what another person was feeling when
a
you were reading her/his message?
...you could feel what a person was feeling when
b
you were reading her/his message?
c ...you took an action upon it?

11). Do you agree or disagree with following statements:

11i. I can trust most of the participants.

(a) Yes (b) No (c) N/A

11ii. Somebody has to be very careful in the online course because some
participants wanted to take advantage of people and situations.

(a) Yes (b) No (c) N/A

11iii. Most of the participants were trying to help.

(a) Yes (b) No (c) N/A

11iv. Nobody trusted no one as regards knowledge exchange and contribution.

(a) Yes (b) No (c) N/A

1 2 3 4 5
In a scale from 1-5, where 1 means
12). very little and 5 a lot, how much do Very Small Great Very
Neit
you thin the participants trusted Small exten exten great
her
each other in the online course? extend d d extend
Individuals who had similar
a
specialisation
Individuals who had similar writing
b
skills

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 243
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

c E-tutors
Individuals who seemed to have
d
advanced knowledge on the subject
e You cant trust anyone

13). In general, do you think that the level of trust during the online
course:

(a) Increased (b) Decreased (c) Remained the same (d) N/A

Β. PARTICIPATION IN THE E-LEARNING


COMMUNITY

14). Do you think that... 1 2 3 4 5


...you knew the community’s netiquette for proper
a
interaction?
b ...you knew the kind of community you were participating?
c ...you liked to work together with the other members?
d ...you felt free to express yourself?
e ...you participated actively in the discussions?
... the e-learning management system helped the e-learning
f
community?
g ...the new tools helped the e-learning community?

15). Do you think that YOUR participation was useful to the e-learning
community?
(a) Yes (b) No (c) N/A

15ai. If YES, can you describe why?

15bi. If NO, can you describe why not?

16. After your registration to the online course, from whom did you learn issues
that were of your interest?

16a After your registration to the online course, how did you learn issues that
were of your interest? Can you describe some ways?

17). Do you thin that there were roles developed between the members?

(a) Yes (b) No (c) N/A

17ai. If YES, what were these roles (please give an example)?

18. Do you think that active participation is necessary?

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 244
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

(a) Yes (b) No (c) N/A

18ai. If YES, why?; 18bi. If NO, why not?

19. Do you think that the members who DID NOT participate actively had to
remain in the course?

(a) Yes (b) No (c) N/A

20). Do you think that new members in the e-learning community can contribute
to existing knowledge?

(a) Yes (b) No (c) N/A

20a. If YES, in what ways?

20β. If NO, why not?

21). Do you think that cooperation with other professional communities (e.g.
programmers, multimedia developers, etc) is necessary?

(a) Yes (b) No (c) N/A

21a. If YES, why?

21b. If NO, why not?

22). The participants were specialised in different areas. Do you think that this
helped collaboration in the online course?

(a) Very (b) Not much (c) A little (d) No (e) N/A

C. LEARNING IN THE COLLABORATIVE E-


LEARNING COMMUNITY

23). Did you learn ways to enhance collaboration using new tools in the
online course?

(a) Yes (b) No (c) N/A

23a.. If YES, how did you learn them?

24). What were the most important examples of collaborative learning for
you?

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 245
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

25). Who did you MOSTLY turn for help when you faced some
problems?

(a) to the e-tutors (b) to other e-learners

(c) nobody (d) to a different resource (e) N/A

D. NEW TOOLS’ USE & USABILITY

26 Frequency for using the tool in Moodle... 1 2 3 4 5


a Participation – participation graphs and Avatar
b Kind of reply – message tag in reply
c Sociogramme VIT Nodes
d Sociogramme VIT Centrality

27 Did you think that the tools was attractive? 1 2 3 4 5


a Participation – participation graphs and Avatar
b Kind of reply – message tag in reply
c Sociogramme VIT Nodes
d Sociogramme VIT Centrality

Was the additional information with the new tools on


28 Moodle tiring?
1 2 3 4 5

a Participation – participation graphs and Avatar


b Kind of reply – message tag in reply
c Sociogramme VIT Nodes
d Sociogramme VIT Centrality

29 How would rate the quality of the graphics? 1 2 3 4 5


a Participation – participation graphs and Avatar
b Kind of reply – message tag in reply
c Sociogramme VIT Nodes
d Sociogramme VIT Centrality

Did you think that the tool design was functional (easy
30 to use)?
1 2 3 4 5

a Participation – participation graphs and Avatar


b Kind of reply – message tag in reply
c Sociogramme VIT Nodes
d Sociogramme VIT Centrality

31 How would you rate the originality of the tool? 1 2 3 4 5


a Participation – participation graphs and Avatar

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 246
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

b Kind of reply – message tag in reply


c Sociogramme VIT Nodes
d Sociogramme VIT Centrality

32 Were the instructions on the tools satisfactory? 1 2 3 4 5


a Participation – participation graphs and Avatar
b Kind of reply – message tag in reply
c Sociogramme VIT Nodes
d Sociogramme VIT Centrality

Did you think that learning how to use the tool is time
33 consuming and tiring?
1 2 3 4 5

a Participation – participation graphs and Avatar


b Kind of reply – message tag in reply
c Sociogramme VIT Nodes
d Sociogramme VIT Centrality

34 Did you think that accessibility of the tool was easy? 1 2 3 4 5


a Participation – participation graphs and Avatar
b Kind of reply – message tag in reply
c Sociogramme VIT Nodes
d Sociogramme VIT Centrality

Were there any failures to complete an activity


35 because of tool failure?
1 2 3 4 5

a Participation – participation graphs and Avatar


b Kind of reply – message tag in reply
c Sociogramme VIT Nodes
d Sociogramme VIT Centrality

Did you think that the tool responded relatively


36 quickly?
1 2 3 4 5

a Participation – participation graphs and Avatar


b Kind of reply – message tag in reply
c Sociogramme VIT Nodes
d Sociogramme VIT Centrality

Did you think that the tool responded to the


37 educational goals?
1 2 3 4 5

a Participation – participation graphs and Avatar


b Kind of reply – message tag in reply
c Sociogramme VIT Nodes
d Sociogramme VIT Centrality

38 Did you think that the tool enhanced motivation for 1 2 3 4 5

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 247
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

communication and collaboration?


a Participation – participation graphs and Avatar
b Kind of reply – message tag in reply
c Sociogramme VIT Nodes
d Sociogramme VIT Centrality

Did you think that the tool was effective to facilitate


39 collaborative learning?
1 2 3 4 5

a Participation – participation graphs and Avatar


b Kind of reply – message tag in reply
c Sociogramme VIT Nodes
d Sociogramme VIT Centrality

40 How satisfied were you using... 1 2 3 4 5


a Participation – participation graphs and Avatar
b Kind of reply – message tag in reply
c Sociogramme VIT Nodes
d Sociogramme VIT Centrality

1) Ε. PROFESSIONAL DEVELOPMENT
WITHIN CoP
41. Which one of the statements describes the common space between discussions
and your own educational practice? Please choose accordingly (more than one
answers are allowed):

(a) I brought ideas/information from my job to the discussions

(b) Some e-learners provided information that made me think on my educational


practice
(c) Some e-learners provided information that triggered ideas for my educational
practice
(d) The discussions will not change my educational practice

(e) I found ideas/information I can use in my educational practice

(f) Other

If Other, can you describe what is was?

42. The best thing(s) in my participation in the study was…


43. The worst thing(s) in my participation in the study was…

44. How many times have the educational authorities from the Greek Ministry of
Education contacted you to ask your opinion on learning technology issues?

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 248
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

(a) Many times (b) Some times (c) A couple of times (d) Never
45. How many times have the educational authorities from the Greek
Pedagogical institute contacted you to ask your opinion on learning technology
issues?

(a) Many times (b) Some times (c) A couple of times (d) Never

46. Do you have any other comments?

Did you have any problems with the questionnaire?

You can use this space for more comments and observations:

OBSERVATION REPORT No.

Name (Optional)

Description of activity Comments


1
2
3
4
..

Please check whether you have replied to all questions.

Thank you very much for your participation in the study.

Niki Lambropoulos
Research Student,
Centre for Interactive Systems Engineering
Faculty of Business, Computing and Information Management
London South Bank University, London, United Kingdom
http://www.lsbu.ac.uk/bcim/research/cise/ * http://nikilambropoulos.org &
1 Dale Grove London N12 8EE U.K.
Tel: +44(0)2084465909
Email: nikilambropoulos@gmail.com * niki@intelligenesis.homechoice.co.uk
Skype: niki.lambropoulos

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 249
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

APPENDIX VI: PARTICIPANTS’ DOCUMENTS

Only samples from the participants’ documents are presented here; these were the
following:
A_VI_1. Invitation to the study

A_VI_2. Netiquette

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 250
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

A_VI_3. Instructions of use Moodle and the Research Pool

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 251
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

APPENDIX VII: THE E-MMERSION DATA ANALYSIS


A_VII_1. Demographics

Demographics
Variables Values # Participants Percent
20-30 7 22.6
30-40 10 32.3
1 Age
40-65 14 45.2
Missing 0 0
Female 12 39
2 Gender Male 19 61
Missing 0 0
1-5 11 35.5
6-10 5 16.1
Working
3 11-20 11 35.5
Experience
40+ 4 12.9
Missing 0 0
1-5 4 12.9
6-10 14 45.2
Use of computer
4 11-20 10 32.3
- software
20+ 3 9.7
Missing 0 0
No use 1 3.2
1 -2 h 17 54.8
Use of new
3-5 h 6 19.4
5 technologies in
6-12 h 4 12.9
class: h/w
12+ h 3 9.7
Missing 0 0
No use 0 0
1 -2 h 13 41.9
Use of new 3-5 h 8 25.8
6 technologies for
6-12 h 5 16.1
education: h/w
12+ h 5 16.1
Missing 0 0
No use 1 3.2
1 -2 h 21 67.7
Use of Internet 3-5 h 4 12.9
7
in class: h/w 6-12 h 2 6.5
12+ h 3 9.7
Missing 0 0
No use 0 0
1 -2 h 8 25.8
Use of Internet
3-5 h 8 25.8
8 for education:
6-12 h 11 35.5
h/w
12+ h 4 12.9
Missing 0 0
0 12 38.7
1-5 16 51.6
9 Use of LMS
6-10 2 6.5
Missing 1 3.2
Simple 7 22.6
OK 19 61.3
Moodle Difficult 2 6.5
10 N/A 2 6.5
(std.d: 0.92)
Other 1 3.2
Missing 0 0

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 252
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

A_VII_2. Crosstabulation: Internet Use in class * Use for educational purposes


Internet use for educational purposes (h/w)
1 -2 h 3-5 h 6-12 h 12+ h Total
no use 0 1 0 0 1
Internet 1 -2 h 8 4 8 1 21
use in
class 3-5 h 0 2 1 1 4
(h/w) 6-12 h 0 0 1 1 2
12+ h 0 1 1 1 3
Total 8 8 11 4 31

A_VII_3. Crosstabulation: Moodle Use * Time using LMS


Time using LMS (years)
Total
0 1-5 6-10 nil
Simple 4 2 1 0 7
OK 6 11 1 1 19
Moodle Difficult 1 1 0 0 2
Use
Don't
1 1 0 0 2
Know
Other 0 1 0 0 1
Total 12 16 2 1 31

E-Learners' perseptions on
Moodle usability level

2, 7%
2, 7% 7, 24%

Easy
OK
Difficult
Don't Know

Missing: 1

18, 62%

A_VII_4. Moodle Usability

A_VII_5. Training in Educational Technologies

Training in Educational Technologies


Training locus # Participants Percent
1 Old academies 1 3.2
2 Universities 4 12.9
3 Postgraduate studies 4 12.9
4 Courses 14 45.2
5 N/A 4 12.9
6 Other 4 12.9
Total 31 100.0

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 253
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

A_VII_6. Reasons for participating in e-learning communities

A_VII_7. Use of e-learning tools


Web Design Pool:

A_VII_8. Messages Quantitative Analysis

MESSAGES QUANTITATIVE ANALYSIS


Richness of text
Forums Topics Threads Depth Date
(# words)
(P) 4/5 522 30/03-11/04/06
1 Ask the expert 3 (P) 2 302 27/03-09/04/06
(P) 2 233 30/03-09/04/06
4/8 59 27/03-07/04/06
2 Problems 3/6 (P) 10 838 03-05/04/06
2 48 27/03-04/04/06
Quality in
3 1/5 6/7 558 29/03-10/04/06
Education
5/5 282 29/03-03/04/06
4 Hypertext - Url 2
6/6 247 27-31/03/06
Files
5 1
management
Files
6 1
characteristics
7 Navigation 1 4/4 863 28/03-03/04/06
8 Menu 1 3/3 431 28/03-15/04/06

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 254
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

9 Website Plan 1 4/4 495 28/03-12/04/06


10 > left-right click 1 4/4 138 28/03-12/04/06
(P) 3/3 235 27/03-04/04/06
11 Layout 2
(P) 1
12 Colour palette 1 2/2 146 28/03-12/04/06
(P) 3/3 108 27/03-15/04/07
13 What to avoid 3 3/3 115 29/03-12/04/06
2
14 Hosting 1 6 538 03-12/04/06
Total 14 23 73 6,158 27/03-15/04/06
Per
0.7/d 1.2/d 3.8/d 324.1/d 19 days
day

A_VII_9. Forums and users view log files

LOG FILES (27/03-15/04/2006 – 19 days)


Percent on total
Type of View Views Average per day
views (2,493)
1 Forum view discussion 758 39.9 30.4
2 Forum view forums 61 3.2 2.4
3 Forum view forum 567 29.8 22.7
4 Forum add post 73 3.8 2.9
5 User view all 59 3.1 2.4
6 User view 171 9 6.9
Total 1,689 88.9 67.7

A_VII_10. Participation in e-learning Communities

A. Active Participation in E-Learning Communities


Importance # Responses Percent
1 Yes 29 100
Missing 2 6.2
Reasons
Knowledge exchange –
1 8 27
active learning
2 The community can exist 6 20
Responsibility for own
3 3 10
learning & self-esteem
Learning is more
4 1 3
interesting
5 Problem solving 1 3
6 Achieving learning targets 3 10
7 Collaborative learning 5 17
8 Win-win situation 1 3
9 Expands learning further 2 7
10 N/A 1 3.2
Missing 4 12.4

B. Time to Develop a Sense of Community


Time Responses Percent
1 1-2 days 18 58
2 3-4 days 2 6
3 5-7 days 3 10
4 Up to 2 weeks 3 10

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 255
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

I don’t feel part of the


5 5 16
community
Missing 0 0

A_VII_11. Online course experience


Online Course Experience
Reasons Responses # Responses Percent
Learning in e-learning 6 18
Interesting & useful
4 12
experience
The new tools 3 9
Interaction and participation 12 37
The course was in Greek 1 3
The best thing e-learning community 1 3
1 Time issues for course
was... 1 3
organisation
New ideas 2 6
Participation in e-learning
2 6
research
My limitations in the use of
1 3
ICT
Total 33 (28/31) 100
Unclear instructions 2 14
Lack of time 7 51
The worst thing Technical problems 2 14
2
was… The course was short 2 14
Fear of unknown 1 7
Total 14 (14/31) 100
Exciting experience 1 5
Life-long learning 2 9
e-learning is the future 1 5
New approaches 1 5
Low implementation 1 5
More e-learning initiatives 2
More evaluation &
3 More comments 4 17
feedback
Better implementation of
4 17
the tools
Better e-learning quality in
3 14
Greece
More chat use 1 5
More Moodle usability 2
Total 22 (6/31) 100

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 256
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

A_VII_12. Messages Analysis: Collaborative E-Learning Episode I


COLLABORATIVE E-LEARNING EPISODE I
Levels of Abstraction
CeLE parameters #Codes Indicators #References
Info, statement, definition 57
Social cues (nice behaviour, thanks, greetings) 19
Question 32
Url 24
Initiation: Problem 20
1 9 185
Question - Information Bullet points 11
I think, I believe 9
Instruction 7
I know, have worked 5
Image 1
Individual solution 28
Example, further explanation 23
2 Explanation 4 57
Because, this is why, thus, therefore 3
Help 3
Yes, I agree, you are right, same 9
3 Agreement 3 Refer-to-a-name for agreement 7 17
It is very interesting 1
But, however, on the contrary 7
3a Conflict 2 7
Disagreement, different
If, might, could, would, should, think 18
4 Exploration 4 Suggestion 4 27
I have an idea, something else, what do you mean 3
alternative 2
Best, it is important 23
Comparison 15
5 Evaluation 5 Worst, unfortunately, no meaning 7 51
I prefer 3
Easiest 3
Mutual solution 4
6 Ideas, Co-construction 2 7
Overall, we agreed, finally 3
7 Other 1 Emphasis (colour, bold etc) 25 25
TOTAL 30 376

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 257
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

A_VII_13. CeLE 1 (AIa-1:stanzas1-25)

Introduction

A Ask the expert! Sophi


I Danis answers your questions a How we can create a website
in two or more languages? 1. Spyros Papadakis 5 Tue, 11 Apr 2006, 11:02 PM

Spyros Papadakis 5 Tue, 11 Apr 2006, 11:02 PM 30


Re: How we can create a website in two or more languages? by Antoniou Konstantinos - Thursday, 30 March 2006, 80
09:10 PM
> Re: How we can create a website in two or more languages? by Sophi Danis – Friday, 31 March 2006, 07:26 122
6 PM
A I a Re: How we can create a website in two or more languages? by Niki Lambropoulos - Friday, 31 March 2006, 09:37 22 544
(1+5)
AM
Re: How we can create a website in two or more languages? by Damian Damianopoulos - Sunday, 9 April 2006, 194
09:51 PM
> Re: How we can create a website in two or more languages? by Mary Frentzou - Tuesday, 11 April 2006, 96
11:02 PM

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 258
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Participant SP1 started with a question as the title of the discussion (AIa-1:stanza3).
Discussion Title: How we can create a website in two or more languages?

Participant SP1 elaborated the subject and indicated the aim of the discussion (AIa-
1:stanza4).
I want to make a website in Greek and English (or and more) Languages.

It was followed by 2 questions (AIa-1:stanzas5-6).


- Which techiques are the best in this case.
- Are there any templates available?
Then the first message ended with participant’s name (social cue) (AIa-1:stanza8).
Spyros
There were five responses from AK1, SD1, NL1, DD1 & MF1.

Response 1 (AK1).
The first was an explanation with an example (AIa-1:stanza14).
The only way I know is having a template and then editing-translating the content and the
buttons. This site (website) has great templates, but you ought to have some experience with
Photoshop to edit them.

Then there was a reference to AK1’s experience and evaluation took place by
referring to best practice and justification of the evaluation (AIa-1:stanza14).
I prefer this way, because I don't have to code or design my webpages from scratch.

A second initiation took place providing further information (AIa-1:stanza15).


Other web sites offer templates in html format,

A conflict stressed a comparison and initiated justification for best practice (AIa-
1:stanza15).
but I'm a little bit lazy and want to get the best results with little effort, so I prefer Photoshop
file format

Participant AK1 closes his message with an emoticon declaring satisfaction (social
cue) (AIa-1:stanza15).

Response 2 (SD1).

The answer from Participant AK1 was agreement based on the same interest and
provision of more information as the initiation of his message (AIa-2:stanza18).
This is a very interesting question, now we cooperate across countires, especially working on
EU educational projects. Sometimes international websites contain a cookie that will search
and detect the regional settings your computer is using and they will present the version that
is compatible with those settings, including the country (regional settings)….

An alternative solution initiated the aim of the next verse by replying explicitly to SP1
(you) (AIa-2:stanza20)
If, however, you want to design a webisite can be both Greek and English…

Based on the previous Participant’s SP1 message explores the issue further (AIa-
2:stanza20).
tehn you will ahve to resort the translation of teh website and use,…

Participant SD1 agrees with a previous message and refers to the name of the

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 259
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

person (social cue) (AIa-2:stanza20).


as mentioned by K, templates;…

Additionally, SD1 provides a different solution (AIa-2:stanza20).


or alternatively, you can build your website and translae it, and use one css template for both.

SD1 justifies and evaluates the proposition with a comparison referring to practical
implementation (AIa-2:stanza20).
This will make your job easire.

SD1 closes the message by providing more information as well as evaluating the
given information (AIa-2:stanzas22-24)
More on CSS can be found here>
http://www.w3.org/Style/CSS/ this is basic
and some more http://www.developertutorials.com/css-2/cover.html

Response 3 (NL1)
The Researcher replied by exploring the issue further, explaining her solution on a
project (AIa-3:stanzas27-28). The researcher tried to get involve in the discussions
as minimum as possible.
A different approach is the bilingual one.
I built the Greek School of London website here <http://dim-lon.europe.sch.gr/> If does not
work > (pic available)

Response 4 (DD1)
DD1 started his message by greeting the person who sent the initial question (social
cue) (AIa-3:stanza30).
Hi, S.

He agreed with a previous message and provided his own individual solution (AIa-
3:stanza31).
Yes you can make a website in two or more languages. A good example is my personal home
page (<http://users.ker.sch.gr/geoker>)

Then he went into an exploration of SD1’s suggestions (AIa-3:stanza32).


In order to produce an identical site in two languages

DD1 compared and evaluated the suggestions (AIa-3:stanza32).


it is best to produce the two templates from a single one.

He explored and explained the suggestions further (AIa-3:stanza32).


You must also use identical graphics, but in the different languages if the graphics involve
speech (cf. site above). If you use FrontPage you can develop the corresponding pages in
parallel. You can even perform exactly the same steps in the construction of the pages by
toggling from one page to the other clicking on the relevant tab and perform the identical
jobs.

After exploring and explaining his suggestions, DD1 made an evaluation on a topic
as regards the design (AIa-3:stanza33).
It is best to give names to your files which facilitate grouping

Then he provided a justification of this evaluation (AIa-3:stanza33).


that is to group the materials which are needed for a page can have a common characteristic
at the beginning.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 260
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

He gave an example of best practice for further exploration (AIa-3:stanza33).


For instance if I want to produce a page connected with Corfu I would give names like
<corfu.htm> for the HTML page. The various graphics would have names like
<corfu.banner.jpg>, <corfu.bottombanner.jpg>, <corfu.sidebarmenu.jpg>,
<corfu.mainframe.htm>, if you see what I mean.

Lastly, he offered more help if needed (AIa-3:stanzas34).


I should be glad to help more if you need any more specific details.

The he finished his message with a greeting and providing his name (social cues)
(AIa-3:stanzas35-36).
Cheers. D.
Response 5 (MF1)
MF1 adopted DD1’s style of structuring the message. The message has a greeting
as an initiation; MF1 refers to a person from a previous message and ends the
message in the same way, indicating empathic feelings reflected in her writing.
MF1 started her message by greeting everyone (social cue) (AIa-3:stanza38).
Hi everybody

Then she agreed with DD1 referring to his name (social cue), the participant who
sent the previous message (AIa-3:stanza39).
I agree with D.

MF1 evaluated his suggestions (AIa-3:stanza34).


The way he describes is the easiest.

Then she explored DD1’s solution with an example (AIa-3:stanza41).


If you want to have both languages on the same page if you use frontpage (at least the older
versions) you copy and paste <meta http-equiv="Content-Type" content="text.html;
charset=iso-8859-7"> you insert it after the <head>.
She tried to provide additional material but her attempt failed due to technical
problems (AIa-3:stanza41).
I tried to upload it as an attachment but it was impossible.

Referring to previous participants’ solutions MF1 explored them by adding her


suggestion (AIa-3:stanza41).
Frontpage 2003 seems to recognize the languages automatically although I am not pretty
sure beccause I haven't worked with it.

Finally, she explored an alternative option to everything said up to now (AIa-


3:stanza43).
If you use dreamweaver you change the page properties to Greek (English is recognized
automatically)

Imitating DD1, MF1 she finished her message with a greeting and providing her
name (social cues) (AIa-3:stanzas45-46).
cheers
M

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 261
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

A_VII_14. Participant SP1: Thought Processes

AIM – Q1
Q2
(AK1 ) explanation
Example
Evaluation (AK1 experience)
Justification (reasons)
Further Information
Conflict to justify
Aim
Evaluation

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 262
Tools & Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

APPENDIX VIII: CONDITIONS OF WORKING AND LEARNING ONLINE


TIME IN EDUCATION AND THE USE OF ICT IN ONLINE EDUCATION
0 years Months 1-5 years 6-10 years 11-20 years 20+ years N/A
Time
Freq. % Freq. % Freq. % Freq. % Freq. % Freq. % Freq. %
1 Employed 2 5.0 10 25.0 18 45.0 9 22.5 1 2.5
2 Use Computers 3 7.5 14 35.0 15 37.5 7 17.5 1 2.5
1-3 2 5.0 3 7.5
3 Use LMS 22 55.0 13 32.5 1 2.5
4-6 1 2.5 37 92.5
Train in ICT
No University Postgraduate ICT course Other Freq. %
1
3 7.5 2 5.0 4 10.0 29 72.5 1 2.5 1 2.5
Train in Moodle
1 26 65.0 2 5.0 1 2.5 8 20.0 1 2.5 2 5.0
Time on the Internet
All day Once–twice a day Every 3 days Freq. %
1
13 32.5 23 57.5 2 5.0 2 5.0
USE & IMPORTANCE OF PROFILES AND FORUMS (LIKERT SCALE)
Very Low Low Neither High Very High Freq. %
1 Use Profiles 3 7.5 1 2.5 9 22.5 6 15.0 8 20.0 13 32.5
Importance 2 5.0 1 2.5 9 22.5 6 15.0 12 30.0 10 25.0
2 Use Forums 0 0.0 6 15.0 4 10.0 8 20.0 13 32.5 9 22.5
Importance 0 0.0 1 2.5 5 12.5 9 22.5 17 42.5 8 20.0

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 263
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

APPENDIX IX: THEMATIC ANALYSIS IN THE MAIN STUDY


A_IX_1. Post-retreat opinions on participation

POST-RETREAT OPINIONS ON PARTICIPATION IN COMMUNITIES (N=34)

LEVELS Sub-levels Responses #


Willing to participate 5
Collaboration / interactions 4
Sense of belonging / bonding 4
Situated problem solving 3
Evolution 3
Implementation of theory & methodology 2
Management Common goals / problems 2
Dynamic grouping 1
Visible in public 1
Provision of help 1
Coordination 1
Social & Learning Viability 1
Activities Encouragement 1
Information exchange 3
Quality 3
More interesting learning 3
Collaborative learning 2
Shared work experience 2
Knowledge New knowledge building 2
Vicarious learning 1
Discovery 1
Common questions 1
Skills acquisition 1
New pedagogical approaches 1
New technologies 1
Tools Use
Functionality 1

A_IX_2. Post-retreat opinions on e-learners’ participation in the project

POST-RETREAT OPINIONS ON E-LEARNERS’ PARTICIPATION (N=31)


On this community
On any community
LEVELS Sub-levels (personal involvement)
Responses # Responses #
Collaborative activities 4 Positive thinking 1
Active participation 1 Active participation 1
Dialogue 1 Transformation 1
Management
Communication 1 Continuity 1
Social
activities Contribution to the project 1
Motivation 1
Provision of help 4 Everyone helps 2
Contribution Small stone to
Own participation 2 1
community building
Learning Management Target to learn 1
activities Following instructions 1
New challenges 1
Sharing abilities 1
Participation in activities 1

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 264
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Problem solving 1
Learn something interesting 1
Learn something useful on
1
a personal level
Learn something useful on
1
a community level
Implementation of new
1
experience
Implementation of new
1
knowledge
Participation in planning 1
Presentation of own
3 Discussions 1
experience/work
Opinions 3 Ideas 1
Questions 2 Opinions 1
Information 2
Propositions 2
Knowledge Discussions 2
Exploration 1
Comments 1
Advice 1
Suggestions 1
Answers 1
Use new tools 1

A_IX_3. Post-retreat opinions on new members’ contribution

Post-Retreat Opinions on New Members’ Contribution

LEVELS Sub-levels Responses #


Active participation 4
Different problems / heterogeneous group 4
Interaction 2
Enthusiasm 5
Exploration 1
Viability / regenerates the community 3
Management New perspectives / abilities 2
Willingness to cooperate 1
Willingness to learn 1
Collaborative Shared interest 1
e- Learning Training 3
Communities Frees the community / expression 2
Feedback to older members 2
Shared ideas / knowledge 5
Fresh knowledge / ideas / experience 16
Knowledge transfer 3
Quality 1
Knowledge
Questions 4
building
Propositions 1
Criticism 1
Life-long learning 1
Knowledge building 1
Tools Use Training in new technologies 2

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 265
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

A_IX_4. Post-retreat opinions on communities

POST-RETREAT OPINIONS ON COMMUNITY EVOLUTION


What hold the community
LEVELS Evolution elements (N=38)
together? (N=40)
Responses # Responses #
Increasing interactivity / help 12 Common interests / goals 22
Communication outside and after
8 E-tutors 5
the course
Common interests / goals 7 Desire for success 4
Online communication 6 Communication 4
Affective elements 6 Enthusiasm / being positive 3
Quick familiarisation 4 Willingness to collaborate 3
Active participation /
Increasing number of participants 4 3
atmosphere
Social Communication – general 3 Subject 2
Level Collaborative atmosphere 2 Mutual help / trust 2
E-learners’ participation in planning 2 Reliability 1
Sense of belonging 2 Immediate success 1
Personal messages / experiences /
first name 2
New colleagues 1
Photos 1
Visiting others’ web pages 1
Knew nobody initially 1
Participation quality / quantity 1
Information & knowledge
11 New knowledge / curiosity 12
exchange
Collaborative activities 8 Effective learning 3
Dialogue development 5 Projects development 2
Learning
Mobility of ideas 2 Collaborative learning 2
Level
Learn to communicate 1 Willingness to learn 1
Vicarious learning 1 Immediate feedback 1
New skills acquisition 1 Problem solving 1
Number of created projects 1
Collaborative tools 4 New technologies 3
Tools
Profile 1 Collaborative tools 2

A_IX_5. Post-retreat opinions on other communities

POST-RETREAT OPINIONS ON OTHER COMMUNITIES (N=34)


LEVELS Responses #
Knowledge contribution / specialisation - reliability 12
Provision of directions 1
Multiple perspectives 9
Everybody wins 1
Effective problem solving 2
Users / teachers’ engagement - pedagogical issues 4
Positive
Individual development 8
Collaboration 1
Further development 3
Quality 1
Knowledge dissemination 2
Time saving 1

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 266
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Simple interfaces are usable from all users so


3
Negative programmers are not needed
Not following pedagogical approaches
Tools Need for technical infrastructure 2

A_IX_6. Post-retreat opinions on new members’ contribution

Post-Retreat Opinions on New Members’ Contribution (N=32)


LEVELS Sub-levels Responses #
Active participation / interaction 6
Enthusiasm 5
Heterogeneous group 4
Training 3
Viability / regenerates the community 3
Management Frees the community / expression 2
New perspectives / abilities 2
Feedback to older members 2
Collaborative Willingness to learn & cooperate 2
e- Learning
Shared interest 1
Communities
Exploration 1
Fresh knowledge / ideas / experience 16
Shared ideas for knowledge building 9
Questions 4
Knowledge
Propositions 1
building
Criticism 1
Quality 1
Life-long learning 1
Tools Use Training in new technologies 2

A_IX_7. Post-retreat opinions on learning

POST-RETREAT OPINIONS ON LEARNING


LEVELS Responses #
E-tutors 30
Other learners 17
Whom (N=36)
Own work 3
More experienced members 3
Forums / chats 17
Emails / newsletters 16
Educational material 10
How (N=37) E-learning platform 6
Internet 4
Tools 4
Search engines 3
Moderators 7
E-learners 6
E-tutors 5
Motivators 5
Roles (N=22) Leaders 4
Supporters 4
Technical support 2
Active participants 2
Observers 1

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 267
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

APPENDIX X: COLLABORATIVE E-LEARNING EPISODES


(EXAMPLES FROM THE MAIN STUDY)

A_X_1. Collaborative e-Learning Episode III

CeLE-III was completed in one day (05/03/2007) and described the stages with
which P18 found the solution to a registration, problem based on her colleagues’
suggestions (CeLE-CIII: Stanzas 4-23). (Numbers on stanzas occurred automatically
in Atlas-ti™.)

CeLE-III Discussion Title: A different kind of question.

> P18 initiated the discussion with a statement about the exercise they had, that was
creating a blog (stanza 5): I have created a blog in Pathfinder

Then P18 describes her aim (stanza 5): but I would like to comment on Blogger.

The steps P18 took (stanza 5): I created an account in Google

P18 refers to the problem (stanza 5): but every time I log out [the system] it does not
recognise the login name and password.

Actions to solve the problem (stanza 5): I ended up having 3 accounts

Result from actions: and I still have the same problem

Question (stanza 5): I am doing something wrong, but what?


Asking for help: Please help!

> P24 asked P18 a question to clarify the problem (stanza 8): Do you add
yourname@gmail.com for username?

> E-tutor P32 explained what he had to do (stanza11): you simply write the user name
you have from gmail

> Then, R provided an example based on P24 and P32’s suggestion (stanza 14): e.g.
login name: nikilambropoulso@gmail.com password:
nbn3vb4325rb43wqbrliqwyfiuQGFLCBAf

> E-tutor P52 asked more clarifying questions (stanza 17-18): Which blog do you want
to use to comment? Are you allowed to post as administrator? Are the login name and
password incorrect?

> Then P52 asked for a following up on the process (stanza 18): Keep us updated on
the process…

> He finished his message with a social cue (stanza 18): pleaaaaaaaaase!

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 268
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

> P18 replied with a social cue (stanza 20): Thank you all for your help.

> He found what the problem was (stanza 20): I found what was wrong with it.

He explained what the problem was (stanza 20-22): When I first got the login name and
password I wrote the mail address, not the user name and this resulted the system to ask for
this address as a user name.

The problem was solved (stanza 22): Now I can login.

> P52 provided feedback (stanza 23): Bravo mate P18!!

P52 sent greetings and thanked for the discussion (stanza 23): be well, thank you for
the message.

Collaborative e-Learning Episode III: Analysis

A_X_2. CeLE-III Locus (CeLE-CIII: Stanzas 4-23)


COLLABORATIVE E-LEARNING EPISODE III: LOCUS
#
Section Replies Total
Forum Discussion Thread Words
A different kind of 43
question - P18 -
Monday, 5 March
2007, 01:59 PM
Re: A different kind of 6
question - P24 -
Monday, 5 March 2007,
04:54 PM
> Re: A different kind of 10
question – P32 Monday,
5 March 2007, 06:25
PM
>> Re: A different kind 6
I: Blogs in of question – R Monday,
C #29 7 (1+6) 133
Education 5 March 2007, 07:07
PM
>Re: A different kind of 21
question – P52 Monday,
5 March 2007, 10:57
PM
>> Re: A different kind 38
of question – P18
Monday, 5 March 2007,
11:12 PM
>>> Re: A different kind 9
of question – P52
Monday, 5 March 2007,
11:48 PM

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 269
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

A_X_3. CeLE-III Analysis

COLLABORATIVE E-LEARNING EPISODES


Levels of Abstraction
CeLE parameters #Codes Indicators #References
Info, statement, definition, aim 3
Social cues (nice behaviour, thanks,
4
Initiation: greetings)
1 Question - 6 Social cue - help 1 12
Information Question 1
Aim 1
Problem 2
2 Explanation 1 Example 1 1
Suggestion 2
4 Exploration 2 5
Question 3
5 Evaluation 1 Ask for result 1 1
Ideas, Co- Result 1
6 2 2
construction Bravo 1
TOTAL 12 21

A_X_4. CeLE-III Code Network: Initiation, Explanation, Exploration, Evaluation,


New ideas & Social cues

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 270
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

A_X_5. Internalisation and externalisation thought process

CeLE-III is relatively small (133 words) with more information on initiation and social
cues. The agreement and conflict stage was missed; three exploration levels were
built on participants’ questions. The following list represents the discussion in a linear
form as a collaborative problem solving process:

Initial post (P18): Statement – Aim – Problem – Question – Social cue (Help)

(P24 ) Exploration
(P32) Exploration: suggestion
(R) Explanation: example
(P52) Exploration – Initiation

(P18) Idea – Social cue


(P18) Evaluation: result
(P52) Co-construction - Social cue

This discussion is a Collaborative e-Learning Episode for a technical problem. The


discussion started with a message that initiated the aim and the problem to solve in
the discussion. A social sue attached an emotional level to the post (P18 seemed
desperate). Three interlocutors located the problem correctly using exploratory
questions as well as an example to help P18. It appears that P18 was based on
these suggestions, and managed to locate the problem and found the solution. This
was verified by P52’s social cues for feedback (bravo). Persistence for measuring
CeLE’s intensity seemed to be on a high level; the discussion was focused on the
particular problem.

A_X_6. Collaborative e-Learning Episode IX

CeLE-IX was completed in 3 days (29-31/03/2007) and referred to the use of specific
e-learning tools in the project creation process. The discussion was initiated outside
the online course between P37 and P22; then the participants shared their
proposition with the rest of the group (CeLE-GIX: Stanzas 2-84). (Numbers on
stanzas occurred automatically in Atlas-ti™.)

CeLE-IX Discussion Title: Implementing e-learning tools in Project Method.

> P37 initiated the discussion by presenting a proposition worked and agreed with
P22 (social cue) (stanzas 2-9): After a conversation with P22 about the way e-learning
tools can be implemented in the project method stages, we propose:

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 271
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

[The attached ppt image described an example using Web 2.0 and Web 1.0 tools:
Free subject selection and initial design was related to videoconferencing;
Research – information search and Elaboration was related to a blog;
Synthesis – production was related to a wiki;
Presentation was related to videoconferencing and a wiki; and
Evaluation was related to videoconferencing and a wiki.]

> O2 was an e-tutor. He replied the same day and tagged the message as INFORM
(29/03/2007, stanzas 10-16). His points were directly disagreeing with P37 (stanza
12): I disagree with ‘Research – information search and Elaboration’ and with
‘Presentation’ as regards the use of tools.

He provided two justifications (stanzas 13-16): In ‘Research – information search and


Elaboration’ it is not the blog because I don’t conduct search with a blog or information
elaboration but I record events, situations etc. In ‘Presentation’ it is not the wiki but the blog
which I present my complete project.

> P37 replied the next day (30/03/2007, stanzas 17-27).

She agreed (stanza 21): But this is exactly what is going to happen in a blog,

She explained the reasons for this decision with an example and stressed her
opinion by highlighting and bolding the main points of her argument (stanza 21):
recording and commenting of the data gathered and focused on events and situations for
their elaboration. Obviously, I don’t do the search via a blog.

Then she explored the topic further and provided alternative points using bullet points
(stanzas 22-27): The presentation in the World Wide Web using a wiki was [suggested] for
the following reasons:

The whole tam will create the context and its contents
The whole team will follow the stages before the final product
The whole team will evaluate [the project] and will be evaluated
And lastly it is recommended for navigation via the links, whereas the blog is somehow still –
unless the presentation is very short.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 272
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

> After 4 and half hours, she made a new proposition as continuing co-construction
based on previous arguments (30/03/2007, stanzas 28-37): [This is] The advanced
diagramme after the dialogue with O2:

The blog is used only for data recording and commenting as regards events and situations
gathered for further development. Search and data collection is conducted by the usual
methods: Web, literature review, educational visits, discussions with specialists etc.

In addition, the presentation can be done in a web-site, if we want to have more multi-media
elements.

[The attached ppt image described an example using Web 2.0 and Web 1.0 tools:
Free subject selection and initial design was related to videoconferencing;
Research – information search and Elaboration was related to WWW, literature
review and a blog;
Synthesis – production was related to a wiki;
Presentation was related to videoconferencing, a wiki and a website; and
Evaluation was related to videoconferencing and a blog.]

> P50 tagged his message as EXPLAIN (30/03/2007, stanzas 38-43).


He agreed with P58 referring to his name(social cue) (stanza 40): I also agree with
P58

He provided his own opinion as a co-constructive idea (stanza 41):


And I believe that chat can be added in the first stage which is the friendliest, the most known
in the community and its use could boost participants’ self esteem......

He signed off with his name (social cue) (stanza 43): P50

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 273
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

> (The following message was not on the logical order for the argument as reply to
messages can be interrupted in discussion forums.) P58 tagged his message as
QUESTION (30/03/2007, stanzas 48-50).

He asked a question for all to agree (stanza 48): Do you agree that it is difficult to have
videoconferencing on the first stage if we refer to a technologically illiterate audience?

He explained his point with an example (stanza 49): Even this online course had to have
videoconferencing in the third week…

He provided an evaluation by exploring its negative aspects (stanza 49):


It is not prohibiting but it can cause problems in the implementation…

Lastly, he signed off with his initials (social cue) (stanza 50): P58

> P13 tagged his message as INFORM (30/03/2007, stanzas 53-58).


He agreed with P58 referring to his name (social cue) (stanza 56): I also agree with
P58.

He explored P37’s argument referring to his name (social cue) (stanza 56): P37, it is
very difficult to start the first stage with videoconferencing.

He continued with an explanation and a co-constructive idea: I propose the division of


the first stage in two sub-stages: Expression of interest and then on a separate basis the
Design. Ideas to be project or having the potential to be projects can be announced via a
database. There, anybody or a class or a school can write about their interests and using a
search engine finding others with common interests. As for the medium of communication
there could be emails and chat.

This has been tested fro many years in Comenius and I think recently in etwinning with great
success. Have a look on http://partbase.eupro.se/frameuk.htm as well.

After this can be proceeded then videoconferencing can be part of the design if needed.

He agreed with the previous message (stanza 56): As for the rest of the stages I agree.

He evaluated P37’s work referring to his name (social cue) (stanza 56): Anyhow, this
was a great job, P37.

Lastly he finished the message with his first name (social cue) (stanza 58): P13.

> P37 tagged his message as EXPLORE and wrote its Greek translation in capital
letters (ΔΙΕΡΕΥΝΗΣΗ) (30/03/2007, stanzas 59-74).
He agreed with the previous messages (stanza 62): I agree with all previous speakers.

He explored the argument further and provided and example based on individual
experience (stanza 63): I just need to stress two parameters:
1) Our students are too familiarised with the new technologies. They wont face any problem
at all with videoconferencing; on the contrary, this environment is very attractive for them.

He was defensive on the initial ideal as the expression of an under construction


proposition (stanza 64):
2) This diagramme presents only a proposition for the implementation and use of tools,
especially web 2.0, we worked on and not all of existing tools.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 274
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

> P14 kept the EXPLORE tag (30/03/2007, stanzas 66-74).

He evaluated and agreed with P37’s work (stanza 68): Very good job P37. As you wrote
in your magic phrase ‘it is just a proposition’. It is accepted and implemented if the
situations, the teachers and the technology allow it.

He summarised (stanza 69): We learned what Blogs and wikis are and how to use them.

He further explored the subject by questioning and evaluating previous statements


(stanza 69): Fine, the important thing is the ways we are going to implement and taking
advantage of them.

Then he co-constructed an idea based on previous statements (stanza 70): In other


words, would it be ‘very erratic’ if we thought for example that as regards an issue in a Blog
or wiki our students knowing and implementing brainstorming presented their ideas?For me,
the issue is to start playing in the ‘field’ they like that is internet and new technologies.

He thanked (social cue) P37 in advance (stanza 72): Since you created this wonderful
presentation, could you send it to me via email?? Thank you.

He finished the message with his full name (social cue) (stanza 74): P14

> P22 replied the next day and tagged his message as EXPLAIN. (It appeared that
he was defensive in his message.) (31/03/2007, stanzas 75-84).
He quoted the first part of P37’s message and stressed the fact that it was an initial
idea by changing the size of the font as well as making it bold (stanza 79): After a
conversation with P22 about the way e-learning tools can be implemented in the project
method stages, we propose:
He agreed with P37 (social cue) (stanza 80):

- At is appears in P37’s first post, this is just a proposition based on the tools we used in the
online course, including skype and msn, tools that most of our students can easily use, and in
many occasions, better than many of our colleagues.

He explored the argument further with bullet points (stanzas 81-84):

- On the question which tools somebody will use to design, develop and complete his work,
this depends on the situation (students’ age, school, infrastructure, level, students’ mood,
etc)

- The order we propose with the tools and its implementation it is not, and it can’t be in any
way, neither strict nor unique. Each one said [that] it depends on what, how, etc.

- The positive thing is that the previous slide created an argument that lead to a creative
dialogue, and this is the essence of the online course, it is not that insignificant that a month
ago most of us did not know or heard about these tools, we hardly knew them and for sure we
had never used them.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 275
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

A_X_7. Internalisation and externalisation thought process

The following thought processes represent the discussion in a linear form as a


collaborative problem solving process:

Initial post (P37): Statement – Aim – Proposition (text and image)


(O2 ) Disagreement
(P37) Agreement – Explanation – Exploration
(P37) New proposition / idea
(P50) Explanation - New proposition / idea

(P58) Question
(P13) Agreement – Initiation – Explanation - New proposition / idea –
Evaluation – Social cue
(P37) Explore
(P14) Evaluation – Agreement – Summarise –
Exploration - New proposition / idea – Social
cue
(P22) Agreement – Exploration

Collaborative E-Learning Episode IX: Analysis

A_X_8. CeLE-IX Locus (CeLE-GIX: Stanzas 2-84)

COLLABORATIVE E-LEARNING EPISODE III: LOCUS


#
Section Replies Total
Forum Discussion Thread Words
Implementing e-learning
II: VC in tools in Project Method. –
G E- #4 7 P37 – Thursday 29 March 67 921
Learning
2007, 08:53 AM
Re [INFORM]:
Implementing e-learning
tools in Project Method – 60
O2 Thursday, 29 March
2007, 10:51 PM
> Re: Implementing e-
learning tools in Project
Method – P37 Friday, 30 96

March 2007, 09:31 AM


>> Implementing e-learning
tools in Project Method –
P37 Friday, 30 March 2007, 67
01:48 PM
>>> Re [EXPLAIN]:
Implementing e-learning 41

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 276
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

tools in Project Method –


P50 Friday, 30 March 2007,
04:33 PM
Re: [QUESTION]
Implementing e-learning
tools in Project Method – 40
P58 Friday, 30 March 2007,
02:47 PM
> Re [INFORM]:
Implementing e-learning
tools in Project Method – 117
P13 Friday, 30 March 2007,
08:48 PM
>> Re [EXPLORE]:
Implementing e-learning
tools in Project Method – 63
P37 Friday, 30 March 2007,
09:40 PM
>>> Re [EXPLORE]:
Implementing e-learning
tools in Project Method – 128
P14 Friday, 30 March 2007,
10:34 PM
>>> Re [EXPLAIN]:
Implementing e-learning
tools in Project Method – 233
P22 Saturday, 31 March
2007, 01:38 PM

A_X_9. CeLE-IX Analysis

COLLABORATIVE E-LEARNING EPISODE IX


Levels of Abstraction
CeLE parameters #Codes Indicators #References
MESSAGETAGs 3
Propositions 2
Social cues (nice behaviour, thanks,
8(+4)∗
Initiation: greetings, names)
1 Question - 7 Question 2 21/3
Information Links 1
Aim 1
Quoting previous message 1
Images 2
MESSAGETAGs 2
2 Explanation 1 3/2
Example 3
Agree 3
3 Agreement Agree to all previous interlocutors 1
Agreement referring to a name 4 12
Disagreement Disagree 1
3a
(Conflict) Disagree with justification 3
MESSAGETAGs 2
Question 2
4 Exploration Suggestion, alternative solution 4 8/2
Individual experience 1
Bullet points 1
5 Evaluation Evaluation 5 10

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 277
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

Negative evaluation 1
Best practice & justification 4
Result 1
Ideas, Co-
6 New idea 4 7
construction
Summary 2
7 Other 1 Emphasis (colour, bold etc) 15 15
Tagged
TOTAL 76/7
9.2%
∗The number in parenthesis refer to a reply to a specific person indicated by her name

A_X_10. CeLE-IX Code Network: Initiation, Explanation, Exploration, Other,


Evaluation, New ideas, and Agreement/Disagreement

CeLE-IX appears to be rich in arguments (921 words), especially on initial


information, exploration as well as agreements and disagreements. New ideas and
knowledge construction seemed to be related with all CeLE stages as explanation
and evaluation were relatively rich. Even though the discussion started as a
disagreement on a previous agreed proposition between two participations, it
cultivated two series of arguments that were mostly based on explanations and
explorations in order to evaluate the previous comments and reach new knowledge
constructions. Both CeLEs diagrammes depict the process inside and outside the
individual as a personal monologue and simultaneously a dialogue with the other co-
learners.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 278
Tools and Evaluation Techniques for Collaborative E-Learning Communities Chapter 6: Main Study

It is interesting to note that the interlocutors were different individuals except P37
who appears to be on the medium activity level. The participation in the discussion
appeared as follows:

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 279
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

Appendix XI: Messages Quantitative Analysis


A_XI_1. Messages Quantitative Analysis in Moodle@GSN
Post Message Analysis in Moodle@GSN (01/03 – 31/03/2007 – 31 days)
Richness
# Initiations
Replies (add post) Discussion of text (#
Section Forums (add forum topic)
Depth words)
Total Analysis Participants Total Participants Total
Introduction (01- 21/03/2007) 59 49 49 182 142 (40) 231 9,075
Timetable (01 – 28/03/2007) 45 15 7 112 80 (32) 127 1,654
A Introduction Social networking (01-24/03/2007) 6 5 1 25 16(9) 30 354
Moodle (01-08/03/2007) 3 2 2 5 3(2) 7 260
Problems (01-08/05/2003) 47 26 22 101 70(31) 127 4,113
Total 5 160 97 81 425 311(114) 522 15,456
B Project Method Projects Archive (01-07/03/2007) 6 1 1 5 4(1) 6 142
Total 1 6 1 1 5 4(1) 6 142
Blogs (01-31/03/2007) 61 42 43 156 127(29) 198 7,110
C Blogs Tools (03-12/03/2007) 8 3 3 7 7(0) 10 498
Blog & HTML (05-12/03/2007) 3 1 1 2 2(0) 3 61
Total 3 72 46 47 165 136(29) 211 7,669
Problems (12-23/03/2007) 15 10 9 27 24(3) 37 1,055
Design (17-18/03/2007) 2 2 2 4 4(0) 6 116
D Wikis
Practicality (13-18/03/2007) 4 2 2 16 13(3) 18 856
Groups (18-24/03/2007) 9 7 7 25 22(3) 31 748
Total 4 30 21 20 72 63(9) 92 2,775
Technical Problems (16-22/03/07) 8 2 0 15 7(8) 12 303
E Videoconferencing
Groups (18-23/03/07) 1 1 0 10 7(3) 16 94
Total 2 9 3 0 25 14(11) 28 397
Project ideas (01-06/03/2007) 4 1 1 2 3(0) 3 236
F Internet Cafe
Other (01-18/03/2007) 4 2 2 4 2(2) 6 232
Total 2 8 3 3 6 5(2) 9 468
Overall 16 285 171 152 698 533(166) 868 26,907
Percentage per day 0.97 01/03 – 31/03/2007 – 31 days 9.1 5.5 4.9 22.5 17.1(5.3) 28 867.9
Percentage 95 participants 3 1.8 1.6 7.3 5.6 9.1 283.2

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 280
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

per participant
Messages Density 2a/N(N-1) 2X868/95(95-1)=0.19

A_XI_2. Messages Quantitative Analysis in the Research Pool


Research Pool (25 – 31/03/2007 – 6 days)
Richness
Overall
# Initiations (add forum) # Replies (add post) of text
Section Forums (Analysis+
(# words)
Participants)
Total Analysis Participants Total Participants Total
News (28-30/03/07) 7 2 5 6 5(1) 7 999
G VC in E-Learning
VC in E-learning (28/03-01/04/07) 9 8 4 70 70(0) 78 8,942
Overall 2 16 10 9 76 75(1) 85 9,941
Percentage per day 0.4 25 – 31/03/2007 – 5 days 3.2 2 1.8 15.2 15(0.2) 17 1,988.2
Percentage
42 participants 0.4 0.2 0.2 1.8 1.79 2 236.6
per participant
Messages Density 2a/N(N-1) 2X85/42(42-1)=0.1

* My initiations were included as some participants’ messages were produced from these initiations.

The tables present the first three quantitative variables, richness of text, depth of discussions, and messages density. (My posts are in
parentheses.) These tables describe the results in Moodle@GSN and the research pool. The first column describes the online course section, the
second column the forums and the dates these forums were introduced and completed. Then the replies are divided into initiations found in logs
as “add forum”, and the actual replies, found in logs as “add posts”. From these messages, the sub-columns present the total number of
messages, the messages for analysis as well as participants’ only messages. The last two columns show the overall number of messages
provided the initiations for analysis including my messages, and the participants’ messages, and the richness of text.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 281
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

APPENDIX XII: MAIN STUDY DATA & REPORTS

A_XII_1. PEDAGOGICAL USABILITY – UTILITY RESULTS

Frequency % Frequency % Frequency % Frequency % Frequency % Mean St.D. N/A


Graphs &
Avatars 2 5 3 7.5 8 20 17 42.5 10 25 3.8 1 0 3.625
MessageTag 1 2.5 3 7.5 8 20 17 42.5 11 27.5 3.9 1 0
VIT Nodes 4 10 6 15 8 20 14 35 8 20 3.4 1.2 0
VIT
1 Instructions Centrality 4 10 7 17.5 7 17.5 14 35 8 20 3.4 1.2 0
Graphs &
Avatars 7 17.5 14 35 7 17.5 7 17.5 5 12.5 2.7 1.3 0 2.4
MessageTag 8 20 14 35 6 15 4 10 2.8 1.2 0
VIT Nodes 14 35 11 27.5 13 32.5 1 2.5 1 2.5 2.1 1 0
Frequency of VIT
2 use Centrality 16 40 12 30 9 22.5 2 5 1 2.5 2 1.3 0
Alignment Graphs &
with Avatars 1 2.5 1 2.5 14 35 16 40 8 20 3.7 0.9 0 3.5
educational
goals MessageTag 1 2.5 1 2.5 16 40 13 32.5 9 22.5 3.7 0.9 0
VIT Nodes 4 10 3 7.5 16 40 10 25 7 17.5 3.3 1.2 0
VIT
3 Centrality 4 10 3 7.5 16 40 10 25 7 17.5 3.3 1.1 0
Graphs &
Avatars 1 2.5 1 2.5 13 32.5 17 42.5 8 20 3.7 0.8 0 3.475
MessageTag 1 2.5 1 2.5 13 32.5 17 42.5 8 20 3.8 0.9 0
Support VIT Nodes 7 17.5 3 7.5 11 27.5 14 35 5 12.5 3.2 1.3 0
collaborative VIT
4 e-learning Centrality 7 17.5 3 7.5 11 27.5 14 35 5 12.5 3.2 1.3 0
Graphs &
5 Learnability Avatars 6 15 10 25 9 22.5 12 30 3 7.5 2.5 1.1 0 2.725

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 282
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

MessageTag 11 27.5 11 27.5 10 25 6 15 1 2.5 2.4 1.1 1


VIT Nodes 6 15 10 25 9 22.5 12 30 3 7.5 3 1.2 0
VIT
Centrality 6 15 11 27.5 8 20 12 30 3 7.5 3 1.2 0
Graphs & 1 2 10 15 12
Avatars 2.5 5 25 37.5 30 3.9 1 0 3.675
MessageTag 1 2.5 2 5 8 20 16 40 13 32.5 4 1 0
VIT Nodes 6 15 4 10 9 22.5 11 27.5 10 25 3.4 1.3 0
VIT 6 3 9 12 10
6 Accessibility Centrality 15 7.5 22.5 30 25 3.4 1.3 0
Graphs & 1 3 20 10 6
Avatars 2.5 7.5 50 25 15 3.4 0.9 0 3.4
MessageTag 2 5 4 10 13 32.5 16 40 4 10 3.47 1 1
VIT Nodes 4 10 5 12.5 11 27.5 10 25 9 22.5 3.38 1.2 1
VIT 4 5 11 11 8
7 Originality Centrality 10 12.5 27.5 27.5 20 3.35 1.2 1
Graphs & 2 3 10 15 10
Avatars 5 7.5 25 37.5 25 3.7 1 0 3.5
MessageTag 1 2.5 3 7.5 9 22.5 16 40 10 25 3.9 1 1
VIT Nodes 6 15 5 12.5 12 30 11 27.5 6 15 3.2 1.2 0
Motivation to VIT 6 4 12 11 7
8 participate Centrality 15 10 30 27.5 17.5 3.2 1.2 0
Graphs & 15 11 7 4 3
Avatars 37.5 27.5 17.5 10 7.5 2.2 1.2 0 2.175
MessageTag 16 40 10 25 8 20 4 10 2 5 2.1 1.2 0
VIT Nodes 14 35 10 25 10 25 5 12.5 1 2.5 2.2 1.1 0
Information VIT 14 10 10 5 1
9 overload Centrality 35 25 25 12.5 2.5 2.2 1.1 0
Graphs & 1 1 14 19 5
10 Functionality Avatars 2.5 2.5 35 47.5 12.5 3.65 0.8 0 3.375
MessageTag 1 2.5 14 35 18 45 7 17.5 3.75 0.8 0
VIT Nodes 5 12.5 5 12.5 15 37.5 10 25 5 12.5 3.1 1.1 0

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 283
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

VIT 5 6 16 8 5
Centrality 12.5 15 40 20 12.5 3 1.1 0
Graphs & 1 2 18 13 5
Avatars 2.5 5 45 32.5 12.5 3.5 0.8 1 3.3875
MessageTag 1 2.5 1 2.5 15 37.5 17 42.5 6 15 3.65 0.8 0
VIT Nodes 5 12.5 4 10 15 37.5 9 22.5 7 17.5 3.2 1.2 0
VIT 5 4 15 9 7
11 VD: Graphics Centrality 12.5 10 37.5 22.5 17.5 3.2 1.2 0
Graphs & 1 5 14 14 6 0.9
Avatars 2.5 12.5 35 35 15 3.4 3.0875
MessageTag 1 2.5 2 5 16 40 13 32.5 7 17.5 3.65 1 1
VIT Nodes 5 12.5 8 20 13 32.5 10 25 4 10 3 1.1 0
VD: VIT 6 7 13 10 4 1.2
12 Attractiveness Centrality 15 17.5 32.5 25 10 2.3 0
Graphs & 19 14 4 3 0.9
Avatars 47.5 35 10 7.5 1.8 0 1.85
MessageTag 22 55 12 30 3 7.5 3 7.5 1.7 0.9 0
VIT Nodes 18 45 13 32.5 4 10 3 7.5 2 5 2 1.1 0
VIT 19 12 4 3 2 1.1
13 Tool failure Centrality 47.5 30 10 7.5 5 1.9 0
Graphs & 2 12 16 10 0.9
Avatars 5 30 40 25 3.8 0 3.55
MessageTag 4 10 9 22.5 16 40 11 27.5 3.8 1.1 0
VIT Nodes 6 15 1 2.5 15 37.5 12 30 6 15 3.3 1.2 0
VIT 6 1 15 12 6 1.2
14 Fast response Centrality 15 2.5 37.5 30 15 3.3 0
Graphs & 1 2 13 16 8 0.9
Avatars 2.5 5 32.5 40 20 3.7 0 3.375
MessageTag 1 2.5 3 7.5 15 37.5 12 30 9 22.5 3.6 1 0
VIT Nodes 6 15 3 7.5 17 42.5 8 20 6 15 3.1 1.2 0
Overall VIT 6 4 15 9 6 1.2
15 satisfaction Centrality 15 10 37.5 22.5 15 3.1 0

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 284
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

A_XII_2. CORRELATIONS IN SPSS

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 285
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

A_XII_3. MOST IMPORTANT CORRELATIONS


IN THE HIERARCHICAL CLUSTERING EXPLORER

USABILITY – UTILITY CORRELATIONS

Rank X axis Y axis Pearson’s’ r


Positive Correlations
7 VD: Attractiveness VIT Centrality VD: Attractiveness VIT Nodes 0.981
8 Accessibility VIT Centrality Accessibility VIT Nodes 0.977
9 Tool failure VIT Centrality Tool failure VIT Nodes 0.977
10 Functionality VIT Centrality Functionality VIT Nodes 0.961
11 Instructions VIT Centrality Instructions VIT Nodes 0.961
12 Satisfaction VIT Centrality Satisfaction VIT Nodes 0.955
13 Info overload MessageTag Info overload Graphs/avatars 0.946
14 Failure MessageTag Failure Graphs/avatars 0.944
15 Motivation VIT Centrality Motivation VIT Nodes 0.930
16 Accessibility MessageTag Accessibility Graphs/avatars 0.918
17 Education goals MessageTag Education goals Graphs/avatars 0.883
18 Satisfaction MessageTag Functionality VIT Nodes 0.843
19 Education goals VIT Nodes Originality VIT Nodes 0.810
20 Education goals VIT Centrality Originality VIT Nodes 0.810
21 Satisfaction VIT Centrality Functionality VIT Centrality 0.801
22 Satisfaction VIT Centrality Originality VIT Nodes 0.801
23 Satisfaction VIT Nodes Functionality VIT Nodes 0.792
24 Support CeL VIT Nodes Motivation VIT Nodes 0.788
25 Support CeL VIT Centrality Motivation VIT Nodes 0.788
26 Satisfaction VIT Centrality Support CeL VIT Nodes 0.784
27 Satisfaction VIT Centrality Support CeL VIT Centrality 0.784
28 Education goals Graphs/avatars Functionality Graphs/avatars 0.782
29 Satisfaction VIT Centrality VD: Graphics VIT Nodes 0.777
30 Satisfaction VIT Centrality VD: Graphics VIT Centrality 0.777
31 Originality VIT Nodes Functionality VIT Nodes 0.771
32 Accessibility VIT Centrality Instructions VIT Centrality 0.763
33 Originality VIT Nodes VD: Graphics VIT Nodes 0.752
34 Originality VIT Nodes VD: Graphics VIT Centrality 0.752
35 Satisfaction VIT Nodes Functionality VIT Centrality 0.751
36 Fast response MessageTag Fast response Graphs/avatars 0.751
37 Satisfaction VIT Nodes Support CeL VIT Nodes 0.748
38 Satisfaction VIT Nodes Support CeL VIT Centrality 0.748
39 Originality VIT Nodes Functionality VIT Centrality 0.740
40 Learnability VIT Nodes Learnability Graphs/avatars 0.737
41 Learnability VIT Centrality Learnability Graphs/avatars 0.737
42 Accessibility VIT Nodes Instructions VIT Centrality 0.733
43 Functionality VIT Centrality VD: Graphics VIT Nodes 0.729
44 Functionality VIT Centrality VD: Graphics VIT Centrality 0.729
45 Accessibility VIT Centrality Instructions VIT Nodes 0.723
46 Functionality VIT Centrality VD: Attractiveness VIT Centrality 0.722
47 Satisfaction VIT Nodes Originality VIT Nodes 0.721
48 VD: Graphics VIT Nodes VD: Attractiveness VIT Centrality 0.718
49 VD: Graphics VIT Centrality VD: Attractiveness VIT Centrality 0.718
50 Satisfaction Graphs/avatars Education goals Graphs/avatars 0.712
51 Support CeL VIT Nodes Originality VIT Nodes 0.712
52 Support CeL Centrality Originality VIT Nodes 0.712
53 Satisfaction VIT Nodes VD: Graphics VIT Nodes 0.710
54 Satisfaction VIT Nodes VD: Graphics VIT Centrality 0.710
55 Motivation MessageTag Motivation Graphs/avatars

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 286
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

Negative Correlations
1714 Satisfaction VIT Nodes Failure VIT Centrality -0.600
1715 Learnability MessageTag VD: Graphics VIT Nodes -0.603
1716 Accessibility VIT Nodes VD: Graphics MessageTag -0.603
1717 Support CeL Graphs/avatars Functionality VIT Nodes -0.605
1718 Satisfaction VIT Nodes Info overload MessageTag -0.605
1719 Satisfaction VIT Nodes Info overload Graphs/avatars -0.607
1720 Motivation VIT Centrality Info overload MessageTag -0.608
1721 Satisfaction VIT Nodes Failure VIT Nodes -0.609
1722 Motivation VIT Nodes VD: Graphics MessageTag -0.611
1723 Satisfaction VIT Centrality Accessibility Graphs/avatars -0.612
1724 Instructions VIT Centrality Frequency Graphs/avatars -0.619
1725 Originality VIT Nodes VD: Graphics MessageTag -0.620
1726 Motivation VIT Nodes Info overload MessageTag -0.620
1727 Satisfaction Graphs/avatars Functionality VIT Centrality -0.621
1728 Education goals MessageTag Accessibility VIT Nodes -0.621
1729 Education goals MessageTag Accessibility VIT Centrality -0.624
1730 Education goals Graphs/avatars Instructions VIT Nodes -0.624
1731 Satisfaction VIT Nodes Accessibility Graphs/avatars -0.628
1732 Motivation VIT Centrality Info overload VIT Nodes -0.628
1733 Motivation VIT Centrality Info overload VIT Centrality -0.628
1734 Satisfaction VIT Centrality Info overload VIT Nodes -0.629
1735 Satisfaction VIT Centrality Info overload VIT Centrality -0.629
1736 Education goals VIT Nodes Info overload MessageTag -0.630
1737 Education goals VIT Centrality Info overload MessageTag -0.630
1738 Support CeL Graphs/avatars Functionality VIT Centrality -0.631
1739 Motivation VIT Nodes Info overload VIT Nodes -0.632
1740 Motivation VIT Nodes Info overload VIT Centrality -0.632
1741 Education goals Graphs/avatars Instructions VIT Centrality -0.633
1742 Education goals Graphs/avatars Accessibility VIT Centrality -0.636
1743 Support CeL VIT Nodes VD: Graphics MessageTag -0.638
1744 Support CeL VIT Centrality VD: Graphics MessageTag -0.638
1745 Instructions VIT Centrality VD: Graphics MessageTag -0.641
1746 Education goals Graphs/avatars Accessibility VIT Nodes -0.642
1747 Learnability VIT Nodes VD: Attractiveness VIT Centrality -0.653
1748 Learnability VIT Centrality VD: Attractiveness VIT Centrality -0.653
1749 Originality VIT Nodes Info overload MessageTag -0.655
1750 Functionality MessageTag VD: Attractiveness VIT Nodes -0.657
1751 Support CeL VIT Nodes Fast response Graphs/avatars -0.657
1752 Support CeL VIT Centrality Fast response Graphs/avatars -0.657
1753 Satisfaction VIT Nodes Failure Graphs/avatars -0.665
1754 Instructions VIT Centrality VD: Graphics MessageTag -0.672
1755 Info overload VIT Nodes VD: Attractiveness VIT Nodes -0.676
1756 Info overload VIT Centrality VD: Attractiveness VIT Nodes -0.676
1757 Accessibility VIT Centrality Learnability VIT Nodes -0.687
1758 Accessibility VIT Centrality Learnability VIT Centrality -0.687
1759 Education goals VIT Nodes Info overload Graphs/avatars -0.688
1760 Education goals VIT Centrality Info overload Graphs/avatars -0.688
1761 Accessibility VIT Centrality VD: Graphics MessageTag -0.689
1762 Info overload MessageTag VD: Attractiveness VIT Nodes -0.690
1763 Originality VIT Nodes Info overload Graphs/avatars -0.698
1764 Learnability VIT Nodes VD: Attractiveness VIT Nodes -0.712
1765 Learnability VIT Centrality VD: Attractiveness VIT Nodes -0.712
1766 Info overload MessageTag VD: Attractiveness VIT Centrality -0.719
1767 Accessibility VIT Nodes Learnability VIT Nodes -0.729
1768 Accessibility VIT Nodes Learnability VIT Centrality -0.729
1769 Info overload VIT Nodes VD: Attractiveness VIT Centrality -0.747
1770 Info overload VIT Centrality VD: Attractiveness VIT Centrality -0.747

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 287
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

A_XII_4. THE BEST THING IN THE PROJECT

THE BEST THING IN THE PROJECT WAS… (N=39)


E-learners
LEVELS Sub-levels
Responses #
Common interests / goals 6
Communication / participation / work
Community Management 11
together
Sense of belonging to something greater 7
Exchange of opinions 1
Implementation of acquired knowledge 2
Presentation of own experience / work 1
Educational material 1
Management
Professional training 2
Learning
Feedback 1
New interests 1
New ideas / knowledge 1
Knowledge building
New skills 1
New technologies 9
Tools Use Working with new technologies 1
Real-time evaluation 2
Excitement 1
Study Hawthorn effect
Successful contribution 1

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 288
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

APPENDIX XIII: RECOMMENDATIONS

Recommendations for researchers


• Open communication channels
• Work on multidisciplinary frameworks to acquire coherent views of the
research context
• Use ethnotechnology to inform design
• Target human-human and human-computer interaction in e-research
• Use social network analysis to triangulate results from qualitative and
quantitative data analysis
• Use real-time evaluation for proactive, just-in-time, and reactive decision
making
• Use time-series design for formative and summative evaluation
• Ensure scalability of methods, tools, and techniques

Recommendations for e-learning engineers


• Open communication channels
• Get involved in multidisciplinary teams
• Acknowledge the post hoc nature of design in evolving situations
• Design for learners as users and users as learners
• Design for pedagogical usability and utility
• Design tools relevant to professional practice
• Offer pragmatic solutions
• Use learner-centred design sensitive to its context on a macro and micro level
• Evaluate design in real-time as circumstances evolve
• Involve all e-learning participants in design
• Acquire opinions from different levels of practitioners’ expertise
• Integrate design, use and evaluation in design
• Make visible and support social and temporal structures of social interaction
• Design for collaborative e-learning tools and activities to
o observe and analyse human-human and human-computer interactions
o support persistence and depth of discussions
o support increasing participation
o provide detailed and accurate reports for each e-learner
• Modify the tools in this study

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 289
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

o Calculation of participation levels based on the highest poster’s


responses
o Enable quoting
o Semantic and temporal arrangement of the “Reply” function in
discussion forums
• Use different methods for pedagogical usability testing specifically designed
for new tools

Recommendations for e-learning practitioners


• Plan for E-learning
o Acknowledge the strategic use of collaborative e-learning
o Facilitate and moderate progressive discourse
o Tackle performance problems associated with personality traits
o Obtain a balance on personalised and social learning
o Support self-organised e-learning
o Plan and routinise collaborative activities as a process of increasing
participation
o Use available technologies effectively
• Plan for E-Learning Communities
o Open communication channels
o Define stakeholders’ intentions and goals
o Create highly targeted and interactive courses to engage e-learners
o Use community assessment
o Enhance social networking to create learning opportunities
o Train e-learners how to work together
o Work towards the different levels of participation
o Allow time for getting-to-know activities
o Focus on development and maintenance of empathy and trust
o Keep the enthusiasm going

Recommendations for the Greek educational authorities


• Open communication channels
o Create an inclusive environment to support cooperation between
national institutions and teachers
o Be proactive, just-in-time, and reactive in decision making
o Shorten decision making periods using new technologies
o Enhance participatory decision-making

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 290
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

o Create groups for research and development beyond administration


• Create policies to support e-learning quality
o Create a central organisation to ensure quality in e-learning
o Train e-tutors and e-learners
o Provide quality short training cycles relevant to the Greek teachers’
profession on a life-long learning basis
o Provide opportunities for Ministry chairs to learn firsthand about the
benefits of e-learning and the changing nature of 21st century learning
o Incorporate e-learning in institutions to support employees’ life-long
learning
o Create opportunities for e-learning experts and e-learners

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 291
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

Glossary

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 292
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

Baseline: The observation of behaviour prior to any treatment designed to alter


behaviour.

Collaboration: A coordinated, synchronous activity as a result of a continued


attempt to construct and maintain a shared conception of a problem.

Collaborative e-Learning Communities (CeLC): Social aggregations that emerge


in online courses when enough people carry on progressive dialogues for the
purpose of learning.

Collaborative E-Learning Episode (CeLE): A communicative discussion episode


with a starting point, a transition and an end point that indicates a collaborative e-
learning cycle.

Collaborative Learning: The type of learning that takes place when learners work in
groups on the same task using progressive dialogue for co-creativity.

Community: A group of people who consciously share a sense of belonging


anchored in common interests and enhanced by social interactions. An online
community is a community where social interactions are facilitated by information
and communication technologies.

Communities of Practice (CoP): Groups of people who share a concern or a


passion for something they do and learn how to do it better as they interact regularly.

Decentralisation: The ability to take into account viewpoints of a given situation

Empathy: A complex psychological inference in which observation, memory,


knowledge and reasoning are combined to yield insights into the thoughts and
feelings of others.

E-Research: The research in online environments.

Ethnography: The branch of anthropology that provides scientific description of


individual human societies.

Ethnotechnology: An ethnographic field for studying design in real environments.

Grounding: Interactions intended to create mutual understanding, knowledge,


beliefs, assumptions or repairing misunderstandings in a group.

Heuristics: A form of usability inspection where usability specialists judge whether


each element of a user interface follows a list of established evaluation variables.

Human Computer Interaction (HCI): The study, planning and design of what
happens when humans interact with computers.

Informal learning: The unofficial and unstructured way of learning.

Instructional Design: The systematic process of activities to solve an instructional


problem with the aid of technologies.

Instructional Engineering: The systematic process of activities to solve an online


instructional problem anchored in Human-Computer Interaction.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 293
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

Learner-Centred Design (LCD): The design that considers the learner/user as the
center of instructional design.

Learning Management System (LMS): A software package designed to manage


learning interventions in technology enhanced learning.

Legitimate Peripheral Participation (LPP): The process of social learning that


occurs in Communities of Practice containing different levels en route for members’
engagement and practice.

Lurking (Passive Participation): The activity of one of the "silent majority" in an


electronic forum that involves posting occasionally or not at all but reading the
group's postings.

Mirror Neurons: The premotor neurons which fire both when an animal acts and
when the animal observes the same action performed by another (especially of the
same species) animal.

Online Communities: Online social aggregations that emerge when enough people
carry on those public discussions long enough to form relationships.

Online Learning: A planned teaching and learning experience that uses a wide
spectrum of technologies to reach learners at a distance.

Pedagogical Usability: A quality attribute that assesses how easy learner/user


interfaces are to use with a purpose of learning.

Sense of e-Learning Community Index (SeLCI): The index to measure the sense
of belonging in an e-learning community.

Social Computing: The incorporation of sociological understandings into interface


design aiming at building systems that fit more easily into the ways we communicate
and work.

Socio-cultural learning: A theoretical framework which emphasises the role of


social interaction in the development of cognition. It supports that cognitive
development is based on the negotiation of meaning that originates from individuals’
actual relationships.

Social Network Analysis (SNA): The mapping and measuring of relationships and
flows between people, groups, organizations, computers or other
information/knowledge processing entities. SNA glossary follows:

Betweennes: The measurement of the node’s prominence according to


its position in the network.

Centrality: Measures who is central (powerful) or isolated in networks.

Clique: A subgroup where actors are connected to each other as a maximal


complete subgraph of three or more nodes (members) adjacent to each other.

Closeness: The measurement of the distance between one node and other
node in a network as the number of other nodes divided by the sum of all
distances between the node and all others.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 294
Tools and Evaluation Techniques for Collaborative E-Learning Communities Appendices

Cohesion: The representation of interactions’ weight (density), participants’


preferences (reciprocity), subgroups (cliques), and similar behaviour (structural
equivalence).

Density: The number of actual ties in a network compared to the total amount
of ties that the network can theoretically support.

Degree Centrality: A directed network where the direction of the


communication is important.

Ego Network: Consists of a focal node and a set of alter nodes adjacent to or
from the focal node.

Equivalence: A description of the actors who have similar patterns of relations


to others in the network and exhibit similar communication behaviour.

Global centrality: The communication nodes between the members of a


network characterised by direction and strength.

In-degree centrality: The number of lines that are incident to a node.

Intensity: The participation levels and persistence in online learning.

Isolates: Nodes whose degree equals 0.

Nodes: The actors or subjects of study.

Out-degree Centrality: The number of lines that are incident from a node.

Reciprocity: The number of ties that are involved in reciprocal relations relative
to the total number of actual ties.

Structural Equivalence: The role-set structure of a network based on the


similarity of tie-profiles among its nodes and is computed by the Euclidean
distance of tie-value from and to all other nodes.

Socio-Technical Design (STD): The design that is influenced by an organisations’


social structure.

Thematic analysis: The process for encoding qualitative information in order to


relate the data to prior ideas.

Time-Short Series Design: A sequence of data points spaced and measured at


short time intervals using methods to understand such time series, the underlying
theory or to make forecasts.

Usability: A quality attribute that assesses how easy user interfaces are to use.

User-Centered Design (UCD): The design that considers the user into the center of
software design.

Vicarious Learning: It is the type of learning that occurs as a function of observing,


retaining and replicating novel behavior executed by others.

Ph.D. Thesis Centre for Interactive Systems Engineering LSBU Page 295

Anda mungkin juga menyukai