Designing and evaluating a debugging environment for

children ages ten to twelve.

Juta Staes

Supervisors: Prof. dr. ir. Francis wyffels, Prof. dr. ir. Joni Dambre
Counsellor: Tom Neutens

Master's dissertation submitted in order to obtain the academic degree of

Master of Science in Computer Science Engineering

Department of Electronics and Information Systems

Chair: Prof. dr. ir. Rik Van de Walle
Faculty of Engineering and Architecture
Academic year 2016-2017
Designing and evaluating a debugging environment for
children ages ten to twelve.

Juta Staes

Supervisors: Prof. dr. ir. Francis wyffels, Prof. dr. ir. Joni Dambre
Counsellor: Tom Neutens

Master's dissertation submitted in order to obtain the academic degree of

Master of Science in Computer Science Engineering

Department of Electronics and Information Systems

Chair: Prof. dr. ir. Rik Van de Walle
Faculty of Engineering and Architecture
Academic year 2016-2017
PREFACE i

Preface

Last summer I worked at Mediaraven where I introduced children to working with robots.
When I encountered this subject in the list, it immediately made me enthusiastic. I am
very glad I got to spend my master dissertation working with children and robots. I hope
I provided a useful debugger that can be used by many children at Dwengo in the future.

I could not have realized this master dissertation without the support and help of several
people. I want to thank my counselor Tom Neutens for spending a lot of time helping me
with the realization of my master dissertation, both when implementing the debugger and
when writing the thesis. I also want to thank my supervisors Francis Wyffels and Joni
Dambre for allowing me to work on the development of a debugger for children.

Additionally, I want to thank my family and friends for supporting me all these years.
In particular, my mom and dad for giving me the opportunity to study and for their
emotional support. Finally, I want to thank my partner Thomas for helping me with
mastering JavaScript and CSS, and for giving me many ideas.

Juta Staes, June 2017

PERMISSION TO LOAN ii

Permission to loan

“The author gives permission to make this master dissertation available for consultation
and to copy parts of this master dissertation for personal use. In the case of any other
use, the copyright terms have to be respected, in particular with regard to the obligation
to state expressly the source when quoting results from this master dissertation.”

Juta Staes, June 2017

 Designing and evaluating
a debugging environment for
children ages ten to twelve
by
Juta Staes
Master’s dissertation submitted in order to obtain the academic degree of
Master of Science in Computer Science Engineering
Academic year 2016–2017
Supervisors: Prof. Dr. Ir. Francis Wyffels, Prof. Dr. Ir. Joni Dambre
Counselor: Tom Neutens
Faculty of Engineering and Architecture
Chair: Prof. Dr. Ir. Rik Van de Walle
Department of Electronics and Information Systems
Ghent University

Summary
In this master dissertation, a debugging environment for children is presented and evalu-
ated. The visual programming language used is Blockly for Dwenguino, and the debugger
has the form of a simulator for the Dwenguino board. The goal of the debugger is to help
children find mistakes when they are programming by providing new ways to execute and
control their program. First, existing programming environments are analyzed to extract
the requirements for the debugger. The implementation of the debugger is then discussed.
To evaluate the debugger, a comparison is made between a control group learning to pro-
gram without a debugger and an experimental group learning to program with the use of
the debugger. The differences between the opinions of the children and their programming
skills are studied.

Keywords
Visual Programming Language, debugging, education, simulator, STEM
 Designing and evaluating a debugging environment
for children ages ten to twelve
Juta Staes
Supervisor(s): Francis Wyffels, Joni Dambre, Tom Neutens
Abstract— Since technology has become a substantial part of our daily
lives, the need for children and teenagers to learn about technology is be-
coming more and more important. One way to answer this need is by teach-
ing children to program. Many successful programming environments for
children have been developed. However no well discussed debugging envi-
ronment exists for a visual programming language (VPL) for children. This
paper presents such a debugging environment for Dwenguino. The goal of
the debugger is to help children find mistakes when they are programming
by providing new ways to execute and control their program. When test-
ing the debugger with children, most of them indicated that the debugger
helped them. When testing their programming knowledge, no significant
increase was observed compared to a control group. The use of the debug-
ger did not increase the performance of the learners for programming but
thanks to the debugger children can learn how to debug a program.
Keywords— Visual Programming Language, debugging, education, sim-
ulator, STEM
I. I NTRODUCTION
N order to be able to follow the evolution of technology, it
I is imperative that children and teenagers can use the tech-
nology, and that they also learn how to understand it. A first
step in this direction has been taken by introducing STEM (Sci-
ence, Technology, Engineering and Mathematics) into educa-
tion. STEM education integrates more technical skills into the
classroom to encourage children and teenagers into pursuing a
career in STEM. Another goal of STEM education is to attract
more girls into technical careers because of the current gender
gap [1].
However, after 10 years of STEM education the “T” of tech-
nology and the “E” of engineering still do not benefit from a lot
of attention in early education [2]. To let children fully benefit
from a STEM education, more attention should be given to tech-
nology and engineering. Teaching children to program robots
satisfies this need perfectly.
When teaching children to program a robot, several skills can
be learned. Early studies with the text-based language Logo
have shown that computer programming can help young chil-
dren with variety of cognitive skills, including number sense,
language skills, visual memory and planning abilities [3]. In
more recent studies a new concept has been introduced to de-
scribe a set of skills learned by programming: computational
thinking. Wing [4] first introduces computational thinking as
a set of thinking skills, habits and approaches that are integral
to solving complex problems using a computer. Furthermore,
handling the robot allows children to develop fine motor skills
and hand-eye coordination and when working in groups children
also naturally learn about social interaction and teamwork.
This paper presents a programming environment for robots
with a debugger that can be used by children ages ten to twelve.
The purpose of the debugger is to help children during program-
ming. The debugger can help children in one of the following
ways: help them find mistakes in their programs and correct
them, help them understand a new program, help them make the
link between a block of code and its corresponding execution.
The debugger also teaches children how to debug a program,
which is a skill essential to programming.
This paper will first present some existing VPLs for children
that were studied to extract the requirements for the debugger.
The debugger that was created is than presented and evaluated.
Research was done in classes to compare the opinions of chil-
dren working with or without the debugger and their program-
ming skills were assessed. The paper presents this research and
the results of using a debugger when teaching children how to
program.
II. R ELATED W ORK
There exists no well discussed debugger for children so in-
stead of looking at debuggers, programming environments for
children were studied to analyze what makes them successful
and what are the necessary requirements when designing a de-
bugger. Children generally learn programming with a VPL. This
has certain advantages: it takes away syntax errors, learners can
understand the data/program structure more intuitively and pic-
tures or animations can concisely convey more meaning than
words [5]. Armoni et al. [6] showed that learning to program
with a VPL as a child is a good basis to learn textual program-
ming later.
VPLs exist with different purposes and for different ages. A
first example is a tangible programming language that doesn’t
involve a computer such as KIBO [2]. In KIBO programming
is done with wooden blocks and a robot that scans the bar codes
on the blocks. KIBO is designed for children between 4 and 7.
Another example is Scratch [7], where children can create inter-
active media, such as games, stories, and simulations. Scratch
is designed for children between 8 and 16. Similar to Scratch is
Alice [8], a 3D programming environment to learn about object
oriented programming. Alice is designed for teenagers.
Based on the analysis of these programming languages, four
requirements were formulated when designing the debugger.
The first one is maintaining a preference for low floor [9]. Pro-
gramming languages for children are designed to be used as a
first step into programming. This means that it should be easy
to achieve something small. The second requirement is to main-
tain the right level of abstraction. Depending on what you want
the children to learn you should hide some of the functionality
that is too complicated [10], such as hiding the RGB compo-
sition of light. Third, to motivate learners you should connect
with their interests or experiences [11]. This way, they are able
to relate the programming work to what motivates them in their
existing life. Finally, the last requirement is to visualize a com-
putation’s result and its ongoing execution [12] so that children
can directly map instructions to the result of what they see on
the screen.
III. D EBUGGER FOR DWENGUINO
The programming environment used is Blockly for Dwen-
guino, which is a VPL influenced by Scratch used to program a
Dwenguino. A Dwenguino is a multi-functional Arduino com-
patible microcontroller board. With a Dwenguino board and
LEGO pieces a robot can be build. An example of such a
robot is shown on Figure 3. The Dwenguino board was created
by Dwengo, a non-profit organization that encourages technol-
ogy and engineering in education. The functions of Dwenguino
include: push buttons, analog inputs, a 2x16 LCD display, a
buzzer, everything for motor driving, servos, a sonar sensor that
can measure distance and LED lights.
The debugger implemented for this study is a simulator for
the Dwenguino board. The simulator shows the execution of
the board on the screen. It includes the board itself, 2 servos, 2
motors and a sonar. The simulator makes it possible to run the
program at a chosen speed. The program can be paused, stopped
and run step by step. If variables are used in the program, their
value is shown on the screen. During the execution of a pro-
gram, the executed block is highlighted. The controls for the
debugger were explicitly kept as minimal as possible to make
the debugger easy to use and understand.
Fig. 1. Blockly program that plays a melody when the north button on the
Dwenguino board is pressed. This program uses a condition as well as a
for loop. The blue block (delay block) shows the block that is currently
executed in the simulation.
Figure 1 and Figure 2 show respectively a program and its
corresponding execution on the simulator. The program plays
a melody when the north button on the Dwenguino board is
pressed. In the program, we can see that the delay block has
a blue color. The blue color is used to indicate the block that is
currently executed. The second image shows how the simulator
looks. All the different components are shown. The simulator
is in its first iteration of the for loop as can be noticed by the
value of variable i (200). The north button is pressed to start the
music, which is indicated by the red color of the button. On the
LCD display the text ”MUSIC” is written as ordered by the first
programming instruction.
Fig. 2. Simulator for Dwenguino during the execution of the program on Fig-
ure 1. The north button is pressed and ”MUSIC” is displayed on the LCD
display.
Aside from the static view of the simulated Dwenguino board.
The simulator also includes two scenarios to simulate driving
robots. The first scenario is a driving robot that has two wheels,
each of them attached to a motor. When the motors of a robot are
turned on, the robot on the screen will move. In this scenario, the
field of the robot is infinite, meaning that when the robot falls off
the screen it will appear on the other side. The second scenario
is a driving robot in a room, with a sonar sensor attached to the
front of the robot. In this case when the robot hits a wall in the
room, it will get stuck.
IV. R ESEARCH M ETHOD
Multiple workshops were organized in primary schools in
Ghent over a period of 3 weeks. In each school, two different
workshops were given to the same class, with approximately 2
weeks in between. Within a school, two classes followed the
workshops, both fifth and sixth grade (children between 10 and
12 years old). In total 6 classes followed the first workshop, and
4 classes followed the second workshop (one school canceled
the second workshop). Each workshop lasted for approximately
1 hour and 40 minutes. During the workshops, children worked
in groups of 3. The groups were chosen arbitrarily. For each
group, there was one laptop and 1 robot. To find an answer to
the research question, a comparative study was made between
children learning to program with the debugger (experimental
group) and children learning to program without the debugger
(control group). A concrete overview of the different workshops
that were given is shown in Table I.
The first workshop that was given served as an introduction to
programming. During this workshop, the children had to solve
small programming exercises. After workshop 1, a question-
naire was filled in by the children that asked them about their
opinion on the workshop. The goal of this questionnaire was to
 TABLE I
The most difficult part for the children was to place the program-
D IFFERENT SETTINGS DURING THE WORKSHOPS
ming blocks in the right order and to understand the difference
School Grade With Participants Participants between the setup and loop part of the Dwenguino start block.
debugger workshop 1 workshop 2 They also frequently misplaced the delay blocks in their pro-
A fifth no 24 23 gram. This finding corresponds with Carver and Klahr’s study
A sixth yes 18 18 [3] where they mentioned that it is especially difficult for chil-
B fifth yes 15 15 dren to find procedural errors (misordered commands). The sec-
B sixth no 18 18 ond observation is that the groups using the debugger only used
C fifth yes 22 0 a limited set of functions from the debugger: start, stop and
C sixth no 24 0 pause. They did not have enough expertise to use and under-
total: 121 74 stand the other functions.
The average scores for both the control and experimental
group for some questions of questionnaire 1 are indicated on
determine whether the use of the debugger changed the motiva- Table II. In the control group, where children did not use the
tion of the learners, and to determine if the children found the debugger, more children claim to like computers, they liked the
debugger useful. The questionnaire contained questions where workshop more and they found it approximately equally diffi-
children could give a score from one to five. The questions asked cult.
them about their programming affinity (whether they liked pro- TABLE II
gramming), their opinion about the workshop (if they liked the A NSWERS ON QUESTIONNAIRE 1: AVERAGE SCORE ON 5 BY CONTROL
workshop), their performance during the workshop (how good GROUP AND EXPERIMENTAL GROUP
they were at programming) and the usefulness of the debugger
(if the debugger helped them). question average average significance
without with de- (p-value)
debug- bugger
ger
likes computers 4.65 4.05 0.001
liked the workshop 4.19 3.98 0.23
difficulty workshop 2.95 2.91 0.84
debugger helped me 3.69

To analyze the significance of the differences between the

Fig. 3. Robot used during the second workshops in the classrooms. The robot
has two wheels, each attached to a motor and one sonar sensor at the front
of the robot.
During the second workshop, children had to program a robot
that avoids obstacles by detecting them. This programming ex-
ercise is supported by the scenarios in the simulator. The robot
is shown on Figure 3. The robot has two wheels, each attached
to a motor and one sonar sensor at the front of the robot that is
used to detect the obstacles. The children did not have to build
the robot, only program it. Two questionnaires were given to
measure the programming skills of the children, one before and
one after the workshop. Each questionnaire contained 6 ques-
tions where children had to identify what a given program did.
For each correctly answered question, 1 point was given. The
goal of these questionnaire was to determine whether the use of
the debugger influenced the programming skills learned by the
children.
V. R ESULTS
The first observation made during the workshop is that in gen-
eral the exercises were rather difficult for the children to solve.
two groups, a two-sample t-test is used with significance level
α equal to 0.05 and without assuming equal variance. When
comparing all the question, the only significant difference is
that children in the control group like computers more. This
could influence the results on the other questions. However,
when comparing the difference in how much the children liked
the workshop, there is no significant difference between the two
groups. This suggests that the use of the debugger has no in-
fluence on the motivation of the learners. When looking at the
usefulness of the debugger, 55% of the children gave a score
of 4 or higher indicating that the debugger helped them. This
indicates that most of the children found the debugger useful.
The scores on the tests of workshop 2 for the control and ex-
perimental group are indicated on Table III. In total 62% of the
children improved their test scores between the two tests and
26% got an equal test score. A scatter plot showing the scores
before and after on the tests for each individual in the control
and experimental group is shown on Figure 4. When comparing
the two groups, the control group obtained a lower pre test score
and a higher post test score. In other words, children program-
ming with the debugger obtained a lower test score on average
at first, but had a higher test score on average after workshop 2.
This result could be explained by the increased difficulty to
learn programming for the children using the debugger. As they
need to learn more things, they might focus less on the program-
ming at first and thus obtain a lower score on the first test. But as
 TABLE III
S CORES OBTAINED ON PRE AND POST TEST MEASURING PROGRAMMING
KNOWLEDGE
average average scored scored
pre score post score better equally
Without 1.88 2.40 56% 27%
debugger
With 1.48 2.58 70% 24%
debugger
seen in this study, this learning barrier is only of short duration,
once the children spent more time programming, they caught up
and even obtained a higher test score on the second test.
To determine whether this result is significant, a paired t-test
is used with significance level α equal to 0.05. and three two-
sample t-test are used with significance level α equal to 0.05
and without assuming equal variance. The first test is used to
compare the learning progress (difference between pre and post
test) for an individual in general. The other three test compare
the control and experimental group in the following three as-
pects: the test scores for the pre test, the test scores for the post
test and the learning progress made between the two tests. For
the learning progress in general, the result is an α coefficient of
1.19e − 05 meaning that there is a significant increase in the test
score between the pre and post test. This implies that children
improved their programming knowledge during workshop 2, re-
gardless of working with or without debugger. When comparing
the control and experimental group, none of the differences in
the test scores is significant. In this study, no significant differ-
ence in mean test score is obtained between the groups working
with or without the debugger. The children using the debugger
thus gained an equal amount of programming skills compared
to the children programming without debugger.
Fig. 4. Scatter plot showing the scores before and after on the tests for each
individual working with or without the debugger
VI. C ONCLUSION
In this paper a debugging environment for children ages ten
to twelve was created and evaluated. The goal of the debug-
ger is to help children when they are programming by providing
new ways to execute and control their program. The debug-
ging environment consisted of a simulator for Dwenguino. The
requirements for this debugger were inspired based on the anal-
ysis of the success of existing programming languages for chil-
dren. Once the debugger was created its effect on the children
was studied by organizing multiple workshops to teach children
programming, comparing a control and experimental group.
The first aspect that was studied was the difference in mo-
tivation between the two groups. No difference was observed
suggesting that the use of the debugger does not influence the
motivation of the learners. The two groups liked the workshop
equally and found it equally difficult. The second aspect that
was analyzed was the usefulness of the debugger. From the ob-
servations during the workshop it was clear that only a limited
set of functions from the debugger were used (start, stop and
pause). When asking the children about their opinion, the an-
swers were divided but most of them indicated that the simulator
helped them when programming. This result suggests that the
simulator can be useful when learning to program but more re-
search should be done. The last aspect that was studied was the
difference in learning progress between the two groups. When
analyzing the test scores of all children, a significant learning
progress is observed between the pre and post test. This means
that the children gained a substantial amount of programming
knowledge during the workshops. Children using the debugger
gained an equal amount of programming skills compared to the
children programming without debugger.
To conclude, the use of the debugger did not show any big
disadvantages, children had the same motivation, most of them
found the simulator to be helpful and the same level of program-
ming skills was obtained after the second workshop. To find out
if the debugger can provide significant better results in program-
ming or debugging skills more research should be done where
children spend more time programming so that they gain more
expertise on how to use the debugger.
R EFERENCES
[1] Vlaamse Overheid. ”Actieplan voor het stimuleren van loopbanen in
wiskunde, exacte wetenschappen en techniek.” (2012)
[2] Sullivan, Amanda, and Marina Umaschi Bers. ”Robotics in the early child-
hood classroom: learning outcomes from an 8-week robotics curriculum in
pre-kindergarten through second grade.” International Journal of Technol-
ogy and Design Education 26.1 (2016): 3-20.
[3] Carver, Sharon McCoy, and David Klahr. ”Assessing children’s LOGO de-
bugging skills with a formal model.” Journal of educational computing re-
search 2.4 (1986): 487-525.
[4] Wing, Jeannette M. ”Computational thinking.” Communications of the
ACM 49.3 (2006): 33-35.
[5] Neag, I. A., D. F. Tyler, and W. S. Kurtz. ”Visual programming versus tex-
tual programming in automatic testing and diagnosis.” AUTOTESTCON
Proceedings, 2001. IEEE Systems Readiness Technology Conference. IEEE
(2001)
[6] Armoni, Michal, Orni Meerbaum-Salant, and Mordechai Ben-Ari. ”From
scratch to real programming.” ACM Transactions on Computing Education
(TOCE) 14.4 (2015): 25.
[7] Resnick, Mitchel, et al. ”Scratch: programming for all.” Communications
of the ACM 52.11 (2009): 60-67.
[8] Cooper, Stephen, Wanda Dann, and Randy Pausch. ”Alice: a 3-D tool for
introductory programming concepts.” Journal of Computing Sciences in
Colleges. Vol. 15. No. 5. Consortium for Computing Sciences in Colleges.
(2000)
[9] Dasgupta, Sayamindu, et al. ”Extending Scratch: New pathways into pro-
gramming.” Visual Languages and Human-Centric Computing (VL/HCC),
2015 IEEE Symposium on. IEEE. (2015)
[10] Resnick, Mitchel, and Brian Silverman. ”Some reflections on designing
construction kits for kids.” Proceedings of the 2005 conference on Interac-
tion design and children. ACM. (2005)
[11] Cooper, Steve, and Steve Cunningham. ”Teaching computer science in
context.” Acm Inroads 1.1 (2010): 5-8.
[12] Utting, Ian, et al. ”Alice, greenfoot, and scratch–a discussion.” ACM
Transactions on Computing Education (TOCE) 10.4 (2010): 17.
CONTENTS ix

Contents

Preface i

Permission to loan ii

Overview iii

Extended abstract iv

Contents ix

List of Figures xii

List of Tables xiv

Used abbreviations xv

1 Introduction 1

2 State of the Art 3

2.1 Need For Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1.1 STEM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1.2 Skills Learned Through Programming . . . . . . . . . . . . . . . . . 5
2.2 Visual Programming Languages . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.2 History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2.3 Programming Environments for Children . . . . . . . . . . . . . . . 10
2.2.3.1 KIBO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
CONTENTS x

2.2.3.2 LEGO Mindstorms . . . . . . . . . . . . . . . . . . . . . . 12

2.2.3.3 Scratch . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.3.4 Greenfoot . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.3.5 Alice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2.3.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.4 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3 Assessing Programming Skills . . . . . . . . . . . . . . . . . . . . . . . . . 21

3 Research Question 24

4 Implementation of the Debugger 26

4.1 Blockly for Dwenguino . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.1.1 Dwenguino . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.1.2 Google Blockly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.2 Debugger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.2.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.2.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.2.3 Design Decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.2.4 Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

5 Research Method 37
5.1 Workshops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.1.1 Classroom Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.1.2 Workshop 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.1.3 Workshop 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.2 Questionnaires . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.2.1 Emotional Measures . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.2.2 Cognitive Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

6 Results 44
6.1 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6.1.1 Workshop 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6.1.2 Workshop 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.2 Emotional Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
CONTENTS xi

6.2.1 General results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

6.2.2 Comparison With or Without Debugger . . . . . . . . . . . . . . . 48
6.2.3 Children’s Opinion on the Debugger . . . . . . . . . . . . . . . . . 51
6.2.4 Correlation Between Test Questions . . . . . . . . . . . . . . . . . . 52
6.3 Cognitive Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
6.3.1 General Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.3.2 Comparison Questionnaires A and B . . . . . . . . . . . . . . . . . 55
6.3.3 Learning Progress . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
6.3.4 Learning Progress With or Without Debugger . . . . . . . . . . . . 58
6.4 Correlation Emotional and Cognitive Measures . . . . . . . . . . . . . . . . 59
6.5 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

7 Conclusion 62
7.1 Further Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Bibliography 66

Appendix 72
LIST OF FIGURES xii

List of Figures

2.1 Example of the user interface of HI-VISUAL [24] . . . . . . . . . . . . . . 10

2.2 Example of a Logo program with program execution [31] . . . . . . . . . . 11
2.3 KIBO robot together with its programming blocks [5] . . . . . . . . . . . . 12
2.4 GUI for programming with LEGO Mindstorms . . . . . . . . . . . . . . . . 13
2.5 GUI for programming in Scratch [29] . . . . . . . . . . . . . . . . . . . . . 14
2.6 GUI of Greenfoot: main view [36] . . . . . . . . . . . . . . . . . . . . . . . 16
2.7 GUI of Greenfoot: programming view [36] . . . . . . . . . . . . . . . . . . 16
2.8 GUI for programming in Alice [38] . . . . . . . . . . . . . . . . . . . . . . 17
2.9 GUI of Splish: programming view [50] . . . . . . . . . . . . . . . . . . . . 21
2.10 GUI of Splish: debugging view [50] . . . . . . . . . . . . . . . . . . . . . . 21
2.11 Questions of an artifact-based interview about Scratch [48] . . . . . . . . . 22

4.1 Dwenguino board with its functions indicated [54] . . . . . . . . . . . . . . 27

4.2 Dwenguino expansion shield with its functions indicated [54] . . . . . . . . 28
4.3 A selection of art robots built with Dwenguino and LEGO pieces [53] . . . 28
4.4 Blockly program that writes ”Hello Dwenguino” on the LCD display and
lets a light blink repeatedly. This program uses only custom Dwenguino
blocks (purple color). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.5 Blockly program that plays a melody when the north button on the Dwen-
guino board is pressed. This program uses a condition as well as a for loop.
The blue block (delay block) shows the block that is currently executed in
the simulation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.6 Simulator for Dwenguino during the execution of the program on Figure 4.5.
The north button is pressed and ”MUSIC” is displayed on the LCD display. 31
4.7 Overview of the different steps taken when executing the debugger for a
simple program. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
LIST OF FIGURES xiii

4.8 A view of the simulation of use case two: a robot driving in a room that
detects the wall and turns. 4 consecutive points in time during the simulation. 36

5.1 Robot used during the second workshops in the classrooms. The robot has
two wheels, each attached to a motor and one sonar sensor at the front of
the robot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

6.1 Picture of a class during workshop 2 . . . . . . . . . . . . . . . . . . . . . 46

6.2 Scatter plot showing the relations between the questions on questionnaire
A. A clear correlation is for example observed between (2) how good they
were feeling at the moment of the workshop and (3) liking the workshop. . 53
6.3 Histogram of test scores obtained per test both including the pre and post
tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
6.4 Scatter plot showing the scores before and after on the tests for each indi-
vidual working with or without the simulator . . . . . . . . . . . . . . . . . 58
LIST OF TABLES xiv

List of Tables

5.1 Different settings during the workshops . . . . . . . . . . . . . . . . . . . . 38

6.1 Answers on questionnaire 1 by control group working without debugger.

Number of children that indicated a score from 1 to 5 for each question . . 49
6.2 Answers on questionnaire 1 by experimental group working with debugger.
Number of children that indicated a score from 1 to 5 for each question . . 50
6.3 Summary of answers on questionnaire 2: total correct answers per question
and average test score obtained per child in each group before and after
workshop 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
6.4 Individual progress made during workshop 2 for each test group . . . . . . 55
6.5 Average test scores per programming concept on questionnaires A and B . 57
USED ABBREVIATIONS xv

Used abbreviations

API Application Programming Interface

CHERP Creative Hybrid environment for computer Programming
GUI Graphical User Interface
IDE Integrated Development Environment
RGB Red Green Blue
STEM Science, Technology, Engineering, and Mathematics
VPL Visual Programming Language
VPLs Visual Programming Languages
INTRODUCTION 1

Chapter 1

Introduction

Since technology has become a substantial part of our daily lives, the need for children
and teenagers to learn about technology is becoming more and important. To be able to
follow the evolution of technology, it is imperative that children and teenagers can use the
technology, and that they also learn how to understand it. The Royal Flemish Academy
of Belgium has published a statement concerning computer science in education [1]. The
statement recommends that both primary and secondary education should include a strong
component of computer science in compulsory education.

A first step in this direction has been taken by introducing STEM (Science, Technology,
Engineering and Mathematics) into education. In 2012 the Flemish government published
a plan for action in STEM education [2]. STEM education integrates more technical
skills into the classroom to encourage teenagers into pursuing a career in STEM. Another
goal is to attract more girls into technical careers because of the current gender gap [2].
A challenge in STEM education is to integrate technology and engineering as much as
sciences and mathematics.

This master dissertation will focus on teaching children ages ten to twelve about technology
and engineering. This is done by teaching children how to program a robot with the help
of a debugger. There exist several computer environments that children can use to learn
programming. However, when it comes to debugging, not many environments present this
functionality. This dissertation will present a simulator for Dwenguino with debugging
functionality in a VPL (Visual Programming Language). The goal of the simulator is to
help children with their learning process. In chapter 2 an overview of existing programming
INTRODUCTION 2

environments for children will be given along with a broader explanation for the need of
computer science in education. Afterwards, the research question and the requirements for
the debugger will be discussed. Thereafter, the research method will be presented together
with its results. The master dissertation will then end with a conclusion and discussion of
further work.
STATE OF THE ART 3

Chapter 2

State of the Art

2.1 Need For Programming

The need for programming in education can be motivated through the introduction of
STEM. This section will first explain how programming fits into STEM and what the
reasons are for introducing STEM into education. Afterwards an overview will be given of
the specific skills that can be learned when programming a robot.

2.1.1 STEM

We live in a world where we are constantly surrounded by technology, from cell phones
and computers to automatic faucets and automatic doors. Traditional education teaches
children about literacy, numeracy and science about the natural world. However not many
attention is paid to science from the human-made world. Introducing children to engi-
neering and technology is just as important as their understanding of the natural world
because technology has become such a large part of our daily lives. Children can develop
interest in engineering and technology because of their natural inclination to design and
build things, and to take things apart to see how they work. Not only is it important that
children should learn about this, it is also important that they learn it from an early age
on because education in early childhood is associated with more durable effects compared
to education on an older age [3].

STEM education was introduced more than 10 years ago to answer this need. More and
2.1 Need For Programming 4

more governments recognize the need to integrate STEM into education [1]. The most
prominent example of this is in the UK, where computer science has become mandatory
starting from the age of five. This decision was made due to both pressure from the industry
and lobbying by interest groups. The government of the UK argued that computer science
develops useful transferable skills, as explained in section 2.1.2, and introduces valuable
principles for a modern, digital world [4].

However, after 10 years the “T” of technology and the “E” of engineering still do not benefit
from a lot of attention in early education [5]. To let children fully benefit from a STEM
education, more attention should be given to technology and engineering. Teaching children
to program robots satisfies this need perfectly. Technology is learned when programming
the robot while engineering is learned when building the robot and interacting with the
physical world through the input and output sensors of the robot. Robotics offers a way
to teach young children about the types of sensors and electronics they encounter in daily
life in a hands-on and engaging way. Teaching foundational programming concepts, along
with robotics, makes it possible to introduce children to important ideas that inform the
design of many of the everyday objects they interact with.

Other benefits of a STEM curriculum are that children who start with it at an early age
demonstrate fewer gender-based stereotypes regarding STEM careers and fewer obstacles
entering technical fields [6, 7]. Attracting more women and more minorities would solve
the problem of skill shortage in critical STEM disciplines. Fewer gender-based stereotypes
would also mean that women in STEM can use more of their potential at work.

With so many advantages from STEM in education, one could ask why it still isn’t im-
plemented on a global scale. One of the challenges in introducing computer science into
education is the lack of skilled teachers in this subject. Among early childhood educators
there is a lack of knowledge and understanding in technology and engineering. There is also
a lack in appropriate pedagogical approaches to bring those disciplines into classrooms [8].
This master dissertation will not focus on the lack of skilled teachers, it however provides a
practical environment for children to learn programming and debugging that can be used
in the classroom.
2.1 Need For Programming 5

2.1.2 Skills Learned Through Programming

Many research has been done to find out about the skills that are learned through program-
ming. Early studies mostly point towards the development of cognitive skills. Cognitive
skills are the core skills your brain uses to think, read, learn, remember, reason, and pay
attention. Early studies with the text-based language Logo - invented in 1967 - have shown
that computer programming can help young children with variety of cognitive skills, in-
cluding number sense, language skills, visual memory and planning abilities [9].

In more recent studies a new concept has been introduced to describe a set of skills learned
by programming: computational thinking. Wing [10] first introduces computational think-
ing as a set of thinking skills, habits and approaches that are integral to solving complex
problems using a computer. More concretely it is also defined as the ability to abstract from
computational instructions (programming languages) to computational behaviors and to
identify potential bugs and places for errors. This concept is widely used among researchers
in this domain and is sometimes interpreted in different ways because it is a collection of
skills.

Three main skills can be extracted summarizing computational thinking when it comes to
young children learning to program: abstraction, automation, and analysis [11]. Abstrac-
tion is the process of transforming the problem into a general approach that can be used
for all instances. For example, when driving a robot, it considers all possible sensor com-
binations and the actions that should happen upon encountering them. The second skill,
automation, is the process in which a computer is instructed to execute a set of tasks. It
is transforming the idea of the driving robot into an actual implementation. The last skill,
analysis, is a reflective practice that refers to the validation of whether the abstractions
made were correct. It could be that a combination of sensors was not considered and this
could be discovered during analysis.

Other studies also point out some more general skills that can be learned when program-
ming. Children can learn about problem solving when translating the desired behavior of
the robot into a program. Applied mathematics can be learned when implementing math-
ematical calculations or basic algorithms. Children also learn to reason systematically
when approaching new problems. Furthermore, they also learn about project management
when they are working on a large program [12]. Finally, when programming, children can
become creative. They can become storytellers by creating their own meaningful projects
2.2 Visual Programming Languages 6

that react in response to their environment.

The skills mentioned above can all be learned when programming alone, in front of a
computer. However, when programming with robots more skills can be learned through
handling the robots. Young children can become engineers by playing with gears, motors,
sensors, and programming loops [13]. Robotic manipulation allows children to develop fine
motor skills and hand-eye coordination [14]. It is also important to note that programming
with robots limits the screen time per day as not all time is spent in front of a computer.
Instead the robot is tested in the real world with which it can also interact.

When children are programming in group, a last important set of skills can be learned.
While working together they naturally learn about social interaction and teamwork. When
they have to make a decision in group, they also learn about negotiation and compromise.
All these skills can be learned in a playful context which makes it easy and fun for children
to learn [15].

2.2 Visual Programming Languages

A VPL is a programming language that visualizes the model of the program in two-
dimensional forms with visual expressions such as pictures or animations, rather then
building the program textually. A VPL converts the data elements of a programming lan-
guage into pictorial ones, the language model such as data structures or program structures
into graphs, and often program behaviors into animation patterns [16].

This section will first discuss why children use VPLs to program. It will then give a brief
overview of the history of VPL followed by a summary of the existing VPLs that children
currently use to program. Afterwards, it will review the current techniques children use for
debugging. Finally, the different approaches for assessing programming skills in a study
will be discussed.

2.2.1 Motivation

Learning to program is not easy. It is especially difficult when you are learning a textual
programming language. The difficulties can be separated into five areas with a certain
2.2 Visual Programming Languages 7

degree of overlap. First there is the general problem of orientation, finding out what
programming is for and what kinds of problems can be solved. Second, there are difficulties
associated with understanding the general properties of the machine that one is learning to
control. When programming a robot, the learner should understand the link between the
behavior of the robot in the physical world and how it relates to the program. Third, there
are problems associated with the notation of the formal language that has to be learned,
both mastering the syntax and the underlying semantics. The syntax of a programming
language consists of knowing which symbol has which function for example knowing when
to use semi-colons. The semantics indicate the meaning of a syntactically correct statement,
for example understanding that a function can have a return type. The fourth difficulty is
to understand standard structures in the programming language such as loops or variables.
Finally, there is the issue of mastering the pragmatics of programming, where a student
needs to learn the skill of how to specify, develop, test, and debug a program using whatever
tools are available [17].

Research was done to find out the most common mistakes that students make when learning
to program. The most common mistakes are type and semantic errors such as invoking
methods with wrong arguments (e.g. wrong types) or trying to invoke a non-static method
as if it were static. However, a lot of syntax errors are also made such as unbalanced
parentheses [18].

VPLs are created with the idea of removing some of these barriers for beginners in program-
ming, these can be both children or people without expertise. The insight of knowing what
the difficulties are when learning to program and which mistakes are most frequently made,
can help when creating a new programming language that tries to be as easy and accessible
as possible. This is exactly the idea behind VPLs. There are three main advantages when
using block-based programming compared to text-based programming.

The first advantage of a VPL is that, by using blocks instead of text, it takes away syntax
errors. VPLs operate with intuitive graphical abstractions which means they require a
lower level of programming expertise [19]. Booth and Stumpf [20] evaluated the relative
usability of blocks and text-based programming for the same task set, finding that - for
people without a background in electronics or programming - block-based environments
supported easier modification and left users with higher satisfaction.

Second, learners can understand the data/program structure more intuitively than a con-
2.2 Visual Programming Languages 8

ventional text-based programming language, because visualized programs look simpler and
more appealing [16]. When first encountering the programming environment, the learner
can easily build a small program, for example writing text on the display of a robot. This
makes the programming language very accessible for new users.

Finally, pictures or animations are more powerful than words as a method of communica-
tion. They can often concisely convey more meaning than words. They can help compre-
hension and remembering, they may provide an encouragement for learning to program
and, if designed properly, they can work across cultures and languages [21].

These advantages show that VPLs provide a better environment for children when learn-
ing to program compared to text-based languages. However, they also have some clear
restrictions. The first restriction is that it is highly challenging to create and maintain a
complex program as the size of the program increases. This both in terms of organizing
the program and when reading or manipulating it [22]. One reason for this is that the
maximum number of visual elements that can be displayed on screen or printed on a page
is limited. Another reason is that visual languages lack the advanced capabilities that
allow textual languages to deal with application complexity, such as namespaces, objects
and object templates. A last reason is that visual programming has a limited ability to
model complex data structures and to describe complex algorithms. For example, repre-
senting both control flow and data flow on the same diagram may increase the complexity
of drawings. As the complexity of the application itself increases, such diagrams become
unmanageable [19].

Another restriction is that there is no tool compatibility. Plain text makes it easier to work
with external tools, such as version control systems. It is also the accepted interchange
format to pass program source to third-party compilers. Finally, VPLs provide lower
usability. Although text permits many syntax errors, it is more flexible and easier to
manipulate than many more rigid representations [22].

To know if VPLs are indeed a good learning method to teach children to program, Armoni
et al. [23] studied the transition from learning VPLs in middle school to learning textual
programming in secondary school. They found that the programming knowledge and ex-
perience of students who had learned a VPL greatly facilitated learning the more advanced
material in secondary school: less time was needed to learn new topics, there were fewer
learning difficulties, and they achieved higher cognitive levels of understanding of most
2.2 Visual Programming Languages 9

concepts. Furthermore, there was increased enrollment, and students were observed to
display higher levels of motivation and self-efficacy.

In conclusion, visual programs are ideal for simple applications and learners without pro-
gramming expertise. Although their inherent limitations make them inadequate for com-
plex problems, this is not a problem for children because they only make simple applica-
tions. VPLs also form a good base to learn textual programming in the future because
children are more motivated and they have less learning difficulties.

2.2.2 History

Visual programming environments can be used both for educational purposes but also
in the professional industry. As early tools in industry we consider LSI-CAD which was
application dependable and HI-VISUAL which was one of the earliest icon-based visual
programming environments [24, 25]. In LSI-CAD - invented in the late 70’s - a logic
circuit was specified by arranging visual objects corresponding to logic gates/circuits and
connecting appropriate input/output terminals of the visual objects on the screen. After
construction of the circuit, it accepts certain input values and then generates output values
corresponding to the input, as mentioned earlier this is very application dependent. On the
other hand, HI-VISUAL - invented in 1984 - is an iconic programming system. Specification
of a program is carried out on the screen by overlapping icons representing objects as is
done by the user in the real world. A programming interface of HI-VISUAL applied to an
office information processing environment is shown on Figure 2.1.

After the introduction of VPLs a lot more programming languages were invented with the
idea of replacing textual programming. An example of a VPL that is still currently used
by professionals is LabVIEW. LabVIEW [26], similar to Matlab, is commonly used for
automation control and data acquisition. Its graphical representation, similar to a process
flow diagram, was created to provide an intuitive programming environment for scientists
and engineers. It is however one of the few examples of VPLs used by the professional
industry.

Besides the introduction of VPLs, Logo also played a huge role in the development of
current VPLs for children. Seymour Papert’s 1980 book Mindstorms [27] presented Logo
as a cornerstone for rethinking approaches to education and learning. Logo is a graphic-
oriented Lisp-based programming language designed for educational use, developed in the
2.2 Visual Programming Languages 10

Figure 2.1: Example of the user interface of HI-VISUAL [24]

late 1960s by Seymour Papert and his colleagues at MIT. The project became more popular
than expected as it was used by millions of junior high school students. To even make it
more accessible, the language was adapted so that it contained English words and a robot
turtle was introduced [28]. An example of a Logo program is illustrated in Figure 2.2. Logo
served as inspiration for multiple VPLs, such as Scratch, as explained in section 2.2.3.3
[29, 30].

2.2.3 Programming Environments for Children

This section will give an overview of some important VPLs in this century for children.
Over the past 25 year many VPLs have been developed. Rather than giving an extensive
list of all programming languages, this section will only mention five. Each of the mentioned
languages has a different approach to how children can program: using a physical robot,
providing a transition from blocks to text or VPLs focused on different age groups. The
languages that will be presented are KIBO: a tangible programming language for children
in pre-kindergarten, LEGO Mindstorms: a building kit for programmable robots, Scratch:
a programming environment for creating your own interactive stories, games, animations,
and simulations, Greenfoot: a programming environment for children ages 14 and up,
providing a transition to textual programming and finally Alice: a programming system
for building 3D virtual worlds.
2.2 Visual Programming Languages 11

Figure 2.2: Example of a Logo program with program execution [31]

2.2.3.1 KIBO

KIBO (formerly known as KIWI or Kids Invent with Imagination) is a robotics kit for
children ages 4-7 developed at Tufts University in 2011 through funding from the National
Science Foundation [32]. KIBO involves hardware (the robot itself) and the software
used to program the robot called CHERP (Creative Hybrid environment for computer
Programming). CHERP is a tangible programming language consisting of interlocking
wooden blocks and thus no computer is needed to program the robot. An example of
the KIBO robot together with a program can be found on Figure 2.3. The robot has an
embedded scanner that allows users to scan barcodes on the CHERP blocks and send a
program to the robot instantaneously.

Sullivan and Bers [5] used the KIBO robot to demonstrate that children, starting from
the age of 4, are able to learn foundational programming concepts and that children as
young as 7 years old are able to master concepts as complex as programming a robot using
conditional statements. Based on numerous testing of the robot, this study formulates
three criteria for a programmable robot for children in pre-kindergarten: robotics parts
should be physically and intuitively easy to connect, programming the robot should need
2.2 Visual Programming Languages 12

Figure 2.3: KIBO robot together with its programming blocks [5]

the minimum amount of computer equipment and children should be able to attach a
variety of crafts and recycled materials to the core robotic parts so that the children can
decorate their robot.

2.2.3.2 LEGO Mindstorms

LEGO Mindstorms [33] are a series of a programmable robot kits designed for children
ages 10 and up, brought on the market in 1998. Each robot is customizable as a kit
contains many LEGO pieces to adapt the robot. A robot can have the following input
sensors: touch, light, angle (rotation), and temperature. It also has different outputs:
motors, infrared emitters, lights and sound. As the basic LEGO Mindstorms kit, that
costs approximately US$200, contains 750 building block pieces, many different projects
can be built with the robots.

LEGO provides two tools for programming the robot. The first is a VPL where program-
ming is done by dragging puzzle pieces (representing program steps) together to produce
a chain that forms a complete program. This GUI (Graphical User Interface) environment
supports basic programming constructs such as loops, subroutines, and concurrency. An
example of the user interface is shown on Figure 2.4. An advantage of LEGO Mindstorms
is that it can also be programmed through a library for generating Visual Basic programs.
This means that the robot can also be controlled by textual programming making it us-
able not only by children but also by college students as demonstrated by Klassner and
2.2 Visual Programming Languages 13

Figure 2.4: GUI for programming with LEGO Mindstorms

Anderson [33].

2.2.3.3 Scratch

Scratch [30] is the most important VPL that exists for children at this day. It was developed
by the Lifelong Kindergarten Group at the MIT Media Lab in 2007. Scratch is a block-
based programming language that enables young people to create interactive media, such
as games, stories, and simulations. One of Scratch’s significant predecessors and sources of
intellectual inspiration is Logo, see section 2.2.2. But it is also based on other VPLs such
as Alice which is discussed further. The idea behind Scratch is that they were not satisfied
with other VPLs as they wanted to create a programming language that appealed to people
who hadn’t previously imagined themselves as programmers. This was done by making
the floor even lower and the walls even wider, meaning that Scratch is very accessible for
people with no expertise, even more than other programming languages. Once people gain
some expertise there is still a lot that can be done and learned as there are many different
2.2 Visual Programming Languages 14

types of projects and people don’t get bored.

Scratch was developed for children between 8 and 16. Using Scratch, young designers “snap
together” programming blocks, just as one might snap together Lego bricks. Scratch blocks
are shaped to fit together only in ways that make syntactic sense. Scratch offers more than
100 programming blocks, offering many possibilities for experienced users. In a Scratch
project, blocks are used to manipulate the attributes of objects, called sprites. Scratch
projects are created by adding one or more sprites and then programming their behaviors.
The interface for programming in Scratch can be found on Figure 2.5 [29]. On this figure
the program lets the cat move with the sound of drum beats and the color of the cat
changes.

Figure 2.5: GUI for programming in Scratch [29]

MIT provides an online web space (https://scratch.mit.edu/) where people can share
their creations with one another. This online Scratch community played an important role
in its success. In the first six years of the site more than 3 million projects have been
2.2 Visual Programming Languages 15

uploaded. The community makes it possible to share your own projects, receive feedback
and advice, but also to be inspired by other people’s work. For many Scratchers this is a
strong motivation.

As Scratch is very popular, a lot of people build extensions to it [34]. An example for this
is S4A (Scratch for Arduino) which allows for simple programming of the Arduino open
source hardware platform. It provides new blocks for managing sensors and actuators
connected to Arduino [35]. This way Scratch can also be used to program a robot.

2.2.3.4 Greenfoot

Greenfoot [36] is a tool for teaching object oriented programming to students from 14 years
old and upwards, and is also suitable for college- and university-level education. Greenfoot
was developed starting from 2004 by the University of Kent. Its development environment
is a combination of a framework for producing Java programs that can be visualized in a
two-dimensional grid and of an IDE (with class browser, editor, compiler, execution control,
debugger etc.) suitable for novice programmers. Novice programmers can this way both
program with visual elements and by textual programming. This makes it possible for
the student to make a transition from visual programming to textual programming in
Java. Greenfoot provides the “main” method and a tiny yet powerful class library that
shields the user from the overwhelming large and complex set of Java APIs. It provides
an Actor/World framework that makes it possible to start by writing short, self-contained
methods without having to build a Java application from scratch [37].

Figure 2.6 shows Greenfoot’s main window, with a project open in the environment. The
main part of the window shows the Greenfoot world, the area where the program executes.
On the right, we see a class diagram that visualizes the classes used in this scenario and
their inheritance relations. By clicking on a class, we can see the Java code where we can
program the behavior of the objects, as shown in Figure 2.7. This makes Greenfoot not
entirely a VPL, as programming is done in Java, but is has visual elements in the sense
that the program’s structure (relations between objects) and its execution are visualized.
In other words, Greenfoot is used to introduce students to textual programming with the
use of graphical elements.
2.2 Visual Programming Languages 16

Figure 2.6: GUI of Greenfoot: main view [36]

Figure 2.7: GUI of Greenfoot: programming view [36]

2.2 Visual Programming Languages 17

2.2.3.5 Alice

Alice is a 3D animation environment, developed in 1997 at Carnegie Mellon University

in which objects and their behaviors are visualized. The goal of the Alice project is to
make it easy for novice programmers to develop interesting 3D environments and to learn
about object oriented programming. The 3D animation assists in providing stronger object
visualization and a flexible, meaningful context for helping students to “see” object oriented
concepts. [38, 39]

Figure 2.8: GUI for programming in Alice [38]

Alice was developed for college students as a first approach to programming. Users can
build programs by dragging and dropping graphical command tiles and selecting parame-
ters from drop-down menus. Figure 2.8 shows an Alice screen as a user creates a simple
animation. Alice provides several built-in action commands. In general, actions can be
2.2 Visual Programming Languages 18

subdivided into two categories: those that tell an object to perform a motion (e.g. move
or rotate) and those that change the physical nature of an object (e.g. appear or destroy)
[40].

2.2.3.6 Discussion

Although each of these programming languages has its own focus, there are some common
principles that can be found among them. Four of them are important in this study as
they are all considered when designing and evaluating the debugging environment. The
first one is maintaining a preference for low floor [34]. All these programming languages
are designed to be used as a first step into programming. This means that they should
not be too complicated so that everybody can learn something. This principle can mainly
be found in Scratch and KIBO, which are languages designed for young, non-technical
learners.

The second principle is to maintain the right level of abstraction. Depending on what you
want the children to learn you should hide some of the functionality that is too compli-
cated. An example of this is that Greenfoot has a compile button as it wants students
to understand compiling in contrast to the other languages that hide compiling. In 2005
Resnick and Silverman [41] highlighted the importance of choosing blocks carefully. They
designed a block to input color with three parameters (RGB) but this turned out to be
too complex and they replaced it by one numerical value instead. As they did not want
the children to understand the RBG composition of light, it was better to hide it.

Third, to motivate learners, you should connect with their interests or experiences [42].
This way, they are able to relate the programming work to what motivates them in their
existing life. Scratch does this by supporting many different types of projects so people
with many different interests and learning styles can all become engaged. Alice does the
same by allowing students to create 3D animations.

Finally, the last principle is to visualize a computation’s result and its ongoing execution
[37]. Because Alice worlds (and those of Scratch and Greenfoot) are so visual, students
can directly map instructions to the result of what they see on the screen. This is a real
power of program visualization. When an animation doesn’t work correctly, the student
almost always knows which are the offending sections of code that drive. This will help
the student to identify and solve their mistakes.
2.2 Visual Programming Languages 19

2.2.4 Debugging

Ko and Myers [43] defined a bug as a combination of one or more errors in the code
(software errors), which may produce errors in execution (run time faults), which in turn
may produce failures in program behavior (run time failures). They defined debugging
as determining what run time faults led to a run time failure, determining what software
errors were responsible for those run time faults, and modifying the code to prevent the
run time faults from occurring.

As mentioned earlier, see section 2.2.2, Logo was a pioneer for programming languages
for children. Early research was done to assess children’s Logo debugging skills. in 1986
Carver and Klahr [9] introduced a debugging model presenting four phases of debugging.
This model is still an accurate representation of the current debugging process. The four
phases are:

1. Program evaluation: run the program and compare the output to the desired behav-
ior.

2. Bug identification: describe the difference between the output and the desired be-
havior. Propose types of bugs that might be the cause of the problem.

3. Bug location: find the location of the code that causes the discrepancy in the program.

4. Bug correction: replace the code with a correct statement that results in the desired
output.

Research into debugging for textual programming started in the 1980s and indicated that
novice programmers have difficulty learning to debug because it requires the use of many
new skills simultaneously [44]. Debugging includes using print statements, commenting
out lines or using a debugger to run through the code. The hardest part about debugging
is identifying the location of the bug. The process of fixing the bug, once it is located,
is considered less difficult [45]. More recent research corroborates these findings. Marzieh
et al. [46] also discovered that a good understanding of programming doesn’t necessarily
implies a good understanding of debugging. This suggest that debugging should be con-
sidered as a skill separated from programming, and it should also be taught to students.
More investigation into novice debugging could lead to important insights for improving
computer science education [47].
2.2 Visual Programming Languages 20

While experienced programmers debug with the use of for example print statements, this
is too complicated for children. Many programming environments for children do not
present a debugger. This means that children have to use other methods to find mistakes
in their programs. Brennan and Resnick [48] asked Scratchers to described their various
testing and debugging practices, which they developed through trial and error, transfer
from other activities, or support from knowledgeable others. Debugging techniques include
read through your scripts, experiment with scripts, try writing scripts again or find example
scripts that work. Carver and Klahr [9] found that it is especially difficult for children to
find procedural errors (misordered commands).

However, when a debugger is introduced much more techniques can be used, such as
running the code step by step. In practice, as this is a newly examined subject, there is
no VPL with a well discussed debugging environment for children. A related study from
2012 presents a debugger for tangible programming called Robo-blocks [49]. Robo-blocks
lets children run the commands of the robot step by step as in a real debugger. During the
execution, LED lights indicate which command is currently executed. This naturally allows
children to associate a command block with the resulting action. Thus, pinpointing the
bug became more manageable. They found that the children could easily use the debugger
and that it helped them to identify and correct bugs.

Another related study represents Splish [50], a debugging environment for a VPL, but it is
not specifically targeted for children. Splish is a VPL that allows users to write programs
that run on a microcontroller board family called Arduino. In Splish users can run the
code step by step and at the same time get information about the input and output of the
Arduino. The user interface of Splish for building a program is shown in Figure 2.9 and
the debug information is shown on Figure 2.10. Based on the interface of the debugging
screen it is clear to see that Splish is not designed for children.

The challenge in this master dissertation is to make a debugger for a VPL that can help
children to identify and correct their bugs and at the same time that can easily be used by
children. None of the above discussed environments fulfill these requirements.
2.3 Assessing Programming Skills 21

Figure 2.9: GUI of Splish: programming view [50]

Figure 2.10: GUI of Splish: debugging view [50]

2.3 Assessing Programming Skills

Major et al. [51] summarized the different types of studies that are being performed by
researchers to investigate the teaching of introductory programming concepts. Brennan et
al. [48] also discussed the different techniques that were used to evaluate Scratch. When
evaluating programming, two different aspects can be considered. The two major aspects
2.3 Assessing Programming Skills 22

are cognitive (how effectively do children learn) and affective (how enjoyable the experience
is, and how motivated by it the children are) [52].

The four different methods will now be presented along with their strengths and limitations.
Questionnaires are by far the most used technique. In a questionnaire, the researchers can
both evaluate the cognitive and affective aspects by combining questions about how much
they liked it with questions about their programming knowledge. Multiple questionnaires
can be filled in along the study to measure the progress. A questionnaire is an easy method
as it requires a short amount of time and can easily evaluate the children’s programming
knowledge. Its limitation is that children fill in the questionnaire with their perception of
the truth and this may be different from the actual truth. An example of this is when
asking them how difficult it was they might think that they performed well while this was
not the case.

The second technique is by performing interviews. An example of an interview about

Scratch is shown on Figure 2.11. In an interview, all aspects of programming can be
interrogated. The advantage of an interview is that the interviewer can easily see if the
child has a deep understanding of programming. The interview gives more conclusive
results compared to the questionnaire. The disadvantage is that it is time consuming.

Figure 2.11: Questions of an artifact-based interview about Scratch [48]

2.3 Assessing Programming Skills 23

The third method is by giving the children programming tasks and grading them. This
approach focuses on the cognitive aspects. Possible programming tasks are: programming
a project, explain what a project does, describe how it could be extended, fix a bug, or
extend a project by adding a feature. With this method, different programming tasks can
be given with a different difficulty level. The strength of this approach is that it offers an
opportunity to explore different ways of knowing, such as critiquing, extending, debugging,
and remixing, as well as fluency with different concepts and practices. The programming
tasks also emphasize process-in-action, rather than process-via-memory. The limitations
are that it is also time consuming. And it is difficult to deal with children who are stuck:
how much assistance should be given?

The last method is project analysis. This approach also focuses on the cognitive aspects.
With this method, children can freely build their own projects. This can for example be
done in Scratch as children have a portfolio containing all the projects they made. To
assess their programming skills, the researchers look at the complexity of the projects that
the children made and with which frequency different types of programming blocks are
used. The advantage is that this method allows the researcher to see which programming
concepts (e.g. variables, loops, conditions) are mastered by the children. A disadvantage
is that this approach is entirely product-oriented, and reveals nothing about the process
of developing projects and, in turn, nothing about the particular computational thinking
practices that might have been employed. It also requires the children to already have
spent a lot of time creating projects.

One important aspect to keep in mind when performing a study is to keep the subject
appealing to the learners, as discussed previously in section 2.2.3. By integrating for
example art into the programming tasks, increased learning is observed and a broader
audience (with differences in age, gender and socio-cultural backgrounds) is captivated
[53].
RESEARCH QUESTION 24

Chapter 3

Research Question

In chapter 2, an overview as given of the motivation to teach children to program and

several programming environments for children were analyzed. Programming for children,
mainly because of Scratch, is a well discussed topic. However, no debugging environment
exists is a VPL that is aimed for children.

This master dissertation will present a simulator of a robot as a debugging environment for
children ages ten to twelve. The debugger will be designed based on the study in chapter 2.
The goal of the debugger is twofold. (1) It should help the children when they are learning
to program. The debugger can help the children during different activities:

• Find mistakes in a program

• Understanding someone else’s program

• Understand the meaning of each programming block individually

• Make the link between the robot’s behavior and the executed code

• Testing a program is faster as no hardware is required

• See the difference between the ideal scenario in the simulator and the possibly faulty
hardware in the real world

• Control the speed of the execution, this allows children to have more time to process
what is happening
RESEARCH QUESTION 25

(2) The use of the debugger will teach children how to debug a program, which is an
important skill when it comes to programming.

To measure whether the debugger actually helps the children, a comparative study will
be performed with a control group that learns to program without a debugger and an
experimental group that uses the debugger. Both the motivation of the learners and their
programming skills will be evaluated. Based on the study an answer will be provided on
the following questions:

1. Do the children demonstrate a more significant learning progress in programming

when using a debugger compared to a control group?

2. Does the debugger change the motivation of the learners?

3. Do the children think that the debugger is useful?

IMPLEMENTATION OF THE DEBUGGER 26

Chapter 4

Implementation of the Debugger

This section will start by giving an overview of the used programming environment (Blockly
for Dwenguino) used in this study. It will then discuss the requirements for the debugger
based on the literature research from chapter 2. Finally, it will give an overview of the
created debugger and its use case scenarios.

4.1 Blockly for Dwenguino

Blockly for Dwenguino is the VPL used in this study. This is a VPL to program the Dwen-
guino, a multi-functional Arduino compatible microcontroller board. With a Dwenguino
board and LEGO pieces a robot can be build. The existing environment was extended
with a simulator for Dwenguino with debugging functionality. This section will describe
the Dwenguino board and the VPL.

4.1.1 Dwenguino

The Dwenguino [54] board was created by Dwengo, a non-profit organization that encour-
ages technology and engineering in education. The board along with its functions is shown
on Figure 4.1. The functions include: push buttons, analog inputs, a 2x16 LCD display,
a buzzer, everything for motor driving, servos, a sonar sensor that can measure distance
and LED lights. Through its expansion connector every pin can be accessed and with its
handy expansion shield, the Dwenguino can be extended with your own circuit or with
4.1 Blockly for Dwenguino 27

many Arduino shields. The expansion shield is shown on Figure 4.2.

Figure 4.1: Dwenguino board with its functions indicated [54]

A possible use case for the Dwenguino board is to build robots that create art [53]. Exam-
ples of art robots built with LEGO pieces and the Dwenguino board are shown on Figure
4.3.
4.1 Blockly for Dwenguino 28

Figure 4.2: Dwenguino expansion shield with its functions indicated [54]

Figure 4.3: A selection of art robots built with Dwenguino and LEGO pieces [53]
4.2 Debugger 29

4.1.2 Google Blockly

Google Blockly is a programming language influenced by Scratch. It is different from other

VPLs as it is not intended for direct use. Instead, Blockly is a set of libraries that greatly
facilitates the development of a Scratch-like programming language. Blockly provides a
set of predefined blocks for common programming functions such as loops and variables.
Additionally, one can easily define its own set of blocks using the Blockly API (Application
Programming Interface) to create a fully customized VPL [55].

Blockly is written in Javascript and is intended to run inside a web browser environment.
One of the defining features of Blockly is that there exist generators that translate the
Blockly code into other programming languages such as JavaScript and Python. On top of
this, custom generators can also be made using the API. This function will be used when
developing the debugger.

Dwenguino for Blockly makes use of the Blockly API. Custom blocks were created to
control the Dwenguino board. Figure 4.4 shows a program built with Blockly that writes
”Hello Dwenguino” on the LCD display and lets a light blink repeatedly. This program
uses only custom Dwenguino blocks (purple color). The meaning of each block can easily
be understood based on the text and the images. As with Scratch, the blocks are shaped
to fit together only in ways that make syntactic sense. Figure 4.5 shows a program that
plays a melody when the north button on the Dwenguino board is pressed. This program
uses, besides the Dwenguino blocks, also predefined blocks from Blockly for conditions and
loops.

When programming in Blockly for Dwenguino, the interface also shows the corresponding
code in C++, which is used to program the Dwenguino textually. This transition is done
by using the generators. This can allow more advanced children to make a connection
between a visual program and the corresponding textual program.

4.2 Debugger

The debugger implemented for this study is a simulator for the Dwenguino board. The
simulator shows the execution of the board on the screen. It includes the board itself, 2
servos, 2 motors and a sonar. The simulator makes it possible to run the program at a
4.2 Debugger 30

Figure 4.4: Blockly program that writes ”Hello Dwenguino” on the LCD display and lets a light
blink repeatedly. This program uses only custom Dwenguino blocks (purple color).

chosen speed. The program can be paused, stopped and run step by step. If variables are
used in the program, their value is shown on the screen.

Figure 4.5 and Figure 4.6 show respectively a program and its corresponding execution
on the simulator. The program plays a melody when the north button on the Dwenguino
board is pressed. In the program, we can see that the delay block has a blue color. The
blue color is used to indicate the block that is currently executed. The second image shows
how the simulator looks. All the different components are shown. The simulator is in its
first iteration of the for loop as can be noticed by the value of variable i (200). The north
button is pressed to start the music, which is indicated by the red color of the button.
On the LCD display the text ”MUSIC” is written as ordered by the first programming
instruction.
4.2 Debugger 31

Figure 4.5: Blockly program that plays a melody when the north button on the Dwenguino
board is pressed. This program uses a condition as well as a for loop. The blue
block (delay block) shows the block that is currently executed in the simulation.

Figure 4.6: Simulator for Dwenguino during the execution of the program on Figure 4.5. The
north button is pressed and ”MUSIC” is displayed on the LCD display.
4.2 Debugger 32

4.2.1 Requirements

At the end of section 2.2.3 four important principles were discussed for designing a pro-
gramming language for children. The four principles were (1) maintain preference for low
floor, (2) maintain the right level of abstraction, (3) connect with children’s interests or
experiences, and (4) visualize a computation’s result and its ongoing execution. These four
principles were considered as requirements when designing the debugger. Each of the four
requirements will now be discussed together with an explanation of how the requirement
was fulfilled.

The first requirement is to maintain a preference for low floor, meaning the simulator should
be easy to use. This was accomplished by only implementing a minimal set of functions
for the simulator, and thus limiting the controls for the simulator. The simulator can be
started, stopped, paused and can be run step by step. The speed of the simulation can
also be changed. Breakpoints were not included in the design because they are inherently
difficult to understand for children and hence they do not respect the low floor requirement.

To respect the second requirement, maintain the right level of abstraction, the concept of
debugging was hidden because debugging is a difficult concept to understand for children.
To the children the debugger was presented as a simulator rather than as a debugger. This
means that the children are in fact debugging without them knowing.

The third requirement, connect with children’s interests or experiences is considered when
creating the actual workshop for the children that was used to test the research question
in this study. This will be discussed in chapter 5.

The last requirement, visualize a computation’s result and its ongoing execution, is realized
by showing the execution of the program through the simulator. While the execution is
ongoing, the corresponding block is highlighted. This makes it possible for the learners to
map programming instructions to the result of what they see on the simulator.

4.2.2 Implementation

The implementation for the debugger can be found on https://github.com/Juta/Blockly

-for-Dwenguino. The implementation of the debugger can be summarized in three parts:
4.2 Debugger 33

(1) implement the debugging process, (2) extend the generator for Blockly to translate
custom blocks into JavaScript and (3) visualize the result on the simulated Dwenguino
board.

To implement the debugging process, an open source library debugjs [56] was used. Debugjs
gets as input the JavaScript code to run and can handle the debugging steps.

The custom blocks that were created in Blockly for Dwenguino did not yet have a generator
for JavaScript. This generator was implemented so that it could be used as input for the
debugger. The main part of the application was to implement the generated functions that
handle the corresponding visualizations.

An example that summarizes the three programming parts and links them together is
shown on Figure 4.7. In this example a simple program that writes text on the LCD
display is first processed into JavaScript code. This code is given as input to a debugjs
machine. This machine is handled step by step. When one step is executed on the machine,
the code to write text on the simulated Dwenguino board is run.

4.2.3 Design Decisions

Throughout the implementation some design decisions had to be made. The most im-
portant one was to choose how the timing of the instructions was done. With the actual
Dwenguino board, instructions run as fast as they can. This is however not possible in a
web view context. When running the simulation all at once the web browser froze. To
prevent this, one instruction is run every 16ms when the speed is chosen to be real-time.
This way the result of every instruction can separately be seen on the screen. The disad-
vantage is that this does not correspond to what happens in the real world. An example
of this is when running a code sample that immediately turns a LED light on and off in
a loop. In the real world, we will not see the light blink because the changes between the
state of the light are too fast. But in the simulator, we will see the light blink.

Some simplifications were also made to simplify the representation of the robot. An ex-
ample of this is that when using the sonar sensor, it is assumed that the sonar is well
connected to the chosen pins as the pin numbers are not taken into account for this func-
tion. When working with the expansions shield, additional sensors can be added to the
Dwenguino board by using the additional pins. The values of these pins are not supported
4.2 Debugger 34

Figure 4.7: Overview of the different steps taken when executing the debugger for a simple
program.
4.2 Debugger 35

by the simulator. This could be implemented but as it is rather advanced it was chosen
not to be included.

One major change was made in the design based on testing. At first the debugger could be
run simultaneously with altering the code (without affecting the debugger), as is done in
a typical debugger. But during the initial testing of the debugger with children it became
clear that they almost always let the simulation run for a long time without paying attention
to it, and without realizing it. Therefore, in the final design it is impossible to code and
debug at the same time. The belief is here that this will enable children to focus on the
debugger rather than only looking at it for a short time. This also makes it clear for the
children what is expected from them by making a clear distinction between programming
and debugging.

4.2.4 Use Cases

Aside from the static view of the Dwenguino board during the simulation, two use cases
were implemented to simulate driving robots. The user can select these use cases under
scenario, indicated on Figure 4.6. When the user selects a scenario this is shown below the
view of the static board, so both can be observed at the same time. The two use cases will
now be discussed.

When choosing the two use cases the goal was to support the workshops in the classrooms
that will be discussed in chapter 5. During the workshops, children had to program a
driving robot with a sonar sensor. The first use case can be used when the children
program a driving robot, while the second scenario can be used when they have a driving
robot that turns when encountering an object. Figure 5.1 shows the robot used for the
workshops, which is the robot where the use cases are based on.

The first use case is a driving robot that has two wheels, each of them attached to a motor.
When the motors of a robot are turned on, the robot on the screen will move. In this first
use case, the field of the robot is infinite, meaning that when the robot falls off the screen
it will appear on the other side.

The second use case is a driving robot in a room, with a sonar sensor attached to the front
of the robot. In this case when the robot hits a wall in the room, it will get stuck. The
4.2 Debugger 36

goal here is to program the robot to turn and in this way avoid getting stuck. An example
of a robot detecting a wall and turning there after, is shown on Figure 4.8.

Figure 4.8: A view of the simulation of use case two: a robot driving in a room that detects the
wall and turns. 4 consecutive points in time during the simulation.
RESEARCH METHOD 37

Chapter 5

Research Method

To provide an answer to the research questions, workshops were given in several classrooms.
This section will first present the content of the workshops and its audience. It will then
discuss the questionnaires that were used to measure the results of the workshops.

5.1 Workshops

5.1.1 Classroom Setting

Multiple workshops were organized in primary schools in Ghent over a period of 3 weeks.
In each school, two different workshops were given to the same class, with approximately
2 weeks in between. Within a school, two classes received the workshops, both fifth and
sixth grade (children between 10 and 12 years old). In total 6 classes followed the first
workshop, and 4 classes followed the second workshop (one school canceled the second
workshop). Each workshop lasted for approximately 1 hour and 40 minutes. During the
workshops, children worked in groups of 3. The groups were chosen arbitrarily. For each
group, there was one laptop and 1 robot. In total 121 children followed the first workshop
and 74 children followed the second.

To find an answer to the research question, a comparative study was made between children
learning to program with or without the debugger. To eliminate background differences
between schools there was a control and experimental group in each school. To eliminate
age differences there were both fifth and sixth graders learning to program with the de-
bugger. A concrete overview of the different workshops that were given is shown in Table
5.1 Workshops 38

5.1.

School Grade With Participants Participants

debugger workshop 1 workshop 2
A fifth no 24 23
A sixth yes 18 18
B fifth yes 15 15
B sixth no 18 18
C fifth yes 22 0
C sixth no 24 0
total: 121 74

Table 5.1: Different settings during the workshops

5.1.2 Workshop 1

Before workshop 1, the children had little to no programming experience. Therefore, the
first workshop consisted of multiple small exercises serving as an introduction to program-
ming. Both the control group, working with the robot and without the debugger, and the
experimental group, working with the robot and the debugger, received the same program-
ming tasks. The schedule of the first workshop was as followed:

1. Introduction to the robot (15min)

2. Small programming tasks (40min)

3. Find mistakes is small programs (30min)

4. Fill in questionnaire (15min)

A small explanation for each part of the workshop will now be provided. The workshop
started with an introduction to teach the children about all the functions of the robot
(LCD display, led lights, buttons, etc.). After this, the children could start programming.
For part two of the workshop, the small programming tasks, a total of 5 programming
tasks were provided. Only 3 of them will be discussed because none of the children had
the time to complete more than 3 tasks. To help the children with the tasks, a sheet of
5.1 Workshops 39

paper was given to them that indicated the programming blocks that would be used for
each task.

The first task was to write their names on the screen of the robot. This task introduced
the children to the setup block for the Dwenguino. The resulting program for task 1 can be
found on Figure 4.7. The second task was to make a light blink. This task familiarized the
children with both the loop in the setup block of the Dwenguino and with the delay block
to handle the timing within the loop. The third task was to make a doorbell, with light
instead of sound. For this task the children had to learn about conditions. In other words,
if a button was pressed, then the light had to go on. The concepts that were introduced
during the first workshop are the same one’s that will be used for workshop 2, namely the
difference between setup and loop, the time behavior of a program and conditions in if
statements.

During part three of the workshop the children had to find mistakes in existing programs.
Three existing programs were given, although most of the groups only solved one problem.
The first program was a counter that showed how many times a button was pressed. But
after 500ms the number disappeared because of a misplaced clear of the LCD screen. The
second and third exercise introduced children to for loop but this will not be considered as
most of them never completed these exercises.

The questionnaire that the children filled in at the end of this workshop was used to
evaluate the opinion of the children about programming and is discussed in section 5.2.1.

5.1.3 Workshop 2

Based on the observations from workshop 1, the program for workshop 2 was formed.
The first observation was that after workshop 1, the children gained a small amount of
programming experience but they still had great difficulty with small programs. As less
then 2h to introduce children to programming is very limited, special attention was paid for
making the second workshop not too difficult. Another observation is that children working
with both the debugger and the robot had the same amount of time to complete their work
although they had more things to do. Therefore, in the second workshop the children were
specifically told to use the simulator first and only after the simulation worked, they could
also test their program on the robot.
5.1 Workshops 40

For the second workshop, one big programming task was chosen. The exercises were still
small but they build on each other to make one resulting program. To choose the subject
of the programming task, multiple scenarios were considered: a robot avoiding obstacles
by detecting them, a robot that followed a line, a robot that created art with paint or
a robot that could draw (Spirograph). The first scenario, a robot avoiding obstacles by
detecting them, was selected because of both timing constraints (a workshop only lasted
1h40 and this makes it challenging to do anything else then programming, such as making
art) and the restricted difficulty of the corresponding programming task.

Figure 5.1: Robot used during the second workshops in the classrooms. The robot has two
wheels, each attached to a motor and one sonar sensor at the front of the robot.

When mentioning the important aspects of designing and evaluating a programming envi-
ronment in section 2.2.3.6, one of the principles was to connect with the learners interests
or experience to motivate them. The belief is that a driving robot that can avoid obstacles
is in fact challenging and appealing for most of the children between ten to twelve years
old. As only one scenario was chosen it was however difficult to find a scenario that would
be appealing to everyone that respected the time and difficulty constraints. The robot
that was used during the workshop was built beforehand to save time so that the children
could focus on the programming task. The robot is shown on Figure 5.1. The robot has
two wheels, each attached to a motor and one sonar sensor at the front of the robot that
is used to detect the obstacles.

The schedule of the second workshop was as followed:

5.2 Questionnaires 41

1. Pre test (15min)

2. Recapitulation of the previous workshop (10min)

3. Demo (5min)

4. Big programming task (1u10)

5. Post test (15min)

Two tests to measure the knowledge of the children were performed. One before the start
of the workshop and one after. In 2 groups (one control and one experimental group) the
pre test was done right before starting, while the post test was done 2h after completing
the workshop. In the 2 other groups the inverse was done, 2h before the workshop the pre
test was filled in and right after the workshop the post test was completed. The reason for
this is strictly practical: to gain time during the 1h40 workshop and to perform the 2 tests
not too close to each other. The content of the pre and post tests are discussed in section
5.2.2.

After the first test, the content of the previous workshop was rehearsed as a small intro-
duction to the workshop. After this, a demo was given to demonstrate the driving robot
that they would program to motivate the children. The programming task itself was split
up into 4 smaller incremental exercises: measure the distance between an obstacle and
display this on the screen, make the robot drive, print “turning” on the screen when an
obstacle is detected within 30cm and finally make the robot turn when an obstacle is too
close. As in the previous workshop, the children received a sheet of paper that indicated
the programming blocks that would be used for each task. When successfully completing
the workshop, the program created should be the same as the one shown on Figures A.1
and A.2, included in the appendix.

5.2 Questionnaires

In section 2.3, different techniques for evaluating programming skills were discussed. In
this study the used technique is questionnaires. This was chosen because questionnaires
can both measure the cognitive and emotional effect of the learners and because they are
not time consuming. The different questionnaires will now be presented.
5.2 Questionnaires 42

5.2.1 Emotional Measures

The first questionnaire was used to measure the emotional perception of the children. This
questionnaire was performed at the end of workshop 1. The goal of this questionnaire was
to provide an answer to research questions (2) “Does the debugger change the motivation
of the learners?” and (3) “Do the children think that the debugger is useful?”.

The questions in this questionnaire were chosen based on a similar study [52] measuring
the opinions of children. In this study children were introduced to programming in Scratch
and along the way their motivation was measured by asking them how they felt.

The questionnaire contained questions where children could give a score from one to five.
The questions asked to all children are:

1. Do you like using computers/tablets?

2. How good did you feel during the workshop?

3. Did you like the workshop?

4. Did you find the workshop difficult?

5. Did you already have previous programming experience?

6. Did you have a lot of problems during the workshop?

7. Did you easily find mistakes in the programs?

8. Do you want to program again in the future?

The questions asked to only the children using the debugger are:

9. Did the simulator help you?

10. Did you use the simulator frequently?

11. Did you find mistakes in the programs thanks to the simulator?
5.2 Questionnaires 43

These questions can be grouped into four groups that ask questions about the same subject:
programming affinity (1, 5 and 8), opinion about the workshop (2 and 3), performance
during the workshop (3, 6 and 7) and the usefulness of the simulator (9, 10 and 11). In
chapter 6, a comparison will be made between the answers of the control group and those
of the experimental group.

5.2.2 Cognitive Measures

During the second workshop a pre and post test were used to measure the progress of the
children, one before and one after the workshop. The goal of these questionnaires was to
provide an answer to research question (1) “Do the children demonstrate a more significant
learning progress in programming when using a debugger compared to a control group?”.

There were two questionnaires, A and B that were alternately used as pre or post test. In
other words, two groups used questionnaire A as pre test and B as post test, while the
other two groups did the inverse. The entire questionnaires can be found in the appendix
from Figure A.3 to A.8.

Each of the questions tests a different concept or mix of concepts in programming. The
different programming concepts that the children learned were discussed in section 5.1.2.
These concepts are: (1) difference between setup and loop in the Dwenguino start block,
(2) the time behavior of a program and (3) using conditions and if statements. The first
three questions of each questionnaire measure respectively the understanding of one of the
three concepts. The fourth question asks about concepts 1 and 2, the fifth question asks
about concepts 2 and 3 and the sixth question asks about concepts 1 and 3.

In chapter 6, the two questionnaires will be compared to each other to see if they are of
the same level of difficulty. The answers on the pre and post test within a same group
will be compared to determine if there is progress. Afterwards a comparison will be made
between the control and experimental group. These results will be analyzed and possibly
linked to the answers on the previous questionnaire within each group.
RESULTS 44

Chapter 6

Results

This chapter discusses the results from the two workshops. First some general observations
that were made during the workshops will be discussed. Afterwards the results of the
two questionnaires will be analyzed and a comparison between the control group and the
experimental group will be made. This chapter will then end with a discussion of the
results regarding the research question.

6.1 Observations

This section will discuss the observations made during the two workshops. These work-
shops will be discussed individually although some observations were made during both
the workshops.

6.1.1 Workshop 1

During workshop 1, most of the children came into contact with programming for the first
time. This meant that a lot of new concepts had to be learned. The main observation
during this workshop was that children had a hard time grasping the concepts of program-
ming. In total 8 programming exercises were provided but on average the children only
completed 4. A lot of children found it difficult to find programming mistakes. This is
very comprehensible as this was a complete new subject for them. As mentioned in sec-
tion 5.1.3, based on this observation the exercises for workshop 2 were revised to be more
6.1 Observations 45

achievable.

Another observation made is that children from the group working with the simulator
didn’t use the simulator very often. On top of learning to program, this group had to
learn how to work with the simulator, and this within the same time. Because of the time
constraints and the added difficulty, they didn’t use the simulator much. Most of them
struggled with understanding the purpose of the simulator and only used the start and
stop button.

During workshop 1 there were some troubles with the hardware as some Dwenguino boards
did not respond to the programmed code. This meant that some children worked with a
broken board at one time, that later was replaced. This had two consequences. The
first one is that children appeared to be less motivated as they were frustrated by the
malfunctioning of the board (the board did not react and did not run any code). The
second consequence is that some children abandoned using the robot and instead ran
everything on the simulator. This particular group was strongly motivated to use the
simulator and as a result understood more of the simulator’s purpose and its controls.
This was however an exception compared to the other groups.

A last observation is that there was some difference from class to class on how the workshop
went. In general, the workshops were rather chaotic and the children needed a lot of help
during the programming exercises. In one school the classes were smaller (15 children)
compared to the other schools. As a result, the children in those classes got more attention
and thus got more help. These classes were more quiet and things ran more smoothly.
This suggests that a class size for the workshops of 15 is better than 20 children for one
teacher. Smaller classes seem to improve the number of successfully completed exercises,
and if more research is to be done it is advised to reduce the class size so that children get
the help they need when encountering problems.

6.1.2 Workshop 2

During workshop 2, the children worked with a driving robot. Children working with the
robots are shown on Figure 6.1. This time more attention was paid to provide enough
material so that in case of failing hardware the material could be replaced. In general
children were more motivated during this workshop. This might be because there was less
frustration caused by malfunctioning robots. But also, because the children got to work
6.1 Observations 46

with a driving robot, something that they were very excited about. Some children even
preferred playing with the robot once they programmed it to drive instead of completing
the programming exercises.

Figure 6.1: Picture of a class during workshop 2

However, in half of the groups, all the programming exercises were completed. Half of
the children thus succeeded in programming a robot that tried to evade obstacles. Three
groups of children even received an additional exercise because they finished early. In that
case, they had to program the robot to first drive backwards and then turn, instead of
turning immediately when encountering an obstacle.

Another observation is that the most difficult part for the children was to place the pro-
gramming blocks in the right order and to understand the difference between the setup
and loop part of the Dwenguino start block. They also frequently misplaced the delays
in their program. This finding corresponds with Carver and Klahr’s study [9] where they
mentioned that it is especially difficult for children to find procedural errors (misordered
commands).

There also was a noticeable difference between groups of the fifth or sixth grade due to the
6.2 Emotional Measures 47

age difference. Older children encountered substantially less problems when programming.
This suggests that Dwenguino for Blockly with the simulator can be used by older children
but preferably not by children younger than 10.

Finally, during this workshop, some attention was paid to how the children used the simula-
tor compared to workshop 1. The children seemed to understand the simulator better and
they used it more. But most of them still only used the start and the stop buttons. During
this workshop, some groups made a link between what happened in the real world and
what happened in the simulator. They understood the difference between it as they had
for example a wheel that was not working. To gain more insight in the use of the debugger,
more research should be done where children gain more experience with programming and
with the debugger to find out if the step by step function and the highlighting of the corre-
sponding code indeed can help them when programming. At this point, most of them did
not use this functionality so the conclusions made will mostly be based on the simulation
capacity of the debugger rather than the debugging tools.

6.2 Emotional Measures

This section will give an overview of the results obtained from the first questionnaire
regarding the opinion of the children after workshop 1. First the difference between the
control and the experimental group will be analyzed and afterwards the answers on the
questions about the simulator will be discussed. The correlation between the different test
questions will also be studied.

6.2.1 General results

The first questionnaire contained questions for both the group working with or without the
simulator about the children’s programming affinity, their opinion about the workshop and
their performance during the workshop. The results of this questionnaire for both these
groups are shown in Table 6.1 and Table 6.2.

Some observations can be made based on looking at the results regarding the answers of
both the control and the experimental group. First, children generally felt good and liked
the workshop which was one of the goals when designing the workshop. Although there
6.2 Emotional Measures 48

is some difference in this from class to class. One class was remarkably less motivated
then others, namely 5th grade from school C. 72% of that class gave a score of 3 or less
on question 3, whether they liked the workshop. The reason for this is unknown, because
it wasn’t asked. It can be due to the execution of the workshop but also because of an
external factor. Workshop 2 was not performed in this class so this was not investigated
further. The reason that the workshop was canceled is because of planning issues and not
because of a lack of motivation.

Second, the questions concerning the children’s performance during the workshop (ques-
tions 4, 6 and 7) all had an average score around 2.75. This means that the difficulty level
of the workshop, as perceived by the children, was good: not too easy and not too difficult.
Some children claimed to have no difficulty with identifying mistakes in the problems. This
however does not correspond with the observations made during the workshop. A possible
explanation is that these children overestimated their own performance.

Third, children like using computers and/or tablets. Most of the children participating
in the workshop did not have any previous programming experience but many of them
indicate that they want to program in the future.

In conclusion, although the workshop was relatively difficult, the children had a positive
learning experience. Most of the children were very excited about programming.

6.2.2 Comparison With or Without Debugger

A comparison is now made between the answers to the questionnaire from the control and
the experimental group. The main goal is to provide an answer to research question three:
“Does the debugger change the motivation of the learners?”. This will be done based on
the different answers on questions 2 and 3. But before answering the research question, all
the differences between the two groups will be pointed out and analyzed to see if there are
any significant differences between the two groups.

In the control group, where children did not use the simulator, more children claim to like
computers, they are happier at the moment of the workshop and they liked it more. They
also claim to have more previous programming experience and said they had less problems
with finding mistakes in the programs. But in most cases the difference is rather small.
This can be seen by comparing Table 6.1 and Table 6.2.
6.2 Emotional Measures 49

# question score score score score score average total variance

5 4 3 2 1 answers
1 likes 49 9 7 0 0 4.65 65 0.44
computers
2 feeling during 26 24 14 1 0 4.15 65 0.65
workshop
3 like the 31 19 14 2 0 4.19 66 0.76
workshop
4 difficulty 7 14 20 17 7 2.95 65 1.34
workshop
5 previous 9 8 10 9 29 2.37 65 2.20
programming
6 problems 3 16 15 15 15 2.64 64 1.48
during
workshop
7 easily found 12 10 22 10 12 3.00 66 1.76
mistakes
8 want to 14 11 13 7 21 2.85 66 2.37
program
again

Table 6.1: Answers on questionnaire 1 by control group working without debugger. Number of
children that indicated a score from 1 to 5 for each question

To analyze the significance of the differences between the two groups, a two-sample t-test
is used with significance level α equal to 0.05 and without assuming equal variance. The
null hypothesis is that there is no difference in the scores the children give on a question
between the two groups. The alternative hypothesis is that there is a difference between
the scores. This test is performed for each of the 8 questions.

The result from these tests is that there is a significant difference in mean between the two
groups concerning how much they like to use computers and/or tablets. It is possible that
this influenced the results on the other questions. To look more into this, the different
classes were analyzed individually. As mentioned earlier one class was remarkably less
motivated and this class also scored less on the question regarding how much they like
6.2 Emotional Measures 50

# question score score score score score average total variance

5 4 3 2 1 answers
1 likes 25 15 9 5 1 4.05 55 1.14
computers
2 feeling during 21 16 15 2 0 4.04 54 0.81
workshop
3 liked the 24 10 15 5 0 3.98 54 1.09
workshop
4 difficulty 6 15 13 10 11 2.91 55 1.68
workshop
5 previous 4 3 5 13 28 1.91 53 1.52
programming
6 problems 6 14 15 6 13 2.89 54 1.77
during
workshop
7 easily found 5 9 16 11 13 2.67 54 1.59
mistakes
8 want to 11 5 18 5 14 2.89 53 2.06
program
again
9 simulator 22 8 12 9 3 3.69 54 1.70
helped me
10 used 15 10 10 11 7 3.28 53 1.98
simulator
frequently
11 found 14 9 6 11 14 2.96 54 2.44
mistakes with
simulator

Table 6.2: Answers on questionnaire 1 by experimental group working with debugger. Number
of children that indicated a score from 1 to 5 for each question

computers. When removing this class from the test set, there is no longer a significant
difference between the two groups. As this class did not participate in workshop 2, these
results have no influence on the results regarding the cognitive measures. This however
6.2 Emotional Measures 51

influenced how much the experimental group, as this class used the simulator, liked the
workshop.

Nonetheless, when comparing the difference in how much the children liked the workshop
and how they felt, there is no significant difference between the two groups. This suggests
that the use of the simulator has no influence on the motivation of the learners. It could
have been that adding the simulator to all the new things that had to be learned made the
learning process less fun. It could also be the case that children preferred using the real
robot and that using the simulator was considered a burden. But none of these scenarios
seems to be valid. Another explanation is that the motivation didn’t change because the
children didn’t use the simulator very often. More research should be done to have more
conclusive results.

6.2.3 Children’s Opinion on the Debugger

To provide an answer to research question three: “Do the children think that the debugger
is useful?”, the children had to answer three questions concerning their opinion of the
simulator: whether the simulator helped them, if they used it frequently and if it helped
them find mistakes. The results on the questions concerning the usefulness of the simulator
are summarized at the end of Table 6.2.

In section 6.1.1 it was mentioned that children didn’t really know how to use the debugger.
They didn’t see the highlights and didn’t use the step by step function. It looked like the
simulator didn’t serve its purpose. However, when asking the children about their opinion,
55% of them gave a score of 4 or higher indicating that the simulator helped them. 47%
of the children indicated that they used the simulator frequently and 43% of the children
said they found mistakes in the programs thanks to the simulator.

Thus, the children didn’t use the simulator as it was designed to be used, probably because
it was too difficult as they were only beginners in programming. But their answers to the
questions indicate that the simulator helped them. A possible explanation is that the failing
of the hardware during workshop 1 made them recognize the utility of the simulator. To
find out why the children indicated that the simulator helped them, more research should
be done as children were not interviewed individually.

In conclusion, the results are promising as most children indicated that the simulator
6.3 Cognitive Measures 52

helped them. This indicates that most children found the simulator useful. However, more
advanced research should be done where children spend more time learning to program
and to work with the simulator. This way they can learn how to use all the functions of the
debugger as they become more advanced. And once they know how the use the simulator,
they should be questioned again about their opinion on it.

6.2.4 Correlation Between Test Questions

To end the analysis of the results on the first questionnaire, the correlation between the
different questions is also studied. To analyze this, a Pearson correlation coefficient is
used, with significance level α equal to 0.05. The result is that many test questions are
correlated. A scatter plot showing the relations between the test questions is shown on
Figure 6.2.

There exists a positive correlation between how much the children like computers, how
much they liked the workshop, how good they are feeling, how much they want to program
in the future and how many programming experience they have. There is also a positive
correlation between how much they liked it and how good they were at finding mistakes
in the programs. A negative correlation exists between how much they liked it and how
difficult they found it or how many problems they had. In other words, the more they
liked computers, the more they liked the workshop and the less problems they had.

6.3 Cognitive Measures

This section will give an overview of the results obtained from the second questionnaire
regarding the programming performance of the children before and after workshop 2. First
the two questionnaires will be compared with each other, followed by an analysis of the
general learning progress of the children. Finally, a separation will be made between the
control and experimental group regarding this learning progress to analyze if the use of the
debugger influenced the learning progress obtained for the children.
6.3 Cognitive Measures 53

Figure 6.2: Scatter plot showing the relations between the questions on questionnaire A. A clear
correlation is for example observed between (2) how good they were feeling at the
moment of the workshop and (3) liking the workshop.

6.3.1 General Results

During the second workshop, a pre and post test were performed on the children. The
groups alternately had questionnaire A or B as pre/post test. 6 questions were asked,
6.3 Cognitive Measures 54

each measuring respectively the understanding of a different programming concept: (1)

difference between setup and loop in the Dwenguino start block, (2) the time behavior of
a program and (3) using conditions and if statements, (1) and (2), (1) and (3) and (2) and
(3). Table 6.3 summarizes the results on these questionnaires. The table shows for each
question how many children answered it correctly. It also shows the average test score
obtained per child counting 1 point for each question that is answered correctly.

One immediate observation can be made. All the groups show an increase in test score
after the workshop. To analyze this further, the individual progress of the children was
studied. Table 6.4 shows how many children had a better/worse test score when comparing
the pre and post test. On average 62% children in this study improved their test score and
26% obtained an equal test score.

# simu- test time score score score score score score total ave- var-
lation 1 2 3 4 5 6 an- rage iance
swers score
1 with A before 4 4 0 6 1 9 15 1.6 1.04
1 with B after 1 5 6 7 7 10 15 2.4 1.57
2 without A before 8 7 0 3 1 16 18 1.94 1.05
2 without B after 0 10 7 6 8 10 18 2.28 0.76
3 with B before 1 4 5 4 2 9 18 1.39 0.33
3 with A after 7 13 5 4 5 15 18 2.72 0.26
4 without B before 1 5 11 8 4 13 23 1.83 0.30
4 without A after 10 13 2 7 4 21 23 2.48 0.79
total: 32 61 36 45 32 103 148 2.09 0.26

Table 6.3: Summary of answers on questionnaire 2: total correct answers per question and
average test score obtained per child in each group before and after workshop 2

A second observation can be made, that is that the average test score for the pre test for
the children in the control group is higher compared to the experimental group. However,
after the workshop the average test score for the experimental group is higher.

To analyze this further, a distinction is first made between questionnaire A and B to see
if they are of the same difficulty. Based on this result a statistical test is performed to
analyze the learning progress further and determine its significance.
6.3 Cognitive Measures 55

# better score equal score worse score

1 50% 22% 28%
2 73% 13% 14%
3 60% 30% 10%
4 67% 33% 0%
total: 62% 26% 12%

Table 6.4: Individual progress made during workshop 2 for each test group

6.3.2 Comparison Questionnaires A and B

To measure the programming skills of the learners two questionnaires were used. These
questionnaires were both used as pre and post tests in different groups. Based on the
difference in correct answers between the results on these questionnaires, the progress is
determined. In total, there are 4 test groups, as observed in Table 6.3. Two groups used
test A as pre test and the two other groups used test B. This section will determine if there
is a difference in difficulty level between the two tests. If there is no significant difference,
then the two tests can be seen as one and analyzed together. Otherwise a distinction
should be made between the progress made in the groups depending on which test was
used as pre test.

The average score obtained on test A (both as a pre or post test) is 2.23 while the average
on test B is 1.95. This suggests that there might be a difference in difficulty. All the
different test scores obtained for both the tests are shown on Figure 6.3. To determine
if the difference is significant, a two-sample t-test is used with significance level α equal
to 0.05 and without assuming equal variance. The null hypothesis is that there is no
difference between the difficulty of the two tests. The alternative hypothesis is that there
is a difference. This test was performed two times, once for the scores on the pre test and
once for the scores on the post test.

The obtained α from the tests are respectively 0.52 and 0.32. When performing a test
on both the pre and post test together, the value of α is equal to 0.12. This means that
statistically the difference in mean test score between the two tests is not significant and
will not be considered when analyzing the learning progress.
6.3 Cognitive Measures 56

Figure 6.3: Histogram of test scores obtained per test both including the pre and post tests

6.3.3 Learning Progress

As mentioned earlier most children obtain a better test score on the post test compared to
the pre test. The difference in score between those two tests is what is considered as the
learning progress. This section will determine if there is significant learning progress for
all the children without taking into account the different groups. The average test score
on the pre test for all the groups together is 1.70 and the average test score on the post
test is 2.47.

To determine if the learning progress is significant, a paired t-test is used with significance
level α equal to 0.05. The null hypothesis is that the test score did not change between
the pre and post test. The alternative hypothesis is that the test score changed.

The result is an α coefficient of 1.19e−05 meaning that there is a significant increase in the
test score. This implies that the children improved their programming knowledge during
workshop 2. In general, the tests were rather difficult as the average test score is 2.47
out of 6 for the post test. But nonetheless it allowed for the difference in programming
6.3 Cognitive Measures 57

performance to be measured.

When looking more specifically at the individual scores obtained on the questions, the
specific programming concepts that were learned, can be analyzed. The programming
concepts questioned were: (1) difference between setup and loop in the Dwenguino start
block, (2) the time behavior of a program and (3) using conditions and if statements. For
each of these concepts there were three questions including the concepts. To measure how
well a concept is understood the average for these three questions is taken.

The average test scores per programming concept are shown in Table 6.5. The scores sug-
gests that the children understood the difference between setup and loop in the Dwenguino
start block the least, followed by conditions and if statements and understood the timing
in a program the best (still not very good). The lowest score is for concept three, which
does correspond to the observations made in section 6.1.2 about workshop 2. However,
when looking at the correlation between the test questions measuring the understanding of
a same concept, there is no correlation. This means that if question 1 is answered correctly
this does not give any guarantee about a correct answer on questions 4 or 5.

# programming corresponding average

concept question score
numbers (on 3)
1 difference between 1,4,6 0.89
setup and loop in the
Dwenguino start block
2 time behavior of a 2,4,5 1.64
program
3 using conditions and if 3,5,6
statements

Table 6.5: Average test scores per programming concept on questionnaires A and B

Based on the fact that there is no significant correlation between the test questions, no con-
clusions can be made with regards to which programming concepts the children understood
the best. But when looking at the total test score a clear increase is observed between the
pre and post test. To conclude, the children did not have a very good understanding of
programming at the end of workshop 2 but meaningful progress was made.
6.3 Cognitive Measures 58

6.3.4 Learning Progress With or Without Debugger

To answer research question one: “Do the children demonstrate a more significant learning
progress in programming when using a debugger compared to a control group?”, a com-
parison is made in learning progress between the control and the experimental group. The
learning progress is considered as the difference in test score between the pre and post test.
A scatter plot showing the scores before and after on the tests for each individual in the
control and experimental group is shown on Figure 6.4.

Figure 6.4: Scatter plot showing the scores before and after on the tests for each individual
working with or without the simulator

When looking at Table 6.3, an observation was made about the average test score for the
different groups. In the control group the average test score obtained on the pre test is equal
to 1.88, while in the experimental group this is 1.48. On the post test the average score
obtained by the control group is 2.39 and by the experimental group 2.58. The difference
in test scores is respectively 0.51 and 1.10. In other words, children programming with a
simulator obtained a lower test score on average at first, but had a higher test score on
average after workshop 2.
6.4 Correlation Emotional and Cognitive Measures 59

This result could be explained by the increased difficulty to learn programming for the
children using the simulator. As they need to learn more things, they might focus less on
the programming at first and thus obtain a lower score on the first test. But as seen in this
study, this learning barrier is only of short duration, once the children spent more time
programming, they caught up and even obtained a higher test score on the second test.
This result suggests that children using the simulator are weaker at first but are slightly
better at programming after overcoming the first difficulties.

To determine whether this result is significant, a two-sample t-test is used with significance
level α equal to 0.05 and without assuming equal variance. This test will be performed three
times: to compare the test scores for the pre test, to compare the test scores for the post test
and to compare the learning progress made between the control and experimental group.
The null hypothesis is the test scores obtained are the same. The alternative hypothesis is
that the test scores obtained are different.

For test one, comparing the scores on the pre test, an α value of 0.10 is obtained. For test
two, comparing the scores on the post test, an α value of 0.48 is obtained. For the last test,
comparing the learning progress, an α value of 0.09 is obtained. This means that none of
the differences in the test scores is significant. In this study, no significant difference in
mean test score is obtained between the groups working with or without the simulator.

To conclude, to determine if there is indeed a difference in programming experience gained

when using the simulator both in an early and later stadium, more research should be
done. Based on the results, the only conclusion that can be made is that children using
the debugger gained an equal amount of programming skills compared to the children
programming without simulator.

6.4 Correlation Emotional and Cognitive Measures

To end the analysis of the results, the correlation between the answers on questionnaire 1
and those on questionnaire 2 were analyzed. This is to find out if what the children say
actually corresponds to how good they are. For example, if they say they had very few
problems during the workshop, does this actually mean they they obtain a good score on
the test?
6.5 Overview 60

But because the first questionnaire was chosen to be anonymous, the only comparison
that can be made is between the mean opinions of the class and the mean test scores
obtained. This is very limited. To analyze this, a Pearson correlation coefficient is used,
with significance level α equal to 0.05. The correlation is studied between the average
answer on each question and the average test score obtained for test 1, this for each class.

The result is that there appears to be no significant correlation. This is probably due to
the averaging out in each classroom. As this analysis was rather limited, no conclusions
can be made. To study the relation between what children say on the test (how good
they feel, how much they like it, how good they think they are) and their test score, more
research should be done.

6.5 Overview

To end this chapter, an overview of the obtained results is given. In total 121 children
participated in this study, 55 of whom programmed with the use of the debugger. Although
the workshops were relatively difficult, in general the children liked the introduction to
programming very much.

When comparing the opinion about the workshop between the group using the simulator
and the group not using it, no difference in motivation is found. They on average liked
the workshop equally and found it equally difficult. Thus, the use of the simulator had no
influence on the motivation of the learners.

Children were also asked about their opinion on the debugger. These opinion variate
strongly but most of the children indicated that the simulator helped them. The average
scores for using the simulator frequently and for finding mistakes thanks to the simulator
are around 3 indicating that the opinions are divided. To find out how useful the simu-
lator really is, more research should be done where children learn all the functions of the
debugger. But this results suggests that children find the simulator useful and it can help
them overcoming problems when programming.

As 62% of the children that participated in both workshops showed an increase in test
score and only 12% had a decrease, thus clear learning progress is observed concerning
programming skills. In other words, many children improved upon their programming
6.5 Overview 61

knowledge during workshop 2. When comparing the control and experimental group no
significant difference in obtained test scores is found. This means that children using
the debugger gained an equal amount of programming skills compared to the children
programming without simulator.
CONCLUSION 62

Chapter 7

Conclusion

In this master dissertation, a debugging environment for children ages ten to twelve was
created and evaluated. The debugging environment consisted of a simulator for Dwenguino.
The simulator had the following controls: start, stop, pause and run one step. During
execution, the values of the variables are displayed and the currently executed block is
highlighted. The controls for the debugger were explicitly kept as minimal as possible to
make the debugger easy to use and understand. Aside from a static view of the Dwenguino
board, two use case scenarios were implemented simulating a driving robot on an infinite
plane and a driving robot with a sonar sensor in a room.

The goal of the debugger is to help children when they are programming by providing
new ways to execute and control their program. The debugger can help children in one
of the following ways: help them find mistakes in their programs and correct them, help
them understand a new program, help them make the link between a block of code and
its corresponding execution. The debugger also teaches children how to debug a program,
which is a skill essential to programming.

In literature, there exists no well discussed debugging environment for children in a Scratch-
like VPL. However, a lot of VPLs for children exist. The requirements for the debugger
were inspired based on the analysis of the success of existing programming languages for
children. The requirements were: maintain preference for low floor, maintain the right level
of abstraction, connect with childrens interests or experiences, and visualize a computations
result and its ongoing execution.

Once the debugger was created, it was evaluated to provide an answer to the three main
CONCLUSION 63

research questions: “Do the children demonstrate a more significant learning progress in
programming when using a debugger compared to a control group?”, “Does the debugger
change the motivation of the learners?” and “Do the children think that the debugger is
useful?”. All three questions evaluate whether the debugger reaches its goal or not.

To provide an answer to these questions, multiple workshops were organized to teach chil-
dren programming. In total 121 children participated in this study. The control group,
consisting of 66 children, learned to program without using the debugger while the experi-
mental group used the debugger. Both groups received two identical workshops. Both their
opinion on the workshop and their programming skills were measured. This was done by
means of three questionnaires: one questionnaire asking about their opinion, one pre test
measuring their programming knowledge after workshop 1 and one post test measuring
their programming knowledge after workshop 2.

The first aspect that was studied was the difference in motivation between the two groups.
No difference was observed suggesting that the use of the debugger does not influence the
motivation of the learners. The two groups liked the workshop equally and found it equally
difficult.

The second aspect that was analyzed was the usefulness of the simulator. From the ob-
servations during the workshop it was clear that the simulator was not used a lot and
that only a limited set of functions from the simulator were used (start, stop and pause).
When asking the children about their opinion, the answers were divided but most of them
indicated that the simulator helped them when programming. This last result suggests
that the simulator can be useful when learning to program but more research should be
done.

The last aspect that was studied was the difference in learning progress between the two
groups. When analyzing the test scores of all children, significant learning progress is
observed between the pre and post test. This means that the children gained a substantial
amount of programming knowledge during the workshops. Although the introduction to
programming was more complicated when using the debugger, this did not reflect in the
results obtained on the tests when comparing both groups, as the difference in obtained test
scores was not significant. This means that children using the debugger gained an equal
amount of programming skills compared to the children programming without debugger.

To conclude, the use of the debugger did not show any big disadvantages, children had the
7.1 Further Work 64

same motivation, most of them found the simulator to be helpful and the same level of
programming skills was obtained after the second workshop. To find out if the debugger
can provide significant better results in programming or debugging skills more research
should be done. An overview of possible further work is now given.

7.1 Further Work

The debugger that was created for this study is not complete. It only supports a limited
set of functions. To obtain a more complete debugger breakpoints could be implemented
as a functionality for more advanced or older children. Furthermore, the debugger only
supports two use cases with a driving robot but it cannot model any other situations, for
example if the wheels would be placed differently. If the debugger were to be extended
research should be done into which are the most useful functions of the debugger for the
children. This while guaranteeing that the debugger is still easy to use.

As noted in this study, children only used a limited set of functions from the simulator. For
the children to learn about all the functions, more time should be spent teaching them to
program. Only a limited amount of time was spent in this study with the children. Once
they gain more expertise, more research can be done into the use of the simulator. Do
they start using the step by step function? Does this help them find mistakes? Do they
find the highlighting of the executed block useful? Based on the opinions of the children
once they are more advanced, the usefulness of the debugger can be re-evaluated.

When comparing the obtained scores on the pre and post test, a difference was observed
between the control and experimental group. The control group scored higher on the pre
test while the experimental group scored higher on the post test. These differences were
found to be insignificant. To determine if there is indeed a learning gap at first when
working with the simulator that later disappears, more research should be done. When
comparing the test scores for the post test, a slightly higher score was obtained for the
experimental group. Ideally the use of the debugger would have resulted in a significant
bigger learning progress compared to the control group. This was not the case. Once
children get more advanced in programming their progress can be measured again to make
a new comparison and see if there is a bigger learning progress at a later time due to the
use of the simulator.
7.1 Further Work 65

A last option to explore is to analyze the debugging skills of children when using the simula-
tor. Do they indeed gain useful debugging skills that can later be used when programming
textually? Are they more successful at identifying and correcting bugs compared to the
control group? The road is now open to start more research on children and debugging.
BIBLIOGRAPHY 66

Bibliography

[1] Giovanni Samaey, Jacques Van Remortel, Hugues Bersini, Yvan Bruynseraede, Jan
Dekelver, Frederik De Laender, Dirk Deschoolmeester, Paul Gistelinck, Bern Martens,
Lennart Martens, et al. Informaticawetenschappen in het leerplichtonderwijs. 2014.

[2] Vlaamse Overheid. Actieplan voor het stimuleren van loopbanen in wiskunde, exacte
wetenschappen en techniek. 2012.

[3] Marina Umaschi Bers, Safoura Seddighin, and Amanda Sullivan. Ready for robotics:
Bringing together the t and e of stem in early childhood teacher education. Journal
of Technology and Teacher Education, 21(3):355–377, 2013.

[4] Neil CC Brown, Sue Sentance, Tom Crick, and Simon Humphreys. Restart: The resur-
gence of computer science in uk schools. ACM Transactions on Computing Education
(TOCE), 14(2):9, 2014.

[5] Amanda Sullivan and Marina Umaschi Bers. Robotics in the early childhood class-
room: learning outcomes from an 8-week robotics curriculum in pre-kindergarten
through second grade. International Journal of Technology and Design Education,
26(1):3–20, 2016.

[6] Ronald J Burke and Mary C Mattis. Women and minorities in science, technology,
engineering, and mathematics: Upping the numbers. Edward Elgar Publishing, 2007.

[7] Linda Rae Markert. Gender related to success in science and technology. The Journal
of Technology Studies, 22(2):21–29, 1996.

[8] Marina U Bers. Blocks to robots: Learning with technology in the early childhood
classroom. 2007.

[9] Sharon McCoy Carver and David Klahr. Assessing children’s logo debugging skills
with a formal model. Journal of educational computing research, 2(4):487–525, 1986.
BIBLIOGRAPHY 67

[10] Jeannette M Wing. Computational thinking. Communications of the ACM, 49(3):33–

35, 2006.

[11] Irene Lee, Fred Martin, Jill Denner, Bob Coulter, Walter Allan, Jeri Erickson, Joyce
Malyn-Smith, and Linda Werner. Computational thinking for youth in practice. Acm
Inroads, 2(1):32–37, 2011.

[12] Chris Rogers and Merredith Portsmore. Bringing engineering to elementary school.
Journal of STEM Education: Innovations and Research, 5(3/4):17, 2004.

[13] MU Bers. Blocks, robots and computers: Learning about technology in early child-
hood. Teacher’s College Press, NY, NY, 2008.

[14] Kenneth TH Lee, Amanda Sullivan, and Marina U Bers. Collaboration by design:
Using robotics to foster social interaction in kindergarten. Computers in the Schools,
30(3):271–281, 2013.

[15] Mitchel Resnick. Playful learning and creative societies. Education Update, 8(6):1–2,
2003.

[16] Hiromichi Endoh and Jiro Tanaka. Integrating data/program structure and their
visual expressions in the visual programming system. In Computer Human Interaction,
1998. Proceedings. 3rd Asia Pacific, pages 453–458. IEEE, 1998.

[17] Benedict Du Boulay. Some difficulties of learning to program. Journal of Educational

Computing Research, 2(1):57–73, 1986.

[18] Amjad Altadmri and Neil CC Brown. 37 million compilations: Investigating novice
programming mistakes in large-scale student data. In Proceedings of the 46th ACM
Technical Symposium on Computer Science Education, pages 522–527. ACM, 2015.

[19] IA Neag, DF Tyler, and WS Kurtz. Visual programming versus textual programming
in automatic testing and diagnosis. In AUTOTESTCON Proceedings, 2001. IEEE
Systems Readiness Technology Conference, pages 658–671. IEEE, 2001.

[20] Tracey Booth and Simone Stumpf. End-user experiences of visual and textual pro-
gramming environments for arduino. In International Symposium on End User De-
velopment, pages 25–39. Springer, 2013.
BIBLIOGRAPHY 68

[21] Geoffrey G Roy, Joel Kelso, and Craig Standing. Towards a visual programming en-
vironment for software development. In Software Engineering: Education & Practice,
1998. Proceedings. 1998 International Conference, pages 381–388. IEEE, 1998.

[22] Michael Kölling, Neil CC Brown, and Amjad Altadmri. Frame-based editing: Easing
the transition from blocks to text-based programming. In Proceedings of the Workshop
in Primary and Secondary Computing Education, pages 29–38. ACM, 2015.

[23] Michal Armoni, Orni Meerbaum-Salant, and Mordechai Ben-Ari. From scratch to
”real” programming. ACM Transactions on Computing Education (TOCE), 14(4):25,
2015.

[24] Masahito Hirakawa and Tadao Ichikawa. Advances in visual programming. In Systems
Integration, 1992. ICSI’92., Proceedings of the Second International Conference on,
pages 538–543. IEEE, 1992.

[25] B Kaučič and T Asič. Improving introductory programming with scratch? In MIPRO,
2011 Proceedings of the 34th International Convention, pages 1095–1100. IEEE, 2011.

[26] Chance Elliott, Vipin Vijayakumar, Wesley Zink, and Richard Hansen. National in-
struments labview: a programming environment for laboratory automation and mea-
surement. JALA: Journal of the Association for Laboratory Automation, 12(1):17–24,
2007.

[27] Seymour Papert. Mindstorms: Children, computers, and powerful ideas. Basic Books,
Inc., 1980.

[28] Brian Harvey. Logo. 2003.

[29] Karen Brennan. Learning computing through creating and connecting. Computer,
46(9):52–59, 2013.

[30] Mitchel Resnick, John Maloney, Andrés Monroy-Hernández, Natalie Rusk, Evelyn
Eastmond, Karen Brennan, Amon Millner, Eric Rosenbaum, Jay Silver, Brian Silver-
man, et al. Scratch: programming for all. Communications of the ACM, 52(11):60–67,
2009.

[31] MIT Logo Foundation. A logo primer. http://el.media.mit.edu/

logo-foundation/what_is_logo/logo_primer.html, 2015.
BIBLIOGRAPHY 69

[32] Amanda Sullivan, Mollie Elkin, and Marina Umaschi Bers. Kibo robot demo: En-
gaging young children in programming and engineering. In Proceedings of the 14th
international conference on interaction design and children, pages 418–421. ACM,
2015.

[33] Frank Klassner and Scott D Anderson. Lego mindstorms: Not just for k-12 anymore.
IEEE Robotics & Automation Magazine, 10(2):12–18, 2003.

[34] Sayamindu Dasgupta, Shane M Clements, Abdulrahman Y Idlbi, Chris Willis-Ford,

and Mitchel Resnick. Extending scratch: New pathways into programming. In Visual
Languages and Human-Centric Computing (VL/HCC), 2015 IEEE Symposium on,
pages 165–169. IEEE, 2015.

[35] Scratch for Arduino. About s4a. http://s4a.cat/, 2015.

[36] Michael Kölling. The greenfoot programming environment. ACM Transactions on

Computing Education (TOCE), 10(4):14, 2010.

[37] Ian Utting, Stephen Cooper, Michael Kölling, John Maloney, and Mitchel Resnick. Al-
ice, greenfoot, and scratch–a discussion. ACM Transactions on Computing Education
(TOCE), 10(4):17, 2010.

[38] Caitlin Kelleher and Randy Pausch. Lowering the barriers to programming: A tax-
onomy of programming environments and languages for novice programmers. ACM
Computing Surveys (CSUR), 37(2):83–137, 2005.

[39] Stavroula Georgantaki and Symeon Retalis. Using educational tools for teaching
object oriented design and programming. Journal of Information Technology Impact,
7(2):111–130, 2007.

[40] Stephen Cooper, Wanda Dann, and Randy Pausch. Alice: a 3-d tool for introductory
programming concepts. In Journal of Computing Sciences in Colleges, volume 15,
pages 107–116. Consortium for Computing Sciences in Colleges, 2000.

[41] Mitchel Resnick and Brian Silverman. Some reflections on designing construction kits
for kids. In Proceedings of the 2005 conference on Interaction design and children,
pages 117–122. ACM, 2005.

[42] Steve Cooper and Steve Cunningham. Teaching computer science in context. Acm
Inroads, 1(1):5–8, 2010.
BIBLIOGRAPHY 70

[43] Andrew J Ko and Brad A Myers. A framework and methodology for studying the
causes of software errors in programming systems. Journal of Visual Languages &
Computing, 16(1):41–84, 2005.

[44] DN Perkins and Fay Martin. Fragile knowledge and neglected strategies in novice
programmers. In first workshop on empirical studies of programmers on Empirical
studies of programmers, pages 213–229, 1986.

[45] Irvin R Katz and John R Anderson. Debugging: An analysis of bug-location strategies.
Human-Computer Interaction, 3(4):351–399, 1987.

[46] Marzieh Ahmadzadeh, Dave Elliman, and Colin Higgins. An analysis of patterns of
debugging among novice computer science students. ACM SIGCSE Bulletin, 37(3):84–
88, 2005.

[47] Sue Fitzgerald, Gary Lewandowski, Renee McCauley, Laurie Murphy, Beth Simon,
Lynda Thomas, and Carol Zander. Debugging: finding, fixing and flailing, a multi-
institutional study of novice debuggers. Computer Science Education, 18(2):93–116,
2008.

[48] Karen Brennan and Mitchel Resnick. New frameworks for studying and assessing the
development of computational thinking. In Proceedings of the 2012 annual meeting
of the American Educational Research Association, Vancouver, Canada, pages 1–25,
2012.

[49] Nussarin Nusen and Arnan Sipitakiat. Robo-blocks: a tangible programming system
with debugging for children. In Proceedings of the 19th international conference on
computers in education. Chiang Mai, pages 1–5, 2011.

[50] Yoshiharu Kato. Splish: a visual programming environment for arduino to accelerate
physical computing experiences. In Creating Connecting and Collaborating through
Computing (C5), 2010 Eighth International Conference on, pages 3–10. IEEE, 2010.

[51] Louis Major, Theocharis Kyriacou, and O Pearl Brereton. Systematic literature re-
view: Teaching novices programming using robots. IET software, 6(6):502–513, 2012.

[52] Amanda Wilson and David C Moffat. Evaluating scratch to introduce younger
schoolchildren to programming. Proceedings of the 22nd Annual Psychology of Pro-
gramming Interest Group (Universidad Carlos III de Madrid, Leganés, Spain, 2010.
BIBLIOGRAPHY 71

[53] Francis Wyffels, Willem Van de Steene, Jelle Roets, Maria-Cristina Ciocci, Juan Pablo
Carbajal, et al. Building artbots to attract students into stem learning. arXiv preprint
arXiv:1608.03405, 2016.

[54] Dwengo vzw. Dwenguino. http://www.dwengo.org/tutorials/dwenguino, 2012.

[55] Cesar Vandevelde, Jelle Saldien, Cristina Ciocci, and Bram Vanderborght. Overview
of technologies for building robots in the classroom. In International Conference on
Robotics in Education, pages 122–130, 2013.

[56] amasad. Debugjs. https://github.com/amasad/debugjs, 2016.

APPENDIX 72

Appendix

Figure A.1: Program used during workshop 2 to let a robot drive and turn when encountering
an obstacle - part 1
APPENDIX 73

Figure A.2: Program used during workshop 2 to let a robot drive and turn when encountering
an obstacle - part 2
APPENDIX 74

Figure A.3: Questionnaire A used during workshop 2 to evaluate programming knowledge of

the children - part 1
APPENDIX 75

Figure A.4: Questionnaire A used during workshop 2 to evaluate programming knowledge of

the children - part 2
APPENDIX 76

Figure A.5: Questionnaire A used during workshop 2 to evaluate programming knowledge of

the children - part 3
APPENDIX 77

Figure A.6: Questionnaire B used during workshop 2 to evaluate programming knowledge of

the children - part 1
APPENDIX 78

Figure A.7: Questionnaire B used during workshop 2 to evaluate programming knowledge of

the children - part 2
APPENDIX 79

Figure A.8: Questionnaire B used during workshop 2 to evaluate programming knowledge of

the children - part 3