Tugas Review Artikel Kelompok 1

REVIEW ARTIKEL INTERNASIONAL
Sebagai Salah Satu Persyaratan untuk Memenuhi Tugas

Karya Tulis Ilmiah Pendidikan Matematika
Dosen Pengampuh : Dr. Hasan S. Panigoro, S.Pd., M.Si
Oleh :
Mamat Abas (703522001)

Nurjanah Namko Ladjali (703522002)
Iken Meyti Katili (703522003)
PROGRAM PASCASARJANA
PROGRAM STUDI PENDIDIKAN MATEMATIKA
UNIVERSITAS NEGERI GORONTALO
2022
Ruang Lingkup Masalah
No Judul/Tahun/Penulis Justifikasi/Novelty Metode Penelitian Hasil Penelitian Ket.
/ Tujuan
1 Author : Lingkup Masalah : Justifikasi atau Metode Penelitian yang 1. Berdasarkan hasil analisis
1. Muhammed Paf 1. Perkembangan Novelty yang digunakan merupakan statistik deskriptif diperoleh
2. Beste Dincer teknologi informasi di ditawarkan dalam metode kuantitatif bahwa rata-rata kemampuan
abad 21 memberikan penelitian ini dengan desain penelitian CT untuk siswa perempuan
Tahun : pengaruh terhadap menemukan korelasional lebih tinggi dibanding siswa
Oktober 2021 Vol. 20 pendidikan dan hubungan yang laki-laki
Issue 4 kemampuan setiap menguntungkan Sampel yang digunakan
individu, tidak hanya atau positif antara 1098 siswa yang dipilih 2. Berdasarkan hasil analisis
Jurnal : itu abad 21 menjadi CT dan CPS. Untuk secara proportional statistik deskriptif diperoleh
TOJET (The Turkish tantangan sendiri bagi meningkatkan CT cluster sampling pada bahwa rata-rata keterampilan
Online Journal of setiap individu sehingga CPS ikut desa germencik prov. CPS untuk siswa perempuan
Educational Technology) meningkat, perlu Aydyn tahun ajaran lebih tinggi dibanding siswa
2. Salah satu tantangan dilakukan treatmet 2018/2019 laki-laki
Judul : diabad 21 yaitu melalui penggunaan
A Study of the bagaimana individu model/metode Intrumen yang 3. Berdasarkan hasil analisis data
Relationship between dapat menyelesaikan pembelajaran yang digunkanan merupakan menggunakan korelasi pearson
Secondary School permasalahan yang tepat. angket dengan indikator momen diperoleh bahwa
Students Computational dipeoleh dalam CT dan CPS mengacu kemampuan berpikir
Thinking Skills and kehidupan sehari-hari peneliti terdahulu komputasional (CT) memiliki
Creative Problem-Solving melalui kemampuan Analisis data hubungan positif dengan
Skills. dan keterampilan yang menggunakan analisis keterampilan pemecahan
dimiliki statistik deskriptif untuk masalah secara kreatif (CPS)
(Hubungan Antara menggambarkan tingkat hal ini bisa dilihat dari
Kemampuan Berpikir 3. CT dan CPS CT dan CPS serta koefisien korelasi yang
Komputasional dan merupakan suatu korelasi pearson momen diperoleh r = 0.636. hal ini
Keterampilan kemampuan dan menunjukkan bahwa semakin
Penyelesaian Masalah keterampilan yang tinggi kemampuan CT maka
Secara Kreatif Pada Siswa dinilai tepat harus semakin tinggi juga
Sekolah Menengah) dimiliki oleh setiap keterampilan CPS Siswa
individu, karena
diharapkan dapat
membantu menemukan
solusi untuk suatu
permasalah dalam
kehidupan sehari-hari
4. Proses pembeljaran
berperan penting dalam
meningkatkan CT dan
CPS
Tujuan :
Mengetahui hubungan CT
dan CPS pada siswa
sekolah menengah.
2 Author : Lingkup Masalah : Justifikasi atau Metode penelitian yang 1. Efisiensi aktifitas pembelajaran
1. Jirattikorn Siphai 1. Matematika memegang novelty yang digunakan merupakan matematika dengan
2. Kanyarat Cojorn peranan penting dalam ditawarkan dalam metode kuantitatif menggunakan proses inkuiri
kehidupan karena penelitian ini yaitu dengan desain penelitian dan CPS untuk meningkatkan
Tahun : membantu dalam potensi siswa dalam pra eksperimen one CT siswa pada matakuliah
Oktober 2022 Vol. 02 No. mengembangkan ide- memecahkan group posttest only statistika diperoleh 76.77/77.45
05 Hal. 39-46 ide kreatif dan berpikir permasalahan design lebih tinggi dari kriteria 70/70
rasional. matematika secara yang telah ditetapkan.
Jurnal : efektif dapat Instrumen yang 2. Kemampuan CT siswa yang
International Journal of 2. Sebagian besar masalah diterapkan digunakan merupakan belajar melalui kegiatan
Sociologies and matematika dalam keterampilan test untuk mengukur pembelajaran matematika
Anthropologies Reviews kehidupan nyata berpikir kreatif, kemampuan berpikir dengan menggunakan proses
bersifat kompleks. dengan terbiasanya komputasional dan inkuiri dengan CPS lebih
Judul : siswa memecahkan keterampilan berpikir tinggi dari kriteria 70%
Development of 3. CT sangat penting masalah maka kreatif melalui tes hasil 3. Capaian pembelajaran siswa
Mathematics Learning untuk pelajar saat ini dapat meningkatkan pembelajaran yang melalui kegiatan pembelajaran
Activities by Using kemampuan tentunya sudah dengan menggunakan proses
Inquiry Process with 4. Pembelajaran komputasional dilakukan uji validitas inkuiri dengan CPS lebih
Creative Problem-Solving menggunakan proses dan reliabilitas. tinggi darikriteria 70%
(CPS) to Promote inkuiri dengan CPS
Computational Thinking dapat digunakan untuk
Skills in Matthayomsuksa meningkatkan
2 Students kemampuan CT
Tujuan :
(Pengembangan Aktivitas Mengembangkan
Pembelajaran Matematika kegiatan pembelajaran
melalui Proses Inquiry matematika
dengan Keterampilan menggunakan proses
Berpikir Kreatif untuk inquiri dengan CPS
Meningkatkan untuk meningkatkan
Kemampuan Berpikir kemampuan CT
Komputasional Pada
Siswa Matthayomsuksa 2)
3 Author : Lingkup Masalah : Justifikasi atau Metode yang digunakan Berdasarkan hasil analisis data
1. Kay-Dennis Boom 1. CT dalam beberapa novelty yang dalam penelitian menggunakan analisis regresi dan
2. Matt Bower penelitian berdasarkan ditawarkan dalam merupakan metode korelasi pearson momen
3. Jens Siemon pemecahan masalah penelitian ini kuantitatif dengan diperoleh bahwa kemampuan
4. Amael Arguel dapat dibagi dua yaitu menemukan desain penelitian berpikir komputasional (CT)
pemecahan masalah hubungan yang korelasional memiliki hubungan positif
Tahun : komputasi positif antara dengan kualitas program
Januari 2022 Vol. 27 Hal. (Pemrograman kemampuan komputer, hal ini menunjukkan
8289-8310 Komputer) dan berpikir bahwa semakin tinggi
pemecahan beragam komputasional kemampuan (CT) siswa maka
Jurnal : masalah yang non- dengan kualitas kualitas program komputer dapat
Education and pemrograman program komputer. dikatakan memiliki kualitas
Information Technologies tinggi.
by Springer 2. Belajar pemrograman
komputer bergantung
Judul : pada orang yang
Relationships between memanfaatkan dan
computational thinking menerapkan pemikiran
and the quality of komputasi
computer programs
3. CT sangat penting
(Hubungan Antara diterapkan dalam
Kemampuan Berpikir berbagai disiplin ilmu
Komputasional dan dan merupakan dasar
Kualitas Program dri pemrograman
Komputer) komputer yang sukses
Tujuan :
untuk mengevaluasi
sejauh mana proses
berpikir komputasional
secara umum serta
berpikir komputasional
terapan dalam pemecahan
masalah.
4 Author : Lingkup Masalah : Sepanjang modul peneliti membagi Setelah total enam jam interaksi
1. Kenia Wiedemann 1. Modul ini terdiri dari instruksional aktivitas penelitian dengan tutorial, termasuk
2. Jie Chao beberapa tutorial CodeR-4MATH, menjadi dua masalah penilaian, sepertiga dari siswa
3. Benjamin Galluzzo mandiri yang siswa belajar dunia nyata yang diberi yang berpartisipasi dalam
4. Eric Simoneau menantang siswa untuk bagaimana nama Meal Plan vs. Pay- implementasi menyatakan bahwa
membuat model solusi mengatasi masalah As-You-Go (M.P. vs. mereka tertarik untuk mengambil
Tahun : untuk masalah praktis dunia nyata PAYG) dan Driving for kelas pemrograman di masa
Maret 2020 seperti biaya makan di menggunakan Gas (DFG), keduanya depan (2 siswa, atau 5%,
Vol. 11, No. 1 perguruan tinggi, biaya pemodelan merupakan bagian dari mengatakan mereka sudah
sebenarnya untuk matematika, dengan modul yang disebut tertarik untuk mengeksplorasi
Jurnal : memiliki mobil, dll. bahasa Lifehacking. Siswa kemungkinan ini di masa depan).
acm Inroads Dengan kata lain, pemrograman (R) mengerjakan modul ini peneliti juga mengamati bahwa
modul ini dirancang sebagai lingkungan dalam total 6 jam untuk siswa lainnya, yaitu mereka
Judul : untuk membawa pemodelan. Selama selama dua setengah yang tidak peduli dengan
Mathematical Modeling masalah terbuka yang proses tersebut, minggu. pemrograman dan yang
with R: Embedding berhubungan dengan siswa secara alami mengatakan minat mereka tidak
Computational Thinking kehidupan nyata. melatih berubah, alasan utama yang kami
into High School Math keterampilan identifikasi adalah konsep diri.
Classes 2. Modul CodeR-4MATH berpikir 21 dari 26 siswa yang
(Pemodelan Matematika dirancang untuk komputasional mengatakan bahwa mereka masih
dengan R: Menanamkan memungkinkan siswa sambil mempelajari (atau kurang) tertarik pada
Pemikiran Komputasi ke belajar bagaimana konsep dasar pemrograman, membenarkan
dalam Kelas Matematika mengatasi masalah pemrograman sentimen mereka karena mereka
di SMA) dunia nyata komputer percaya bahwa beberapa orang
menggunakan secara alami pandai matematika
pemodelan matematika, dan yang lainnya tidak.
sambil memiliki bahasa
pemrograman sebagai
lingkungan pemodelan.
Tujuan :
Merancang dan menguji
modul pembelajaran yang
dapat membantu guru
matematika sekolah
menengah untuk
menanamkan pemikiran
komputasi ke dalam kelas
mereka.
5 Author : Lingkup Masalah : penelitian ini Pretest dan posttest Analisis hasil pretest dan posttest
1. Shiau-Wei Chan 106 Siswa Sekolah menunjukkan dirancang dan dibangun menunjukkan bahwa kinerja
2. Chee-Kit Looi Menengah Pertama (usia kemungkinan untuk berdasarkan topik kelompok eksperimen sama
3. Weng Kin Ho 13 tahun) dari sekolah membawa CT ke Number Patterns dalam dengan kelompok kontrol.
4. Wendy Huang menengah di Singapura mata pelajaran silabus Matematika Temuan ini tidak mendukung
5. Peter Seow ikut serta dalam penelitian sekolah yang ada. Menengah Singapura. hipotesis bahwa
6. Longkai Wu ini. Sebuah kuasi- Pengembangan Penelitian ini mengintegrasikan CT dalam
eksperimental non- instruksi dan menggunakan waktu pelajaran dapat menghasilkan
Tahun : ekuivalen desain penilaian CT lima hari untuk peningkatan pembelajaran
Agustus 2021, Vol. 07 kelompok digunakan di berfungsi sebagai melengkapi matematika. Namun, peningkatan
Hal. 2405-8440 mana 70 siswa ditugaskan sumber daya bagi pengumpulan data yang drastis diamati pada siswa
ke dalam kelompok guru sekolah, terungkap. Hari pertama individu dari kelompok
Jurnal : eksperimen, dan 36 siswa pengembang dan hari terakhir eksperimen, sementara tidak ada
Heliyon by Elsevier ditugaskan ke dalam kurikulum, peneliti, digunakan untuk peningkatan yang jelas atau
kelompok kontrol. administrator, dan pengujian, sementara ekstrim untuk siswa dari
Judul : Kelompok eksperimen pembuat kebijakan. tiga hari lainnya kelompok kontrol. Studi ini
Learning Number diberi intervensi dengan Sumber daya ini digunakan untuk memberikan beberapa bukti
Patterns Through aktivitas yang diinfuskan memberi mereka pengajaran. empiris baru dan kontribusi
Computational Thingking CT baik di dalam maupun seperangkat praktik Pengumpulan data praktis untuk infus praktik CT di
Activities : A Rasch Model di luar komputer, yang lebih jelas dan dilaksanakan selama kelas matematika.
Analisis sedangkan kelompok konkret untuk periode matematika di
kontrol tidak menerima memandu sekolah. Pada awal
(Mempelajari Pola intervensi semacam itu. pengembangan penelitian, peneliti
Bilangan Melalui kurikulum dan memberikan pretest
Aktivitas Berpikir Tujuan : penerapan konsep selama 15 menit kepada
Komputasi : Analisis untuk mengetahui CT di kelas siswa kelompok
model Rasch pengaruh pembelajaran eksperimen dan
Matematika + C terhadap kelompok kontrol. dan
prestasi belajar siswa pada Setelah instruksi, kedua
tipikal soal tes mata kelompok diberikan
pelajaran matematika posttest 15 menit.
SMP topik pola bilangan. Data yang dikumpulkan
dianalisis menggunakan
versi kredit parsial dari
model Rasch.
6 Author : Lingkup Masalah : penelitian ini Deskripsi kuantitatif : “Tanpa keraguan abstraksi”
1. Ibrahim Cetin Ruang lingkup pada membahas : mendeskripsikan dan adalah gagasan mendasar dari
2. Ed Dubinsky penelitian ini yaitu tentang 1. perbedaan mengaitkan abstraksi pemikiran komputasi. Peneliti
keterkaitan refleksi pandangan reflektif dengan teori telah menggunakan abstraksi
Tahun : abstraktif dengan terkait abstraksi APOS sebagai teori reflektif untuk menggambarkan
Juli 2017 kemampuan komputasi dan bagaimana pembelajaran abstraksi dalam pemikiran
Vol. 47 Hal.70-80 menggunakan teori APOS Piaget matematika untuk komputasi. Dengan menggunakan
(Action, proses, objek, mengkonseptuali mempelajaran konsep dekomposisi genetik,
Jurnal : Skema) dalam sasikan abstraksi pemahaman siswa peneliti telah memulai penerapan
Journal of Mathematical pembelajaran matematika reflektif dalam pemikiran Teori APOS untuk mempelajari
Behavior by Elsevier 2. Teori APOS komputasi pemahaman siswa tentang
Tujuan : sebagai teori pemikiran komputasional.
Judul : membangun jembatan pembelajaran Penerapan gagasan abstraksi
Reflective Abstraction in teoretis antara pemikiran matematika, reflektif, dan karenanya Teori
Computational Thinking komputasi dan Teori pedagogi APOS, dalam pemikiran
APOS dan menunjukkan pendukungnya komputasional akan memiliki
(Abstraksi Reflektif dalam bahwa abstraksi reflektif dan peran konsekuensi baik untuk Teori
Pemikiran Komputasi) dapat digunakan dalam pemikiran APOS maupun pemikiran
konteks pemikiran komputasional komputasional. Teori APOS
komputasi. dalam pedagogi dibangun dalam konteks
3. sifat abstraksi pendidikan matematika untuk
dan peran menguji mekanisme dan struktur
abstraksi reflektif mental yang dikembangkan untuk
dalam pemikiran mempelajari mata pelajaran
komputasi matematika. Memperluas dan
menerapkan teori untuk
pendidikan ilmu komputer secara
signifikan memperkuat teori
sebagai teori umum pembelajaran
7 Author : Lingkup Masalah : Dalam penelitian (menggunakan desain Penelitian ini memberikan bukti
1. Anaclara Gerosa 102 anak (laki-laki = 52; ini, peneliti korelasional cross- tentang hubungan antara
2. Victor Koleszar usia rata-rata keseluruhan membandingkan sectional. peneliti kuesioner CT yang ditetapkan
3. Gonzalo Tereja = 5,6 tahun) menghadiri penilaian pemikiran menilai anak-anak sebelumnya dan berbagai
4. Leonel Gomez-Sena tingkat 5 (TK) di sekolah komputasi yang selama tiga minggu kemampuan kognitif pada anak
5. Alejandra umum di Montevideo. dilaporkan (setengah sampel dinilai TK. Hasil menunjukkan
Tingkat sosiokultural sebelumnya dengan pada Oktober 2018, pemikiran komputasi pada anak
Tahun : untuk sampel dicirikan baterai sembilan tes sedangkan sisanya usia dini sebagian besar
Agustus 2021 Vol. 2 Hal. sebagai menengah ke atas kognitif, yang dinilai pada Maret dijelaskan oleh keterampilan
2666-5573 menurut administrasi meliputi kecerdasan 2019). Anak-anak pengurutan temporal peserta dan
Jurnal : Computers and pendidikan publik cair, memori kerja, berpartisipasi dalam tiga kemampuan numerik mengenai
Education Open by Uruguay. Kriteria inklusi perencanaan, sesi evaluasi hingga 25 pemahaman mereka tentang
Elsevier terdiri dari anak usia 4-6 pengurutan, rotasi menit untuk mencegah besaran simbolis, yang
tahun dengan mental, kosa kata, kelelahan atau selanjutnya membangun gagasan
Judul : perkembangan yang khas. dan prekursor kebosanan. Semua anak yang ada dari bukti sebelumnya
Cognitive abilities and 1 anak dikeluarkan dari matematika awal dinilai di pagi hari, mengenai asosiasi CT dengan
computational thinking at sampel kami karena seperti transcoding antara jam 9 dan 11 penalaran dan keterampilan
age 5: Evidence for memiliki gangguan numerik dan pagi. Tiga peneliti matematika awal. Selain itu,
associations to sequencing perkembangan yang perbandingan berpartisipasi dalam peneliti membandingkan hasil
and symbolic number didiagnosis. Informed besaran simbolik. pengumpulan data. yang didapat dengan kinerja
comparison consent diperoleh dari Regresi linier efek Pengukuran berbasis anak-anak selama intervensi
(Kemampuan Kognitif orang tua/pengasuh, dan campuran tablet diterapkan secara robotika pendidikan
dan Pemikiran komputasi penelitian ini disetujui diimplementasikan bersamaan dalam menggunakan robot yang dapat
pada usia 5 tahun : bukti oleh Komite Etik dengan CT sebagai kelompok, mengikuti diprogram melalui objek nyata,
Untuk Asosiasi Penelitian Sekolah variabel dependen rasio 4:1 antara anak- menemukan korelasi positif yang
Pengurutan dan Psikologi di Universidad untuk anak dan orang dewasa. signifikan antara kinerja ER dan
Perbandingan angka de la República, Uruguay mengeksplorasi Penilaian berbasis kertas CT anak-anak.
simbolis) hubungan antara diterapkan secara
Tujuan : variabel yang individual dalam rasio
menjembatani digunakan 1:1
kesenjangan yang ada
dengan mengeksplorasi
hubungan antara CT dan
berbagai keterampilan
kognitif dalam sampel
anak-anak taman kanak-
kanak
8 Author : Lingkup Masalah : Penelitian ini Penelitian ini Hipotesis nol (H01) menyatakan
1. Samri Chongo Penelitian ini merupakan memperkenalkan menggunakan bahwa tidak ada hubungan antara
2. Kamisah Osman penelitian yang pendekatan baru pendekatan kuantitatif keterampilan CT dan prestasi
3. Nazrul Anuar Nayan menggunakan pendekatan yakni STEM untuk dengan melakukan matematika.
Tahun : kuantitatif dengan mencari diterapkan dalam survei untuk Pengujian hipotesis nol kedua
2020, Vol & No : 31(2) hubungan yang bersifat kegiatan mengevaluasi tingkat (Ho2 ) menunjukkan tidak ada
Hal : 159-163 kausal. Computational pembelajaran keterampilan CT, perbedaan yang signifikan antara
thinking (CT) sering matematika yang hubungannya dengan laki-laki dan perempuan dalam
Jurnal : dikaitkan dengan ilmu mana dalam prestasi belajar hal keterampilan CT (t = 0,055;
Science Education komputer dan matematika meningkatkan matematika (PT3), dan df = 126, >.05)
International pada umumnya. Namun, keterampilan perbedaan keterampilan
keterlibatan ilmu Computational CT berdasarkan gender Hasil analisis uji-t ditunjukkan
Judul : komputer menurut gender Thingking. pada Tabel 6, dan hubungan
Level of Computational terlihat tidak seimbang. H01 : Tidak ada antara keterampilan CT gender
Thinking Skills among hubungan yang lemah. Hasil ini menunjukkan
Secondary Science Tujuan : signifikan antara bahwa jenis kelamin tidak
Student: Variation across mengidentifikasi tingkat keterampilan CT dengan memainkan peran penting dalam
Gender and Mathematics CT, hubungan antara prestasi belajar keterampilan CT siswa. Hasil ini
Achievement keterampilan CT dan matematika. sependapat dengan Korkmaz dan
prestasi matematika, dan Oluk (2016) yang sebelumnya
(Tingkat Keterampilan perbedaan keterampilan H02 : Tidak ada melaporkan tidak ada perbedaan
Berpikir Komputasi di CT berdasarkan jenis perbedaan yang yang signifikan dalam
kalangan Menengah Siswa kelamin. signifikan keterampilan keterampilan CT.
Sains: Variasi Gender dan CT berdasarkan jenis
Matematika Pencapaian) kelamin Temuan ini menunjukkan bahwa
faktor gender tidak
mempengaruhi keterampilan CT,
sedangkan prestasi matematika
terkait dengan keterampilan CT
9 Author : Lingkup Masalah : Penelitian ini Penelitian ini Temuan telah mengungkapkan
1. Ayşe Çiftçi Agar pemikiran memperkenalkan menggunakan desain bahwa ada perbedaan yang
2. Mustafa Sami Topçu komputasional (CT) pendekatan baru kuasieksperimental, signifikan dalam skor post-test
berhasil yakni STEM untuk yaitu kelompok kontrol pada 'skala keyakinan selfefficacy
Tahun : diimplementasikan dalam diterapkan dalam hanya diberikan terhadap pengajaran CT'
2022 lingkungan pendidikan, kegiatan pendidikan STEM, mendukung kelompok
keyakinan selfefficacy pembelajaran sedangkan kelompok eksperimen. Ukuran efek yang
Jurnal : pengajaran CT guru harus matematika yang eksperimen diberikan dihitung untuk perbedaan antara
Research in Science & dikembangkan. Penelitian mana dalam pendidikan STEM nilai rata-rata pre-test dan post-
Technological Education yang ada pada topik ini meningkatkan terintegrasi C. Alat test dari siswa kelompok
umumnya difokuskan keterampilan pengumpulan data yang eksperimen ternyata besar. Hasil
pada pelatihan guru in- Computational digunakan dalam ini menunjukkan bahwa
Judul : service di CT untuk Thingking. penelitian ini adalah pendidikan STEM yang
Improving early mengembangkan self- 'skala keyakinan efikasi terintegrasi dengan CT
childhood pre-service efficacy mereka. Selain itu juga diri terhadap pengajaran berkontribusi pada
teachers’ computational Sebaliknya, penelitian saat penelitian ini pemikiran pengembangan keyakinan
thinking teaching self- ini bertujuan untuk difokuskan pada komputasional. Untuk selfefficacy pengajaran CT guru
efficacy beliefs in a STEM berkontribusi pada guru prajabatan analisis, digunakan uji t prajabatan
course pengembangan keyakinan PAUD yang pada sampel independen dan
self-efficacy pengajaran penelitian uji t sampel
(Meningkatkan pemikiran CT guru pra-layanan anak sebelumnya lebih berpasangan.
komputasi guru pra- usia dini melalui kepada guru
jabatan anak usia dini pendidikan STEM inservice.
yang mengajarkan terintegrasi CT. Studi saat
keyakinan efikasi diri ini menyelidiki efektivitas
dalam kursus STEM) pendidikan STEM dan
pendidikan STEM
terintegrasi CT dalam
mengembangkan
keyakinan self efficacy
pengajaran guru
pralayanan anak usia dini
Tujuan :
Dalam penelitian ini,
pengaruh pendidikan
STEM terintegrasi CT
pada keyakinan
selfefficacy pengajaran
CT dari guru pra-layanan
anak usia dini
dieksplorasi.
10 Author : Lingkup Masalah : Penelitian ini Model yang diusulkan Hasil ini menunjukkan bahwa
1. Chien Hsiang Liao Sebuah model konseptual memperkenalkan dengan lima variabel baik pemikiran komputasional
2. Chang-Tang Chiang diusulkan untuk menguji atau menegaskan dan enam hipotesis maupun kesenangan memainkan
3. I-Chuan Chen hubungan struktural antara bahwa mahasiswa dinilai dengan peran penting dalam eksplorasi
4. Kevin R. Parker pemikiran komputasi, non-STEM dapat menggunakan survei diri dan efikasi diri digital. Selain
kemanjuran diri, belajar berpikir kuantitatif. Semua itu, efikasi diri digital dan
eksplorasi diri, komputasi secara konstruksi diukur eksplorasi diri juga memiliki
Tahun : kenikmatan dan kepuasan efektif dengan dengan beberapa item, pengaruh positif yang signifikan
2022 (19:43) belajar dalam kursus dukungan dan skala tipe Likert terhadap kepuasan belajar.
pendidikan liberal perangkat tujuh poin yang berkisar Temuan ini memiliki implikasi
Jurnal : berbasis AppInventor. pembelajaran yang dari "sangat setuju" untuk mempengaruhi hasil belajar
International Journal of tepat. Zaman hingga "sangat tidak mahasiswa non-STEM-utama,
Educational Technology Tujuan : sekarang yang setuju" digunakan berpikir komputasi instruktur
in Higher Education Mengeksplorasi hubungan semakin digital kursus berpikir nasional, dan
antara pemikiran membuat pekerja kebijakan yang relevan berpikir
Judul : komputasi dan kepuasan pengetahuan komputasi
Exploring the relationship belajar untuk mahasiswa nonSTEM akan
between computational non-STEM-jurusan diminta untuk
thinking and learning berinteraksi dengan
satisfaction for non-STEM profesional
college students terhadap teknologi
informasi di
(Menjelajahi hubungan berbagai domain.
antara pemikiran
komputasi dan kepuasan
belajar untuk mahasiswa
non-STEM)
See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/355219924
A Study of the Relationship between Secondary School Students'

Computational Thinking Skills and Creative Problem-Solving Skills
Article in Turkish Online Journal of Educational Technology · October 2021
CITATIONS READS
2 223
2 authors:
Muhammed Paf Beste Dinçer

Education Aydın Adnan Menderes University
3 PUBLICATIONS 6 CITATIONS 20 PUBLICATIONS 83 CITATIONS
SEE PROFILE SEE PROFILE
Some of the authors of this publication are also working on these related projects:
Computational thinking and creative problem solving skills View project
All content following this page was uploaded by Muhammed Paf on 15 October 2021.
The user has requested enhancement of the downloaded file.

TOJET: The Turkish Online Journal of Educational Technology – October 2021, volume 20 issue 4
A Study of the Relationship between Secondary School Students' Computational

Thinking Skills and Creative Problem-Solving Skills
Muhammed PAF*
Republic of Turkey Ministry of National Education, 09100 Aydin, Turkey
muhammedpaf@gmail.com
ORCID: 0000-0002-4033-4115
Beste DİNÇER
Faculty of Education, Education Programs and Teaching, Aydin Adnan Menderes University, 09100 Aydin, Turkey
bestedincer@gmail.com
ORCID: 0000-0002-9264-3665
*Correspondency: bestedincer@gmail.com
This research is based upon the master thesis titled "The Relationship between Secondary School Students Computational Thinking Skills
and Creative Problem Solving Skills" which is authored by Muhammed PAF and supervised by Dr. Beste DİNÇER.
ABSTRACT
The primary goal of this research is to investigate the relation between computational thinking skills and creative
problem-solving skills in secondary school students over the 2018-2019 academic year (5th, 6th, 7th, and 8th
grades). The study's sample size is made up of 1098 secondary school pupils. The T-test, one-way ANOVA, and
multiple linear regression analysis were used. As can be observed from the research findings, the students'
numerical thinking abilities and creative problem-solving skills average scores were strong in terms of total
scores and sub-dimensions. According to the results, the mean scores of students' numerical thinking skills and
creative problem-solving skills differed considerably in favour of girls. As the pupils' grade level progressed,
statistically significant disparities in their computational thinking abilities scores appeared. Another major
conclusion from the study is that there is a substantial link between students' thinking skills, creative problem-
solving skills, and their capacity to keep up with technological advances. Furthermore, a somewhat favourable
and substantial connection between students' computational thinking skills and creative problem-solving skills
was discovered. Due to the findings of multiple linear regression analysis, sub-dimensions of creative problem-
solving skills highly predicted computational thinking skills.
Keywords: creative problem-solving computational thinking; 21st century skills
INTRODUCTION
Scientific, social, economic, and technical advancements in various sectors appear to be changing the individual
traits required and the abilities that must be held correspondingly in the twenty-first century. It may be argued
that, particularly in the 2000s, when information technology evolved fast, an education strategy based on
knowledge transfer was insufficient for individual growth. Many different styles of thinking arise as a child's
brain grows and new life experiences are gained (Relkin, 2018). Modern times require individuals with high-
level skills instead of individuals who memorize the information transmitted to them (Saracaloğlu, Yenice &
Karasakaloğlu, 2009).
It is an indisputable reality that information technologies have a direct influence on individual behavior and
modify our requirements in many aspects of our life. These innovations have an influence on education systems,
and many talents that people should acquire must be adjusted. As a result, persons in the twenty-first century can
think creatively and critically, generate unique answers to issues they meet, and adapt these solutions to new
contexts. Individual and social requirements change because of the great advancement of information
technology, making it critical for individuals to acquire certain abilities, referred to as 21st century skills.
According to Pakman (2018), 21st century talents include computational and creative thinking, as well as
algorithmic thinking. Kuleli (2018) noted that in the twenty-first century, students and instructors must be
technologically literate, problem solvers, researchers, and collaborators. Individuals entering the corporate world
nowadays are expected to have certain abilities such as digital literacy, entrepreneurship, creativity, problem
solving, and critical thinking (Kölemen, 2017). As Tanrıöğen and Sarpkaya (2011: 5) mentioned in their book,
the major approach to build the targeted society and lifestyle in the twenty-first century, which is defined as the
era of science, technology, and innovation, is via education through qualified manpower. Individuals that can
think and regulate their cognitive processes are considered to adapt better to this altering environment (Dinçer,
Copyright © The Turkish Online Journal of Educational Technology

1
2009). Individuals who create information, find proper information, and use information efficiently are needed in
this setting.
In the report published by OECD (2018), it is stated that certain jobs will be replaced by new occupational
categories during the next ten years. According to recent studies, many occupational groups that exist today will
not exist in the future, and other occupational groups will arise. Many occupational categories that exist today
will not exist in the future, according to current research, and other occupational groups will arise. Therefore,
along with fundamental qualities such as algorithmic thinking skill, inovative thinking skill, logical cause, and
effect relationship, and being a productive individual, being able to develop individuals with 21st century
abilities should be our major aims.
Furthermore, the primary goal of education has been to raise individuals who can create, criticize, determine,
question, and solve issues, and in recent years, the importance of teaching methods for higher order thinking
abilities has grown. As a result, it is recognized in educational institutions that it is critical for students to prepare
for new situations that may develop and to gain the necessary skills. As a result, when newly updated curriculum
is evaluated, it is stressed that the content of all courses is designed to create persons with high level thinking
skills. In this perspective, it is an important issue how these important skills that are emphasized in the revised
curriculum have an impact on students.
More than in previous years, the Ministry of National Education (MNE) made a major and dramatic modification
in the programs in 2005. It was discovered in the Ministry of National Education's revised curriculum that
fostering computational thinking of students was one of the objectives of computer science (MNE, 2018a) and
information technologies and software classes (MNE, 2018b). In this context, it can be said that computational
thinking and creative problem solving are among the skills that individuals should have. Wing (2016) states
computational thinking as a key skill for children in the 21st century. Therefore, “computational thinking skills
should be among the basic skills that students should have in the 21 st century (ISTE, 2016; Yıldız, Çiftçi &
Karal, 2017).
Being able to think in a computational way in daily life helps to learn the basic structure of the emerging
problems and to perceive the repeated mistakes better. In addition, computational thinking skills can be taught
with skills such as social interaction, communication, and working as a team. Problem solving and computational
thinking are related skills and can be transferred to other numerical fields such as mathematics (Çiftçi, Çengel &
Paf, 2018). As a result of this disclosed information, computational thinking has been an important skill to be
examined in recent years. It is stressed in the literature that there is a continuous link between computational
thinking, creative thinking, and problem solving, all of which are referred to be 21st century abilities. Individuals
with creative thinking talents are also excellent problem solvers, according to this statement. Studies on 21st
century abilities stress the need of developing students' creativity and innovative skills (Fox, 2011).
In the light of this information, computational thinking and creative thinking have become key skills of the 21 st
century. When the individuals, having creative thinking skills, encounter a problem in their daily lives, they can
produce fast and creative solutions to this problem. For this reason, it is expressed as a requirement that students
acquire higher order thinking skills since a young age.
In this context, answers to the following sub-problems were sought in the study:
(1) What level is the computational thinking skills of secondary school students participating in the
research?
(2) Is there a substantial difference in computational thinking skills among secondary school pupils based
on gender, class, or the degree to which they are following technology developments?
(3) What level is the creative problem-solving skills of secondary school students?
(4) Do creative problem-solving skills of secondary school students show a significant difference according
to gender/class, the state of following the technological developments?
(5) Is there a correlation between creative problem-solving talents and computational thinking skills in
secondary school students?
(6) Do sub-dimensions of creative problem-solving skills predict computational thinking skills in a
meaningful way?
METHOD
RESEARCH DESIGN
The goal of this study was to examine the relation between computational thinking skills and creative problem-
solving skills in secondary school pupils. This goal was achieved using a relational (correlational) model within

2
the scope of the investigation.
POPULATION AND SAMPLE

The population of the study comprises of secondary students studying in the Germencik district of Aydn
province during the 2018-2019 academic year. According to the theoretical sample size table in determining the
number of samples, the research universe consisting of 2000 people should have 322 with levels of α = .05
significance and 5% tolerance; It is stated that it can represent 1661 people at the level of α = .01 significance
and 1% tolerance (Can, 2014: 28). Accordingly, the number of samples in this study represents 49% of the
universe. While selecting the sample, two classes (A-B) were taken from each school as a cluster to increase the
possibility of the study population to represent the sample group and a single-stage cluster sample was made. In
single-stage cluster sampling, the main population (schools) is first divided into clusters, and the desired number
of clusters (classes) are drawn randomly from these clusters (İslamoğlu & Alnıaçık, 2016). Since the number of
classes and students in each school is not equal, two classes from all schools were included in the sample except
two village schools. In this way, the study's sample group consists of 1098 secondary school students chosen by
proportionate cluster sampling from the Germencik district of Aydın province during the 2018 – 2019 academic
year.
On analysis of the distribution of high school pupils volunteering in the study, 570 of them girls (51.9%), 528 of
them were boys (48.1 %). 269 of the children in the research were in the fifth grade (24.5 percent), 282 in the
sixth grade (27.5 percent), 348 in the seventh grade (31.7 percent), and 199 in the eighth grade (18.1%) as it was
presented in Table 1.
Table 1. Demographic Features of the Students

Variable Group N %
Male 528 48.1
Gender
Female 570 51.9
5 th Class 269 24.5

6 th Class 282 25.7
Class level
7 th Class 348 31.7
8 th Class 199 18.1
Status of Following Technological Yes 697 63.5

Developments No 381 34.7
Unknown 20 1.8
Total 1098 100
DATA COLLECTION TOOLS

The Computational Thinking Skills Scale (for Secondary School Students) developed by Korkmaz, Çakır and
Özden, (2015) as well as the "Creative Problem-Solving Features Inventory" adapted to Turkish by Baran-Bulut,
İpek, and Aygün (2018) were used to collect the data.
COMPUTATIONAL THINKING SKILLS SCALE

The scale, designed by Korkmaz, Çakır, and Özden (2015), and tested on secondary school students has a total of
22 items. Original scale developed in accordance with the university level, Korkmaz et al. (2015), validity and
reliability at secondary school level were also examined.
In addition, because of confirmatory factor analysis, it was stated that the observed values of the scale model
differed between .51 and .87 and showed an acceptable level of agreement. The validity and reliability of the
scale were tested again with the data obtained within the scope of the research. As a result of the confirmatory
factor analysis, it was observed that the standardized regression loads received different values between .52 and
.73.
In accordance with the confirmatory factor analysis, fit indices were examined, and the values were determined
as GFI = .995, AGFI = .947, CFI = .939, IFI = .939, NNFI = .927, RMSEA = .038, SRMR = .055. The model
formed in this direction has been found to provide excellent fit indices. The Cronbach Alpha reliability
coefficients calculated for the scale and its sub-dimensions are in the sub-dimension of "Creativity".62, in the
sub-dimension of "Algorithmic Thinking".71, in the sub-dimension of "Cooperation".73, and "Critical

3
Thinking".69, "Problem Solving" sub-dimension.75, and across the scale.83. Accordingly, when the value
related to the scale is examined, it is seen that the measurement tool has high reliability.
CREATIVE PROBLEM-SOLVING INVENTORY

The Creative Problem-Solving Inventory, designed by Lin (2010) and converted to Turkish by Baran-Bulut,
İpek, and Aygün (2018), has 40 items and five variables. The inventory has dimensions for convergent and
divergent thinking, motivation, environment, general knowledge, and skills. As a result of the analysis, it is
claimed that the 40-item inventory translated to Turkish meets the goal. The scale's validity and reliability were
once again assessed using data collected as part of the study. Confirmatory factor analysis showed that
standardized regression loads received different values between .31 and .76. In accordance with the confirmatory
factor analysis, the fit indices were examined, and the values were determined as GFI = .891, AGFI = .878, CFI
= .888, IFI = .888, NNFI = .876, RMSEA = .045, SRMR = .071.
The model formed in this way has been found to have acceptable fit indices. The Cronbach Alpha reliability
coefficients calculated for the scale and its sub-dimensions are as follows: “Divergent Thinking”.80,
“Convergent Thinking”.76, “Motivation”.74, “Environment” sub-dimension.87, “General Knowledge and
Skills” sub-dimension.76, “for the whole scale was .93. In this direction, when the value related to the scale was
analysed, it was seen that the measurement tool has a very high reliability.
DATA ANALYSIS
A statistical package application was used to analyze the research data. When interpreting the data, p<.05
significance level was taken as the basic criterion. More than one parameter was taken as a basis for the
examination of the normal distribution of data.
In the examination of the test result, the degree of closeness of the average, mode and median values of the data;
skewness and kurtosis coefficients, normal distribution (histogram), Normal Q-Q graphs were examined, and it
was concluded that the data showed normal distribution. If it is examined in more detail; the average, mode and
median values of the data were found to be very close to each other. Since this is a feature stated in the normality
assumptions (Leech, Berrett & Morgan, 2005), it was accepted among the normality parameters within the scope
of this research. Altman and Bland (1995) argued that when the sample is above certain limits, the assumption of
normality can be neglected and when the size of the sample within the scope of this study is analyzed (N =
1098), it is seen that this assumption is sufficient to be fulfilled. In addition, according to George and Mallery
(2019: 211) and Hair, Black, Babin and Anderson (2014: 39), it is acceptable for the normality assumption to be
within the ± 1 range of skewness and kurtosis values. The scale's skewness and kurtosis values of the mean
scores for the computational thinking skills scale -.276 to -.523; for creative problem-solving skills inventory -
.142 to -.524. In this context, it was seen that the data were in the range of ± 1. and the skewness and kurtosis
coefficients, which were accepted as the biggest indicator for normality assumption, were accepted as another
important parameter. In addition to this situation, normal distribution graph (histogram) and Normal Q-Q graph
were examined, and the data was found to have a distribution that is acceptable as near to normal.
Considering the meeting status of the normal distribution assumptions described above and the size of the sample
number (N = 1098), it was accepted that the data used in this study showed a normal distribution. As a result,
parametric statistical approaches were used for data processing activities. The sample t-test, ANOVA, and
Pearson correlation test were used to analyse the data in this way. Five-point Likert-type grading intervals were
used to evaluate students' computational thinking and creative problem-solving abilities. As a result, the ranges
1.00 - 1.79 are considered "very low," 1.80 - 2.59 are considered "low," 2.60 - 3.39 are considered "mid," 3.40 -
4.19 are considered "high," and 4.20 - 5.00 are considered "extremely high."
FINDINGS
Table 2 displays descriptive data connected to the students' computational thinking ability levels in relation to the
research's first sub-problem.
Table 2. Descriptive Statistics of Average Scores of Students' Computational Thinking Skill Levels and Sub-
dimensions
Sub-Dimensions N SS
Creativity 1098 4.07 .72

Algoritmic thinking 1098 3.70 .81
Collaboration 1098 4.08 .87
Critical thinking 1098 3.67 .87

4
Problem solvin 1098 3.53 .95

Computational Thinking(total) 1098 3.78 .58
Table 2 shows that the average scores of the students' computational thinking ability levels are typically high (X ̅
= 3.78). Furthermore, when the scores on the computational thinking abilities sub-dimensions were investigated,
it was discovered that the greatest score was in the cooperation dimension (X ̅ = 4.08), and the lowest score was
in the problem-solving dimension (X ̅ =3.53).
THE DIFFERENCE IN COMPUTATIONAL THINKING SKILL LEVELS AMONG

SECONDARY SCHOOL STUDENTS BASED ON GENDER VARIABLE
In accordance with the first variable of the second sub-problem of the research, a t-test for unrelated samples
from parametric tests was performed to evaluate whether the computational thinking skills of secondary school
pupils varied significantly by gender. Table 3 contains information on the test findings obtained in this context.
Table 3. The t-Test Results Related to the Differentiation Status of the Students' Computational Thinking Skill
Levels and Sub-Dimensions According to the Gender Variable
Sub-Dimensions Gender N SS t Sd P
Female 570 4.18 .63

Creativity Male 528 3.95 .79 -5.32 1096 .000*
Female 570 3.74 .76

Algoritmic Thinking Male 528 3.65 .85 -1.80 1096 .073
Female 570 4.22 .80

Colloberation Male 528 3.92 .92 -5.74 1096 .000*
Male 570 3.73 .87

Critical thinking Female 528 3.60 .86 -2.37 1096 .018*
Male 570 3.68 .95

Problem Solving Female 528 3.37 .94 -5.39 1096 .000*
Computational Male 570 3.89 .56

-6.31 1096 .000*
Thinking in General Female 528 3.67 .59
*p<.05
When the test results given in Table 3 were examined, that the average computational thinking scores of female
students (X ̅ = 3.89) were significantly higher than those of male students (X ̅= 3.67) was found. There was a
significant difference in favour of girls (t (1096) = -6.31, p<.05). In this case, it can be stated that the gender
variable has a significant effect on students' computational thinking skills generally.
Apart from the average scores of algorithmic thinking (t (1096) =1.80, p>.05) sub-dimension; creativity (t (1096) =
-5.32, p<.05), collaboration (t (1096) = -5.74, p<.05), critical thinking (t (1096) = -2.37 It is observed that, p<.05)
and problem solving (t (1096) = -5.39, p<.05) sub-dimensions made a significant difference in favor of girls.
DIFFERENTIATION OF SECONDARY SCHOOL STUDENTS' COMPUTATIONAL

THINKING SKILL LEVELS BASED ON CLASS VARIABLE
In accordance with the second variable of the second sub-problem of the research, one-way analysis of variance,
which is one of the parametric tests, was done to evaluate if the computational thinking skills of secondary
school pupils varied significantly by class. Information on the test findings obtained in this context was
presented in Table 4.

5
Table 4. ANOVA Results Regarding Differentiation of Average Scores of Students' Computational Thinking
Skill Levels and Sub-Dimensions by Class Variable
Sub-Dimensions Groups Sum of df Mean F p Difference
Squares Squares
Between 3.088 3 1.029 1.984 .115
Groups
Creativity Within Groups 567.559 1094 .519
Total 570.647 1097
Between .521 3 .174 .265 .850
Groups
Algoritmic Thinking Within Groups 715.773 1094 .654
Total 716.294 1097
Between 10.682 3 3.561 4.732 .003
Groups 5-7*
Colloberation Within Groups 823.169 1094 .752 5-8*
Total 833.851 1097
Between 1.114 3 .371 .495 .686
Groups
Critical thinking Within Groups 820.657 1094 .750
Total 821.771 1097
Between 30.705 3 10.235
Groups 5-6* 5-7*
Problem Solving Within Groups 967.828 1094 .885 11.569 .000
5-8*
Total 998.534 1097
Computational Between 6.571 3 2.190 6.568 .000
Groups 5-7*
Thinking in General Within Groups 364.856 1094 .334 5-8*
Total 371.426 1097
*p< .05
As it was shown in table 4, the average scores of the secondary school students studying at different 4 grade
levels regarding their computational thinking skills were compared with the One-way variance analysis for
unrelated samples according to the class variable (5 th, 6th, 7th and 8th grades). The results showed that the
averages of students in 5 th
grade is as (x̄ = 3.66), averages of students in 6th grade is (x̄ = 3.77), averages of
students in 7th grade (x̄ = 3.85) and averages of students in 8th grade have (x̄ = 3.86) statistically significant
differences between at least two (F (3-1094) = 6.56, p<.05). The effect size (ղ 2 = .02) calculated that this
difference is low. As a result of the Tukey multiple comparison test, it was observed that the difference was
between the 5th and 7th grade students and the 5th and 8th grade students, and the differentiation was in favour of
the 7th and 8th grades, respectively.
When the analysis results given in Table 4 were examined, creativity (F (3-1094)= 1.99, p>.05), algorithmic
thinking (F (3-1094)= .27, p>.05) and critical thinking mean scores (F (3-1094)= .50, p >.05) subscales did not make a
significant difference according to the class variable; collaboration (F (3-1094)= 4.73. p< .05) and problem solving
(F (3-1094)= 11.57, p< .05) it was found that there was a significant difference. As a result of the Tukey multiple
comparison test, the problem solving sub-dimension between the 5th grades (x̄= 3.96) and 7th (x̄= 4.17) and 8th
grades (x̄= 4.18) in the collaboration sub-dimension of the significant difference It was observed that it was
between 5th grade (x̄= 3.26) and 6 th (x̄= 3.53), 7th (x̄ = 3.66) and 8th grades (x̄= 3.68). When the average scores
were analysed in this direction, it was seen that the differentiation occurred in favour of the upper classes.

6
DIFFERENTIATION OF SECONDARY SCHOOL STUDENTS' COMPUTATIONAL

THINKING SKILL LEVELS BASED ON THE VARIABLE TO FOLLOW THE
TECHNOLOGICAL DEVELOPMENTS
In accordance with the third variable of the second sub-problem of the research, a t-test was performed for
unrelated samples from parametric tests to determine whether the computational thinking skills of
secondary school students differ significantly depending on the state of the following technological
developments. Data related to the test results carried out in this context are given in Table 5.
Table 5. T-Test Results Related to the Differentiation of the Average Scores of the Computational Thinking
Skill Levels and Sub-Dimensions of the Students According to the Status of Following Technological
Developments
Sub-Dimensions Groups N SS T sd p
Yes 697 4.12 .68

Creativity No 381 3.96 .78 3.56 1076 .000*
Yes 697 3.76 .80

Algoritmic Thinking No 381 3.58 .81 3.55 1076 .000*
Yes 697 4.12 .84

Colloberation No 381 3.97 .92 2.62 1076 .009*
Yes 697 3.73 .89

Critical thinking No 381 3.56 .80 2.98 1076 .003*
Yes 697 3.57 .97

Problem Solving No 381 3.46 .92 1.75 1076 .080
Computational Yes 697 3.83 .57

Thinking in General No 381 3.68 .59 4.02 1076 .000*
*p<.05
As it is shown in table 5, according to the state of following the technological developments, significant
differences were observed in all dimensions apart from problem solving sub-dimension of students'
computational thinking skills scores. A significant difference was observed in favour of the students who stated
that they followed the technological developments between the mean scores of students who indicated (x̄ = 3.68)
(t (1076) =4.02, p<.05) in general. In this case, it can be said that the state of following technological developments
had a significant effect on students' computational thinking skills.
Additionally, the average scores of problem solving (t (1076) = 1.75, p>.05) sub-dimension did not make a
significant difference according to the state of following the technological developments; but creativity (t (1076)
=3.56, p<.05) algorithmic thinking (t (1076) = 3.55, p<.05), collaboration (t (1076) = 2.62, p<.05) and critical
thinking (t (1076) = 2.98, p<.05) sub-dimensions created significant differences in favour of students who state that
they follow technological developments.
FINDINGS ON SECONDARY SCHOOL STUDENTS' CREATIVE PROBLEM-SOLVING

SKILL LEVELS
In line with the third sub-problem of the research, descriptive statistics about students' creative problem-solving
skill levels are given in Table 6.
Table 6. Descriptive Statistics of Students' Creative Problem-Solving Skill Levels

Sub-Dimensions N SS
Divergent Thinking 1098 3.70 .78

Convergent Thinking 1098 3.86 .72

7
Motivation 1098 3.75 .81

Environment 1098 3.98 .78
General knowledge and Skills 1098 3.56 .86
Creative problem solving Skill 1098 3.81 .62
in General
In Table 6, it was shown that the average scores of the students regarding their creative problem-solving skill
levels were high (x̄= 3.81). In this context, it could be stated that students' creative problem-solving skill levels
were at high level (x̄= 3.81). In addition, when the average scores of the creative problem-solving skills, sub-
dimensions were examined, it was seen that the highest average score was in the environment (x̄= 3.98), and the
lowest average score was in the general knowledge and skills (x̄= 3.56) dimension.
GENDER DIFFERENCE IN SECONDARY SCHOOL STUDENTS' CREATIVE PROBLEM-

SOLVING SKILL LEVELS
In accordance with the first variable of the fourth sub-problem of the research, a t-test was performed on
unrelated samples from parametric tests to evaluate if there was a significant difference in creative problem-
solving skills among secondary school students based on gender. Findings related to the test results in this
context are given in Table 7.
Table 7. t-Test Results of Differentiation of Secondary School Students' Creative Problem-Solving Skill Levels
by Gender
Sub-Dimensions Gender N SS t sd P
Female 570 3.72 .78

Divergent Thinking Male 528 3.69 .77 -.55 1096 .585
Female 570 3.93 .71

Convergent Thinking Male 528 3.80 .73 -2.98 1096 .003*
Female 570 3.79 .81

Motivation Male 528 3.70 .82 -1.85 1096 .065
Female 570 4.09 .76

Environment Male 528 3.86 .77 -5.04 1096 .000*
Female 570 3.53 .84

General knowledge and Male 528 3.60 .87 1.30 1096 .193
Skills
Creative Problem Female 570 3.86 .61

Solving -2.91 1096 .004*
Skill İn General Male 528 3.75 .63
*p< .05
When the test results given in Table 7 were examined, a significant difference was observed between creative
solving average scores of female students (x̄ = 3.86) and those of male students (x̄= 3.75) (t (1096) = -2.91. p < .05)
in favour of female students. Additionally, divergent thinking (t (1096) = -.55, p>.05), motivation (t (1096) = -1.85,
p>.05) and general knowledge and skills (t (1096) = 1.30, p>.05) sub-dimension did not make a significant
difference according to gender; but convergent thinking (t (1096) = -2.98, p<.05) and the environment (t (1096) = -
5.04, p<.05) sub-dimensions had significant differences in favour of girls. In this case, it can be stated that
gender variable has some significant effects on students' creative problem-solving skills.

8
DIFFERENCE IN CREATIVE PROBLEM-SOLVING SKILL LEVELS AMONG SECONDARY

SCHOOL STUDENTS BASED ON CLASS VARIABLE
In accordance with the second variable of the fourth sub-problem of the research, ANOVA, one of the parametric
tests, was used to evaluate if the creative problem-solving skills of secondary school pupils varied significantly
by class. Table 8 summarizes the findings linked to the test results in this context.
Table 8. ANOVA Results Regarding Differentiation of Average Scores of Students' Creative Problem-Solving
Skills Levels and Sub-Dimensions by Class Variable
Sub- Groups Sum of df Mean F p
Dimensions Squares Squares
Between Groups .655 3 .218 .363 .780
Divergent Within Groups 658.568 1093 .602
Thinking Total 659.222 1097
Between Groups .229 3 .076 .145 .933

Convergent Within Groups 574.374 1094 .525
Thinking Total 574.602 1097
Between Groups .919 3 .306 .462 .709

Motivation Within Groups 725.039 1094 .663
Total 725.958 1097
Between Groups .516 3 .172 .285 .837

Environment Within Groups 661.306 1094 .604
Total 661.823 1097
General Between Groups 3.059 3 1.020 1.394 .243

Knowledge and Within Groups 800.386 1094 .732
Skills Total 803.445 1097
Creative Between Groups .353 3 .118 .301 .825

Problem Within Groups 427.549 1094 .391
Solving Total 427.902 1097
Skill in General
When the analysis results given in Table 8 are examined, there were no statistically differences between
divergent thinking (F(3-1094)= .36, p>.05), convergent thinking (F (3-1094)= .15. p>.05), motivation (F (3-1094)= .46,
p>.05), environment (F (3-1094)= .29, p>.05) and general knowledge and skills (F (3-1094)= 1.39, p >.05) of creative
thinking scores and the class variable. It could be said that class level did not make significant effect on the
students’ creative problem-solving skills.
DIFFERENTIATION OF SECONDARY SCHOOL STUDENTS' CREATIVE PROBLEM-

SOLVING SKILL LEVELS BASED ON TECHNOLOGICAL DEVELOPMENT STATUS
In accordance with the third variable of the fourth sub-problem of the research, a t-test was performed on
unrelated samples from parametric tests to determine whether secondary school students' creative problem-
solving skills make a significant difference based on the state of the following technological developments.
Table 9 summarizes the findings relating to the test results obtained in this context.
Table 9. T-Test Results Regarding the Differentiation of the Average Scores of the Students' Creative Problem-
Solving Skill Levels and Sub-Dimensions According to the Variable Follow-Up Technological Status Variable
Sub-Dimensions Groups N SS t sd p
Yes 697 3.81 .76

Divergent Thinking No 381 3.52 .77 5.99 1076 .000*
Yes 697 3.92 .69

Convergent Thinking No 381 3.75 .77 3.78 1076 .000*

9
Yes 697 3.81 .81

Motivation No 381 3.63 .81 3.48 1076 .001*
Yes 697 4.04 .77

Environment No 381 3.86 .79 3.79 1076 .000*
Yes 697 3.64 .84

General knowledge and No 381 3.43 .85 3.84 1076 .000*
Skills
Creative Problem Yes 697 3.88 .61
Solving Skill in General No 381 3.67 .63 5.39 1076 .000*
*p< .05
When the test results given in Table 9 were examined, significant differences were found in all dimensions
calculated as divergent thinking (t (1076) = 5.99, p<.05), convergent thinking (t (1076) = 3.78, p< .05), motivation (t
(1076) = 3.48, p< .05), environment (t (1076) = 3.79, p< .05) and general knowledge and skills (t (1076) = 3.84, p<.05)
of creative problem solving skills of the students according to following up technological developments. It could
be said that significant differences were in favour of students who state that they follow technological
developments in the sub-dimensions.
SECONDARY SCHOOL STUDENTS' COMPUTATIONAL THINKING SKILLS AND

CREATIVE PROBLEM-SOLVING SKILLS: FINDINGS AND COMMENTS
In accordance with the fifth sub-problem of the research, Pearson moments multiplication correlation analysis
was used to investigate the link between students' computational thinking skills and creative problem-solving
skills and sub-dimensions. Table 10 presents the results of the correlation study.
Table 10. Pearson Moments Product Correlation Analysis Results Related to the Correlation between Students'
Computational Thinking Skills and Creative Problem-Solving Skills
Sub- Divergen Converg Motivati Environ GKS CPS
Dimensions t T. ent T. on ment
Correlation .473** .478** .446** .382** .365** .537**
Creativity p .000 .000 .000 .000 .000 .000
N 1098 1098 1098 1098 1098 1098
Correlation .488** .462** .465** .378** .493** .558**
Algoritmic .000 .000
Thinking p .000 .000 .000 .000
N 1098 1098 1098 1098 1098 1098
Correlation .326** .382** .323** .351** .239** .417**
Colloberation p .000 .000 .000 .000 .000 .000
N 1098 1098 1098 1098 1098 1098
Correlation .567** .492** .538** .410** .458** .611**
Critical .000 .000
thinking p .000 .000 .000 .000
N 1098 1098 1098 1098 1098 1098
Correlation .184** .206** .174** .133** .174** .213**
Problem .000 .000
Solving p .000 .000 .000 .000
N 1098 1098 1098 1098 1098 1098
Computationa Correlation .554** .553** .529** .448** .474** .636**
l .000 .000
Thinking p .000 .000 .000 1098 1098 .000
Skills N 1098 1098 1098 1098
in General
** p<0.01 GKS = General Knowledge and Skills, CPS = Creative Problem Solving Skills in General
Table 10 evaluated the Pearson Moments Product Association Coefficient in evaluating the correlation between
secondary school students' computational thinking skills and creative problem-solving skills, because the
variables matched the requirements of normalcy. As a result, a somewhat positive and significant association (r

10
=.636, p.01) was discovered between students' computational thinking skills and creative problem-solving
ability.
When the relationship between the computational thinking skills and the sub-dimensions of creative problem-
solving skills were examined in line with the findings, it was found that there was a moderate positive and
significant correlation with divergent thinking sub-dimension (r =. 554, p<.01), convergent thinking sub-
dimension (r = .553, p<.01), motivation sub-dimension (r = .529, p <.01), environment sub-dimension (r = .448,
p<.01), and with the general knowledge and skills sub-dimension (r = .474, p<.01) and computational thinking
skills.
In other words, as students' creative problem-solving skills increase, their computational thinking skills also have
tendency to increase. Similarly, between critical thinking and divergent thinking sub-dimensions of the highest
relationship among sub-dimensions; the lowest relationship was observed between problem solving and
environmental sub-dimensions (r=.133, p<.01).
MULTIPLE LINEAR REGRESSION ANALYSIS RESULTS IN SUB-DIMENSIONS OF

COMPUTATIONAL THINKING SKILLS AND CREATIVE PROBLEM-SOLVING SKILLS
Multiple linear regression analysis was used to see if computational thinking skill can be predicted based on
creative problem-solving sub-dimensions. The problem of multiple coupling is the most difficult condition in
multiple regression analysis. By examining the tolerance and VIF values, it was determined that there would be
no multi collinearity problem (Leech, Barrett, & Morgan, 2005).
Table 11. Multiple Linear Regression Model Summary on predicting Computational Thinking Skill with sub-
dimensions of creative problem-solving skills
Sub-Dimensions Std. Error β
Constant 1.536 .084
Divergent Thinking .159 .026 .211
Convergent Thinking .157 .028 .195
Motivation .089 .025 .125
Environment .089 .021 .119
General knowledge and Skills .102 .020 .150
** p<0.01
The sub-dimensions of creative problem solving skills substantially predicted computational thinking skills,
according to the analysis results (F (5-1092)= 155.209, p 0.01). All sub-dimensions contribute considerably to
the model's development as well. According to the beta values in Table 11, divergent thinking is the most
important contributor to the model's creation, followed by convergent thinking, general knowledge and skills,
motivation, and environment sub-dimensions, in that order. The R2 value that has been modified based on the
analysis results is 0.413. This demonstrates that the model explains 41% of computational thinking skills.
DISCUSSION AND CONCLUSION

In the study, students' computational thinking and creative problem-solving abilities were explored in connection
to several factors (gender, class level, and computer ownership), and whether there was a significant relationship
between variables was investigated. The computational thinking ability levels of the students were disclosed
within the framework of the first sub-problem of the investigation. The average scores of the students'
computational thinking ability levels and sub-dimensions were high, according to the results.
The highest mean score was found as collaboration and the lowest mean score was problem solving. In their
study, Korkmaz, Çakır and Özden (2015) reached the conclusion that students' computational thinking skills are
quite high, and the lowest average in terms of sub-dimensions is in the problem-solving dimension. Similarly,
Oluk (2017) concluded that students' computational thinking skill levels were high Korkmaz et al. (2015), in
another study, individuals' perceptions of computational thinking skill levels were half high and half medium; it
was stated that the highest average was collaboration, and the lowest averages were algorithmic thinking and
problem solving. Çakır (2017) concluded that students' computational thinking skills were above average, and
the highest average was collaboration, and the lowest average was problem solving. In this context, the fact that

11
students' computational thinking skills are at a high level and that students are computational thinkers is an
important finding for modern days of the 21 st century.
According to the results obtained in line with the second sub-problem of the study, a meaningful difference was
found in favour of girls between the average scores of female students' computational thinking skills and the
average scores of male students. When the average scores of the sub-dimensions were examined, that the average
scores of the female students were higher than male students and a significant difference was observed in favour
of the girls in all sub-dimensions except for algorithmic thinking.
Sarıtepeci (2017) stated that the computational thinking skill levels of women were higher, but this situation did
not create a significant difference. According to a similar result, Oluk (2017) stated that the average of female
students is higher than that of boys. Some studies in the literature differ with the results achieved. Gonzalez et al.
(2017) concluded that their scores on computational thinking skills were higher in favor of men. Kuleli (2018)
also found that the gender variable did not make any difference on the computational thinking skills. Oluk and
Korkmaz (2016) and Turan (2019) found that the gender variable did not make a difference in computational
thinking skills in their studies.
As a consequence of the results obtained in the context of the class variable, it was discovered that as students'
grade levels grew, so did their mean scores for computational thinking skills and sub-dimensions, with a
substantial difference between classes. According to Gonzalez et al. (2016), as students' grade levels improved,
so did their computational thinking skills. According to Korucu et al. (2017), kids' computational thinking skills
fluctuate considerably across grade levels. Some research provides outcomes that differ from those found in the
literature. Korkmaz et al. (2015) and Oluk (2017) state that there was a decrease in their computational thinking
skills as their grade levels progress.
The outcomes regarding the condition of following the technological improvements show that mean scores of the
computational reasoning skills differ in favor of the pupils who express that they follow the accomplished
technological improvements. It can be said that the state of following technological developments in this
direction has a significant effect on students' computational thinking skills. Çiftçi et al. (2018) stated that in their
study, there was a negative relationship between following technological developments and self-efficacy
regarding programming, and prospective teachers with high skills follow the developments in the field less. This
study compared to the other students who follow technological developments in the computational thinking skills
were found to be higher. In addition, it is striking that the results obtained in the studies with different sample
groups differ in the literature. It is thought that accessing and using technology correctly is an expected result
that will have a positive effect on students' computational thinking skills, but the differentiation situation in some
studies may be due to the profile of the sample group.
According to the results obtained in accordance with the study's second sub-problem, there was a substantial
difference in favor of females between the average scores of female students and the average scores of male
students in terms of creative problem-solving ability. Unlike the research findings, Toraman (2017) found that
male students are more likely than female students to achieve a creative solution. Unlike previous research,
Zeytun (2010) revealed that teacher applicants' judgments of creativity and problem-solving skills are not
gendered. In this study, the fact that female students' computational thinking and creative problem-solving skills
were statistically higher than men reveals that gender variable is a significant factor. Accordingly, it can be
thought that skills affect each other positively.
The third sub-problem of the research focused pupils' innovative problem-solving abilities. According to the
findings, the average scores of the students for their creative problem-solving ability levels and sub-dimensions
were high. A high degree of creative problem-solving ability among students was attained in this study, which
was a desirable conclusion. In this setting, students' high levels of creative problem-solving abilities are crucial
in terms of giving innovative solutions to challenges faced by pupils.
According to the class variable, students' creative problem-solving skills increased on average within the
framework of general and sub-dimensions. In the context of the variable of following technology advancements,
it was discovered that the average scores for creative problem-solving skills differ in favor of students who claim
to follow technical advances. It was shown that students that adhere to technology advances in their creative
problem-solving skills and sub-dimensions scored higher on average. In this respect, it is possible to assert that
the status of the following technical advances has a substantial impact on research.

12
The link between students' computational thinking ability levels and creative problem-solving talents was
investigated in the study's fifth sub-problem. There is various research in the literature that look at the relation
between problem solving skills and computing skills (Sarıtepeci, 2017; Gonzalez et al., 2017). However, no
skills have been discovered that explicitly investigates the link between computational thinking and creative
problem-solving. A somewhat favourable and substantial link was discovered between students' computational
thinking skills and creative problem-solving skills in this setting. In other words, as students' computational
thinking skills improve, so do their creative problem-solving abilities. The discovery that the two variables have
a positive influence on each other lends credence to the idea that computational thinking talent is fundamentally
articulated as a problem-solving process (Kalelioğlu, Gülbahar, and Kukul, 2016).
SUGGESTIONS
As a consequence of the research, the following recommendations for practitioners and researchers could be
made.
 According to the findings, as pupils' grade levels improved, so did their computational thinking skills.
To assist this development, it is formally recommended that computational thinking abilities be
included into the curriculum beginning with preschool.
 It has been observed that having a computer is effective on students' skills. Accordingly, it can be
suggested to increase / improve the technological equipment of educational environments.
 It may be suggested that courses such as information technologies and software, computer science,
where computational thinking skills are directly related, should be taught from an early age.
 This study was carried out with learners on the secondary school level. By broadening the area of the
study, it may be proposed that studies be conducted at the elementary, secondary, and university levels.
REFERENCES
Altman, D. G., & Bland, J. M. (1995). Statistics notes: Absence of evidence is not evidence of absence. BMJ,
311(7003), 485–485. doi:10.1136/bmj.311.7003.485
Ananiadou, K., & Claro, M. (2009). 21st Century skills and competences for new millennium learners in OECD
Countries. OECD Education Working Papers, 41.
Barak, M. (2009). Idea focusing versus idea generating: A course for teachers on inventive problem solving.
Innovations in Education and Teaching International, 46(4), 345-356. DOI:
10.1080/14703290903301743
Barak, M. (2013). Impacts Of Learning Inventive Problem-Solving Principles: Students’ Transition From
Systematic Searching To Heuristic Problem Solving. Instructional Science, 41(4), 657-679.
Baran-Bulut, B., İpek, A. S. ve Aygün, B. (2018). Yaratıcı Problem Çözme Özellikleri Envanterini Türkçeye
Uyarlama Çalışması. Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi, 18 (3), 1360-1377.
Brennan, K., & Resnick, M. (2012). New frameworks for studying and assessing the development of
computational thinking. In Annual American Educational Research Association meeting, Vancouver,
BC, Canada.
Büyüköztürk, Ş., Kılıç Çakmak, E., Akgün, Ö.E., Karadeniz, Ş. ve Demirel, F. (2018). Bilimsel Araştırma
Yöntemleri (25. Baskı). Ankara: Pegem Yayınları
Can, A. (2014). SPSS ile Bilimsel Araştırma Sürecinde Nicel Veri Analizi. Ankara: Pegem Akademi.
Cramond, B., Martin E. C., & Shaw L. E. (1990). Generalizability of Creative Problem Solving Procedures to
Real-Life Problems. Journal for the Education of the Gifted, 13(2), 141-155.
https://doi.org/10.1177/016235329001300203
Cuny, J., Snyder, L. & Wing. J. M. (2010). Demystifying computational thinking for non-computer scientists.
Çakır, E. (2017). Ters Yüz Sınıf Uygulamalarının Fen Bilimleri 7. Sınıf Öğrencilerinin Akademik Başarı,
Zihinsel Risk Alma Ve Bilgisayarca Düşünme Becerileri Üzerine Etkisi. Yayımlanmamış Yüksek
Lisans Tezi. Ondokuz Mayıs Üniversitesi, Samsun.
Çiftci, S., Çengel, M., ve Paf, M. (2018). Bilişim Öğretmeni Adaylarının Programlama İlişkin Öz -
Yeterliklerinin Yordayıcısı Olarak Bilişimsel Düşünme ve Problem Çözmeye İlişkin Yansıtıcı Düşünme
Becerileri. Ahi Evran Universitesi Kırşehir Eğitim Fakültesi Dergisi, 19(1), 321–334.
Demirci, N. (2014). Sistematik Yaratıcı Problem Çözme Etkinliklerinin İlköğretim 7. Sınıf Öğrencilerinin Işık
Konusundaki Kuramsal, Deneysel ve Günlük Yaşam Problemlerini Çözmelerine Etkisi. Yayınlanmamış
Doktora Tezi. Marmara Üniversitesi Eğitim Bilimleri Enstitüsü, İstanbul.
Dinçer, B. (2009). Öğretmen Adaylarının Düşünme Stilleri Profillerinin Çeşitli Değişkenler Açısından
Değerlendirilmesi. Yayımlanmamış Yüksek Lisans Tezi, Adnan Menderes Üniversitesi Sosyal Bilimler
Enstitüsü, Aydın.
Dinçer, B., ve Paf, M. (2018, Eylül). Bilgisayar ve Öğretim Teknolojileri Eğitimi Bölümü Öğrencilerinin
Bilişimsel Düşünmeye İlişkin Farkındalıkları. Sözlü Bildiri. II. Uluslararası Eğitim Araştırmaları ve
Öğretmen Eğitimi Kongresi (ERTE Congress) Bildiri Özetleri Kitabı, Aydın.

13
Fox, M. O. (2011). Implementing 21st Century Skills: A Paradox In A Traditional World Of Education?.
Unpublished Doctoral Thesis, College of Saint Elizabeth Morristown, New Jersey.
George, D. & Mallery, M. (2019). IBM SPSS Statistics 25 Step by Step: A Simple Guide and Reference, 15.0
update (15a ed.). Boston: Pearson.
Hair, J. F., Black, W. C., Babin, B. J., Anderson (2014). Multivariate Data Analysis: Pearson Education Limited
(7 ed.).England: Pearson Education Limite.
ISTE (2015). CT leadership toolkit. Retrieved from http://www.iste.org/ docs/ct-documents/ct-leadershipt-
toolkit.psd?sfvrsn=4
İslamoğlu, A. H. ve Alnıaçık, Ü. (2016). Sosyal Bilimlerde Araştırma Yöntemleri. (5.baskı) Beta Yayıncılık:
İstanbul.
John P. Casey (1965) Creativity and the Solving of Social Problems. The Journal of Educational Research, 59:4,
154-159, DOI: 10.1080/00220671.1965.10883322
Kalelioğlu, F., Gülbahar, Y., & Kukul, V. (2016). A Framework For Computational Thinking Based On A
Systematic Research Review. Baltic Journal Of Modern Computing, 4(3), 583-596.
Korkmaz, Ö., Çakır, R., Özden, Y. M., Oluk, A. ve Sarıoğlu, S. (2015). Bireylerin Bilgisayarca Düşünme
Becerilerinin Farklı Değişkenler Açısından İncelenmesi. Ondokuz Mayıs Üniversitesi Eğitim Fakültesi
Dergisi, 34(2), 68-87.
Korkmaz, Ö., Çakır, R., ve Özden, Y. M. (2015). Bilgisayarca Düşünme Beceri Düzeyleri Ölçeğinin (BDBD)
Ortaokul Düzeyine Uyarlanması. Gazi Eğitim Bilimleri Dergisi, 1(2), 67-86.
Korkmaz, Ö., Çakir, R., & Özden, Y. M. (2017). A validity and reliability study of the computational thinking
scales (CTS). Computers in Human Behavior, 72, 558–569. http://doi.org/10.1016/j.chb.2017.01.005
Kuleli, Ç. S. (2018). Öğretmen Adaylarının Çevrimiçi Öğrenmeye Hazırbulunuşluk Düzeyleri Ve Bilgi İşlemsel
Düşünme Becerilerinin Değerlendirilmesi.Yayınlanmamış Yüksek Lisans Tezi. Düzce Üniversitesi
Sosyal Bilimler Enstitüsü, Düzce.
Leech L. N., Barrett C. K. & Morgan A. G. (2005). SPPS For Intermediate Statistics - Use and
Interpretation(Second Ed.). London: Lawrence Erlbaum Associates.
Lin, C. (2010). Analyses of attribute patterns of creative problem solving ability amongupper elementary
students in Taiwan.Unpublished document, St. John's Unıversıty, New York.
MEB (2018a). Bilişim Teknolojileri ve Yazılım Dersi Öğretim Programı (Ortaokul 5 ve 6.Sınıflar). Retrieved
from http://mufredat.meb.gov.tr/ProgramDetay.aspx?PID=374
MEB (2018b). Bilgisayar Bilimi Dersi (Kur 1-2) Öğretim Programı. Retrieved from
http://mufredat.meb.gov.tr/ProgramDetay.aspx?PID=335
OECD. (2018). The future of education and skills: Education 2030. Retrieved from
https://www.oecd.org/education/2030/E2030%20Position%20Paper%20
Oluk, A. (2017). Öğrencilerin Bilgisayarca Düşünme Becerilerinin Mantıksal Matematiksel Zekâ ve Matematik
Akademik Başarıları Açısından İncelenmesi. Yayımlanmamış Yüksek Lisans Tezi. Amasya
Üniversitesi Fen Bilimleri Enstitüsü, Amasya.
Pannels, C. T. (2010). The Effects Of Training Preservice Teachers In Creative Problem Solving And Classroom
Management. Unpublished Doctoral Thesis, University of Oklohoma, Oklohoma.
Papert, S. (1996). An exploration in the space of mathematics educations. International Journal of Computers for
Mathematical Learning, 1(1), 95–123. doi:10.1007/bf00191473.
Relkin, E. (2018). Assessing Young Children’s Computational Thinking Abilities. Unpublished Master's
Thesis, Tufts University, Boston.
Román-González, M., Pérez-González, J. C., & Jiménez-Fernández, C. (2017). Which cognitive abilities
underlie computational thinking? Criterion validity of the Computational Thinking Test. Computers in
Human Behavior, 72(2017), 678-691.
Saracaloğlu, S. A., Yenice, N., Karasakaloğlu, N. (2009). Öğretmen Adaylarının İletişim Ve Problem Çözme
Becerileri İle Okuma İlgi Ve Alışkanlıkları Arasındaki İlişki. Yüzüncü Yıl Üniversitesi, Eğitim
Fakültesi Dergisi., 6(2), 187–206.
Sarıtepeci, M. (2017). Ortaöğretim düzeyinde bilgi-işlemsel düşünme becerisinin çeşitli değişkenler açısından
incelenmesi. 5th International Instructional Technologies & Teacher Education Symposium
Proceedings Kitabı. 11-13 Ekim. İzmir: Dokuz Eylül Üniversitesi, 218-226.
Sayın, Z. ve Seferoğlu, S.S. (2016, Şubat). Yeni Bir 21. Yüzyıl Becerisi Olarak Kodlama Eğitimi ve Kodlamanın
Eğitim Politikalarına Etkisi. Akademik Bilişim Konferansı Bildiri Kitabı, Adnan Menderes
Üniversitesi, Aydın.
Tanrıöğen, A. ve Sarpkaya, R. (Editörler), (2011). Eğitim Bilimine Giriş (Üçüncü Baskı). Ankara: Anı
Yayıncılık.
P21 (The Partnership for 21st Century Skill). (2009). Curriculum and instruction: A 21st century skills
implementation guide. Retrieved from http://www.battelleforkids.org/networks/p21

14
Toraman, Ç. (2017). Lise Öğrencilerinin Yaratıcı Çözü Elde Etme Düzeyleri: İstatistiksel Söylem
Analizi.Yayınlanmamış Doktora Tezi. Eskişehir OsmanGazi Üniversitesi Eğitim Bilimleri Enstitüsü,
Eskişehir.
Turan, B. (2019). Ortaokul Öğrencilerinin Geliştirdiği Oyun ve Robot Projelerinde Probleme Dayalı
Öğrenmenin Problem Çözme ve Bilgi İşlemsel Düşünme Becerilerine Etkisi. Yayımlanmamış Yüksek
Lisans Tezi, Van Yüzüncü Yıl Üniversitesi Eğitim Bilimleri Enstitüsü, Van.
Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33.
http://doi.org/10.1145/1118178.1118215
Wing, J. M. (2016). Computational thinking, 10 years later. Retrieved from https://www.microsoft.com/en-
us/research/blog/computational-thinking-10-years-later/
Yeldan, İ. (2016). Sistematik Yaratıcı Problem Çözme Etkinliklerinin, Ortaokul 6. Sınıf Öğrencilerinin Kuvvet
ve Hareket Konusundaki Akademik Başarılarına, Yaratıcı Problem Çözme Becerilerine Etkisi.
Yayınlanmamış Yüksek Lisans Tezi. Marmara Üniversitesi Eğitim Bilimleri Enstitüsü, İstanbul.
Yıldırım, A. (2014). Okul Öncesinde Yaratıcı Problem Çözme Etkinliklerinin Yaratıcılığa Etkisi (5 Yaş Örneği).
Yayımlanmamış Doktora Tezi, Hacettepe Üniversitesi Eğitim Bilimleri Enstitüsü, Ankara.
Zeytun, S. (2010). Okul Öncesi Öğretmenliği Öğrencilerinin Yaratıcılık ve Problem Çözme Düzeyleri
Arasındaki İlişkinin İncelenmesi. Yayımlanmamış Yüksek Lisans Tezi, Dokuz Eylül Üniversitesi
Eğitim Bilimleri Enstitüsü, İzmir.

15
View publication stats

International Journal of Sociologies and Anthropologies Science Reviews (IJSASR), 2 (5):
September-October 2022, page 39-46, ISSN: 2774-0366
Website: https://so07.tci-thaijo.org/index.php/IJSASR/index
DOI: https://doi.org/10.14456/jsasr.2022.37
Development of Mathematics Learning Activities by Using Inquiry Process with Creative Problem-
solving (CPS) to Promote Computational Thinking Skills in Matthayomsuksa 2 Students
Jirattikorn Siphai and 2Kanyarat Cojorn

1
1
Master of Education (Teaching of Science and Mathematics), Faculty of Education, Mahasarakham University
2
Faculty of Education, Mahasarskham University
1
E-mail: mrpeemo@hotmail.com, ORCID ID: https://orcid.org/0000-0002-5841-2185
2
Coresponding author e-mail: kanyarat.c@msu.ac.th, ORCID ID: https://orcid.org/0000-0002-1237-2798
Received 13/09/2022 Abstract: - Computational thinking skills are essential for today’s learners. The learning
Revised 19/09/2022 management model using the inquiry process with Creative Problem-solving (CPS) can
Accepted 20/09/2022 be used to develop computational thinking skills, creative thinking, analytical thinking,
and problem-solving in a systematic manner for students. The purpose of this research
was to develop the mathematics learning activities using a creative problem-solving
process to promote computational thinking skills of grade 8 students in a study about of
efficiency to mathematics learning activities according to the specified criteria of 70/70,
comparison to computational thinking skills, and comparison to achievement according
to criteria of 70% with grade 8 students of 41 people in Mahasarakham University
Demonstration School. The data came from learning management plans of 7 lessons, and
the achievement test to data analysis was analyzed by percentage, mean, standard
deviation, and t-test. The study revealed that the efficiency of mathematics learning
activities has to process and result from efficiencies of 76.77/77.45 to higher than the
threshold of 70/70, the computational thinking skills, and the achievement of the students
to higher than the threshold of 70% were significant at the .05 significant level.
Keywords: Efficiency to Mathematics Learning Activities; Computational Thinking Skill; Achievement; Grade 8 Students
Introduction
Mathematics is a subject related to numbers, calculations, and the use of cause and effect to solve
problems that plays a very important role in human life and helps in the development of human thought
processes to have creative ideas and rational thinking. It also helps to analyze problems and situations
in detail, which will lead to accurate and appropriate forecasting, planning, and decision-making to
solve the problems. The study of mathematics in the Basic Education Core Curriculum 2008 (Revised
2017) aims to enable all youths to continually learn mathematics according to their potential and provide
opportunities for youths to apply the knowledge, skills, and necessary mathematical processes to
improve their quality of life, focusing on essential skills for learning in the 21st century and preparing
students to be ready to learn things and to pursue a career upon graduation or continue their studies at
a higher level (Ministry of Education, 2017).
Computational thinking skills are essential for today’s learners. Because students have to face
problems in the real world, they need to consider the problems, be able to manage the information
related to the problems, test the problem-solving plan to check for errors, and adjust the plan for better
( McKenna, 2017) . Nowadays, most mathematical problems are complex in terms of analyzing
problems, calculating, or finding solutions. Computational thinking is a problem-solving skill that
applies the principles of computer science, consisting of abstraction, decomposition, pattern
recognition, and algorithm to systematically solve complex mathematical problems.
After studying teaching and learning approaches that encourage students to develop their
computational thinking skills, the researchers realized that the learning management model using the
inquiry process with Creative Problem-solving (CPS) can be used to develop computational thinking
skills, creative thinking, analytical thinking, and problem-solving systematically for students. It is
because inquiry-based learning management is a learning method that encourages students to learn by
themselves and try to find solutions on their own, and CPS allows students to practice thinking of new
solutions that can be used to solve problems systematically. Inquiry-based learning management
focuses on student-centered learning. It allows students to practice and have the freedom to think and
express their ideas that can be used to solve problems and create new knowledge. It also allows students
[39]
Citation: Siphai, J., & Cojorn, K., (2022). Development of Mathematics Learning Activities by Using Inquiry Process with Creative Problem-solving
(CPS) to Promote Computational Thinking Skills in Matthayomsuksa 2 Students. International Journal of Sociologies and Anthropologies Science
Reviews (IJSASR), 2 (5), 39-46; DOI: https://doi.org/10.14456/jsasr.2022.37
to fully practice their math skills. It is a 6-step activity process, including a review of previous
knowledge, teaching new content, summary, skill training, knowledge application, and evaluation
(Institute for the Promotion of Teaching Science and Technology, 2003). However, it is not enough for
the development of computational thinking skills because it lacks development in understanding
problems in terms of thinking of creative solutions to problems and solving problems in a systematic
way. Therefore, the researchers studied the techniques or processes which can be applied to inquiry-
based learning management on mathematics that can develop computational thinking skills in all
components.
Creative Problem-solving is a model to help you solve problems and manage change creatively.
It gives you a set of easy-to-use tools to help translate your goals (Donald, Scott, & K. Brian,2010) and
a process that allows people to apply both creative and rational thinking to find solutions to everyday
problems. (Dandy, 1986 ; Isaksen, D. & Treffinger, 1994)
The researchers applied CPS in organizing inquiry-based learning activities to develop
computational thinking skills. It is a thought process in solving complex problems derived from new
ideas, consisting of convergent thinking based on previous knowledge and experience and divergent
thinking from creative thinking, which appropriately promote each other for application in creative
problem-solving. Treffinger, Isaksen, and Dorval ( 2003) presented CPS which can be divided into 4
steps: understanding the challenge, generating ideas, preparing for action, and planning your approach.
The researchers thought that each step of CPS can improve computational thinking skills in terms of
understanding the problem, thinking of creative solutions to the problem, and systematically solving
the problem.
According to the aforementioned reasons, the researchers were interested in applying
mathematics learning activities on statistics of Matthayomsuksa 2 using the inquiry process with CPS
in solving more complex problems to promote students’ computational thinking skills so that they can
develop their potential in solving mathematical problems and effectively apply the gained skills and
knowledge to their daily life.
Research Objectives
1. To develop the mathematical learning activities on statistics by using the inquiry process with
CPS to efficiently promote computational thinking skills of Matthayomsuksa 2 students according to
the 70/70 criterion
2 . To compare the computational thinking skills on statistics by using the inquiry process with
CPS of Matthayomsuksa 2 students with the 70 percent criterion
3 . To compare the learning achievement on statistics by using the inquiry process with CPS of
Matthayomsuksa 2 students with the 70 percent criterion
Research Hypothesis
1. The computational thinking skills of the students learning through the mathematics learning
activities by using the inquiry process with CPS are higher than the 70 percent criterion of the full score.
2. The learning achievement of the students learning through the mathematics learning activities
by using the inquiry process with CPS is higher than the 70 percent criterion of the full score.
Methodology
1. Population and Sample
1.1 The population consisted of 163 Matthayomsuksa 2 students from 5 classes of the Sciences
and Mathematics Program studying at Mahasarakham University Demonstration School (Secondary).
1.2 The samples were 41 Mathayomsuksa 2/3 students studying Sciences and Mathematics
Program at Mahasarakham University Demonstration School (Secondary) obtained by a cluster random
sampling method.
[40]
2. Data Collection Instruments

2.1 The instruments used in the experiment were 7 lesson plans based on the inquiry process
with CPS, a total of 7 hours. The mean scores of the appropriateness assessed by the experts were
between 4.61-4.78, indicating that the lesson plans are the most appropriate.
2.2 The instruments used for data collection were as follows.
2.2.1 A computational thinking skills assessment, consisting of 3 situations: In each
situation, the students’ computational thinking skills were assessed in all four components:
decomposition, abstraction, pattern recognition, and algorithms. The Item-Objective Congruence Index
was 0.67-1.00. The difficulty was 0.54-0.68. The discrimination was 0.59-0.78. The reliability was 0.94.
2.2.2 A learning achievement test of Mathematics 4 on statistics: It was a 20-question,
multiple-choice test with 4 choices for each question. The Item-Objective Congruence Index was 0.67-
1.00. The difficulty was 0.45-0.75. The discrimination was 0.59-0.98. The reliability was 0.84.
3. Research Design
This research aimed to develop the mathematics learning activities by using the inquiry
process with CPS with computational thinking skills of Mathayomsuksa 2 students on statistics. Pre-
experimental research using one group posttest only design was employed (Worakham, 2016).
4. Data Analysis
The data analysis in this research was conducted by using the completely revised research instruments
with the samples who were Mathayomsuksa 2/3 students at Mahasarakham University Demonstration
School (Secondary) studying Mathematics 4 in the second semester of the academic year 2 0 2 1 . The
data analysis process was as follows.
4.1. The efficiency of the mathematics learning activities using the inquiry process with CPS
of Mathayomsuksa 2 students was analyzed to meet the 70/70 criterion by using the formula for
efficiency calculation based on the concept of Brahmawong (2013). The efficiency of the process was
calculated from the percentage of the scores from class assignments, including worksheets and tests at
the end of each lesson plan. The efficiency of the outcome was calculated from the percentage of the
scores from the computational thinking skills test and the learning achievement test of Mathematics 4
on statistics.
4.2. The computational thinking skills were analyzed from the results of Mathayomsuksa 2
students’ computational thinking skills test on statistics using the one-sample t-test, compared with the
specified criterion (70 percent).
4.3. The learning achievement was analyzed from the learning achievement test of
Mathematics 4 on statistics of Mathayomsuksa 2 students using one sample t-test, compared with the
specified criterion (70 percent).
Results
This research aimed to develop the mathematics learning activities by using the inquiry process
with CPS with computational thinking skills of Mathayomsuksa 2 students on statistics. The research
results can be summarized as follows.
1. The efficiency of the mathematics learning activities by using the inquiry process with CPS to
promote computational thinking skills in Matthayomsuksa 2 students on statistics was 76.77/77.45,
which was higher than the set criterion of 70/70. Results as in table 1
[41]
Table 1 The efficiency of the mathematics learning activities

Scores Full score X S.D. Percentage
1. Activity sheet scores and mini-tests at the 126 96.73 6.89 76.77
end of the plan.
2. Scores for taking the Computational
Thinking Skills Test and post-test 65 49.83 2.58 77.45
The efficiency of the mathematics learning activities (E1/ E2) was 76.77/77.45
2. The computational thinking skills of Matthayomsuksa 2 students who studied through the
mathematics learning activities by using the inquiry process with CPS were higher than the 70 percent
criterion with statistical significance at the level of 0.05. Results as in table 2
Table 2 The computational thinking skills.

Test N X S.D.
t p
Computational thinking skills 41 33.93 1.85 8.403* .000
* significance at the level of .05
3. The learning achievement of Matthayomsuksa 2 students who studied through the mathematics
learning activities by using the inquiry process with CPS was higher than the 70 percent criterion with
statistical significance at the level of .05. Results as in table 3
Table 3 Learning achievement.

Test N X S.D.
t p
Learning achievement 41 15.90 1.04 11.667* .000
* significance at the level of .05
Discussion
1. The efficiency of the mathematics learning activities by using the inquiry process with CPS to
promote computational thinking skills in Matthayomsuksa 2 students on statistics was 76.77/77.45,
which was higher than the set criterion of 70/70. It is because the teaching and learning management
developed by the researchers focused on encouraging the students to solve problems on their own. The
students discussed and exchanged their ideas within the group. The teacher presented the problems to
each group. Then the students applied CPS to solve the problems. Once the students learned to
understand the problems, the teacher asked each group to generate ideas on how to solve the problems.
There was not a definite answer to the problems, but a wider choice of solutions. However, what must
be considered is choosing the most suitable solution for that period (Cinnamon and Matulef, 1979). The
teacher instructed each group to write the clear problem-solving steps in a well-structured and effective
way (Ministry of Education, 1998). After that, the problem-solving steps were used to solve the
problems. After the problems were solved, the teacher used question-and-answer conversations about
the solutions with the students until the students in each group concluded. After the students solved the
problems in the form of group activities, the teachers presented new problems different from the
situations in the group activities to each student. The students were encouraged to practice solving
problems on their own by using CPS created from the group activities. The process began with
understanding the problems, generating solutions, choosing the most suitable solution and writing clear
problem-solving steps, and using them to solve the problems. Once the students practiced solving
[42]
problems until they became proficient, the teacher presented more complex or everyday situations to
the students. The students solved the problems using the phenomena they were facing or experiencing
and were ready to challenge their thinking (Suksri-ngam, 2003). As a result, the students can apply the
knowledge gained to solve more complex problems. The students applied the CPS process to solve the
problems. This activity is a teaching method that focuses on the process of seeking knowledge that
helps students to discover truths by themselves and have direct experience in learning ( Laohapaiboon,
1999). It is consistent with a study by Indiana Creative Problem-Solving Initiative Blumberg Center
(2003) on problem-solving abilities. The students learned about CPS. It was found that there was an
increase in the problem-solving ability of those who learned about CPS. In addition, Tananta (2019)
conducted a study titled “A development of mathematics learning activities by using inquiry process
with Polya’s problem-solving to promote mathematics problem-solving ability in Matthayomsuksa 5
students. The study results revealed that the students who learned by using the inquiry process with
Polya’s problem-solving on vector had mathematical learning achievement higher than the 70 percent
criterion at a 0.05 level of significance.
2. The computational thinking skills of Matthayomsuksa 2 students who studied through the
mathematics learning activities by using the inquiry process with CPS were higher than the 70 percent
criterion with statistical significance at the level of .05. It is because the learning process focused on the
development of problem-solving abilities by training the students to search for knowledge. The teacher
asked questions to encourage the students to use the thought process until they discovered feelings or
correct solutions by themselves, and summarized them as principles, rules, or methods for solving
problems and applied them to control, improve, change and create things widely. This is an important
component of the inquiry process teaching (Moolkum and Moolkum, 2002), and another factor is the
creative problem-solving (CPS) process developed by the researchers. CPS is the process of solving
problems creatively and making it easy to achieve the set goals (Treffinger, Isaksen, and Dorval, 2003).
It mostly occurs in situations where there is no definite solution to the problem. This gives a wider
choice of solutions. But what must be considered is choosing the most suitable solution for that period
(Cinnamon and Matulef, 1979). This concept is following the learning activities in this research as the
problems were given to the students, so they can understand the situations. Then they would isolate the
components of the problems, cut out unnecessary parts of the solutions and find or apply the solutions
that can be used to solve the problems and select the most suitable way to solve the problems, and
clearly write the correct sequence of the solutions and use it to solve the problem correctly.
Computational thinking is a way of thinking about defining a problem, and the solution can be presented
in a step-by-step manner (Aho, 2012). This corresponds to (Mitchel and Kowalk. 1999; Poompachati,
2009), stating that creative problem-solving (CPS) is a way of thinking and acting. It consists of
(Creative) which refers to peculiar and unique ideas that the creator must have at least one for finding
the solution; (Problem) which is a situation that expresses a challenge, opportunity, or something that
needs attention, and (Solving) which is a way of planning to answer the question, organize a meeting
or judge the problem. In addition, the learning activities developed by the researchers together with
CPS are consistent with the research of Gonzalez et al. (2016) which found that computational thinking
is related to problem-solving and understanding of representations of things, and being able to use
reasons to solve problems. A study by Leonard et al. (2016) examined the improvement of
computational thinking abilities of 124 junior high school students. It was found that the learning
process aiming at students to solve problems through designing and creating a workpiece can promote
computational thinking. Sittikhetkron and Sawangmek ( 2 021) conducted a study on the development
of computational thinking skills through 5Es inquiry learning activities with the board game and
Formula Coding on the population in the pandemic for Grade 12 students. It was found that the learning
activities consisted of engagement with pandemic news, exploration of pandemic data to design the
prevention and solving by using formula coding with Microsoft Excel program, explanation of
population graphs from changed trend, elaboration of population dynamics illustrated by Covidea board
game before organizing group discussion for concluding population and pandemic. Moreover, the
[43]
students’ CT skills were at the highest level, corresponding to the increase in the development of
computational thinking skills while studying, which was also at the highest level. Jenjit (2003)
researched geometry teaching activities using creative problem-solving for high school students with
mathematical abilities. The results were as follows. 1) More than 75% of the students with mathematical
abilities studying through geometry teaching activities using creative problem-solving passed the 75
percent criterion with statistical significance at the level of .01. 2) After studying geometry teaching
activities using creative problem-solving, the students had creative problem-solving behaviors,
consisting of flexibility, originality, and elaboration at a high level in all aspects.
3. The learning achievement of Matthayomsuksa 2 students who studied through the mathematics
learning activities by using the inquiry process with CPS was higher than the 70 percent criterion with
statistical significance at the level of .05. It is because the developed activities focused on self-study.
The students had to finish worksheets and shared their learning with friends and teachers. Therefore,
they can discover knowledge from both learning materials and the practice of finding creative answers
with the clear purpose of problem-solving. This is consistent with the concept of Moolkum and
Moolkum (2002) who discussed inquiry-based learning management, which is a learning process that
focuses on developing problem-solving abilities through training students to know how to search for
knowledge. In this study, the students used the thought process until they discovered feelings or correct
solutions by themselves, summarized them as principles, rules, or methods for solving problems, and
applied them to control, improve, change and create things widely. In addition, in the learning process,
worksheets were used, and the students did activities both in groups and individually, allowing them to
exchange ideas with friends. They also discussed their different opinions within the class. Moreover,
the mathematics learning activities using the inquiry process focused on stimulating the students to
learn and search for knowledge using the reasoning process (Boonchuvong, 1995). The activities also
focused on the students’ thinking processes in solving complex problems using new ideas to be applied
in creative problem-solving ( Treffinger, Isaksen, and Dorval, 2003) . These activities allowed the
students to know what to look for and what information that can be used to help them find answers. The
students were also encouraged to apply the gained knowledge to benefit and use in daily life. Thus, their
academic achievement was better. This is consistent with a study by Khaoprae and Cheausuwantavee,
studying mathematical analytical thinking ability and learning achievement on statistics among ninth-
grade students through the inquiry cycle learning management. The study results pointed out that the
learning achievement on statistics among ninth-grade students after the inquiry cycle learning
management was statistically higher than the 70 percent criterion at a 0.05 level of significance. It also
corresponds to a study by Pimmun (2014), investigating the effect of synthesization between inquiry
cycle (5Es) and Polya’s problem-solving process on the topic of the application of linear equations with
one variable. The study revealed that the efficiency of the Inquiry Cycle (5Es) and Polya’s problem-
solving process learning management was 80.56/ 80.56. In addition, the mathematics achievement of
the students that learned by the Inquiry Cycle (5Es) and Polya’s problem-solving process was higher
than the criterion at the .05 level of significance.
[44]
Recommendations
4. Recommendations
4. 1 General recommendations: The study revealed that the efficiency of mathematics
learning activities have to process and result in efficiencies of 76.77/77.45 to higher than the threshold
of 70/70, the computational thinking skills, and the achievement of the students to higher than the
threshold of 70% was significant at the .05 significant level. the general recommendations for the
application of research findings are as follows.
4.1.1 Since students are not familiar with the mathematics learning activities by using the
inquiry process with CPS, teachers should study the steps of the learning activities and explain the steps
in solving problems clearly. Also, the problems in the worksheets should not be too complicated.
4 . 1 . 2 During the activities, teachers must carefully observe the behavior of the students
and advise those who have doubts.
4. 1. 3 Because the mathematics learning activities by using the inquiry process with CPS
focus on encouraging students to build their own knowledge, teachers should serve as facilitators or
mentors, giving advice and asking students to stimulate them to think rather than directly telling them
the answers. As a result, students are encouraged to adjust their thinking and better understand the
content.
4.1.4 The mathematics learning activities by using the inquiry process with CPS focus on
enabling students to solve problems in a step-by-step manner, so teaching and learning take quite a lot
of time. Therefore, teachers should plan and manage time properly.
4. 1. 5 The worksheets for group activities should be concise and not contain too much
content because it may cause students to be confused and fail to understand the activities, causing
boredom, and it may take too much time to complete the worksheets.
4. 2 Recommendations for further research: The recommendations for further research
findings are as follows.
4.2.1 Learning management activities that promote and develop the ability to write problem-
solving steps should be studied.
4.2.2 Online materials that can be used with other learning activities that promote and develop
computational thinking skills should be researched.
References
Aho, A. V. (2012). Computation and Computational thinking. The computer journal, 55(7), 832-835.
doi:10.1093/comjnl/bxs074
Boonchuvong. (1995). Principles of teaching. Bangkok: Suan Dusit Rajabhat Institute.
Brahmawong, C. (2013). Developmental testing of media and instructional package. Silpakorn
Educational Research Journal. 5(1), 5-20.
Cinnamon, K. M., Matulef, N. J., & Silverman, R. (1979). Creative Problem-solving (Applied Skills
Training Series). San Diego: Pfeiffer & Co.
Dandy Lion Publications. (1986). Primarily Problem-solving: Creative Problem-solving Activities.
San Luis Obispo, CA 93401.
Donald J. Treffinger, Scott G. Isaksen, & K. Brian Dorval.(2010). Creative Problem-solving (CPS
Version 6.1™) A Contemporary Framework for Managing Change. Creative Problem Solving
Group, Inc. P. O. Box 53169 P. O. Box 648 Sarasota, FL 34232 Orchard Park.
Gonzalez, M. R. Gonzalez, J.P., & Fernandez, J. (2016). Which cognitive abilities underlie
computational thinking? criterion validity of the computational thinking test. Computers in
human behavior. doi:10.1016/j.chb.2016.08.047
Indiana Creative Problem-Solving Initiative Blumberg Center (2003). Indiana Creative Problem-
solving Initiative-Center for Creative Learning accessed August 20, 2003. [Online]
http://www.indetate.edu/ce/Blumberg/epl.html.
[45]
Institute for the Promotion of Teaching Science and Technology. (2003). Management of science
learning, basic education course. Bangkok: Institute for the Promotion of Teaching Science
and Technology.
Isaksen,D., & Treffinger. (1994). Creative Approaches to Problem-solving. Kendall/Hunt,
Jenjit, A. (2002). Geometry teaching activities using creative problem-solving for high school
students with mathematical abilities. Dissertation in Mathematics Education, Faculty of
Education, Graduate School, Srinakharinwirot University Prasarnmit.
Laohapaiboon, P. (1999). Science Teaching Approaches. Bangkok: Thai Wattana Panich.
Leonard, J., Buss, A., Gamboa, R., Mitchell, M., Fashola, O.S., Hubert, T., & Almughyirah, S.
(2016). Using robotics and game design to enhance children's self-efficacy, STEM attitudes,
and computational thinking skills. Science education technology, 25, 860-876.
McKenna, J. (2017). Computational thinking in STEM classroom. [Online]
https://robomatter.com/blog-ct-in-stem-classroom/
Ministry of Education. (1998). Educational Development Manual, B.E. 2541. Bangkok: Ministry of
Education.
Ministry of Education. (2017). Indicators and Mathematics Core Learning Content (Revised Edition,
B.E. 2560, according to Basic Education Core Curriculum, B.E. 2551). Institute for the
Promotion of Teaching Science and Technology.
Mitchell, W.E and Kowalik, T.F. (1999). Creative Problem Solving. NUCEA: Genigraphict Inc.
Moolkum, S., & Monlkum O. (2002). 21 Ways to Manage Learning to Develop a Thinking System.
Bangkok. Parbpim Printing.
Pinmun. J. (2014). The effect of synthesization between inquiry cycle (5Es) and Polya’s problem-
solving process on the topic of application of linear equation with one variable for
Mathayomsuksa II students. Master’s thesis, Graduate School, Burapha University.
Poompachati, P. (2009). Enhancing to creative problem-solving capability for young children.
Silpakorn Educational Research Journal. 13(2), 56-64.
Sittikhetkron, W., & Sawangmek, S. (2021). Development of computational thinking skills through
5Es inquiry learning activities with board game and Formula Coding on the poupoation in
pandemic for Grade 12 students. Journal of Education Naresuan University. 23(3), 286-300
Suksri-ngam, P. (2003). Understanding inquiry approach. Mahasarakham University, Mahasarakham
Tananta, P. (2019). A development of mathematics learning activities by using inquiry process with
Polya’s problem-solving to promote mathematics problem-solving ability in Matthayomsuksa 5
students. International conference on education and global studies. 3(1).
Treffinger, J. D., Isaksen, G. S., & Dorval, K. B. (2003). Creative problem-solving (Cps version 6.1) A
contemporary framework for managing change. Journal of Creative Problem Solving Group.
[Online] http://vm-
webgd.led.ufsc.br/moodledata/webgd_community/954911716_CREATIVE%20PROBLEM%2
0SOLVING.pdf [2011,September 3].
Worakham, P. (2016). Educational Research. 8th edition. Maha Sarakham: Tasila Printing.
[46]
Education and Information Technologies (2022) 27:8289–8310
https://doi.org/10.1007/s10639-022-10921-z
Relationships between computational thinking and the

quality of computer programs
Kay-Dennis Boom1 · Matt Bower2 · Jens Siemon1 · Amaël Arguel3
Received: 5 September 2021 / Accepted: 27 January 2022 / Published online: 3 March 2022
© The Author(s) 2022
Abstract
Computational thinking – the ability to reformulate and solve problems in ways that
can be undertaken by computers – has been heralded as a foundational capability
for the 21st Century. However, there are potentially different ways to conceptualise
and measure computational thinking, for instance, as generalized problem solving
capabilities or as applied practice during computer programming tasks, and there
is little evidence to substantiate whether higher computational thinking capabilities
using either of these measures result in better quality computer programs. This
study examines the relationship between different forms of computational thinking
and two different measures of programming quality for a group of 37 pairs of pre-
service teachers. General computational thinking capabilities were measured using
Bebras tests, while applied computational thinking processes were measured using
a Computational Thinking Behavioural Scheme. The quality of computer programs
was measured using a qualitative rubric, and programs were also assessed using the
Dr Scratch auto-grading platform. The Test of Nonverbal Intelligence (3rd edition,
TONI-3) was used to test for confounding effects. While significant correlations
between both measures of computational thinking and program quality were detect-
ed, regression analysis revealed that only applied computational thinking processes
significantly predicted program quality (general computational thinking capability
and non-verbal intelligence were not significant predictors). The results highlight
the importance of students developing applied computational thinking procedural
capabilities more than generalized computational thinking capabilities in order to
improve the quality of their computer programs.
Keywords Computational thinking · Visual programming · Scratch · Program

quality
Extended author information available on the last page of the article
1 3
8290 Education and Information Technologies (2022) 27:8289–8310
1 Introduction
1.1 Context of the problem
Since its first major appearance in 2006 by Wing, computational thinking has been
intensively discussed in the field of computer science education (Tang, Chou, & Tsai,
2020). CT can be regarded as the ability to reformulate problems in ways that com-
puters can then be used to help solve those problems (International Society for Tech-
nology in Education [ISTE] & the Computer Science Teachers Association [CSTA],
2011). The value proposition of computational thinking capabilities in a digital age is
that they can help people solve a range of problems that lead to personal satisfaction
and success, not only in the technology area but also life more broadly. However, the
conjecture that possessing computational thinking knowledge, or applying computa-
tional thinking skills while solving problems, leads to higher quality solutions, has
rarely been empirically validated.
One aspect of computational thinking that is often emphasized by advocates is
that it is not simply computer programming capability. Research about the effects
of computational thinking knowledge and/or skill can be divided into the area of
effects regarding computational problem solving (e.g. computer programming) and
effects regarding diverse non-programming problems or tasks. For example, a wide
range of problems, from finding the shortest route between map locations to design-
ing an online shopping platforms, rely on people applying computational thinking
processes while they are writing computer programs to solve those problems. How-
ever, computational thinking skills (such as problem decomposition, pattern recogni-
tion, algorithmic thinking and abstraction) can also be used to solve a range problems
that do not involve computer programming, such as finding a way through a maze
or specifying the steps in a dance sequence. While learning computer programming
relies on people utilizing and applying computational thinking as part of the process
they undertake, instructional settings will often use computational thinking founda-
tions to teach subjects and ideas that do not involve computer programming (e.g.
Bull, Garofalo, & Hguyen, 2020). In fact, a literature review conducted by Tang et al.
(2020) concluded that there were far more computational thinking effects analyzed
in subject areas not related to computer science (n = 240) than for effects related to
computer science (n = 78). However, we note that while computational thinking can
be applied in a range of disciplines, it is considered absolutely essential and funda-
mental to successful computer programming (Angeli & Giannakos, 2020; Lye &
Koh, 2014). Yet, we could not find any studies amongst the literature that examined
whether or not computational thinking capabilities did in fact relate to higher quality
computer programs.
The purpose of this study is to evaluate the extent to which general computa-
tional thinking knowledge, as well as computational thinking processes applied dur-
ing problem solving tasks, influence the quality of computer-programming solutions.
This was achieved by comparing university students’ computational knowledge (as
measured by Bebras tests) and the computational thinking processes observed while
they wrote computer programs with the quality of the final computing products that
1 3
Education and Information Technologies (2022) 27:8289–8310 8291
they produced. The findings of this study have implications for how computational
thinking is framed, conceptualized and emphasized within education and society.
2 Literature review
2.1 Defining computational thinking and its subcomponents
Computational thinking is generally seen as an attitude and skill for solving prob-
lems, designing complex systems, and understanding human thoughts and behav-
iors, based on concepts fundamental to computer science (Lye & Koh, 2014). Recent
reviews of computational thinking definitions and components by Shute, Sun and
Asbell-Clarke (2017) and Ezeamuzie and Leung (2021) point out the lack of consis-
tent definition regarding what is meant by computational thinking, though with some
terms being more popular (such as abstraction, algorithm design, decomposition, and
pattern recognition as generalisation), particularly when academics devise explicit
definitions with relation to their research. Some inconsistency between definitions
of components can occur, at times not because there is disagreement about what
computational thinking involves, but because other frequently used terms such as
‘sequencing’, ‘conditional logic’ and ‘loops’ can conceptually fall within overarching
categories (in this case, ‘algorithm design’).
In this study, we will draw upon generally accepted core components of compu-
tational thinking as being comprised of problem decomposition, pattern recognition
(generalisation) algorithmic thinking and abstraction, which accords with other defi-
nitional work from the research field (Angeli & Giannakos, 2020; Cansu & Cansu,
2019; Tsai, Liang, Lee, & Hsu, 2021). We acknowledge that there are other aspects
of computational thinking that are identified in some studies, such as ‘parallelism’,
‘data collection’, and ‘modelling’, as outlined by Shute et al. (2017), however, as
Ezeamuzie and Leung (2021) points out, these sorts of other terms are relatively
uncommon, and they are not processes utilised for all computational thinking prob-
lems. Selecting problem decomposition, pattern recognition (as generalisation), algo-
rithmic thinking and abstraction as the components of computational thinking in this
study also corresponds with approaches adopted in industry (for example, Csizmadia,
Curzon, Dorling, Humphreys, Ng, Selby, & Woollard, 2015; McNicholl, 2019).
One crucial part of any computational thinking task is problem decomposition, the
division of a problem into smaller chunks. Problem decomposition has been identi-
fied as a general problem solving strategy well before the advent of computational
thinking (Anderson, 2015). In computational problems, decomposition is particularly
important because of its relationship to modularity, where the complexity of a task
can be simplified by identifying smaller parts that can each be addressed separately
(Atmatzidou & Demetriadis, 2016). For example, when programming a multimedia
story, one might first identify the different scenes that occur, and then break each
scene into a series of actions by the characters.
Another component of computational thinking is abstraction, in terms of ignoring
unimportant details and instead focusing on relevant information. From a psycho-
logical perspective, abstraction is a thought process that is used to achieve organised
1 3
thinking (Shivhare & Kumar, 2016). In computational problems, abstraction enables

people to concentrate on the essential, relevant, and important parts of the context
and solution (Thalheim, 2009). For instance, when writing a multimedia story to
have characters dance about a screen, a person may recognise that their program only
needs to attend to the coordinates of the characters and their size, and the routines
that they write can be applied to numerous characters irrespective of their colours or
costumes.
A further critical thought process when engaging in computational thinking is
pattern recognition. Pattern recognition involves being able to infer rules based on
observations and apply these rules to instances that have never been encountered
(visa vi Posner & Keele, 1968). Pattern recognition is crucial when solving computa-
tional problems, because rules inferred based on observations can then be translated
into instructions that can be used to solve problems. For instance, when a person
realises that a square can be drawn by drawing a straight line and then turning 90
degrees four times, then they can easily and efficiently specify a set of instructions
for a computer (or human) to execute the process. It is important to note that pattern
recognition is closely related to abstraction as a form of ignoring irrelevant details,
but is generally regarded as distinct by virtue of distilling those aspects of a situation
that repeat or reoccur in certain ways.
The fourth computational thinking category in this study is algorithmic thinking.
An algorithm is a well-defined procedure or ‘recipe’ that defines how inputs can be
used to achieve a specific objective (Cormen et al, 2014; Sipser, 2013). Algorithmic
thinking has roots in cognitive psychology in the form of scripts, that help people
to know how to behave in social or behavioural contexts (for instance, going to a
restaurant or playing a game, see Schank & Abelson, 1977). When solving computa-
tional problems, algorithmic thinking enables people to translate their abstract ideas
and the patterns that they recognise into a set of procedures, for instance having a
robot trace out a square and then dance on the spot. For the purposes of this study,
algorithmic thinking also includes the thinking required to resolve errors that occur
in early versions of algorithm designs (the process known in computing as ‘debug-
ging’), thus overcoming issues associated with delineating these two intrinsically
interrelated processes.
2.2 Ways of measuring computational thinking
When defining a skill, the question arises whether it is possible to measure and differ-
entiate it from other, possibly overlapping or more general skills. For computational
thinking, the existing measurement methods that can be broadly divided into assess-
ment of computational thinking as knowledge that is applied or tested (input), the
assessment of computational thinking as a skill observed during a problem solving
activity (process) and (theoretically also) the assessment of computations thinking by
analyzing the result of a task (output). All measures are subsequently used as indica-
tors for the existence and the grade/level of the respective type of the computational
thinking competence.
The most internationally well-known instruments for measuring general computa-
tional thinking knowledge are the Bebras Challenges. The main idea behind Bebras
1 3
Challenges has been to create abstract, non-computing problems that require specific
cognitive abilities rather than technical knowledge or coding experience (Dagienė &
Sentance, 2016). Examples of Bebras tasks can be found at https://www.bebras.org/
examples.html. Different studies have shown that abilities such as breaking down
problems into parts, interpreting patterns and models, and designing and implement-
ing algorithms are needed to solve Bebras problems (Lockwood & Mooney, 2018;
Araujo, Andrade, Guerrero, & Melo, 2019). There are also other approaches to mea-
suring general computational thinking knowledge, both within computer program-
ming contexts and also other disciplinary contexts, many of which have been applied
in the training of teachers. For instance, Zha, Jin, Moore, and Gaston (2020) used
multiple choice knowledge quizzes about computational thinking and Hopscotch
coding to measure the impact of a team-based and flipped learning introduction to
the Hopscotch block coding platform. In a study exploring the effects of a 13 week
algorithm education course on 24 preservice teachers, Türker & Pala (2020) used the
“Computational Thinking Skills Scale” (CTSS, from Korucu, Gencturk & Gundogdu,
2017) comprising the computational thinking facets creativity, algorithmic thinking,
collaboration, critical thinking and problem solving. Suters and Suters (2020) report
on a paper-and-pencil based computational thinking knowledge assessment to mea-
sure the effects of an extend summer institute for middle school mathematics teach-
ers (n = 22) undertaking training in computer programming with Bootstrap Algebra
and Lego® Mindstorms® robotics. The content assessment consisted of items that
integrated mathematics common core content with facets of computational thinking,
in line with research endeavors recognizing the need to contextualize computational
thinking within specific disciplines (Gadanidis, 2017; Grover & Pea, 2013; Weintrop
et al., 2016). All of these approaches to computational thinking knowledge assess-
ment share an emphasis on short, often multiple choice, closed questioning to mea-
sure computational thinking, rather than examining the computational thinking that
arises as part of authentic and more extended problem solving contexts.
In a second variant of possible computational thinking measurement, the pro-
cess of solving a context-dependent task – mostly typically a programming task – is
observed and analyzed with regard to the abilities which are considered to be part of
computational thinking skill. Skill analysis based on observations is a comparatively
underdeveloped field. Brennan & Resnick (2012) seminally investigated the com-
putational thinking processes and practices that children undertook while designing
their programs using the visual programming platform Scratch, noting that “fram-
ing computational thinking solely around concepts insufficiently represented other
elements of students learning” (p. 6). Their qualitative observations and interviews
identified computational thinking practices such as being incremental and iterative,
testing and debugging, reusing and remixing, and abstracting and modularizing.
However, their results were not reported based on any sort of observational coding
of participants, so that there is no indication of time spent on each of these processes
while solving computing problems.
While analysis of learner pre- and/or post- interview narratives has been previ-
ously conducted to determine evidence of computational thinking (Grover, 2011;
Portelance & Bers, 2015), we were not able to find any computational thinking analy-
ses involving systematic examination of narratives emerging from participants while
1 3
they were solving authentic programming problems. However, there are examples
of observing and thematically categorizing computer programming processes and
narratives (Bower & Hedberg, 2010; Knobelsdorf & Frede, 2016). These approaches
provide a basis for in-situ observation and subsequent qualitative analysis of pro-
gramming activity for computational thinking constructs such as problem decon-
struction, abstraction, pattern recognition and algorithmic thinking, and our study is
based on these more systematic observational approaches.
2.3 Measuring the quality of computer programs
There are a range of qualities that can be used to evaluate the quality of computer
programs, such as the extent to which the code functionally achieves its intentions,
avoids unnecessary repetition, is well organized, and so on (Martin, 2009). Much of
the research relating to evaluating the quality of computer programs examines how
to ways of auto-assessing student work (for instance, Ihantola, Ahoniemi, Karavirta,
& Seppälä, 2010; Pieterse, 2013). However, automated tools struggle to accurately
assess computational thinking (Poulakis & Politis, 2021), and recent work points
out the need to look beyond raw functionality and ‘black-box’ testing of outputs, to
examine the inner working of code and algorithms (Jin & Charpentia, 2020). Some
research also examines the extent to which computational thinking is evident with
the final programming product itself, by virtue of the code fragments that are used
and their sophistication. Brennan and Resnick (2012) examined whether aspects of
computational thinking were present in students’ block-based Scratch programs.
Grover et al. have manually evaluated computational thinking evident in students’
Scratch programs, though without providing detail of the process and rubrics (Gro-
ver, 2017; Grover, Pea, & Cooper, 2015). An increasingly renown innovation, Dr
Scratch, combines automated assessment, examination of the inner workings of pro-
grams, and analysis of computational thinking to provide a measure of program qual-
ity for Scratch programs (Moreno-León & Robles, 2015). One study has established a
strong correlation (r = 0.682) between the Dr Scratch automated assessment of com-
putational thinking evident within students’ Scratch programs and manual evaluation
of computational thinking within Scratch programs by human experts (Moreno-León
et al., 2017). However, the computational thinking within a computer program is not
necessarily a proxy for overall program quality, and the extent to which program
quality relates to the computational thinking knowledge and computational thinking
processes of program authors is an open question.
2.4 Research question
Thus, having established the lack of empirical evidence to suggest that general com-
putational thinking knowledge or in-situ computational thinking processes is related
to computing performance, and armed with potential ways to operationalize and
measure computational thinking knowledge, computational thinking processes, and
quality of computer programs, this study examines the following research questions:
1 3
1. Is the quality of computer programming solutions that people produce related to

their general computational thinking knowledge?
2. Is the quality of computer programming solutions that people produce related to
the applied computational thinking processes that they undertake?
3 Method
3.1 Participants
The sample for this study was drawn from 74 pre-service teachers completing a digi-
tal creativity and learning course at an Australian university. Among them 68% were
female, 30% male and 2% preferred not to say. On average, participants were 23.9
years old (SD = 5.2). In terms of language proficiency, 97% indicated that they spoke
English fluently or were native speakers. In terms of prior knowledge, 97% had no or
only little prior programming experience and none of the participants were familiar
with the Scratch programming environment that was used for the study.
3.2 Instruments
3.2.1 Measuring computational thinking knowledge
To measure computational thinking knowledge as it arises in general problem solv-

ing contexts, participants solved an online version of adapted Bebras tasks. All tasks
were chosen from the Australian versions of the Bebras contests from 2014 (Schulz
& Hobson, 2015) and 2015 (Schulz, Hobson, & Zagami, 2016). Only tasks from
the oldest available age group were selected (i.e., for adolescents 16 to 18 years
of age and school levels 11 and 12, respectively). The tasks were slightly revised
and presented without any iconic beavers or other comical pictures in order to be
more appropriate for the university participants in this study. Although there is still a
considerably age gap between the targeted age group of the tasks and the actual age
of participants, it was not expected that this difference would cause any problems
(e.g., ceiling effects) because pre-service teachers on the whole were not expected
to be familiar with or particularly adept at computational thinking tasks. The scor-
ing of participant performance was based on the recommended scoring system of
the founder of the Australian version of the Bebras tasks (Schulz et al., 2016). There
were eight tasks considered as easy level (worth two points), seven medium (three
points), and five hard tasks (four points) resulting in 20 tasks in total with a maximum
achievable score of 57.
3.2.2 Observing computational thinking processes
To enable participants to demonstrate how much time they spent on computational

thinking-relevant processes while programming, participants were set a Scratch pro-
gramming task. Scratch itself was developed in 2003 at MIT Media Laboratory and
publicly launched in 2007 (Resnick et al., 2009). It is one the first and one of the
1 3
most popular open-source visual programming environments. In visual programming

environments, users connects code blocks with each other instead of actual writing a
code as common in real programming languages.
To prepare students for the task, they were given 45 min to review the Scratch tuto-
rials available from within the Scratch platform. They were also allowed to access
these tutorials during the programming task. The task itself was defined as follows:
“Program a story or a game where a hero has to overcome a challenge in order to
defeat the villain(s).”
This task was chosen because it is somewhat open-ended and can be solved in the
chosen Scratch development framework without prior programming knowledge. Fur-
thermore, computational thinking subskills (problem decomposition, pattern recogni-
tion, abstraction, algorithm design) would most likely have to be used to solve the
task. The way in which the Scratch programming environment, task, and participants
may influence the generalizability of results is considered in the Discussion section
of this paper.
To reliably assess the amount of time they spent on computational thinking pro-
cesses during their Scratch programming session, a computational thinking behavior
scheme (CTBS) was developed. The CTBS was based on event sampling, involving
analysis of how often and for how long specific behavioral cues occur. Based on the
literature review, four components were identified as main features of computational
thinking and which are the latent constructs in the CTBS: decomposition, abstraction
(as in ignoring unimportant details), pattern recognition, and designing and apply-
ing algorithms (see operationalization of these constructs in Table 1 below). Two
Table 1 Operationalization of Computational think- Behavioural indicator (manifest variables)

computational thinking con- ing components (latent
structs in the Computational variables)
Thinking Behavioural Scheme
Decomposition Put problem into pieces / building sub
(CTBS)
tasks or problems
Identifying the immediate next step
Discussing if then relations of the story or
game (is related to programming elements)
Abstraction Focusing on important information; ne-
glecting unimportant details
Simplifying anything (problem, sub prob-
lem, functions, code bocks, etc.)
Pattern recognition Identifying similar characteristics (sub
problems, functions, code blocks, etc.)
Use of copy-paste
Aha moments (must be related to an event
when student understood relationship
between things)
Algorithm design Putting code chunks together
Testing and judging algorithm (i.e., click-
ing on run or double click on sequence or
actively observing a running sequence)
Debugging - try to find error and adjust
algorithm
1 3
researchers coded five entire videos independently to assess inter-rater reliability. As

a result, at least two third of the events were identified by both raters. The range of
the frequency of agreement for the five videos laid between 66.67% and 72.50% and
the κ coefficients ranged from 0.58 to 0.67. Overall, the reliability can be interpreted
as moderate (Landis & Koch, 1977). Note that the CTBS measured the time spent on
computational thinking components, not the correctness of the computational think-
ing processes. It is to be expected that during the process of solving computational
problems that people may not always immediately have correct thoughts about the
right course of action, and this study sought to examine relationship between the time
spent on computational thinking processes and the quality of programming products.
3.2.3 Measuring program quality
To measure participants’ program quality, two measures were used. For one, a rubric
scheme loosely based on “Clean code” of Martin (2009) was developed specially for
this study. The program quality criteria were based on five categories: richness of
project, variety of code usage, organization and tidiness, functionality of code and
coding efficiency. Richness of project described how much was happening in the
Scratch project. Lower scores were given when only one element was programmed to
perform only one behaviour, while Scratch projects consisting of several programmed
elements that were related to each other received higher scores. The variety of code
usage depended on the kinds of code blocks were used. Scratch projects were rated
lower when they mainly consisted of simple code chunks such as motion or looks and
high when more advanced chunks like control or sensing were used. The category
organization and tidiness took into account the extent to which the control section in
Scratch was organized, with more organized Scratch projects receiving higher scores.
Functionality was assessed based on whether the intention of the Scratch project
was clear and whether it worked as intended. Projects received higher scores when
they ran smoothly and the intention was easy to understand. The category efficiency
Figure 1 Two examples of the same function but coded differently. An example with unnecessary dupli-
cates is shown on the left and a more efficient version is seen on the right
1 3
described the usage of code controlling the flow of execution, and the number unnec-
essary duplications. Lower scores were given to projects having many such dupli-
cates, while more generalized and more abstract code scripts received higher scores.
An example of a program with unnecessary duplication is shown in Fig. 1 (left),
compared to a more efficiently represented code block in Fig. 1 (right).
The five code quality categories were all rated on a scale including 0 (not evident),
1 (poor), 2 (satisfactory), 3 (good), up to 4 (excellent). A weighted mean over all cat-
egories was calculated to provide a general assessment. The weight for each category
was based on their importance for program quality, resulting in extent and richness,
variety, and functionality being weighted 20% each, efficiency 30%, and organization
and tidiness 10% to the weighted mean. Quality criteria and the (weighted) scoring
system of the scheme were discussed with two computer science education profes-
sionals to uphold the content validity of the measure. In addition, one of the CS
education professionals rated the Scratch projects to obtain reliability assessment.
Inter-rater reliability was high with ICC(3,1), 95% CI [0.87, 0.96].
The second measure for program quality was based on Dr Scratch (Moreno-León
& Robles, 2015). Dr Scratch provides a measure of program quality based on seven
dimensions relevant to CS: abstraction and problem decomposition, parallelism,
logical thinking, synchronization, algorithmic notions of flow control, user interac-
tivity and data representation. Dimensions are judged as 0 (not evident), 1 (Basic), 2
(Developing), and 3 (Proficient). Scores are aggregated over all dimensions resulting
in a total evaluation score (mastery score) from 0 to 21. Mastery scores between 8
and 14 are regarded as general developing; lower than 8 is regarded as generally
basic, and more than 14 as general proficient. High correlations between Dr Scratch
mastery scores and experts judgments of program quality can be used as an indica-
tor of satisfactory criterion validity (Moreno-León, Román-González, Harteveld, &
Robles, 2017). While Dr Scratch focuses primarily on computational thinking ele-
ments as opposed to other aspects of computer programming (e.g. organization of
code, efficiency), it is based on the final computer programming solution that is pro-
duced, and thus provides an interesting alternative measure of program quality for
this computational thinking study.
3.2.4 Test of nonverbal intelligence
To take account for potential confounding effect, participants’ nonverbal intelligence

was also measured. For this the Test of Nonverbal Intelligence (3rd edition; TONI-3,
Brown, Sherbeernou, & Johnson, 1997) was used. The TONI-3 is a classic culture
fair test (i.e., minimally linguistically demanding) and as in many of them partici-
pants need to recognize a correct figure in a set of abstract and geometrical pictures.
The test consists of 45 items and has an average testing time of 15 min and has a
satisfactory level of psychometrical properties (Banks & Franzen, 2010).
3.3 Procedure
Initially, participants completed the Bebras Computational Thinking Knowledge test

and the test of nonverbal intelligence (TONI-3 online). One week later, the second
1 3
phase took place at university’s classrooms and participants attempted the task in
Scratch. To collect rich video material with many verbal and nonverbal indicators for
the research team to analyze, participants were organized in pairs. It was hoped that
working in pairs would encourage participants to talk and engage more with each
other. The pairs were formed based on similar Bebras scores to minimize any effects
due to large differences in competences. In total, 37 pairs were formed and filmed
while working on the task, forming the corpus for the analysis.
3.4 Analysis
All statistical analysis was conducted using the R statistics programming environ-
ment. In order to acquire a sense of the data, basic descriptive statistics including
means and standard deviations were calculated for all five measures (Bebras scores,
time spent on different computational thinking processes, program quality rubric
score, Dr Scratch score, TONI-3 non-verbal intelligence score). Because participants
worked on the programming task in pairs, all programming assessments based on the
rubric scheme, the additional Scratch evaluation assessment based on Dr Scratch,
and the assessment of how much time participants spent on computational think-
ing behavior based on CTBS, were paired values. Scores on the Bebras tasks and
TONI-3 test were averaged for each pair. Of the 37 pairs of participants who agreed
to complete the Bebras test and have their final programs used in the study, 27 agreed
to be video recorded for the purposes of the CTBS analysis, and 32 pairs agreed to
complete the TONI-3 test.
Spearman’s ρ were computed between all five measures using all available data,
to determine whether the underlying variables were directly correlated. Finally, in
order to account for the possibility of moderating variables, two regression models
were estimated with the two program quality measures as outcomes (program quality
rubric score and Dr Scratch score). These two regression models used the Bebras task
scores, the CTBS, and the TONI-3 IQ scores as predictors, so that it was possible to
detect if any of these were moderating variables.
4 Results
4.1 Descriptive statistics and measurement outcomes
The average score for the measure of general computational thinking capability
(Bebras task) was 57.03% (SD = 18.6%). The range was from 21% to one participant
who achieved 100%. Results indicated a medium level of test difficulty, with no
serious problems due to ceiling or floor effects. In TONI-3-IQ, participants achieved
an average intelligence score of 113.12 (SD = 14.17). The mean of this sample was
slightly higher than the expected value of the population (µ = 100, see, for example,
Sternberg, 2017), which can be explained by the fact the sample was drawn from
university students. The time participants needed to complete the Bebras tasks (Md
= 55 min) and the TONI-3-IQ (Md = 22 min) roughly aligned with the expected time
of 60 min (Dagienė & Futschek, 2008) and 15 min (Brown et al., 1997), respectively.
1 3
Table 2 shows that while writing their programs, coded participants spent nearly
half of their time on computational thinking behaviors, with algorithmic design hav-
ing the largest contribution and little time spent on decomposition and pattern rec-
ognition. Pattern recognition was observed in less than half of all pairs. No sign of
abstraction in the sense of neglecting information was observed for any pair.
Table 3 contains an overview of scores achieved by the pairs of participants in
the rubric scheme for program quality. The full range of the rating scales (0 to 4)
was used. The distributions of all five dimensions had their center at around 2 (i.e.,
satisfactory level).
Table 2 Overview of Coded Computational thinking Pairs events % of time spent on CT-
Events and Time Spent on component relevant behavior
Computational Thinking
M (SD) Max - Min
Behavior
Decomposition 27 310 7.77 (5.35) 22.61
– 1.03
Abstraction - - - -
Pattern recognition 17 53 1.43 (1.05) 3.75
– 0.18
Algorithmic design 27 1,072 37.46 61.06
(12.26) – 10.39
Computational thinking 27 1435 46.66 70.42
overall (14.96) – 15.74
Table 3 Overview of Rubric Programming dimensions M (SD)

Scheme for Programming
Extension 1.86 (0.89)
Quality
Variety 2.19 (1.02)
Organization 1.84 (0.87)
Functionality 1.92 (0.95)
Efficiency 2.08 (1.21)
Note: pairs = 37 Weighted mean 2.00 (0.91)
Table 4 Overview of Dr Scratch Project Scores

Dr Scratch dimension Absolute frequency of level M (SD) Mdn
0 1 2 3
Abstraction and problem decomposition 2 35 - - 0.95 (0.23) 1
Parallelism 5 21 4 7 1.35 (0.95) 1
Logical thinking 15 20 2 - 0.65 (0.59) 1
Synchronization 14 11 1 11 1.24 (1.26) 1
Flow control - 9 28 - 1.76 (0.43) 2
User interactivity - 11 25 1 1.73 (0.51) 2
Data representation 1 22 14 - 1.35 (0.54) 1
1 3
In comparison, participants’ Scratch projects typically only achieved a basic rating

according to Dr Scratch, with only two dimensions rated as developing (See Table 4).
4.2 Correlations between variables
As a first step towards analyzing which of the two computational thinking measures
(general computational thinking knowledge as measured by Bebras versus compu-
tational thinking processes as observed in practice) have a greater relationship to
program quality, Spearman’s ρ were computed (see Table 4). Correlation between
the two measures of program quality (weighted means based on the developed rubric
scheme and Dr Scratch mastery scores) revealed a significant relationship, ρ = 0.61,
p < .001. Based on common interpretation of effect sizes (Cohen, 1988), this cor-
relation can be interpreted as large. The large correlation between the two measures
of program quality reveals a degree of consistency in their assessment of student
programs.
Significant positive correlations were found between general computational think-
ing knowledge (Bebras scores) and both measures of program quality, with a bor-
derline small to medium effect sizes (see Table 5). Significant positive correlations
between time spent on computational thinking processes while programming and
both measures of program quality, with quite large effects.
Because of some potential (partial) conceptual overlaps between nonverbal intel-
ligence and computational thinking, the correlations between the TONI-3 IQ and
computational thinking measures were calculated as well. On one hand, the correla-
tion between the TONI-3 IQ and the Bebras scores was significant and positive with
a medium to large effect size, ρ = 0.49, p = 0.002. On the other hand, correlation
between TONI-3-IQ and time spent on computational thinking processes while pro-
gramming was not statistically significant, ρ = 0.09, p = 0.346
4.3 Regression analysis
As explained in the Methodology section, two regression models were estimated with
both program quality measures as outcomes and the both computational thinking
measures and the TONI-3 IQ scores as predictors. Standardized parameter estima-
tions and tests of significance of the regression model are shown in Table 6. The
regression models only partly supported the findings from the correlations. The pos-
Table 5 Spearman’ ρ Correla- Program- Dr Scratch N

tions Between Programming ming rubric mastery (pairs)
Quality, Dr Scratch and Differ- scheme score
ent Measures
ρ p ρ p
Bebras score 0.32 0.027 0.29 0.041 37
Time of computational think- 0.65 < 0.57 0.001 27
ing behavior 0.001
Note: one-sided p-values IQ based on TONI-3 0.29 0.055 0.19 0.149 32
1 3
Table 6 Regression Models

Program quality Dr Scratch mastery score
Predictors β t-value (SE) p β t-value (SE) p
Bebras score −0.41 −1.95 (1.24) 0.066 −0.14 −0.62 (4.27) 0.542
Time on computation- 0.74 4.31 (0.01) < 0.001 0.70 3.86 (0.03) <
al thinking behavior 0.001
(overall)
TONI-3 IQ 0.36 1.82 (0.01) 0.084 0.11 0.53 (0.05) 0.599
R² (R²adj) 0.50 (0.42) 0.44 (0.36)
F(3,20) 6.60 0.003 5.29 0.008
Note: N = 24. The intercept is omitted for better overview
itive correlation between the Bebras score and both measures of program quality
vanished when taking into account the effect of TONI-3 IQ. The only significant
predictor for both measures of programming quality was the computational thinking
process scores.
Post hoc analyses for both regression models were performed for power estima-
tion. Based on the given parameters (N = 24, number of predictors = 3, effect size =
R2pro.qual = 0.50, R2DrScratch = 0.44, and α = 0.05), a power of > 0.99 for both models
was achieved. Because of the small sample size, assumptions about linear multiple
regressions such as homoscedasticity, multicollinearity, and residuals were rigor-
ously checked. No serious violations of any assumption could be found, though it
should be noted that the residuals when the outcome was programming quality were
not normally distributed, based on Shapiro-Francias test, with W’ = 0.88, p = 0.011.
In conclusion, the power of both regression models were sufficiently high enough and
the regression coefficients can be interpreted as “best linear regression estimations”.
5 Discussion
The general computational thinking knowledge scores (Bebras) and the computational
thinking procedural performance (as indicated by the CTBS), were both positively
correlated with both program quality measures (the rubric scheme and Dr Scratch
mastery score). Therefore, a general interpretation could be that the higher the level
of both general computational thinking knowledge and applied computational think-
ing in practice, the better the program quality. However, this interpretation would be
premature because regression analyses revealed that only one — the applied compu-
tational thinking in practice — was a significant predictor of program quality when
controlling for other variables such as the level of nonverbal intelligence and general
computational thinking knowledge. The reason why the two different computational
thinking measures predict programming differently might lie in different perspec-
tives underlying the two different measures of computational thinking, and how these
might mediate the relationship with program quality.
The Bebras tasks focus on general and conceptual aspects of computational think-
ing. Correlations between the Bebras score and the TONI-3-IQ were between mod-
erate and strong. As for the most instruments for nonverbal intelligence, TONI-3 is
based on pictures in which participants need to identify similar instances and recog-
1 3
nize patterns. Many of the Bebras tasks are designed in a similar fashion. The original
idea behind the Bebras tasks was to create a test about CS concepts “independent
from specific systems” to avoid contestants being dependent on prior knowledge of
any specific IT system (Dagienė & Futschek, 2008, p. 22). This may have led to some
items being similar to those of nonverbal intelligence tests.
As found in some prior studies, this also caused confusion for some Bebras con-
testants. Vaníček (2014) asked participants for their opinions about the Bebras tasks.
Some questioned the purpose and validity of the test, stating “I wonder what the con-
test questions have to do with informatics. Maybe nothing at all?”. If (at least some)
Bebras tasks are similar to those of nonverbal intelligence tests and there is a high
and significant positive correlation between both measures, it is possible that both
tests measure similar constructs. This would explain why the relationship between
the Bebras scores and program quality vanished when controlled for TONI-3-IQ.
The Bebras tasks are validated by several studies (Dagienė & Stupuriene, 2016; Dol-
gopolovas, Jevsikova, Savulionienė, & Dagienė, 2015; Lockwood & Mooney, 2018)
but none of these studies controlled for any potential confounding effects on simi-
lar psychological constructs such as nonverbal intelligence. We could only find one
study in which the potential relationship between the Bebras tasks and nonverbal
intelligence has been discussed, with similar findings to our study (Román-González,
Pérez-González, & Jiménez-Fernández, 2017). Thus, it is possible that the Bebras
tasks indeed measure computational thinking but mainly the facet of abstract think-
ing and pattern recognition.
It is possible that these abstract parts of computational thinking alone are not a
good predictor of programming quality because extensive cognitive effort is required
to transfer the skills for application in different situations and settings. Even though
some similar skills are required to solve both kinds of tasks (the Bebras tasks as well
as the programming task in this study), it would require a high level of transfer-
ability from these abstract logical quizzes to real applied programming situations.
Moreover, according to the authors of the Bebras tasks, participants need to apply
the same cognitive abilities as needed for programming tasks such as problem decon-
struction, thinking abstractly, recognizing patterns, and being able to understand,
design, and evaluate algorithms (Dagienė & Sentance, 2016). However, the content
of the Bebras tasks (as for many ‘unplugged’ methods) is very different from real
programming tasks. This may lead to general computational thinking as measured
by Bebras tasks not providing a good predictor of program quality above and beyond
that which is captured and controlled for by general measures of intelligence (such
as the TONI-3-IQ).
In our opinion, the results can be well explained in terms of the thesis of dispropor-
tion between application extensity and intensity of use (Weinert, 1994). This theory
asserts that, the more general a rule or strategy is, the more minor its contribution to
solving challenging, content-specific problems. This could also apply to the com-
putational thinking skills of the Bebras tasks. The measured skills are very general
and partly overlap with general facets of intelligence. Their contribution to solving
a challenging, content-specific problem might therefore be rather small and statisti-
cally hard to detect. At least, this would be one possible interpretation of the rather
1 3
weak correlation and the lost connection in the regression analysis with regard to
general computational thinking knowledge and program quality.
In contrast to the Bebras tasks, the focus of the CTBS lies on participants’ applied
computational thinking processes in practice. Correlations indicated that the more
participants spent on applied computational thinking processes, the better the pro-
gramming quality of their Scratch project. It must be pointed that this was mostly
due to algorithmic design, with algorithmic design being the most frequently applied
computational thinking activity measured. As stated before, participants were work-
ing on their code from the start of the session and so there is a logical interpretation
that the longer and the more participants spent on algorithmic design, the better the
quality of their programs. Even after controlling for other measures, this relationship
was still significant and persisted in both regression models with the programming
quality rubric and Dr Scratch project evaluation as outcome, respectively. What is
even more remarkable is that computational thinking processes were significantly
correlated with program quality even though the correctness of the computational
thinking processes was not assessed in this study. That is to say, that the more time
spent thinking about computational thinking components while solving the comput-
ing problem led to better quality programming solutions, even when at times that
computational thinking may not have necessarily been ‘right’. This is in line with the
learning concept of ‘productive failure’, where thinking deeply about problems and
exploring incorrect solutions can ultimately lead to greater learning overall (Kafai,
De Liema, Fields, Lewandowski & Lewis, 2019).
These results indicate that the computational thinking process capabilities observed
by the CTBS are more strongly related to program quality than computational think-
ing knowledge as measured by Bebras. While the Bebras Challenge is undoubtedly
a valuable competition for students worldwide, the results from this study indicate
the ability to solve Bebras problems may not be a good indicator of the ability to
solve authentic informatics problems that involve computer programming. In fact,
the result challenges the premise that generalised computational thinking knowledge
underpins the ability to solve authentic programming problems to any substantial
extent. The capacity to apply computational thinking processes in-situ has been
shown in this study to be far more relevant and influential in terms of being able to
derive high quality programming solutions than solving general computational think-
ing knowledge problems. To this extent, from a pedagogical perspective, educators
who wish to use computational thinking as a basis for improving the ability of their
students to solve programming problems should focus on developing students’ abili-
ties to apply computational thinking processes in practice (algorithm design, prob-
lem decomposition, pattern recognition, abstraction) rather than their computational
thinking knowledge in a more detached and decontextualized sense.
5.1 Limitations of the Study
In this study, students worked together in pairs as a naturalistic way to provoke social
interaction and make otherwise unobservable thoughts accessible. This contributed
to the authenticity of the study, with pair programming often occurring in industry
and education. Moreover, pair-programming settings have been used in prior stud-
1 3
ies in terms of measuring computational thinking and programming knowledge for

novices (Denner, Werner, Campe, & Ortiz, 2014; Wu, Hu, Ruis, & Wang, 2019).
However, this approach involved some inherent challenges. It was not possible to
perfectly group pairs according to identical levels of computational thinking, intelli-
gence, or programming quality. Some might argue that the results and overall conclu-
sion might have been different if all measures were obtained and analyzed solely on
an individual basis. However, gauging individual measures of computational think-
ing programming skills from a behavioral perspective also involves challenges, as
it is difficult to encourage individual participants to verbalize their thinking for the
entire duration of the programming process. We believe that the benefits of analyz-
ing computational thinking arising from a more naturalistic setting outweigh those
from measuring the computational thinking of individuals, in terms of the validity of
results.
It is also worth mentioning that the CTBS and the programming quality instrument
were designed specifically for the purpose of this study. That means these instruments
have not been tested in other studies yet. Interrater reliability assessments indicated
a satisfactory level of agreement, but the results based on CTBS and programming
quality rubric scheme deserve to be interpreted with caution. Some indicators of
the computational thinking behavior are dependent on the environment used. For
instance, the computational thinking component algorithmic design category of the
scheme and encompasses all utterance and actions with the purpose of designing an
algorithmic solution to a problem. The programming task in this study was designed
in Scratch, for which the only way to create algorithmic solutions was to drag and
drop code chunks together. If another programming environment were used, or
indeed different programming problems, or even other cohorts of participants, other
indicators may be identified. This potentially limits the generalization of the results
of the study.
6 Conclusion and future work
Computational thinking is promoted as the literacy of the 21st century and is already
implemented in various curricula all over the world. Some refer to computational
thinking even as the foundation of programming and CS (Lu & Fletcher, 2009). Thus,
the goal of this study was to analyses the role of computational thinking in promoting
high quality programming products. Results showed that the answer to the question
of how computational thinking is related to program quality depends on whether
computational thinking is seen as a set of general conceptual understandings or a set
of procedural skills in use. The results of our study found that computational thinking
as general conceptual knowledge (such as that used to solve Bebras challenges) was
not significantly related to program quality. On the other hand, we found that com-
putational thinking as a set of procedural skills applied in practice was significantly
related to programming quality, even when controlling for general intelligence. Thus,
when discussing the role of computational thinking in developing computer program-
ming capacity, we suggest that educators and policy makers focus on the importance
1 3
of cultivating computational thinking procedural capabilities rather than in more

abstract, knowledge-based and context free forms.
There are several potential avenues for research to build upon the results of
this study. Visual programming environments such as Scratch are usually used to
introduce computational thinking or programming concepts to people who have no
knowledge about programming, as was the case in this study. In future, researchers
could analyse how computational thinking is applied when experienced program-
mers solve a programming task in a programming language such as Java or C++. The
way programmers approach problems develops over time as they gain more knowl-
edge (Teague & Lister, 2014). It is possible that the level of computational thinking
for experienced programmers differs from the level of novices, which might mediate
the relationship between both concepts. A range of different tasks could be examined,
for instance, to gauge differences in computational thinking prevalence and relation-
ships to program quality for tasks with more closed solutions as opposed to being
more open-ended in nature. Future research could attempt to analyze all concepts
individually rather than in pairs as an alternative way to examine the relationship
between the constructs in question. In any case, it is intended that the frameworks and
methods presented in this paper provide a strong foundation for these future analyses.
Acknowledgements None.
Funding Open Access funding enabled and organized by CAUL and its Member Institutions
Declarations
None.
Conflict of interest None.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License,
which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long
as you give appropriate credit to the original author(s) and the source, provide a link to the Creative
Commons licence, and indicate if changes were made. The images or other third party material in this
article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line
to the material. If material is not included in the article’s Creative Commons licence and your intended use
is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission
directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/
licenses/by/4.0/.
References
Anderson, J. R. (2015). Cognitive psychology and its implications (8th.). New York, NY: Worth Publishers
Angeli, & Giannakos, M. (2020). Computational thinking education: Issues and challenges. Computers in
Human Behavior, 105, 106185. https://doi.org/10.1016/j.chb.2019.106185
Araujo, A. L. S. O., Andrade, W. L., Guerrero, D. D. S., & Melo, M. R. A. (2019). How many abili-
ties can we measure in computational thinking? A study on Bebras challenge. Proceedings of the
50th ACM Technical Symposium on Computer Science Education (pp. 545–551). https://doi.
org/10.1145/3287324#issue-downloads
1 3
Atmatzidou, & Demetriadis, S. (2016). Advancing students’ computational thinking skills through educa-
tional robotics: A study on age and gender relevant differences. Robotics and Autonomous Systems,
75, 661–670. https://doi.org/10.1016/j.robot.2015.10.008
Banks, S. H., & Franzen, M. D. (2010). Concurrent validity of the TONI-3. Journal of Psychoeducational
Assessment, 28(1), 70–79. https://doi.org/10.1177/0734282909336935
Brennan, K., & Resnick, M. (2012). New frameworks for studying and assessing the development of
computational thinking. In Proceedings of the 2012 annual meeting of the American Educational
Research Association, Vancouver, Canada
Bower, M., & Hedberg, J. G. (2010). A quantitative multimodal discourse analysis of teaching and learning
in a web-conferencing environment–the efficacy of student-centred learning designs. Computers &
Education, 54(2), 462–478
Brown, L., Sherbeernou, R. J., & Johnson, S. K. (1997). Test of nonverbal intelligence-3. Austin, TX:
PRO-ED
Bull, G., Garofalo, J., & Hguyen, N. R. (2020). Thinking about computational thinking. Journal of Digital
Learning in Teacher Education, 36(1), 6–18. https://doi.org/10.1080/21532974.2019.1694381
Cansu, F. K., & Cansu, S. K. (2019). An overview of computational thinking. International Journal of
Computer Science Education in Schools, 3(1), 17–30. https://doi.org/10.21585/ijcses.v3i1.53
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, N.J.: L. Erl-
baum Associates
Cormen, T. H., Leiserson, C. E., Rivest, R. L., & Stein, C. (2014). Introduction to algorithms (3rd ed.).
Cambridge, MA, London: MIT Press
Csizmadia, A., Curzon, P., Dorling, M., Humphreys, S., Ng, T., Selby, C., & Woollard, J. (2015). Compu-
tational thinking - A guide for teachers. Swindon: Computing at School. http://eprints.soton.ac.uk/
id/eprint/424545
Dagienė, V., & Futschek, G. (2008). Bebras international contest on informatics and computer literacy:
Criteria for good tasks. In R. T. Mittermeir & M. M. Sysło (Eds.), Informatics Education - Support-
ing Computational Thinking: Third International Conference on Informatics in Secondary Schools
- Evolution and Perspectives, ISSEP 2008 Torun Poland, July 1-4, 2008 Proceedings (pp. 19–30).
Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-540-69924-8_2
Dagienė, V., & Sentance, S. (2016). It’s computational thinking! Bebras tasks in the curriculum. In A.
Brodnik & F. Tort (Eds.), Lecture Notes in Computer Science. Informatics in Schools: 9th Interna-
tional Conference on Informatics in Schools: Situation, Evolution, and Perspectives, Proceedings
(Vol. 9973, pp. 28–39). Cham: Springer Verlag. https://doi.org/10.1007/978-3-319-46747-4_3
Dagienė, V., & Stupuriene, G. (2016). Bebras - A sustainable community building model for the concept
based learning of informatics and computational thinking. Informatics in Education, 15(1), 25–44.
https://doi.org/10.15388/infedu.2016.02
Denner, J., Werner, L., Campe, S., & Ortiz, E. (2014). Pair programming: Under what conditions is it
advantageous for middle school students? Journal of Research on Technology in Education, 46(3),
277–296. https://doi.org/10.1080/15391523.2014.888272
Dolgopolovas, V., Jevsikova, T., Savulionienė, L., & Dagienė, V. (2015). On evaluation of computational
thinking of software engineering novice students. In A. Brodnik & C. Lewin (Eds.), IFIP TC3 Work-
ing Conference “A New Culture of Learning: Computing and next Generations”. Vilnius, Lithuania:
Vilnius University
Ezeamuzie, & Leung, J. S. C. (2021). Computational thinking through an empirical lens: A sys-
tematic review of literature. Journal of Educational Computing Research, Vol. 59, https://doi.
org/10.1177/07356331211033158
Gadanidis, G. (2017). Five affordances of computational thinking to support elementary mathematics edu-
cation. Journal of Computers in Mathematics and Science Teaching, 36(2), 143–151
Grover, S. (2011). Robotics and engineering for middle and high school students to develop computational
thinking. In Annual Meeting of the American Educational Research Association, New Orleans, LA
Grover, S. (2017). Assessing algorithmic and computational thinking in K-12: Lessons from a Middle
School Classroom. In P. J. Rich, & C. B. Hodges (Eds.), Emerging Research, Practice, and Policy
on Computational Thinking (31 vol., pp. 269–288). Cham: Springer International Publishing. https://
doi.org/10.1007/978-3-319-52691-1_17
Grover, S., & Pea, R. (2013). Computational thinking in K–12: A review of the state of the field. Educa-
tional Researcher, 42(1), 38–43
1 3
Grover, S., Pea, R., & Cooper, S. (2015). Designing for deeper learning in a blended computer science
course for middle school students. Computer Science Education, 25(2), 199–237. https://doi.org/10.
1080/08993408.2015.1033142
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010). Review of recent systems for automatic
assessment of programming assignments. Proceedings of the 10th Koli Calling International Confer-
ence on Computing Education Research, 86–93. https://doi.org/10.1145/1930464.1930480
International Society for Technology in Education [ISTE] & the Computer Science Teachers Association
[CSTA] (2011). Operational definition of computational thinking for K–12 education. Retrieved from
https://csta.acm.org/Curriculum/sub/CurrFiles/CompThinkingFlyer.pdf
Jin, K. H., & Charpentier, M. (2020). Automatic programming assignment assessment beyond black-box
testing. Journal of Computing Sciences in Colleges, 35(8), 116–125
Kafai, Y. B., De Liema, D., Fields, D. A., Lewandowski, G., & Lewis, C. (2019). Rethinking debugging as
productive failure for CS Education. In S. Heckman & J. Zhang (Eds.), Proceedings of the 50th ACM
technical symposium on Computer Science Education. New York, NY: ACM
Knobelsdorf, M., & Frede, C. (2016, August). Analyzing student practices in theory of computation in
light ofdistributed cognition theory. In Proceedings of the 2016 ACM Conference on International
Computing Education Research (pp. 73–81).
Korucu, A. T., Gencturk, A. T., & Gundogdu, M. M. (2017). Examination of the computational thinking
skills of students. Journal of Learning and Teaching in Digital Age, 2(1), 11–19. Retrieved from
https://eric.ed.gov/?id=ED572684
Landis, R., & Koch, G. G. (1977). The measurement of observer agreement for categorial data. Biometrics,
33, 159–174
Lockwood, J., & Mooney, A. (2018). Computational thinking in secondary education: Where does it fit?
A systematic literary review. International Journal of Computer Science Education in Schools, 2(1),
pp. 1–20. https://doi.org/10.21585/ijcses.v2i1.26
Lu, J. J., & Fletcher, G. H. L. (2009). Thinking about computational thinking. In S. Fitzgerald (Ed.),
Proceedings of the 40th ACM technical symposium on Computer science education. New York, NY:
ACM
Lye, S. Y., & Koh, J. H. L. (2014). Review on teaching and learning of computational thinking through
programming: What is next for K-12? Computers in Human Behavior, 41, 51–61. https://doi.
org/10.1016/j.chb.2014.09.012
Martin, R. C. (2009). Clean code: A handbook of agile software craftsmanship. Pearson Prentice Hall
McNicholl, R. (2019). Computational thinking using Code.org. Hello World, 4, 36–37. https://issuu.com/
raspberry314/docs/helloworld04
Moreno-León, J., & Robles, G. (2015). Dr. Scratch: a web tool to automatically evaluate scratch projects.
In J. Gal-Ezer, S. Sentance, & J. Vahrenhold (Eds.), Proceedings of the Workshop in Primary and
Secondary Computing Education, London, United Kingdom, November 09 - 11, 2015 (pp. 132–133).
New York: ACM. https://doi.org/10.1145/2818314.2818338
Moreno-León, J., Román-González, M., Harteveld, C., & Robles, G. (2017). On the automatic assess-
ment of computational thinking skills. In G. Mark, S. Fussell, C. Lampe, m. schraefel, J. P. Hour-
cade, C. Appert, & D. Wigdor (Eds.), CHI’17: Extended abstracts: proceedings of the 2017 ACM
SIGCHI Conference on Human Factors in Computing Systems : May 6-11, 2017, Denver, CO, USA
(pp. 2788–2795). New York, New York: The Association for Computing Machinery. https://doi.
org/10.1145/3027063.3053216
Pieterse, V. (2013). Automated assessment of programming assignments. Proceedings of the 3rd Computer
Science Education Research Conference, 13, 4–5. https://doi.org/10.5555/2541917.2541921
Portelance, D. J., & Bers, M. U. (2015). Code and tell: Assessing young Children’s learning of computa-
tional thinking using peer video interviews with ScratchJr. In M. U. Bers & G. L. Revelle (Eds.), IDC
‘15: Proceedings of the 14th international conference on interaction design and children (pp. 271–
274). New York: ACM
Posner, M. I., & Keele, S. W. (1968). On the genesis of abstract ideas. Journal of Experimental Psychol-
ogy, 77, 353–363. https://doi.org/10.1037/h0025953
Poulakis, E., & Politis, P. (2021). Computational Thinking Assessment: Literature Review. Research
on E-Learning and ICT in Education: Technological, Pedagogical and Instructional Perspectives,
111–128
Resnick, M., Silverman, B., Kafai, Y., Maloney, J., Monroy-Hernández, A., Rusk, N., & Silver, J.
(2009). Scratch: Programming for all. Communications of the ACM, 52(11), 60. https://doi.
org/10.1145/1592761.1592779
1 3
Román-González, M., Pérez-González, J. C., & Jiménez-Fernández, C. (2017). Which cognitive abilities
underlie computational thinking?: Criterion validity of the Computational Thinking Test. Computers
in Human Behavior, 72, 678–691. https://doi.org/10.1016/j.chb.2016.08.047
Schank, R. C., & Abelson, R. P. (1977). Scripts, plans, goals and understanding: An inquiry into human
knowledge structures. Hillsdale, NJ: L. Erlbaum Associates. Artificial intelligence series
Schulz, K., & Hobson, S. (2015). Bebras Australia computational thinking challenge tasks and solutions
2014. Brisbane, Australia: Digital Careers
Schulz, K., Hobson, S., & Zagami, J. (2016). Bebras Austrlia computational thinking challenge - tasks and
solution 2016. Brisbane, Australia: Digital Careers
Shivhare, & Kumar, C. A. (2016). On the Cognitive process of abstraction. Procedia Computer Science,
89, 243–252. https://doi.org/10.1016/j.procs.2016.06.051
Shute, V. J., Sun, C., & Asbell-Clarke, J. (2017). Demystifying computational thinking. Educational
Research Review, 22, 142–158. https://doi.org/10.1016/j.edurev.2017.09.003
Sipser, M. (2013). Introduction to the theory of computation (3rd ed.). Boston: Cengage Learning
Sternberg, R. J. (2017). Human intelligence. Encyclopaedia Britannica. Retrieved from https://www.bri-
tannica.com/topic/human-intelligence-psychology/Development-of-intelligence#ref13354
Suters, L., & Suters, H. (2020). Coding for the core: Computational thinking and middle grades
mathematics. Contemporary Issues in Technology and Teacher Education (CITE Jour-
nal), 20(3). Retrieved from https://citejournal.org/volume-20/issue-3-20/mathematics/
coding-for-the-core-computational-thinking-and-middle-grades-mathematics/
Tang, K. Y., Chou, T. L., & Tsai, C. C. (2020). A content analysis of computational thinking research: An
international publication trends and research typology. Asia-Pacific Education Researcher, 29(1),
9–19. https://doi.org/10.1007/s40299-019-00442-8
Teague, D., & Lister, R. (2014). Longitudinal think aloud study of a novice programmer. In J. Whalley
(Ed.), Proceedings of the Sixteenth Australasian Computing Education Conference - Volume 148.
Darlinghurst, Australia: Australian Computer Society, Inc
Thalheim, B. (2009). Abstraction. In L. Liu, & M. T. Özsu (Eds.), Springer reference. Encyclopedia of
database systems, 1–3. New York, NY: Springer
Tsai, Liang, J. C., Lee, S. W. Y., & Hsu, C. Y. (2021). Structural validation for the developmental model
of computational thinking. Journal of Educational Computing Research, Vol. 59, https://doi.
org/10.1177/07356331211017794
Türker, P. M., & Pala, F. K. (2020). A Study on students’ computational thinking skills and self-efficacy
of block-based programming. Journal on School Educational Technology, 15(3), 18–31 (14 Seiten).
Retrieved from https://imanagerpublications.com/article/16669/
Vaníček, J. (2014). Bebras informatics contest: Criteria for good tasks revised. In Y. Gülbahar & E. Karataş
(Eds.), Informatics in Schools. Teaching and Learning Perspectives: 7th International Conference
on Informatics in Schools: Situation, Evolution, and Perspectives, ISSEP 2014, Istanbul, Turkey,
September 22-25, 2014. Proceedings (pp. 17–28). Cham: Springer International Publishing. https://
doi.org/10.1007/978-3-319-09958-3_3
Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L., & Wilensky, U. (2016). Defining
computational thinking for mathematics and science classrooms. Journal of Science Education and
Technology, 25(1), 127–147
Weinert, F. E. (1994). Lernen lernen und das eigene Lernen verstehen. [Learning how to learn and under-
standing the own learning]. In K. Reusser, & M. Reusser-Weyeneth (Eds.), Verstehen. Psycholo-
gischer Prozess und didaktische Aufgabe [Understanding. Psychological processes and didactical
tasks.] (pp. 183–205). Bern: Huber
Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35. https://doi.
org/10.1145/1118178.1118215
Wu, B., Hu, Y., Ruis, A. R., & Wang, M. (2019). Analysing computational thinking in collaborative pro-
gramming: A quantitative ethnography approach. Journal of Computer Assisted Learning, 35(3),
421–434. https://doi.org/10.1111/jcal.12348
Zha, S., Jin, Y., Moore, P., & Gaston, J. (2020). Hopscotch into Coding: introducing pre-service teachers
computational thinking. TechTrends, 64(1), 17–28. https://doi.org/10.1007/s11528-019-00423-0
Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps
and institutional affiliations.
1 3
Authors and Affiliations
Kay-Dennis Boom1 · Matt Bower2 · Jens Siemon1 · Amaël Arguel3
Matt Bower
matt.bower@mq.edu.au
1
Department of Vocational and Business Education, University of Hamburg, Hamburg,
Germany
2
School of Education, Macquarie University, Building 29WW Room 238, Balaclava Rd
North Ryde, 2109, NSWSydney, Australia
3
CLLE, University of Toulouse, CNRS, Toulouse, France
1 3
CONTRIBUTED ARTICLES
ARTICLES
Mathematical
Modeling with R:
Embedding
Computational
Thinking into High
School Math Classes
By Kenia Wiedemann, The Concord Consortium, Jie Chao, The Concord Consortium,
Benjamin Galluzzo, Clarkson University and Eric Simoneau, Boston Latin School
M athematical modeling is routinely used to represent,

analyze, and simulate natural and human-made
systems. Job ads frequently require computational
puter science (CS) courses [5,12]. The good news? The number
of schools offering CS in the U.S. is on the rise, and the number
of states adopting policies to promote K-12 CS education is also
thinking skills, and new positions open practically daily. increasing [13]. However, it is estimated that only a little over a
However, not enough people apply for these jobs. Many third of public high schools in the U.S. currently offer CS classes
students graduating from high school have no experience and only sixteen states have adopted a policy to give all high
in computer science, believing that because they are not school students access to CS courses [12].
experienced programmers, they are not cut out for jobs Occupations under the computer and information technol-
mentioning terms like machine learning or big data. A ogy umbrella described by the Bureau of Labor Statistics (BLS)
new initiative to eliminate the misconception that to think have a projected growth that goes from 5% to a whopping 32%
computationally is a synonym to programming proposes by 2028 (12% on average) [5] with one notable exception: com-
embedding computational thinking into curricular subjects. puter programmers, whose jobs are expected to decrease 7% by
We discuss one such effort and its encouraging results from 2028 (Figure 1). To quote directly from reference [3], “computer
a curriculum implementation in a U.S. high school. programming can be done from anywhere in the world, so com-
panies sometimes hire programmers in countries where wages
are lower,” strongly suggesting that learning how to program a
INTRODUCTION computer is an important skill to get someone inserted in mar-
Numbers and graphs from mathematical models make the ket of computer and information technology, but it is not the
headlines daily, banks fight over potential new clients advertis- whole story. Newcomers must bring a differential, something
ing different interest rates, APRs, and other confusing financial in addition to coding skills. These new analysts, developers, sys-
terms. It may explain why nine out of ten high schoolers’ par- tem architects, and alike need to demonstrate they can think
ents say they would like their children to have access to com- out of the box and think computationally.
acm Inroads • inroads.acm.org 33
ARTICLES
Mathematical Modeling with R: Embedding Computational Thinking into High School Math Classes
One strategy that can help to build a bridge across this gap The purpose of this research is to design and test instruc-
between computer and information tech professional supply tional modules that can help high school math teachers to em-
and demand is to engage students in computational thinking bed computational thinking into their classrooms. The CodeR-
(CT), a concept that predates coding. It is the thought process 4MATH modules are designed to allow students to learn how
involved in formulating problems and designing solutions that to tackle real-world problems using mathematical modeling,
can be executed by information-processing agents [16]. CT skills while having a programming language as the modeling envi-
are naturally exercised during the process of using mathematical ronment. In this study we investigate why, and to what extent,
modeling to find solutions to often ill-formulated real-life prob- engagement in the project activities by high school students
lems. Embedding CT into mandatory curricular subjects, other enrolled in regular math courses contributes to reshaping their
than explicit CS classes, extends the reach of CT as well as CS level of interest in further exploring computing related learn-
to student populations that otherwise would not have access to ing opportunities in the future. Notably, student performance
it. Not surprisingly, developing computational thinking across demonstrates increased competency in mathematical modeling
all disciplines and educational levels has become a priority for and computational thinking.
scholars and international agencies that make a call to integrate
these concepts across the K-12 and undergraduate curricula. THE INSTRUCTIONAL MODEL
The curriculum modules were developed to engage students in
mathematical modeling, using their knowledge of math con-
cepts to solve real-life problems, using two main pedagogical
approaches: context-based [e.g., 7,11] and faded scaffolding
[e.g., 4,14, and references therein]. In a context-based approach,
everything is taught in the context of the initial open-ended
problem. Students learn new concepts and tools only when
they need to use them, giving them the motivation to learn
something new. Scaffolding is targeted assistance provided to
students (either by their teacher or by prompts and hints in
self-paced tutorials, for example) as they perform a task. Faded
scaffolding can be understood simply as giving support to stu-
dents when the assistance is needed and removing it when it is
no longer needed, giving students the chance to exercise newly
acquired skills to solve more complex problems.
The iterative cycle of mathematical modeling involves iden-
tifying and selecting parameters to represent a situation, choos-
ing mathematical representations to define those parameters
Figure 1. Projected change in employment from 2018 to 2028. Almost all
and their relationships, performing mathematical operations
occupations under the computer and information technology umbrella to draw conclusions, interpreting and validating the findings
described by the BLS have projected growth for the next few years against the situation, and iteratively improving the model (Fig-
(with the exception of computer programmers). Yet, many companies ure 2). Studies have shown that computing activities helped
report difficulties finding these skilled workers. Exercising computational students develop a deeper understanding in a variety of math-
thinking may help to build a bridge across this gap between computer
and information tech professional CS skills’ supply and demand.
ematics domains [e.g., 1,9]. When dealing with a problem
(real-world or otherwise) to be solved mathematically, it is cer-
The Computing with R for Mathematical Modeling (CodeR- tain that one can tackle the problem using any media. The abil-
4MATH) project, funded by the U.S. National Science Founda- ity to teach our understanding of the problem to a computer is
tion (NSF), has been working to create a collection of learning, referred here as computerizing the problem (cycle 3b in Figure
assessment, and tutoring resources. Throughout the CodeR- 2). If we don’t fully understand part of the system we are trying
4MATH instructional modules, students learn how to tackle to describe, the computational results can (and often do) provide
real-world problems using mathematical modeling, with pro- us with a useful test for our initial knowledge and hypotheses, or
gramming language (R) as the modeling environment. During rather what the mathematical results are, either expected or not,
the process, students naturally exercise computational thinking providing insights on how to further refine our algorithm.
skills while learning basic concepts of computer programming. During the classroom activities, we invited students to an-
Developed and supported by the R Foundation for Statistical alyze and solve problems that are relatively open-ended by de-
Computing, R is both a programming language and an open- sign, to encourage students to brainstorm, make assumptions,
source environment for statistical analysis, and a favorite among create algorithms, and hypothesize possible outcomes. Stu-
statisticians and data scientists. We developed a series of learning dents can then translate their algorithms into a computer code,
tasks that encourage students to exercise computational think- creating a mathematical model to test their hypotheses for
ing, working on topics that are relevant and relatable to them. many different scenarios. Based on the model outputs, students
34 acm Inroads 2020 March • Vol. 11 • No. 1

ARTICLES
Figure 2. Computational Mathematical Modeling Cycle. Mathematical modeling is an iterative process that
involves several steps from identifying and selecting variables to represent a situation to validating the
results, repeating the process until the model reaches an acceptable format given time and processing
constraints. Figure 2 is adapted from computational mathematical modeling cycle [3].
may revisit their initial assumptions, add or remove parame-

ters, modify both the algorithm and the code and run new tests,
repeating the process until they are satisfied with the model.
SETTING THE STAGE

School and teachers - The module was implemented in a public
high school located in a fairly prosperous community, with a
median annual household income in 2019 above US$170,000
[15]. The school offers a comprehensive four-year program with
over 1,500 students enrolled, but only 10% or fewer of students
enrolled in a CS course at some point. Asked the reason for the
low enrollment rate, one teacher explains: “It is because of ca-
pacity. You don’t have the number of teachers, or no sections are
approved to run. So we have wait lists for those classes.” The two
participating teachers had 8 and 18 years of experience teaching
mathematics at the high school level. One teacher had taught
introductory CS courses using Python as a programming lan-
guage, while the other teacher had no experience in teaching
CS or programming. Despite their limited experience in pro- Figure 3. Intended major among students. The female/male ratio of the
gramming, both teachers were enthusiastic about the opportu- students was 2:1 in this study, and the vast majority intended to pursue
nity to engage their students in using computer programming careers in the humanities or social sciences.
to solve mathematical modeling problems.
The students - The population in this study is formed by
42 students enrolled in Discrete Mathematics (DM). The DM METHODS
classes consisted of seniors in the second semester of their CURRICULUM AND IMPLEMENTATION
final year of high school, enrolled in DM to fulfill their math- We split the activity into two real-world problems that we
ematics requirements before graduation. There were 28 fe- named Meal Plan vs. Pay-As-You-Go (M.P. vs. PAYG) and
males and 14 males, and the vast majority of them intended to the Driving for Gas (DFG), both part of a module we called
pursue careers in the humanities or social sciences (48% and Lifehacking. The module was designed to present open-ended
24%, respectively). Six students said they were interested in problems relatable to someone who is entering the adult life,
pursuing careers in STEM such as marine biology and neu- like figuring out the costs of eating in college or the real costs
rosciences; however, none indicated an interest in majoring of owning and driving a car. The activities were presented to
in CS or mathematics (Figure 3). While three students men- students as online tutorials; therefore, students could work
tioned that they had had some contact with programming in on them from any computer with access to internet. Students
the past, 39 students (93%) declared not to have any prior ex- worked on this module in a total of 6 hours throughout two and
perience with CS or programming. a half weeks.
ARTICLES
Either because of socioeconomic status or personal interests, their pocket on a per meal basis. We decided for this approach
students may think that building a mathematical model to help (suggesting they should work on a solution for multiple people
them with a one-single-time-decision it’s just not worth the ef- with multiple interests and values) to address the problem we
fort. Paper and pencil and a hand calculator would do. While stu- described in session 2.1, that is, they should develop a mathe-
dents may still be willing to do some work for academic purposes matical model that could (1) maximize the number of college
(to get a good grade, for example), they would likely be more in- students they could help in a given time and (2) find the best
clined to embrace a modeling exercise if they could see the model solution for each particular case. Step-by-step, students were
as a tool to help other people while avoiding the tedious work guided through the math modeling process, brainstorming,
of repeating the same calculations from scratch for each person. making assumptions about their clients’ (the college students)
The tutorials were designed using RMarkdown [2] and the eating habits, budgets, lifestyles, and personal preferences.
tools from the learnr R package [11]. The tools in the learnr They gradually refined the model by revisiting their previous
package allowed us to create code snippets, pre-populate them premises and adding new parameters such as a variable num-
with templates (complete or incomplete codes), add hints and ber of outings per week (dinner with friends, for example), or
solutions, and more. Once the student was logged in, any mod- assuming that their client could have free meals for certain pe-
ification in the snippets was saved after they logged out, so they riods, maybe while visiting their family.
could continue the exercise from the point they left. Figure 4 Students worked on this activity during five sessions over
shows an example of one of the tutorial pages and the kind of two weeks and were strongly encouraged to think aloud and
platform students were working on, designed to be self-paced discuss in groups. Teachers and researchers followed the group
with all instructional elements. Students were free to modify discussion closely, taking notes of behaviors, comments, and
the pre-populated code snippets, complete coding exercises, drafts that students may have been writing on paper before add-
check hints and solutions, and write and run their own code. ing their ideas into code. At the beginning of each new session,
the teachers reviewed what they had accomplished, addressed
any problems they had encountered, and set the goals for the
MEAL PLAN VS. PAY AS YOU GO day. Students worked in small groups through the tutorial at
In the first part of the activity, students were invited to wear their own pace. All students were able to complete the activity
the shoes of a college advisor who has to help college students by the fifth session. After this activity, students were presented
decide between purchasing a meal plan (with unlimited ac- with a new problem called Driving for Gas (described in the
cess to dining halls with an upfront fixed cost) or paying out of following section) as an assessment of their learning.
Figure 4. Example of learning activity on CodeR4MATH. The tutorials were designed

using RMarkdown and the tools from the learnr package. And designed to be
self-paced, containing all instructional elements.

ARTICLES
THE ASSESSMENT: DRIVING FOR GAS RESULTS

The Driving for Gas activity was adapted from an activity de- STUDENTS’ MODELING ASSESSMENT
scribed in the Guidelines for Assessment and Instruction in As described in the Methods section, we aimed to assess stu-
Mathematical Modeling Education (GAIMME) report [8]. dents’ modeling skills using the Driving for Gas activity. The full
The original problem challenges students to create mathemat- assessment code, written in R, is shared in Figure 7. First, we
ical models to help drivers decide whether it is worthwhile to asked the students to run the code and summarize the model’s
drive further to gas stations that sell cheaper gas. To assess the outputs. Then, we asked them to suggest modifications and to try
students’ modeling skills within a short period, we adapted and implement their suggestions into the original code. From the
the problem into a model improvement task. In addition to initial 42 participants, only 36 students returned the assessment.
the original problem statement, students were also presented Of course, it would be too ambitious to assume that students
with a model written as a simple R program. Students were would become self-sufficient computer programmers after 6
asked to describe how the problem was defined and what vari- hours of exposure to their first coding experience. The ability to
ables were considered, comment on the assumptions made, quantify every student’s understanding of programming con-
interpret the model’s outputs, and improve the model by de- cepts was limited to the students’ open answers and comments
scribing additional factors to incorporate and modifying the R to the model (the R code). Students were encouraged to write
program. Teachers and researchers further elicited students’ down their thoughts and experiment with the code freely, that is,
interest by suggesting that a computer code that would save they were not under any pressure of being graded. On one hand,
drivers money and time could become a smartphone or com- that allowed the students to brainstorm as they find fit, where
puter app that could become profitable in the future. Students many out-of-the-box ideas may emerge. On the other hand, giv-
received the suggestion very well and proceeded with their en the limit in time, they didn’t feel compelled to either get final
individual work. quantitative answers or write sharp, focused comments.
Students completed the assessment using RStudio®, a free Defining which concepts students have (or have not) learned
and open-source integrated development environment for R, was determined qualitatively. If a student made a reasonable sug-
which gives them all functionalities they experienced when gestion on how to modify the code in order to improve it (with-
using snippets, and much more. We asked students to submit out any judgment on the meaning of improvement, which would
their answers individually. The activity requested that students vary for each student), we added 1 to the students-who-under-
(1) run the template, paying attention to what each part of the stood-programming-concepts bucket. This way our qualitative
model (R code) was doing, (2) interpret the results (a graph), evaluation of students’ responses could be transferred to a quan-
(3) comment and criticize aspects of the model, (4) propose im- titative system, more or less binary. Using this method, their
provements, such as the introduction of new assumptions and comments to the code suggested that half of students (18 out of
variables (or the removal of old ones), and (5) modify the code the 36 respondents) understood components of the model repre-
to implement those improvements. sented in the R program, and that they have achieved some level
of understanding of fundamental programming concepts such as
THE EXIT QUESTIONNAIRE the meaning of variables in programming, vectors (as a variable
At the end of the unit, the students completed an exit question- that contains a list of elements), and programming functions (as
naire about their impressions, thoughts, and suggestions about secondary codes that are given a name and that perform certain
the modeling activities and their learning experience. The ques- tasks when called in the main code). One student, for example,
tionnaire was designed to help researchers understand how made comments to the code, shown on Figure 5.
the modeling activities had influenced the students’ interest in
computing and how their backgrounds, such as gender and ac-
ademic interest, might mediate the impacts.
Students were asked to provide honest feedback for the re-
search team to improve the curriculum. Among other things,
students were asked to define mathematical modeling in their
own words, to rate their interest in pursuing programming-re-
lated courses in the future, and the reasons behind their sen- Figure 5. Code snippet from a sample of student work. Alterations in
the original R code were added by a student with no prior experience in
timent toward the activity and computer programming. They
programming.
were also asked to list the aspects they liked and disliked about
the module. The code chunk in Figure 5 suggests that the student may
As it is common practice in educational research, classroom have decided to weigh in an additional cost that relates linearly
observations and discussions with students and teachers were with the distance added to go to another gas station (although
part of the implementation of the CodeR4MATH tutorial. they did not comment on the code where the 0.225 factor came
However, the discussion and conclusions presented in this pa- from). They then add this factor to calculate the cost_gs2_
per are based on the analysis of the modeling assessment and real variable and proceed to create a data frame with results
the exit questionnaire. and then later (not pictured here) to plot the results. Although
ARTICLES
it may seem a very simple modification to the experienced pro- be more realistic to make. Also, I changed the graph to 10-50
grammer, these students have never had computer program- [dollars] for the y-axis, so it’s easier to read.” Their comments
ming classes before. suggest their understanding of concepts such as variables and
The main output from the original code provided to the functions in a programming environment, along with a level
students was given by the final graph shown in Figure 6. From of appreciation of the ease of testing a model represented in a
the 36 students who delivered the assessment, 15 seemed to computer code.
have read the model (the R code) as a statement rather than an
open problem and may have misinterpreted the output graph, The connection to real-life questions
assuming that the second station was at a fixed distance of
30 miles from the first one. “The results are strange,” explains and the ability to look at them
one of the students. “At first, I would have easily made the trip
across town to Gas Station 2, however, little did I realize that through a different perspective but
Gas Station 2 is 30 miles away and ends up being the same
price as Gas Station 1.” Actually, the model describes that the
still using the mathematical
price per gallon is fixed for each gas station, while varying the skills they have already acquired at
distance between the two stations (from 1 to 30 miles), to test
what would be the real cost of driving to a second station de- school appealed to students.
pending on its distance from where the driver (and the first
station) is.
IMPACT OF THE CURRICULUM MODULE ON STUDENTS’
INTEREST IN COMPUTER PROGRAMMING
In the exit questionnaire, we also asked students how the mod-
eling activities had influenced their interest in taking computer
programming courses in the future. For this question, we pro-
vided them a type of Likert scale going from ‘I became less in-
terested’ to ‘I became very interested’ and also provided them
a field to write down their own thoughts in case they found
the options didn’t suit their impressions. All 42 students picked
one of the first five options. We immediately followed with an
open-ended question, asking them to explain what aspects of
the modeling activities may have made them more or less inter-
ested in taking programming courses in the future.
As described in the Introduction, the participants in this
study were seniors, and only two of them had taken any CS
course during their high school years. They were enrolled in a
standard-level mathematics course to fulfill their math require-
ment; their mathematics achievement was “average or below
average,” according to their teachers, and the vast majority of
them intended to pursue careers in the humanities or social
sciences. All these characteristics projected a development
Figure 6. Output from the mathematical model presented in the “Driving path away from computing education. Yet many of the students
for Gas” assessment. Students were asked to run a simplified model whose
came to see the value of computing and found interest in the
code was written in R and interpret its output, given by this simple plot.
field that they previously perceived as irrelevant.
As we mentioned, students were also asked to describe the A total of 14 students (one third of participants) said they
modifications they found necessary to make to the model. became interested in taking programming classes in the future
Students seemed to be forthcoming and were not shy about (sum of Little more interested and Much more interested cat-
brainstorming suggestions to make the model more realistic. egories in Figure 8). Half of the students said that they were
“I’d want to factor in the amount of time the person has to go not interested in it before the activity and that their inclination
get gas and the amount of money they have to spend,” suggest- did not change. Two students (5%) said that they were already
ed one student. “That would make the model more precise and somewhat interested in taking programming classes in the fu-
personalized to the needs of the person using it.” Another stu- ture and that didn’t change either. Lastly, five students (12% of
dent managed to add a few modifications to the code, explain- the total) said that they became less interested after this activity.
ing their thinking, “I changed the prices of the gas on both gas The reasons that students indicated to have contributed to their
stations, making it more drastic rather than just 25 cents. I also increased or decreased interested in pursuing computer pro-
decreased the distance from gas station 2, so the trip would gramming are discussed below.

ARTICLES
Figure 7. “Driving for Gas” Assessment. Driving for Gas assessment as presented to
the students. They were required to comment on the R code, suggest modifications
and refinements, and finally implement them to the code.
Figure 8. Impact of the mathematical modeling activity of students’

interest in computer programming. Many students came to see the value
of computing and found interest in a field that they previously perceived
as irrelevant to them. One-third of the students said that they became
interested in taking programming classes in the future.
THE REASONS BEHIND INCREASED OR MAINTAINED themes for which students became interested in computer pro-
INTEREST IN PROGRAMMING gramming were:
Students were also asked to explain what about the learning Coding is powerful
experience made them more or less interested in taking com- Solving real-world problems with math modeling
puter programming courses in the future. We categorized their and coding
open-ended responses into broad themes. The three main New experience
ARTICLES
Positive reason 1: Coding is powerful REASONS BEHIND SOME STUDENTS’ DISINTEREST IN

Eight students said that coding is a powerful tool that makes data PROGRAMMING
analysis much easier and faster. The majority of students who saw Not all students felt compelled to explore more program-
coding as a powerful tool was disproportionately male (75% with- ming-related courses though. Twenty-one participants (50%)
in this subgroup). They also mostly intended to major in fields said that they were not interested in computer programming
that require quantitative skills (e.g., STEM and social sciences). “I before the activity and that they were still not after the exer-
just found it really fascinating how coding can make life that much cise. Five students stated that their interest in programming
easier,” said a student interested in majoring in business. “When I had actually declined. The reasons these students pointed out
learn more and more about it, I get more into it. Something I don’t for not becoming interested (or becoming less interested) in
experience every day, it’s nice and in a way fun.” Another student, programming vary, but the vast majority were related to their
who intended to major in marketing, said that they appreciated self-concept in relation to CS and mathematics itself. In fact,
“how you can use coding to solve big mathematical problems.” An- the responses from over 80% of students in this group (still not
other student, who claimed to be interested in STEM, said that or less interested in programming) fall under this self-concept
they became more interested in programming after the activity category, and the disproportionate majority of these are females
because “coding can make keeping track of data much easier.” (a 6:1 female-male ratio, compared to 2:1 for the whole popula-
tion that responded to the questionnaire).
Positive reason 2: Coding can be used to solve real-
world problems Negative reason 1: Self-concept
The connection to real-life questions and the ability to look at Some students seem to believe they are simply not good at
them through a different perspective but still using the math- mathematics or computer sciences. “I’ve never been a heavy
ematical skills they have already acquired at school appealed math person,” said a student interested in the humanities, “and
to students. Several previous studies (e.g., [9] and references although computer coding is intriguing and seems complex and
therein) have already shown that solving real-life problems of- fascinating, I have no real interest to take another class.” Still,
ten increase students’ performance in CS courses. “I thought it they recognize that being exposed to the experience was help-
was very interesting,” a student said. “It opened up my mind to ful to them, adding that “I liked that it was unlike anything we
other ways in which math can be used in the real world.” have done all year, it was new and yes a little confusing but also
Students also appreciated the process of mathematical mod- interesting.” Another student says that “Coding is just something
eling, which relates directly to the exercise of CT skills. “The I’m generally not interested in, but I think it was good that I was
coding exercises required that I defined all variables and then in- introduced to it, just in case I may need to use it.”
tegrate them into some sort of data structure, which is a valuable We can say that even if for part of the students, the interest in
process to utilize even without coding,” said one of the students. programming has not been triggered, they became aware of the
Also, the teachers have appreciated that students could experi- value of learning about the process of math modeling and coding.
ence the thinking process of solving an open-ended problem. “I
think getting them to think of a big picture for me was probably Negative reason 2: More scaffolding, please!
the best piece,” one of the teachers said. “It was just getting them The activities were designed to pose semi-open questions to the
to think bigger than themselves. How do I take this problem, and students, in the sense that students were guided with instructions
how do I project it, and what are the issues I need to think about?” and examples, and their teacher-led discussions in the classroom.
“I think that connecting, getting them to brainstorm, getting Because the tutorials were self-paced, beginners could take their
them to be more aware, that was my favorite part,” another time to walk through the activities, while more able students had
teacher said. “Having them [students] think that there are things the opportunity to go quickly through the basics and dedicate
out there that you can add value [to], that people’s days can be more time to refining their models. A few students, however, in-
better because of your model.” dicated that they would prefer more scaffolding and more direc-
tion from their teachers, showing some level of discomfort with
Positive reason 3: It was a new experience the curriculum design. “Even when I asked questions, it was still
Finally, the exposure to something that they had never done extremely unclear on what to write within the code and what I was
before also seems to have captivated students’ interest. “It was supposed to be analyzing,” said a student who was interested in the
a new and unique experience that showed a new way of thinking performing arts. When asked what they did not like about the ac-
about modeling,” said a student, “I knew nothing about comput- tivity (another question in the exit questionnaire), they pointed out
er programming, so it gave me a look into what it can be.” that “when working with code or when creating code, you aren’t nec-
Students seem to have found it compelling to have been ex- essarily looking for one specific answer. That really tripped me up
posed to a new use of a technology with which they are already and was something I didn’t understand until the last day of coding.”
familiar (their computers) to do something different and more ap-
plied. “I don’t take any classes that require [me] to do anything on Negative reason 3: Coding is boring
my computer besides type essays or Google things,” another student Two students who intended to major in the humanities re-
said, “so using my technology in a different way was cool for me.” marked that, for them, programming was boring and not useful.

ARTICLES
“I found it boring, and there was no creative part of it,” said one as mathematics course. However, in close collaboration with
of them. They continued by saying that “I didn’t find mathemat- the teachers, we observed strong potential for non-CS teachers
ical modeling to be that much of use.” A second student said, “it to develop competencies to integrate CS content in their class-
was very confusing, and I did not think this had anything to do rooms using the designed tutorial, without typical, extensive
with this math class.” professional development experience. While one of the teach-
Except for the students who felt they would prefer another ers had no prior experience in teaching CS, they both showed
delivery format, with more scaffolding (likely from their teach- the same high autonomy in using the tutorial and guiding their
ers), all other reasons given by students for not liking computer students to complete the activity. Students’ performance in the
programming seem to be primarily due to preconceived ideas assessment and their responses to the exit questionnaire were
about the subject or about themselves. These students appar- comparable for both classrooms. Despite the small sample
ently see programming, and even mathematics, as subjects re- size in this study, these results strongly suggest that the class-
served to the tech-savvy or to the math person, believing that room-readiness of the tutorial materials is critical to ensure not
some people are naturally good at mathematics and others are only that students will have a worthy learning experience, but
simply bad at it, and put themselves in the latter group. also to boost teachers’ confidence in repeating the implementa-
tion or even adapting the curriculum with their own inputs or
FUTURE TOPICS OF INTEREST style, whether or not they have training in teaching CS.
Finally, we asked students about their topics of interest, that The overwhelming majority of students who said they were
is, subjects they would like to learn more about and that they not interested in programming (or became less interested) were
would like to deal with in an activity like this. Forty-five percent females (~78%). Although we do not focus on the gender gap, it
of participants (19 students) said they would like to explore is- did not pass unnoticed. However, it would be incautious to affirm
sues related to daily life problems, such as finances, student life, that gender may have played a major role in this result, because
or personal time management. of confounding factors such as the fact that most of them were
“Budgeting in everyday life, taxes, and more,” suggested one interested in careers in the humanities. They formed the vast
of the students. “Topics that adults deal with in everyday life, majority of students whose reasons behind their lack of interest
but that aren’t necessarily explained and/or taught to students in programming are extended to the sciences in general. Several
within the school curriculum.” This student was interested in responses included phrases like “I am not a math person,” or “It
pursuing a career in the performing arts. These participants doesn’t play to my strengths,” and “I don’t think it is for me.” It
identified possible topics of interest as being related to a per- would be too ambitious to expect that educators could decon-
son’s daily life, in the lines of the Lifehacking curriculum mod- struct the I-am-not-a-math-person mindset created by young-
ule. Another student who declared their intention to follow a sters who reached their high school years. However, we hope that
career in psychology said, “Since I am going to college, another the results from this experience can provide some insights on
topic on it and saving money would be engaging,” and said that how to exercise computational thinking in mathematics class-
they became more interested in exploring programming-relat- rooms to help students abandon the erroneous preconceived
ed courses in the future. idea that exact sciences (and programming in particular) are out
of their reach. The mathematical modeling process is naturally
aligned with computational thinking, and math modeling activ-
DISCUSSION ities regularly naturally motivate the use of computational tools.
There is no substitute for a motivating environment where Our results reinforce that, although it may seem counter-intui-
students feel empowered to do great things with the resourc- tive, presenting students with CT approaches to solution building
es that they already have in hand. The use of code snippets to (e.g., visualizing a set of data, finding a multitude of solutions in
reduce syntax-related stress, along with engaging and relatable moments for comparison and/or analysis) in their regular math
real-world problems seem to have helped many students feel classes have great potential to lead to better learning outcomes.
comfortable and curious about the possibilities of math model-
ing and programming, while exercising computational thinking
skills. While the activity was not able to embed appreciation for SUMMARY
modeling and/or programming to all participants, we were pleas- Computational thinking is a collection of thought processes
antly surprised to find that one-third of them indicated they ac- involved in formulating problems and their solutions so that
tually became more interested in pursuing programming-related the solutions are represented in a form that can be carried out
courses in the future after this experience. This is a positive and by an information-processing agent; a computer program ex-
encouraging outcome, considering that most of these students ecuted by a computer, for example. Of course, computational
had not taken computer science courses during their high school thinking is a highly desirable skill set, and an excellent way
years and the vast majority of them had not seen how computing to exercise this way-of-thinking is through math modeling
was relevant to their everyday life and intended majors. and computer programming. However, students are leaving
This study was not intended to investigate how non-CS teach- high school without having much experience—if any—with
ers could be prepared to teach CS content in their courses such programming or, more broadly, with computer sciences. That
ARTICLES
may happen for several reasons. In one scenario, students the erroneous preconceived idea that exact sciences (and pro-
simply don’t have classes being offered at their school. In an- gramming in particular) are out of their reach. 
other situation, a school may offer computer science classes,
but students are either too busy with their regular classes or Acknowledgments
This research was supported by the National Science Foundation under grant number
think that because of their career preferences for the future, 1742083. We are grateful to the numerous discussions with mathematics teachers
programming (or mathematics itself ) is “not for them.” To during curriculum implementations, workshops, conferences, and alike. Their support,
knowledge, and passion make our work possible.
overcome these barriers, the CodeR4MATH project proposes
to integrate computational thinking into regular, mandatory
References
mathematics classes. We propose to give students the oppor- 1. Akpinar, Y. and Aslan, Ü. Supporting Children’s Learning of Probability Through
tunity to create, visualize, and analyze mathematical models Video Game Programming. Journal of Educational Computing Research, 53, 2
(2015).
using the programming language R as the modeling environ- 2. Allaire, J.J., Xie, Y., McPherson, J. Luraschi, J., Ushey, K., Atkins, A., Wickham, H.,
ment. At the same time, activities like the one presented in Cheng, J., Chang, W., and Iannone, R. (2019). rmarkdown: Dynamic Documents for
R. R package version 1.14; https://rmarkdown.rstudio.com. Accessed 2019 July 26.
this study, provide teachers with several instructional tools 3. Blum, W. and Leiβ, D. (2007). How do students and teachers deal with modelling
that they can use in the classroom to promote integration problems? In C. Haines, P. Galbraith, W. Blum, & S. Khan (Eds.), Mathematical
modelling (ICTMA 12): Education, Engineering and Economics. (Chichester, UK:
between CT and CS into their regular math classes, without Horwood Publishing, 2007), 222-231.
having to open up room to a foreign subject. 4. Bruner, J. S. From communication to language - a psychological perspective.
Cognition, 3, 3 (1974), 255-287.
Here we described the results from the implementation of 5. Bureau of Labor Statistics Employment Projections data for 2018-2028; https://
our pilot module called CodeR4MATH on a population of 42 www.bls.gov/ooh/computer-and-information-technology/computer-and-
information-research-scientists.htm. Accessed 2019 October 30.
senior high school students enrolled in discrete mathematics. 6. Google Inc. & Gallup Inc. Trends in the State of Computer Science in U.S. K-12
The module is composed of several self-paced tutorials that Schools; https://news.gallup.com/reports/196379/trends-state-computer-science-
schools.aspx. Accessed 2019 July 26.
challenge students to model solutions for practical problems 7. Gilbert, J.K. On the Nature of “Context” in Chemical Education, International Journal
like the cost of eating in college, the real costs of owning a car, of Science Education, 28, 9 (2006), 957-976.
8. Guidelines for Assessment & Instruction in Mathematical Modeling Education
deciding between careers, etc. In other words, the module was (2016); https://www.siam.org/Publications/Reports. Accessed 2019 August 6.
designed to bring open-ended problems that are relatable to 9. McMaster, K., Anderson, N., and Rague, B. Discrete math with programming: better
together. ACM SIGCSE Bulletin, 39, 1 (2007),100-104
someone entering adult life. All but three students had no prior 10. R Core Team. R: A language and environment for statistical computing.
experience with programming, and the whole group had never R Foundation for Statistical Computing, Vienna, Austria; https://www.R-project.
org/. Accessed 2019 July 26.
enrolled in computer science classes before. 11. Rose D.E. Context-Based Learning. In: Seel N.M. (eds) Encyclopedia of the Sciences
Our results are undoubtedly encouraging. After a total of six of Learning. (Springer, Boston, MA, 2012).
12. Schloerke, B., Allaire, J.J., and Borges B. learnr: Interactive Tutorials for R. R
hours of interaction with the tutorials, including an assessment, package version 0.9.2.1; https://CRAN.R-project.org/package=learnr. Accessed 2019
a third of the students who participated in the implementation July 26.
13. State of Computer Science Education (2018); https://code.org/files/2018_state_of_
declared that they became interested in taking programming cs.pdf. Accessed 2019 July 26.
classes in the future (2 students, or 5%, said they were already 14. Tawfik, A.A., Law, V., Ge, X., Xing, W., and Kim, K. The effect of sustained vs.
faded scaffolding on students’ argumentation in ill-structured problem solving.
interested in exploring this possibility in the future). Computers in Human Behavior, 87 (2018), 436–449.
Students cited a few reasons for becoming interested in pro- 15. United States Census Bureau (2018); https://www.census.gov. Accessed 2019
August 9.
gramming that we put into three major groups: they felt that 16. Wing, J. Computational Thinking. Communications of the ACM 49, 3 (2006), 33-35.
coding is a powerful tool, that it can be used to work on and
solve real-world problems, and some students simply became Kenia Wiedemann
interested because the whole experience was out-of-the-ordi- Research Associate
The Concord Consortium
nary use of technology, which was something new to them. 25 Love Lane, Concord, MA USA
We also observed that for the rest of the students, that is, those kwiedemann@concord.org
who did not care about programming and that said their interest Jie Chao
did not change, the primary reason we identified was self-con- Research Scientist
The Concord Consortium
cept. Twenty-one out of the 26 students who said that they were 25 Love Lane, Concord, MA USA
still not (or less) interested in programming, justified their senti- jchao@concord.org
ment because they believe that some people are naturally good at Benjamin Galluzzo
mathematics and others are simply not, identifying themselves as Associate Professor
Institute for STEM Education
not being a math person. Although it is common sense that there Clarkson University
is no such thing as a math person, it is unfortunate that these 8 Clarkson Avenue, Potsdam, NY USA
bgalluzz@clarkson.edu
youngsters may leave high school (whether or not they decide
Eric Simoneau
to go to college after that) without developing true appreciation CEO at 33 Sigma Labs
for exact sciences. It may be difficult for a high school teacher to 22 College Avenue, Arlington, MA USA
Mathematics Teacher at Boston Latin School
change this mindset on students who reached high school and 78 Avenue Louis Pasteur, Boston, MA USA
are preparing to ingress in the adult life. However, we hope that mrsimoneau@gmail.com
the results from this classroom experience can provide some in-
sights to teachers so that they can help their students to abandon DOI: 10.1145/3380956 ©2020 ACM 2153-2184/20/03 $15.00

Heliyon 7 (2021) e07922
Contents lists available at ScienceDirect
Heliyon
journal homepage: www.cell.com/heliyon
Research article
Learning number patterns through computational thinking activities: A

Rasch model analysis
Shiau-Wei Chan a, *, Chee-Kit Looi b, Weng Kin Ho b, Wendy Huang b, Peter Seow b, Longkai Wu b
a
Faculty of Technology Management and Business, Universiti Tun Hussein Onn Malaysia, 86400, Parit Raja, Batu Pahat, Johor, Malaysia
b
National Institute of Education, Nanyang Technological University, 1 Nanyang Walk, 637616, Singapore
A R T I C L E I N F O A B S T R A C T
Keywords: Despite the increasing presence of computational thinking (CT) in the mathematics context, the connection be-
Number patterns tween CT and mathematics in a practical classroom context is an important area for further research. This study
Computational thinking intends to investigate the impact of CT activities in the topic of number patterns on the learning performance of
Rasch model
secondary students in Singapore. The Rasch model analysis was employed to assess differences of ability between
Quasi-experiment
Mathematics education
students from the experimental group and control group. 106 Secondary One students (age 13 years old) from a
secondary school in Singapore took part in this study. A quasi-experimental non-equivalent groups design was
utilized where 70 students were assigned into the experimental group, and 36 students were assigned into the
control group. The experimental group was given intervention with CT-infused activities both on- and off-
computer, while the control group received no such intervention. Both groups were administered the pretest
before the intervention and the posttest after the intervention. The data gathered were analyzed using the partial
credit version of the Rasch model. Analysis of pretest and posttest results revealed that the performance of the
experimental group was similar to the control group. The findings did not support the hypothesis that integrating
CT in lessons can result in improved mathematics learning. However, the drastic improvement was observed in
individual students from the experimental group, while there is no obvious or extreme improvement for the
students from the control group. This study provides some new empirical evidence and practical contributions to
the infusion of CT practices in the mathematics classroom.
1. Introduction and vice versa (Weintrop et al., 2016; Ho et al., 2017). It can also
acquaint learners with the practice of mathematics in the real world
In recent years, computational thinking (CT) has become a topic of (Weintrop et al., 2016) and cultivate students' ability to acquire knowl-
high interest in mathematics education (Broley et al., 2017) because edge and apply it to new situations (Kallia et al., 2021). Taking into ac-
some, such as English (2018), have considered CT and mathematics to be count these advantages, many researchers and educators began to
natural companions. In a similar vein, Gadanidis et al. (2017) argued that integrate CT in the mathematics classroom. Thus, there is a growing body
a natural connection existed between CT and mathematics in terms of of literature on CT and mathematics education, and this includes review
logical structure and in the capability to explore and model mathematical studies conducted by Barcelos et al. (2018), Hickmott et al. (2018), and
relationships. Kallia et al. (2021) claimed that both CT and mathematical Kallia et al. (2021).
thinking approach thinking by adopting concepts of cognition, meta- While acknowledging that coding or digital making activities are the
cognition, and dispositions central to problem-solving. In addition, they most prevalent means of learning CT, the authors of this paper have
also recognize and foster socio-cultural learning opportunities that shape joined a growing set of researchers and educators who are stretching the
ways of thinking and practicing that reflect the real world. boundaries of how CT can be learned or used to enrich existing school
The key motivation for introducing CT practice into mathematics subjects. In this project, CT is defined as the thought process involved in
classrooms is the fast-changing nature of mathematics in professional formulating problems and developing approaches to solving them in a
field practice (Bailey and Borwein, 2011). The infusion of CT into manner that can be implemented with a computer (human or machine)
mathematics lessons can deepen and enrich the learning of mathematics, (Wing, 2011). This process involves several problem-solving skills,
* Corresponding author.
E-mail address: swchan@uthm.edu.my (S.-W. Chan).
https://doi.org/10.1016/j.heliyon.2021.e07922
Received 20 May 2021; Received in revised form 27 July 2021; Accepted 31 August 2021
2405-8440/© 2021 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-
nc-nd/4.0/).
S.-W. Chan et al. Heliyon 7 (2021) e07922
including abstraction, decomposition, pattern recognition, and algo- sub-problems can be solved easily. Meanwhile, pattern recognition was
rithmic thinking. The term Math þ C refers to the integration of CT and the action of finding common patterns, features, trends, or regularities in
mathematics in the design and enactment of lessons (Ho et al., 2021). data. It was common in mathematical practice to find out the patterns
Some of the Singapore secondary schools have started to teach CT in from the disordered data. Besides, abstraction was the procedure of
the mathematics curriculum. For instance, a recent study was conducted formulating the general principles that create these identified patterns. It
by Lee et al. (2021) with secondary students in the School of Science and happened when a problem in the real-world setting was expressed in
Technology, Singapore (SST). Descriptive qualitative research was per- mathematical terms. Algorithm design was the development of accurate
formed with 51 Secondary 2 students of mixed to high ability in the step-by-step instructions or recipes to solve the problem at hand or
study. Students were asked to solve a mathematical problem about similar problems. The solution to the problem can be computed by using
quadratic functions by writing a program using Python programming a computer program.
language. The students also had to complete the individual student's Many efforts and initiatives have been conducted to bring CT into the
reflections and worksheets that had a series of questions involving the mathematics curriculum and cover various mathematical topics, such as
four components of CT, i.e., decomposition, pattern recognition, geometry and measurement (Sinclair and Patterson, 2018; Pei et al.,
abstraction, and algorithm design. The results showed that CT can assist 2018), algebra (Sanford, 2018), number and operations (Sung et al.,
students to strengthen the learning of mathematics process and synthe- 2017), calculus (Benakli et al., 2017), and statistics and probability
size their mathematical concepts. In addition, student responses to CT (Fidelis Costa et al., 2017). Weintrop et al.’s (2016) taxonomy of CT
questions and survey questions also appear to strongly indicate that math practices for science and mathematics, which included data practices,
questions support students to apply CT skills. modeling & simulation practices, computational problem-solving prac-
Nevertheless, there are some challenges when infusing CT into the tices, and systems thinking practices, has provided comprehensive de-
mathematics classroom. One of the challenges is the insufficiency of CT scriptions of how CT was applied across science, technology,
expertise. Most of the teachers do not have computing or computer sci- engineering, and math (STEM) subjects. Engineers and scientists use CT
ence backgrounds as they do not take computer science courses during and mathematics to construct accurate and predictive models, to analyze
their studies. They even do not receive any training or are exposed to the data, as well as to execute investigations in new ways (Wilkerson and
pedagogies on how to teach CT effectively, so they lack confidence in Fenwick, 2016).
teaching CT in math class. Besides, students from backgrounds with little
experience in computers and students with learning difficulties found it 2.2. Earlier studies of the impact of CT on mathematics performance
difficult to keep up with the course (Choon and Sim, 2021).
This study provides an initial exploration of the link between CT and A recent quasi-experiment was performed by Rodriguez-Martínez
mathematics at the secondary level. It aims to determine the effect of et al. (2020) to investigate the impact of Scratch on the development of
Math þ C lessons on students’ performance on typical test problems in CT and the acquisition of mathematical concepts of sixth-grade students.
the lower secondary mathematics topic of number patterns. Based on the 47 students from a primary school in Spain were divided into an exper-
findings from the previous studies, it is hypothesized that the CT research imental group (24 students) and a control group (23 students). There
done in this study would have a positive impact on the math learning were two phases in this experiment, namely the programming phase and
performance of secondary students in Singapore. This can be examined the mathematics phase. The programming phase was associated with the
from the comparison of the ability between the experimental students instructions in Scratch and emphasized acquiring the basic concepts of
and the control students. By using the Rasch model analysis, we can CT such as conditionals, iterations, events-handling, and sequences.
analyze the results as a pattern among the scores of individual students, Meanwhile, the mathematics phase stressed instruction on calculating
not only aggregated data. Specifically, this study was guided by one the least common multiple (LCM) and the greatest common divisor
overarching research question: What are the differences of ability be- (GCD) and solving problems involving these concepts. The experimental
tween the students from the experimental group and the control group? group employed Scratch as a pedagogical tool, while the control group
worked in a traditional paper-and-pencil environment. The pretest and
2. Literature review posttest were administered before and after the instruction. The findings
seem to demonstrate that Scratch can be employed to augment both
2.1. Computational thinking and mathematics students’ CT and mathematical concepts.
Fidelis Costa et al. (2017) implemented a quasi-experiment to explore
The earliest mention of the phrase ‘CT’ appeared in Mindstorm: the influence of CT on the math problem-solving ability of 8th-grade
Children, computers, and powerful ideas, in which Papert (1980) briefly students. The students were selected randomly into the experimental
used the term to describe a kind of thinking that might be integrated into group with CT and the control group without CT. The experimental group
everyday life. The term CT was recently refreshed by Wing (2006), in was trained using the mathematics questions that were prepared to be
which she argued that everyone should learn CT, that CT involves more aligned with CT, while the control group was given traditional
“thinking like a computer scientist”, and that computer science should questions prepared by mathematics teachers. The findings were statisti-
not be reduced to programming. CT was further defined by Wing (2011) cally significant, indicating that the integration of CT and mathematics
as “the thought processes involved in formulating problems and their through appropriate adjustments of classroom practice can have a posi-
solutions so that the solutions are represented in a form that can be tive impact on students' problem-solving abilities.
effectively carried out by an information-processing agent” (p. 1). Another quasi-experiment was conducted by Calao et al. (2015) to
A popular breakdown of CT into identifiable practices is Hoyles and test whether the usage of coding in mathematics classes could have a
Noss (2015)'s and Tabesh (2017)'s description of CT as including pattern positive effect on 42 sixth-graders’ mathematical skills. The sample
recognition (seeing a new problem as related to previous problems), included 24 students from the experimental group and 18 students from
abstraction (seeing a problem at diverse detail levels), decomposition the control group. The experimental group students received the inter-
(solving a problem comprises solving a set of smaller problems), and vention which comprised three months of Scratch programming training,
algorithm design (seeing tasks as to smaller associated discrete steps). To while the control group received no such intervention. Both the experi-
operationalize the definition of CT in this project, Ho et al. (2021) mental group and the control group performed pre-intervention and
adapted these four processes in the mathematics context. Decomposition post-intervention tests. A rubric had been developed to assess students’
referred to the procedure by which the mathematical problem was skills and performance in the mathematical processes including
broken down into smaller sub-problems. Through decomposition, the reasoning, modeling, exercising, and formulation, and problem-solving.
complicated or complex problem became more manageable as smaller The findings demonstrated that the experimental group trained in
2
Scratch has a statistically significant improvement in the understanding analyze the parts of a number sequence. Regarding algorithm design, the
of mathematical knowledge. students were led to understand the steps or processes to obtain a solu-
tion for number sequence. The students ought to generate the general
3. Materials and methods formula of the number sequence for the abstraction.
The lesson objectives for the unplugged Math þ C activities were: (a)
3.1. Instructional design of CT activities recognizing simple patterns from various number sequences, and (b)
determining the next few terms and find the general formula of a number
Singapore students learn to find patterns in number sequences since sequence. During the unplugged Math þ C lesson, the students were
primary year one. In the secondary year one topic of Numbers and asked to guess the next two terms for the sequence “2, 5, 8,11, 14, …, …”.
Algebra, students are expected to learn how to recognize and represent The term “number sequence” and “terms” were introduced. These
patterns or relationships by finding an algebraic expression for the nth numbers followed specific rules or patterns, e.g., start with 2 and add 3 to
term. each term to get the next term. Next, the students did the exercises in the
textbook. The teacher highlighted the rules governing number sequences
3.1.1. Instructional design of plugged Math þ C activities using a spreadsheet need not an addition but can be subtraction, multiplication, and division
The instructional design of plugged Math þ C activities using a and can involve negative numbers. The students completed the ‘Practice
spreadsheet was demonstrated in Table 1. There were four-lesson design Now 1’ in the textbook to write down the next two terms. Upon
principles for integrating CT into number patterns lessons anchored on completion, the students were called one by one to answer the questions.
the CT practices, i.e., complexity principle, data principle, mathematics Then, the students were given the Math þ C worksheets and had to
principle, and computability principle (Ho et al., 2021). The first prin- complete pages 1 and 2 to find the general formula for number patterns.
ciple was the complexity principle. The task related to the learning of the The teacher stressed the patterns in the last column titled “further
Number Patterns topic was sufficiently complex to be decomposed into breakdown the numbers”, namely, what does the nth term mean with
sub-tasks. If the task was routine and can be solved easily using a regards to the position of the number and the relationship between the
well-known and simple approach, then the decomposition cannot be position and the numbers that changed in the last column. The students
applied well. The second principle was the data principle. The task ought had to find the general formula based on the worksheet and simplify the
to include the quantifiable and observable data which could be utilized, expression. Next, the students completed the questions on pages 3 and 4
transformed, treated, and stored. Besides, the mathematics principle was in the Math þ C worksheets. The teacher discussed the solutions with the
regarded as the third principle. We needed to identify whether the task students. Lastly, the teacher recapped the lesson and assigned homework
gave rise to a problem or situation that could be mathematized. Mathe- to the students.
matization was the construction of the problem using mathematical
terms. It involved changing a real-world problem setting accurately and 3.2. Lesson in the control group
abstractly to a mathematical problem. The task should be formulated
abstractly so that it could be reasoned, described, and represented In the lessons for the control group, the students were not given Math
meticulously. The last principle was the computability principle where þ C intervention. They were taught the topic of number patterns using
we were required to ensure that the solution to the task could be executed the two worksheets as demonstrated in Table 3. In the worksheet, the
on a computer via a finite process (Ho et al., 2021). students were required to find the next two terms for the number se-
In this lesson, we believe that CT can help students deepen their quences. The teacher asked the students to come to the front to show
understanding of number patterns in two ways. First, recursive re- their answers on the whiteboard. Next, the students were given the
lationships can be easily coded in any programming language. This al- general term of a number sequence and they had to find the terms of the
gorithm method always provides an accurate (numerical) solution to the sequence. The students were guided on how to obtain the 1001st term for
problem, even if there is no closed formula—the power of the numerical the sequence of 2, 4, 6, 8, … For this question, the general term is Tn ¼ 2n,
method. Secondly, the graphical features in the spreadsheet are T1001 ¼ 2 (1001) ¼ 2002. Then, the students were taught how to generate
employed to analyze the relationship between n and Tn (Ho et al., 2021). the general term from the number sequences in the worksheet. The stu-
The computational tool used in the plugged Math þ C activities was a dents had to complete the questions in the worksheets. After that, the
spreadsheet. A spreadsheet was chosen as it was readily accessible and teacher discussed the solutions with the students and gave them home-
available in most of the schools. It has been widely applied in education work. Further, the teacher also gave worksheet 2 to the students and
as a computational tool to promote CT skills (Sanford, 2018). In this discussed it with them.
study, it was utilized to teach number sequences which enabled the
learners to make sense of recursive and explicit formulas. The spread- 3.3. Participants
sheet served as the valuable element to visualize the Left-Hand-Side
(LHS) and Right-Hand-Side (RHS) of “sum ¼ product” characteristics In this study, a quasi-experimental non-equivalent groups design was
in a synchronized way, to comprehend and differentiate between the employed where the students were non-randomly chosen to be in the
explicit forms and recursion forms of LHS ¼ RHS characteristics, as well experimental group or the control group (Lochmiller, 2018). There were
as to link the visualizations which lead to the understanding of formal 106 Secondary One students (who have the age of 13 years old) from a
proof and mathematical induction (Caglayan, 2017). Singapore secondary school who participated in this study. These stu-
dents were chosen as this was the first year they entered secondary school
3.1.2. Instructional design of unplugged Math þ C activities which would have provided us with the longest track for students' CT
Unplugged Math þ C activities played a transitional role in that skills development. Three intact classes were involved: two classes were
teachers and students who have never used computers in learning assigned to the experimental group and one class was assigned to the
mathematics can still acquire or use some aspects of CT without the use of control group. These three classes were taught by three different teachers
the computer. Math þ C worksheets were created to be used in the un- as the teachers who taught mathematics subjects were different. There
plugged Math þ C activities. The CT practices were utilized in the Math þ were 70 students in the experimental group and 36 students in the
C worksheets, namely pattern recognition, decomposition, algorithm control group. In the experimental group, there were 37 males and 33
design, and abstraction. The sample items of Math þ C worksheets were females. Meanwhile, there were 20 males and 16 females in the control
demonstrated in Table 2. Through guided questions, the students were group. The sample size of both groups was large enough to ascertain that
required to use pattern recognition to recognize the common difference the results of the results were well established and justified as Gall et al.
of the number sequence. For the decomposition, the students had to (1996) claimed that there should be at least 15 participants in
3
Table 1. Instructional design for plugged Math þ C activities using a spreadsheet.
Teaching Move CT Practices Description of Instructional Design

Induction Pattern The students worked in pairs and set up driver-navigator roles. The navigator was the one who handles the computer part, and the driver was the
recognition one who gave instructions. The students were given hands-on worksheets to complete the tasks. The students were shown a video about patterns.
Then, the students recalled prior knowledge about number patterns.
Development I Algorithm Design The students were taught the concepts of CT, a way of thinking about and solving problems so that we can make use of the computers to help us. The
students were informed about the lesson objectives, i.e., (a) learn how to generate number patterns using computers, and (b) construct the general
term of a given number pattern.
Pattern The teacher demonstrated how to use the spreadsheet software. The teacher introduced the spreadsheet as a large array of boxes that we called
Recognition ‘cells’ and each cell was a place to store a piece of datum. Each cell was named by its coordinates, which we learned in mathematics. For example,
the default position of the cursor rested in the cell named A1. After that, the teacher provided guidance to the students to enter the word
‘Figure Number (n)’ in cell A1. A cell can be populated with English words which were data called ‘Strings’. The teacher showed how to enter the
number ‘1’ in A2 and a cell can be populated with numerical data called ‘Numbers’. Then, the teacher demonstrated how to enter the formula ¼ A2
þ 1.
Algorithm Design The teacher hovered the cursor near the corner of cell A3, and the Fill Handle appeared with a small cross. The students dragged the Fill Handle
downwards till it reached cell A11, i.e., cell range A2:A11 as displayed in the figure below. The students were taught about the recursion method for
number patterns.
Development Algorithm Design The students created the algorithm that calculates the nth term of the number pattern based on the questions in the hands-on worksheet as exhibited
II in the figure below.
Then, the students produced the final product in the spreadsheet as shown in the figure below.
Decomposition The students broke down the problem into two smaller problems: Problem 1: What is the starting number? Where to key in?; Problem 2: How do I
use the recursive method to generate the number pattern? The students used the spreadsheet to find T10 and T100.
(continued on next page)
4
Table 1 (continued )
Development Pattern When the number n gets larger, the recursive method is troublesome to use, even though we have a computer. If we have a formula in terms of n,
III Recognition such a direct method will be faster. To find the direct relationship between n and Tn, the students drew a scatter plot as a graphical representation to
show the patterns as exhibited in the figure below.
Pattern The students were shown a ‘staircase’ structure in the scatter plot as illustrated in the figure below.
Recognition
After that, the students filled in the blanks based on the ‘staircase’ scatter plot as displayed in the figure below.
Abstraction The students generated the general formula using abstraction skills as shown in the figure below. The nth term of the arithmetic sequence was given
by Tn ¼ T1 þ ðn 1Þd; where T1 is the first term of the sequence and d is the common difference. Finally, the student constructed the general
formula of 4n þ 2.
Algorithm Design The students checked the abstracted formula using the spreadsheet as revealed in the figure below. It was found that the answer to the direct
method was the same as the recursive method.
(continued on next page)
5
Table 1 (continued )
Consolidation Algorithm Design The students did the hands-on exercise in the worksheets. Lastly, the teacher recapped the lesson.
Table 2. Sample items of Math þ C worksheets.
CT Practices Sample Items

Pattern recognition 13, 17, 21, 25, …
What is the difference between each term in the sequence?
Decomposition
Algorithm design Let's further investigate.

1. What do you notice about the highlighted numbers in the last column?
2. What is the relationship between the position number and the highlighted number?
Abstraction What is the general formula that we can use to find any number in the sequence that you see in the previous page?
experimental and control groups for comparison. Following Institutional tests was an arithmetic sequence, and there was only one question of
Review Board (IRB) regulations, students’ consent to participate in this quadratic sequence and geometric sequence. In this study, all the items
study was requested. Their data were kept confidential and anonymous, were under the numeral category, except for Q4A (figural). The skills
for instance, A in A15 referred to the experimental group, 15 represented tested and the sample items in the number patterns were demonstrated in
the student ID, B in B23 referred to the control group, 23 represented the Table 4. There was only one item (Q4A) with a dichotomous response
student ID. which had two values of 0 and 1. The other items were the polytomous
responses which had more than two values such as 0, 1, 2; 0, 1, 2, 3, and
so on (Bond and Fox, 2015).
3.4. Instrumentation The construct validity of the pretest and posttest were identified
through assessing unidimensionality. The unidimensionality assumption
The pretest and posttest were designed and constructed based on the was investigated using the Principal Components Analysis of Rasch
topic of Number Patterns in the Singapore Secondary Mathematics syl- measures and residuals. It can be asserted that the data is fundamentally
labus. They were validated by the experts who were two authors of this one-dimensional if the Rasch measurement indicates a relatively elevated
paper to make sure that the items were appropriate to the targeted percentage of explained variance (at least 40 percent) and the first re-
construct and assessment objectives. The scope covered in the pretest and sidual components of the unexplained variances are less than 2 eigen-
posttest included recognizing simple patterns from various number se- values (Linacre, 2012). All the unexplained values are less than 15%,
quences, determining the next few terms and finding a formula for the showing supporting unidimensionality (Bond and Fox, 2015). The values
general term of a number sequence, and solving problems involving of raw variance explained by measures were 55.6% and 59.9% for pretest
number sequences and number patterns. The pretest and posttest and posttest respectively. They were viewed as strongly measured vari-
comprised short answer items including fill-in-the-blank questions. Both ances as they were greater than 40%. It means that the construct validity
tests had eight items respectively and were labeled as Q1, Q2, Q3, Q4A, was good for both tests.
Q4B, Q4C, Q5A, and Q5B. The majority of the question types for both
6
Table 3. Sample items of two worksheets.
Worksheet Sample Items

1 Write down the next two terms in the following number sequence.
(a) 1, 3, 5, 7, ___, ___
(b) 6, 10, 14, 18, ___, ___
(c) 8, -3, -2, ___, ___
(a) If the nth term of a sequence is Tn ¼ 5n þ 4, find the first 5 terms of the sequence.
(b) If the nth term of a sequence is Tn ¼ n^2–3, find the first 4 terms of the sequence.
(c) If the nth term of a sequence is Tn ¼ - 4nþ9, find the 35th and 50th of the sequence.
Find the general term, Tn.
1, 3, 5, 7, … Tn ¼ ?
Write down the nth term in the following number sequence.
(a) 5, 9, 13, 17, 21, …
(b) 7, 12, 17, 22, 27, …
(c) 2, 4, 14, 20, 26
2 The diagram shows the first three patterns of dots in a sequence.
(a) Draw the 4th pattern of dots.

(b) How many dots are there in the 5th pattern?
Table 4. Test items.
Type Skill Tested Sample Items

Arithmetic sequence (Numeral) Identify terms of simple number sequences when given the Q1. Fill in the missing terms in the following sequence:
initial terms 41, ___, 29, 23, ___, 11
Quadratic sequence (Numeral) Identify terms of quadratic sequences when given the rule Q2. Find the 8th term of the sequence: 1, 4, 9, 16, …
Geometric sequence (Numeral) Identify terms of geometric sequences when given the rule Q3. Write down the next two terms of the number sequence:
4, 16, 64, 256, ____,____
Arithmetic sequence (Figural task with (a) Use visual cues established directly from the structure Q4. The diagram indicates a sequence of identical square cards.
successive configurations) of configurations to illustrate the pattern.
(b) Identify terms of arithmetic sequences when given the
rule
(c) Generate the rule of a pattern
(d) Obtain an unknown input value when given the
formula and an output value
(a) Draw Figure 4.

(b) Complete the following table.
Figure Number Number of squares

1 5
2 8
3 11
4
5
6
(c) Calculate the number of squares in the 130thfigure. Explain or show how you
figured out.
Arithmetic sequence (Numeral) (a) Use only cues established from any pattern when listed Q5. Consider the following number pattern:
as a sequence of numbers or tabulated in a table. 4¼23-2
(b) Generate the rule of a pattern 10 ¼ 3 4 - 2
18 ¼ 4 5 - 2
28 ¼ 5 6 - 2
⋮
208 ¼ n ðn þ 1Þ 2
(a) Write down the equation in the 6th line of the pattern.
(b) Deduce the value of n. Explain or show how you figured out.
The reliability of the pretest and posttest were determined using item 2013). Item separation was an estimate of the separation or spread of
reliability and separation indices. The item reliability showed the repli- items along with the measured variable (Bond and Fox, 2015). The item
cability of item placements along the trait continuum (Bond and Fox, separation of 4.06 for the pretest and 4.20 for the posttest indicated the
2015). It was noticed that the item reliability for pretest and posttest items were well separated by the students who took the test (Chow,
were 0.94 and 0.95, which were regarded as high (>0.90) (Qiao et al., 2013) as it was greater than 2 as suggested by Bond and Fox (2015). The
7
Outfit mean-square (MNSQ) of pretest and posttest item mean was 1.29 3.6. Rasch model analysis
and 1.01 were within the acceptance ranges of 0.5–1.5 (Boone et al.,
2014). This implied that the items were productive and acceptable for a In this study, the pretest and posttest results were analyzed using the
good measurement. partial credit version of the Rasch model. The partial credit model (PCM)
was used as it enables the likelihood of having different numbers of
3.5. Procedure response levels for different items on the same test (Bond and Fox, 2015).
It was a “unidimensional model for ratings in two or more ordered cat-
This study utilized five days to complete the data collection as egories” (Engelhard, 2013, p. 50). The software used for PCM was
revealed in Table 5. The first day and last day were used for testing, while Winsteps version 3.73. To conduct the PCM analysis, a scoring rubric for
another three days were used for instruction. The data collection was pretest and posttest was created as shown in Table 6. There were four
implemented during the mathematics period in the school. At the categories of code for PCM, i.e., code A for a maximum score of 2, code B
beginning of this study, the researcher administered the 15-minutes for a maximum score of 1, code C for a maximum score of 3, and code D
pretest to the students of the experimental group and control group. for a maximum score of 4.
Then, the experiment group was given the intervention by involving the There were some missing data in the pretest and posttest as some of
unplugged Math þ C activities and plugged Math þ C activities using a the students skipped one or more items without giving any answer. Such
spreadsheet. For the unplugged activities, the students were required to a situation may due to the item was difficult to understand or may not be
solve the problems in the Math þ C worksheets. During the computer printed on the test paper. However, the Rasch model was rarely influ-
lesson, the students worked in pairs using laptops. The control group enced by any missing data and did not require to have all the items to be
received no intervention and was given mathematics instruction the answered as it involved a single trait where the response measures can be
usual way by the teacher. After the instruction, both groups were calculated based on the items completed (Boone et al., 2014).
administered the 15-minute posttest. To determine the differences of abilities between the students from
The difference between the experimental group and the control group the experimental group and the control group, the average person mea-
can be examined from two aspects. The first aspect was that the teachers sure for all the students was computed. As Winsteps was employed to run
in the experimental group taught the students by using Math þ C a Rasch analysis of data, all persons were represented on the same linear
worksheets in the unplugged Math þ C activities where the mathematical scale. The logit units of Rasch measurement express where each person
problems were specially designed to align with CT practices, i.e., pattern was positioned on that same variable. Therefore, the person can be
recognition, abstraction, algorithm design, and decomposition. But for compared to other persons, for example, Mary has a higher ability than
the control group, the teacher utilized the routine mathematical prob- Julia (Boone et al., 2014). The logit value person (LVP) was conducted to
lems to teach the students without the use of CT practices. The second compare the level of students' abilities from the experimental group and
aspect was that the students from the experimental group were instructed control group in the pretest and posttest. In other words, students'
using a computational tool which was a spreadsheet, but the students mathematical knowledge in number patterns from both groups was
from the control group were taught without any computational tool. measured and compared. The Wright map was also utilized to present the
students’ abilities and item difficulties comprehensively. Three criteria,
i.e., Outfit mean-square (MNSQ), Outfit z-standardized (ZSTD), and
Point-measure correlation (Pt-Measure Corr) were used for person-fit
analysis (Bond and Fox, 2015).
Table 5. The procedure of data collection.
Day Time Activities Remark 4. Findings and discussion

1 10:40am–12:00pm - Administer pretest 30 students from
- Conduct unplugged experimental group involved The Rasch analysis demonstrated that the average mean of pretest for
Math þ C lesson the experimental group and the control group was 1.22. In the posttest,
1:30pm–1:45pm Administer pretest 40 students from the average mean of the posttest for the experimental group was 1.49,
experimental group involved and the control group is 1.48. The experimental group had only 0.01 logit
2 10:40am–12:00pm - Administer pretest 36 students from control higher than the control group. This means that the abilities between the
- Conduct traditional group involved
experimental group and control were similar and not much different. The
lesson
results of this study were not consistent with the results of previous
3 12:00pm–12:50pm Conduct unplugged 30 students from
Math þ C lesson experimental group involved
studies with a positive impact of CT on students’ understanding of
mathematical knowledge such as Calao et al. (2015), Fidelis Costa et al.
12:00pm–12:50pm Conduct traditional 36 students from control
lesson group involved (2017).
12:50pm–2:15pm Conduct plugged Math þ 40 students from The possible reason for this occurrence may be due to the students in
C computer lesson using experimental group involved the experimental group being involved in CT activities for a short
spreadsheet and they work in pairs using intervention time. The plugged Math þ C activities using a spreadsheet
laptops had only one session and the duration was less than one and a half hours,
4 8:40am–10:00am Conduct unplugged 40 students from while the unplugged Math þ C activities were less than two hours in
Math þ C lesson experimental group involved
duration. Experimental students might have only participated in certain
9:20am–10:40am Conduct plugged Math þ 30 students from
features of the problem-solving process in CT activities, but these features
C computer lesson using experimental group involved
spreadsheet and they work in pairs using are not enough to reveal significant differences with control students.
laptops This was supported by the statement from Wright and Sabin (2007) who
9:20am–10:40am Conduct traditional 36 students from control said that learning may not happen if sessions were too short. The warrant
lesson group involved to suggest that CT elements of the Math þ C treatment influenced stu-
5 9:20am–9:35am Administer posttest 36 students from control dents’ learning needs to be further investigated through interviews.
group involved Another possible reason was that the students might not so familiar
11:20am–11:35am Administer posttest 40 students from with the operations in the spreadsheet required for the plugged Math þ C
experimental group involved activities. Such a situation might hinder the effects of the intervention in
12:00pm–12:15pm Administer posttest 30 students from the experimental group. According to cognitive load theory, the design of
experimental group involved
teaching materials needs to follow some principles in order to effectively
8
Table 6. Scoring rubric for pretest and posttest.
Question Code Score Description Examples of Answers in Pretest Examples of Answers in Posttest
Q1 A 2 Complete or correct response 35, 17 1, -17
1 Only correctly answer one of 35, 16 1, 17
the solutions
0 Incorrect response 30, 16 3, 6
Q2 A 2 Complete or correct response 64 45
1 Provide evidence of correct þ3, þ5, þ7, þ9… 26 (þ2, þ3, þ4…)
way to find the number
sequences, but obtain
incorrect solution
0 Incorrect response 40 T0 ¼ -1
Tn ¼ -1þ2n
T9 ¼ -1 þ 2 (9) ¼ 17
Q3 A 2 Complete or correct response 1024, 4096 729, 2187
1 Only correctly answer one of 1024, 6020 -
the solutions
0 Incorrect response 556, 664 17496, 3779136
Q4(a) B 1 Complete or correct drawing
4
0 Incorrect drawing
6
Q4(b) C 3 Complete or correct response 14, 17, 20 14, 17, 20

2 Only correctly answer two of 14, 17, 21 14, 17, 21
the solutions
1 Only correctly answer one of 14, 18, 22 14, 18, 21
the solutions
0 Incorrect response 15, 18, 21 16, 22, 29
Q4(c) D 4 Complete or correct response 124 3 ¼ 372, 372 þ 20 ¼ 392, The pattern was No. The general equation is 2þ3n. So, 136-2 ¼ 134. 1343 ¼ 44
and justification plus three all the way so I subtract the figure 2
. The result to that equation is a decimal hence there is no figure
number that had been shown on the table with 3
figure number 130 and I get 124 then I multiply it number in the sequence that contains 136 bricks.
by the pattern, 3
3 Obtain correct response, but - No. 131 divided by 3 is a decimal.
partially correct justification
2 Obtain correct response, but 392. 130 2 ¼ 260, 260 þ 132 ¼ 392 No. 5 þ 3 (95-1) ¼ 137
incorrect justification 5 þ 3 (44-1) ¼ 134
1 Provide evidence of correct 130 3 ¼ 390, 390 þ 5 ¼ 395 -
way to find the solution, but
obtain incorrect solution
0 Incorrect response and 394. I calculated with my calculator and I got this Yes. Seeing that the number of bricks increases by 3, 136-3 ¼ 133.
justification So, 136 bricks can be possible.
Q5(a) A 2 Complete or correct response 54 ¼ 7 8 - 2 96 ¼ 3 62–2 6
1 Partially correct response 7 8–2 or 54 96
0 Incorrect response 6 x (6 þ 1) – 2 ¼ 40 3 62–2 5
Q5(b) D 4 Complete or correct response n ¼ 14. Guess and check method n ¼ 11. I used guess and check on my calculator. I started with 3
and justification 14 x (14 þ 1) -2 ¼ 208 8 8–2 8 until I got 3 11 11–2 11 ¼ 341
3 Obtain correct response, but 14. I guessed the answer 11. I used guess and check to figure it out
partially correct justification
2 Obtain correct response or 208 ¼ 14 15-2 11. I used algebra.
justification, but incorrect
response or justification
1 Provide evidence of correct - -
way to find the solution, but
obtain incorrect solution
0 Incorrect response and 208–4 ¼ 204, 204 2 ¼ 408 3 x n2 – 2 x n ¼ 341
justification 3n2 – 2n ¼ 341 n2 ¼ 341
n ¼ 18.466
9
utilize spreadsheet to learn mathematics. If you need to learn specific

spreadsheet skills first to be useful in learning mathematics, then the Table 8. Logit value person (LVP) analysis for control group.
sequencing order is critical. The interactivity and intrinsic cognitive load Group Very High High Moderate Low
of the elements of spreadsheets and math tasks are high. If two learning LVP > þ1.62 þ 1:62 LVP þ þ 1:13 LVP þ LVP þ
tasks are performed at the same time, the cognitive load is likely to in- 1:13 0:64 0:64
crease. Therefore, the learning of these two tasks may be constrained; Pretest 6 14 8 8
hence, in order to maximize the learning of mathematics, you should first Posttest 10 14 8 4
master and consolidate spreadsheet skills (Clarke et al., 2005). When the
students were not competent enough in the use of the spreadsheet, the
students from both groups also were found to be above the item logit
students had to learn some about the spreadsheet in addition to number
average (0.00), except for one student (A30) from the experimental
sequences. This would cause them not to assign all their cognitive load to
group. It can be said that the overall performance of the students from
the learning contents and consequently inhibit the intervention.
both groups was over the expected performance.
The logit value person (LVP) analysis for the experimental group can
In Figure 1 and Figure 2, item Q5B was the most difficult item to be
be seen in Table 7. The students were classified into four levels of abil-
solved by the students from the experimental group and control group as
ities, namely very high level, high level, moderate level, and low level.
the item difficulty was above the person measure average. This means
The grouping of students’ abilities was based on the mean (1.17) and
that the probability for the students to solve this item accurately was less
standard deviation (0.54) of all LVPs. For instance, the LVP value for a
than 50%. Item Q5B asked the students to explain how to deduce the
very high-level group was gained from the sum of the mean and standard
value of n. Hence, it can be asserted that the students always faced dif-
deviation, i.e. 1.71. It was noticed that there were 13 out of 70 students
ficulties in generating algebraic rules from patterns as argued by Stacey
(19%) from the experimental group had a very high of CT in the pretest.
and MacGregor (2001). The easiest item in both tests was item Q4A as it
Meanwhile, 22 of them (31%) possessed a high ability of CT. 26 of them
was positioned below the item logit average and at the lowest part of the
(37%) had a moderate ability and 9 of them (13%) had a low ability. For
logit scale. It was categorized as an arithmetic sequence (Figural task
the posttest, the high ability was achieved by the 31 students from the
with successive configurations). Students were required to draw the
experimental group, i.e. 44%. This followed by 17 of them (24%) had a
fourth identical square card in item Q4A. All the students from both
very high ability, 15 of them (21%) had a moderate ability and 7 of them
groups were able to solve this type of item easily. This finding was
(10%) had a low ability.
contrasted with the study of Becker and Rivera (2006) that claimed that
Table 8 demonstrates the logit value person (LVP) analysis for the
students fail to recognize the figural patterns.
control group. Each group's level of abilities was computed based on the
From Figure 1, item Q1 and item Q4B in the pretest were located
mean value of 1.13 and standard deviation value of 0.49. For example,
below the item logit average. This implied that these two items were easy
the value of LVP for the low-level group was obtained from the difference
for the students to solve. Item Q1 asked the students to fill in the missing
between the mean and standard deviation, which was 0.64. 6 out of 36
terms in the arithmetic sequence. This item involves operations on a
students (17%) from the control group had a very high ability of CT in the
single specific number in order to find the next term in the sequence. 69
pretest, while 14 out of 36 students (39%) had a high ability of CT. 8 of
out of 70 experimental group students (99%) were able to solve item Q1,
them (11%) had a moderate ability and 8 of them (11%) had a low
while all the control group students were able to solve this item correctly.
ability. There were 10 students from the control group who had very high
Meanwhile, item Q4B required the students to identify terms of arith-
ability in the posttest, which was 28%. Students with high ability were 14
metic sequences when given a rule. Item Q4B was solved correctly by 68
(39%), students with moderate ability were 8 (22%) and students with
experimental group students (97%) and all the control group students.
low ability were 4 (11%). From Tables 7 and 8, it can be said that some
Item Q2 asked the students to recognize the eighth term of quadratic
students from both groups have improved their abilities from low and
sequences, while item Q3 required the students to find the next two terms
moderate ability to high and very high ability. The percentage for
of geometric sequences. These two items were positioned at the logit 0.00
experimental group students with high and very high ability increased by
which indicated that half of the students obtained correct answers and
13% and 5% respectively. Meanwhile, the percentage for control group
another half of the students obtained an incorrect answer. Furthermore,
students with very high ability enhanced by 11%, but the percentage for
the students were requested to write down the equation in the sixth line
high ability remained the same.
of the pattern based on the number pattern shown for item Q5A. Seven of
Figure 1 and Figure 2 display the Wright map which visually pre-
the experimental group students (10%) answered this item wrongly,
senting the distribution of items and person-measures in an equal logit
while one of the control group students (3%) answered it wrongly. For
scale (Bond and Fox, 2015). The left side of the Wright map showed the
item Q4C, 30 out of 70 students from the experimental group (43%)
difficulty of items, while the right side showed the ability of the students.
solved it incorrectly and 11 out of 36 students from the control group
Logit 0 was the average of the test items. The items with higher difficulty
(31%) solved it incorrectly. Such a situation was probably due to the
levels were at the top left of the logit scale, while the items with lower
students might not know how to generate the rule of a pattern and obtain
difficulty levels were at the bottom left of the logit scale. By comparing
an unknown input value.
Figure 1 and Figure 2, it can be seen that the logit scale increase from
Regarding the posttest, three items (Q3, Q1, Q4B) were also situated
þ2.0 to þ3.0. Meanwhile, the person average for the pretest was þ1.22
below the item logit average as shown in Figure 2. All the students from
logits and the person average for the posttest was þ1.49 logits. The
the experimental group solved these three items correctly, except for
increment was þ0.27 logits. This indicated that the performance of the
item Q4B. 69 of them (99%) were able to solve item Q4B. Meanwhile, all
students from both groups became greater and a number of them were
these items were solved successfully by all the students from the control
able to solve the difficult items. In Figure 2, we can see that all the
group. Item Q5A was answered by two students from the experimental
group (3%) and all the students from the control group unsuccessfully.
For item Q2, 5 students from the experimental group and one student
Table 7. Logit value person (LVP) analysis for experimental group. from the control group solved it wrongly. Meanwhile, 18 experimental
Group Very High High Moderate Low
group students (26%) and 10 control group students (28%) were unable
LVP > þ1.71 þ 1:71 LVP þ þ 1:17 LVP þ LVP þ to answer item Q4C.
1:17 0:63 0:63 Several students in the experimental group had poor performance in
Pretest 13 22 26 9 the pretest as their logits were below the mean item logit 0.00 including
Posttest 17 31 15 7 A19 (-0.90 logits), A66 (-0.39 logits), A05 (-0.09 logits), and A49 (-0.08
10
Figure 1. The Wright map for pretest.
logits). After they were exposed to the intervention, their performance “misfitting” students from the experimental group. These two students
was improved significantly and their logits were located above the mean (B18 and B28) had Outfit MNSQ, Outfit ZSTD, and Pt-Measure Corr that
item logit 0.00 in the posttest, i.e., A19 (þ0.40 logits), A66 (þ0.27 log- did not in the range. It means that they had an unusual response pattern
its), A05 (þ1.92 logits), and A49 (þ0.88 logits). There were also students in the pretest. These unusual response patterns can be further scrutinized
in the experimental group whose performance augmented drastically, for by looking at the Guttman Scalogram as exhibited in Figure 3. In other
instance, students A53 and A59. Their performance in the pretest was words, we can examine the causes of these unusual response patterns that
located at þ0.10 and þ0.84 logits respectively. But after they partici- did not fit the model through the Guttman Scalogram.
pated in the CT activities, their performance in the posttest increased A Guttman Scalogram comprises a unidimensional set of items that
almost three times until maximum logits, i.e., þ3.24. On the other hand, were ranked in order of difficulty (Bond and Fox, 2015), where item 4
for the students from the control group, there was only one student B15 (Q4A) was the easiest item and item 8 (Q5B) was the most difficult item.
who improved by þ2.12 logits. It can be said that there was no obvious or It can be observed that student B18 obtained correct for the difficult
extreme improvement for the students from the control group compared items, but did wrongly for the easier item. For student B28, he or she also
to the experimental group. managed to solve the difficult items but obtained the wrong answer for
As seen in Figure 1 and Figure 2, there were some big gaps between the easiest item in the pretest. It was most probably the student who
the items of the pretest and posttest, indicating the need for additional made the careless mistake when solving the item.
items to fill the gaps. The Wright map also demonstrated the redundancy Regarding the posttest, three students were considered as misfit
of the item or item with the same difficulty level such as items Q2 and Q3 persons, i.e., A22, A33, and B09. This was because of their Outfit MNSQ,
in the pretest. Q4A was the item-free person in the pretest and too easy Outfit ZSTD, and Pt-Measure Corr that did not fulfill the range. In the
for the students to solve it correctly. In the posttest, there was three items Guttman Scalogram in Figure 4, these three students tended to answer
free person which means all the students were able to answer correctly, the difficult items correctly but solved the easy items wrongly. Most
i.e. Q1, Q3, and Q4A. Besides, there were no items that cover the top of probably they knew how to solve the item, but made a careless mistake
the scale where the best-performing students were located. This sug- during the test. All these misfit students were considered as the person
gested that the more difficult items are required to assess the full range of under-fits the model (Aghekyan, 2020).
person abilities.
To detect the “misfitting” students, person-fit statistics were 5. Conclusion, implications and recommendations
employed including Outfit MNSQ, Outfit ZSTD, and Pt-Measure Corr. The
person-fit indicated how well the responses given by the students This study focused on examining the differences in performances
matched with the model used to produce the level of attainment (Walker between the students from the experimental group and the control group
and Engelhard, 2016). In the pretest, there were two most “misfitting” in learning the mathematics topic of number sequences. It was found that
students in the control group, i.e., B18, and B28. There were no the academic performances of the students in the experiment group were
11
Figure 2. The Wright map for posttest.
Figure 3. Guttman Scalogram of responses for pretest.
similar to that of the control group. This indicated that the intervention of
Figure 4. Guttman Scalogram of responses for posttest.
plugged Math þ C activities using a spreadsheet and unplugged Math þ C
activities did not have much impact on the learning performance of the
students in number patterns. Thus, the CT activities did not influence the the students in the control group experienced no obvious or extreme
learning of the students in mathematics. The results did not support the improvement.
hypothesis that the quasi-experimental done in this study would have a Although the results did not meet the expectations, this study still
positive impact on the learning performance of secondary students in provides some new empirical evidence and practical contributions to the
Singapore. This situation might be due to the short intervention time for integration of CT practices in the mathematics classroom. It also adds to
the experimental group students who are not so familiar with spreadsheet the literature review on the effectiveness of didactic activities that in-
operations required in the learning of number sequences. Nevertheless, it volves CT in mathematics instruction, especially at the secondary level.
can be observed that the performance of several students from the Despite several challenges such as teacher education and institutional
experimental group improved from pretest to posttest drastically, while willingness are not fully invested in integration (Pollak and Ebner, 2019),
12
this research shows the possibilities to bring CT into existing school Funding statement
subjects. The development of the CT instructions and assessments serve
as resources for the school teachers, curriculum developers, researchers, This work was supported by the Singapore Ministry of Education’s
administrators, and policymakers. These resources give them a clearer project grant (OER 10/18 LCK) and by Universiti Tun Hussein Onn
and concrete set of practices to guide curriculum development and Malaysia and the UTHM Publisher’s Office via Publication Fund E15216.
classroom implementation of CT concepts.
The Rasch analysis performed in this study allows the researchers Data availability statement
and instructors to identify whether the tests work for this pool of par-
ticipants and able to differentiate participants according to their ability Data will be made available on request.
level, as well as uncover outlier responses for person-fit analysis. The
use of the Wright map also provides direction for the researchers to Declaration of interests statement
perform refinement and resolve some negative issues in further studies
such as gaps between the items, and item redundancy. For further The authors declare no conflict of interest.
investigation, it is recommended to add more items to reduce the
existing gaps between the items and revise the items that had the same Additional information
level of difficulty, as well as create items that can fully assess the
abilities of the students. No additional information is available for this paper.
This study has developed valid and reliable pretest and posttest for
measuring mathematical knowledge of number patterns, which can be References
used for any kind of pedagogical intervention on this topic, including
ones involving CT. This work proposes a path to address the blank spot of Aghekyan, R., 2020. Validation of the SIEVEA instrument using the Rasch analysis. Int. J.
a lack of validated tools in mathematics education, specifically at the Educ. Res. 103.
Bailey, D., Borwein, J.M., 2011. Exploratory experimentation and computation. Not. AMS
lower secondary level. The report of acceptable validity and reliability
58 (10), 1410–1419.
evidence ensure the quality of the research findings. This allows the in- Barcelos, T.S., Munoz, R., Villarroel, R., Merino, E., Silveira, I.F., 2018. Mathematics
structors and researchers to utilize these assessments in the classroom learning through computational thinking activities: a systematic literature review.
J. Univers. Comput. Sci. 24 (7), 815–845.
confidently and strengthens the wide distribution of instruments as they
Becker, J.R., Rivera, F., 2006. Establishing and justifying algebraic generalization at the
tend to employ valid and reliable tools (Tang et al., 2020). sixth grade level. In: Novotna, J., Moraova, H., Kratka, M., Stehlíkova, N. (Eds.),
There were several limitations found in this study. One limitation was Proceeding of the 28th Annual Meeting of the North American Chapter of the
that it was hard to get more time for the intervention due to the tight International Group for the Psychology of Mathematics Education, 2. Charles
University, Prague, pp. 95–101.
timetables in the school. Therefore, a longer duration is needed in future Benakli, N., Kostadinov, B., Satyanarayana, A., Singh, S., 2017. Introducing
studies to promote the students’ learning in mathematics, as well as to computational thinking through hands-on projects using R with applications to
obtain additional evidence. Furthermore, there was only a small number calculus, probability and data analysis. Int. J. Math. Educ. Sci. Technol. 48 (3),
393─427.
of students involved in this study which could not be representative of Bond, T.G., Fox, C.M., 2015. Applying the Rasch Model: Fundamental Measurement in the
the whole population. Therefore, it is suggested that the number of Human Sciences, third ed. Routledge, New York, NY.
participants in future studies can be increased. Further studies need to be Boone, W.J., Staver, J.R., Yale, M.S., 2014. Rasch Analysis in the Human Sciences.
Springer, Dordrecht, The Netherlands.
executed in terms of different mathematics topics, grade level, age, Bortz, W.W., Gautam, A., Tatar, D., Lipscomb, K., 2020. Missing in measurement: why
school, computational tools, and so forth. identifying learning in integrated domains is so hard. J. Sci. Educ. Technol. 29 (1),
Another limitation was that this study only involved only a single data 121–136.
Broley, L., Buteau, C., Muller, E., 2017. February). (Legitimate peripheral) computational
source for the assessment, which was the pretest and posttest. Such a
thinking in mathematics. In: Dooley, T., Gueudet, G. (Eds.), Proceedings of the Tenth
situation might limit the final verification result (Barcelos et al., 2018). It congress of the European Mathematical Society for Research in Mathematics
is suggested to conduct the triangulation of the method by com- Education. DCU Institute of Education and ERME, Dublin, pp. 2515–2523.
Caglayan, G., 2017. A multi-representational approach to teaching number sequences:
plementing the quantitative method with the qualitative method such as
making sense of recursive and explicit formulas via spreadsheets. Spreadsheets Educ.
interviews and observation to obtain more comprehensive and richer 10 (1), 1–21.
data in the research (Neuman, 2014). Calao, L.A., Moreno-Leon, J., Correa, H.E., Robles, G., 2015. Developing mathematical
Additionally, we only developed the instrument to measure stu- thinking with Scratch: an experiment with 6th grade students. In: Conole, G.,
Klobucar, T., Rensing, C., Konert, J., Lavoue, E. (Eds.), Design for Teaching and
dents' mathematical knowledge in number patterns, but there was no Learning in a Networked World. Springer, Toledo, Spain, pp. 17–27.
specific instrument to measure CT. Hence, in future studies, it is rec- Choon, F., Sim, S., 2021. Computational thinking in mathematics (grade 2-6): developing
ommended that we develop assessments to measure both CT and CT skills and 21st century competencies. In: Looi, C.K., Wadhwa, B., Dagiene, V.,
Liew, B.K., Seow, P., Kee, Y.H., Wu, L.K., Leong, H.W. (Eds.), Proceedings of the 5th
mathematics as the identification of domains assessed by specific APSCE International Computational Thinking and STEM in Education Teachers
assessment tasks may instrumental for the interplay of integrated Forum 2021. National Institute of Education, Singapore, pp. 60–62.
domains in an assessment context, thereby gaining a more compre- Chow, J., 2013. Comparing students’ citizenship concepts with likert-scale. In: Zhang, Q.,
Yang, H. (Eds.), Pacific Rim Objective Measurement Symposium (PROMS) 2012
hensive understanding of students ’strengths and weaknesses in Conference Proceeding. Springer, Berlin.
various domains (Bortz et al., 2020). Clarke, T., Ayres, P., Sweller, J., 2005. The impact of sequencing and prior knowledge on
learning mathematics through spreadsheet applications. Educ. Technol. Res. Dev. 53
(3), 15–24.
Engelhard Jr., G., 2013. Invariant Measurement: Using Rasch Models in the Social,
Declarations Behavioral, and Health Sciences. Routledge, New York, NY.
English, L., 2018. On MTL’s second milestone: exploring computational thinking and
mathematics learning. Math. Think. Learn. 20 (1), 1─2.
Author contribution statement Fidelis Costa, E.J., Sampaio Campos, L.M.R., Serey Guerrero, D.D., 2017. Computational
thinking in mathematics education: a joint approach to encourage problem-solving
Weng Kin Ho: Conceived and designed the experiments; Performed ability. In: Proceedings of 2017 IEEE Frontiers in Education Conference (FIE).
Indianapolis, IN, USA: IEEE.
the experiments.
Gadanidis, G., Hughes, J.M., Minniti, L., White, B.J.G., 2017. Computational thinking,
Wendy Huang: Performed the experiments; Wrote the paper. grade 1 students and the binomial theorem. Digital Exp. Math. Educ. 3 (2), 77–96.
Shiau-Wei Chan: Analyzed and interpreted the data; Wrote the paper. Gall, M.D., Borg, W.R., Gall, J.P., 1996. Educational Research: an Introduction. Longman
Peter Seow, Longkai Wu: Contributed reagents, materials, analysis Publishing.
Hickmott, D., Prieto-Rodriguez, E., Holmes, K., 2018. A scoping review of studies on
tools or data. computational thinking in K–12 mathematics classrooms. Digital Exp. Math. Educ.
Chee-Kit Looi: Performed the experiments; Wrote the paper. 1─22.
13
Ho, W.K., Toh, P.C., Tay, E.G., Teo, K.M., Shutler, P.M.E., Yap, R.A.S., 2017. The role of Rodriguez-Martínez, J.A., Gonzalez-Calero, J.A., Saez-Lopez, J.M., 2020. Computational
computation in teaching and learning mathematics at the tertiary level. In: Kim, D., thinking and mathematics using Scratch: an experiment with sixth-grade students.
Shin, J., Park, J., Kang, H. (Eds.), 2017 ICJSME Proceedings: Future Directions and Interact. Learn. Environ. 28 (3), 316–327.
Issues in Mathematics Education, 2, p. 441─448. Sanford, J., 2018. Introducing computational thinking through spreadsheets. In:
Ho, W.K., Looi, C.K., Huang, W., Seow, P., Wu, L., 2021. Computational Thinking in Khine, M.S. (Ed.), Computational Thinking in the STEM Disciplines, p. 99─124.
Mathematics: to be or not to be, that is the question. In: Toh, T.L., Choy, B.H. (Eds.), Sinclair, N., Patterson, M., 2018. The dynamic geometrisation of computer programming.
Mathematics – Connection And beyond: Yearbook 2020 Association Of Mathematics Math. Think. Learn. 20 (1), 54─74.
Educators. World Scientific Publishing, Singapore, pp. 205–234. Stacey, K., MacGregor, M., 2001. Curriculum reform and approaches to algebra. In:
Hoyles, C., Noss, R., 2015. Revisiting Programming to Enhance Mathematics learning. Math þ Sutherland, R., Rojano, T., Bell, A., Lins, R. (Eds.), Perspectives on School Algebra.
Coding Symposium Western University. Western University, London, Ontario, Canada. Kluwer Academic Publishers, Dordrecht, Netherlands, pp. 141–154.
Kallia, M., van Borkulo, S.P., Drijvers, P., Barendsen, E., Tolboom, J., 2021. Sung, W., Ahn, J., Black, J.B., 2017. Introducing computational thinking to young
Characterising computational thinking in mathematics education: a literature- learners: practicing computational perspectives through embodiment in mathematics
informed Delphi study. Research in Mathematics Education. education. Technol. Knowl. Learn. 22 (3), 443─463.
Lee, T.Y.S., Tang, W.Q.J., Pang, H.T.R., 2021. Computational thinking in the mathematics Tabesh, Y., 2017. Computational thinking: a 21st century skill [Special issue]. Olymp.
classroom. In: Looi, C.K., Wadhwa, B., Dagiene, V., Liew, B.K., Seow, P., Kee, Y.H., Info. 11, 65─70.
Wu, L.K., Leong, H.W. (Eds.), Proceedings of the 5th APSCE International Tang, X., Yin, Y., Lin, Q., Hadad, R., Zhai, X., 2020. Assessing computational thinking: a
Computational Thinking and STEM in Education Teachers Forum 2021. National systematic review of empirical studies. Comput. Educ. 148, 1–22.
Institute of Education, Singapore. Walker, A.A., Engelhard, G., 2016. Using person fit and person response functions to
Linacre, J.M., 2012. Winsteps 3.75. Rasch Model Computer Program, Chicago, Il, examine the validity of person scores in computer adaptive tests. In: Zhang, Q. (Ed.),
pp. 1991–2012. Pacific Rim Objective Measurement Symposium (PROMS) 2015 Conference
Lochmiller, C.R., 2018. Complementary Research Methods for Educational Leadership Proceedings. Springer, Singapore.
and Policy Studies. Palgrave Macmillan, Cham, Switzerland. Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L., Wilensky, U., 2016.
Neuman, W.L., 2014. Social Research Methods: Qualitative and Quantitative Approaches, Defining computational thinking for mathematics and science classrooms. J. Sci.
seventh ed. United States of America: Pearson Education Limited. Educ. Technol. 25 (1), 127─147.
Papert, S., 1980. Mindstorms: Children, Computers, and Powerful Ideas. Basic Books, New Wilkerson, M., Fenwick, M., 2016. The practice of using mathematics and computational
York, NY. thinking. In: Schwarz, C.V., Passmore, C., Reiser, B.J. (Eds.), Helping Students Make
Pei, C., Weintrop, D., Wilensky, U., 2018. Cultivating computational thinking practices Sense of the World Using Next Generation Science and Engineering Practices.
and mathematical habits of mind in Lattice Land. Math. Think. Learn. 20 (1), 75─89. National Science Teachers’ Association Press, Arlington, VA.
Pollak, M., Ebner, M., 2019. The missing link to computational thinking. Future Internet Wing, J.M., 2006. Computational thinking. Commun. ACM 49 (3), 33─35.
11 (263), 1–13. Wing, J., 2011. Computational Thinking – what and Why? The Magazine of Carnegie
Qiao, J., Abu, K.N.L., Kamal, B., 2013. Motivation and Arabic learning achievement: a Mellon University School of Computer Science, Research Notebook. March 2011. The
comparative study between two types of Islamic schools in gansu, China. In: LINK.
Zhang, Q., Yang, H. (Eds.), Pacific Rim Objective Measurement Symposium (PROMS) Wright, B.A., Sabin, A.T., 2007. Perceptual learning: how much daily training is enough?
2012 Conference Proceeding. Springer, Berlin. Exp. Brain Res. 180, 727–736.
14
Journal of Mathematical Behavior 47 (2017) 70–80
Journal of Mathematical Behavior

journal homepage: www.elsevier.com/locate/jmathb
Reflective abstraction in computational thinking MARK

a,⁎ b
Ibrahim Cetin , Ed Dubinsky
a
Mathematics and Science Education Department, Abant Izzet Baysal University, 324 Education Faculty, Bolu, 14280, Turkey
b
651 Raven Ave., Miami Springs, FL, 33166, USA
AR TI CLE I NF O AB S T R A CT
Keywords: Computational thinking has become an increasingly popular notion in K-12 and college level
Computational thinking education. Although researchers have accepted that abstraction is a central concept in compu-
Mental structures tational thinking, they are quick to disagree on the meaning of it. A focus on reflective ab-
Mental mechanism straction has led to the development of APOS Theory in Mathematics education. This has resulted
Reflective abstraction
in many cases of improved student learning in Mathematics (Arnon et al., 2013). Our main aim in
Essence
APOS theory
this paper is to construct a theoretical bridge between computational thinking and APOS Theory
and show that reflective abstraction can be used in the context of computational thinking.
1. Introduction
Wing (2006) made computational thinking popular and proposed that it is not a skill only useful for computer scientists, but
rather a fundamental skill that should be learned by everyone. She argued that “to reading, writing and arithmetic, we should add
computational thinking to every child’s analytical ability” (Wing, 2006, p. 33). Participants of the workshop on The Scope and Nature
of Computational Thinking (National Research Council, 2010) considered the question “Why should students learn computational
thinking?” They contended that computational thinking is helpful in (i) succeeding in a technological society, (ii) maintaining
interest in the information technology profession, (iii) maintaining and enhancing economic competitiveness, (iv) supporting inquiry
in other disciplines, e.g. biology, physics, earth sciences and psychology, (v) and enabling personal empowerment. These arguments
were influential in a broad spectrum of the academic community and computing courses have become increasingly popular in K-12
education.
Actually the notion of computational thinking is not new to the education community. Although computational thinking has been
given significant recognition only recently, it has been present in academic discourse under different forms for decades. Of the
numerous definitions for the computational thinking found in the literature we will utilize the definition of Cuny, Snider, and Wing
(2010): “the thought process involved in formulating problems and their solutions so that the solutions are represented in a form that
can be effectively carried out by an information-processing agent” (as cited in Wing, 2011, p. 20).
Despite the popularity of computing courses today, students’ difficulties in learning computer science related concepts are well
documented in the literature (Boticki, Barisic, Martin, & Drljevic, 2013; Robins, Rountree, & Rountree, 2003). One of the central skills
that students should gain in computational thinking is abstraction (Wing, 2006). Piaget developed the notion of reflective abstraction
to describe the children’s construction of abstract logico-mathematical structures (Beth & Piaget, 1966). He mainly distinguished
three types of abstraction: empirical, pseudo-empirical, and reflective abstraction. Our main purpose in this study is to show that
reflective abstraction can be used as a tool in the study of computational thinking. For this purpose we will discuss: (i) different views
related to abstraction and how Piaget conceptualized reflective abstraction; (ii) APOS Theory as a mathematical learning theory, its
⁎
Corresponding author.
E-mail addresses: ibretin@ibu.edu.tr (I. Cetin), edd@math.kent.edu (E. Dubinsky).
http://dx.doi.org/10.1016/j.jmathb.2017.06.004
Received 4 November 2016; Received in revised form 2 June 2017; Accepted 20 June 2017
Available online 11 July 2017
0732-3123/ © 2017 Elsevier Inc. All rights reserved.
I. Cetin, E. Dubinsky Journal of Mathematical Behavior 47 (2017) 70–80
supporting pedagogy and the role of computational thinking in pedagogy; and (iii) the nature of abstraction and the role of reflective
abstraction in computational thinking.
2. Reflective abstraction
In this section we will discuss different views of abstraction in mathematics, describe Piaget’s notion of reflective abstraction,
discuss the role of reflective abstraction in mathematics and give examples of different views of abstractions.
2.1. Abstraction
The term abstraction means different things to different people and in different contexts. The main meanings we shall consider in
this chapter are what we will call, extraction, decontextualization, and essence. The most common, but not necessarily the most
important meaning of abstraction of a concept in computer science and mathematics, is extraction, that is, the idea of considering
common features of several (the more the merrier) examples and building a structure or category which has all of these features.
Thus, these features are all present in all of the examples, but some or all of the examples may also have other features not necessarily
possessed by all of the examples. Consider, for instance, all pieces of programming code which have the following features: a control
mechanism which is either a variable or is internal and is set at an initial value; variables whose initial values are set; a sequence of
lines of code which includes one or more commands that update the control mechanism but may or may not be reached by the flow of
control; operations on one or more of the variables that may or may not change their values; a test on the control mechanism; and a
transfer mechanism which transfers control out of the sequence of code or returns to the beginning of the sequence of lines of code
depending on the result of the test of the counter variable. Such a piece of code is called a loop and the abstraction consists of
considering any sequence of code having all of the features in the above paragraph to be a loop. Of course, a loop may have other
features not shared by all loops, such as printing out the values of some of the variables, changing the values of one or more variables
outside of the loop, etc.
The idea of extraction is very close to Piaget’s notion of empirical abstraction. According to Piaget (Beth & Piaget, 1966), empirical
abstraction consists in creating a category by deriving common characteristics from a class of objects. It is largely based on per-
ception, as opposed to reflection. For example, a child may develop the concept of “dog” after seeing many examples, called “dog” by
adults, of dogs with four feet, a tail that it wags and a propensity to bark. This conception is good enough to identify animals not
previously seen as dogs, although not with perfect accuracy as such a child may go to the zoo and consider that hyenas and lion cubs
are dogs. In mathematics, a child may develop a primitive notion of numbers such as “three” by considering many collections of three
physical objects and extracting the common property of “threeness”. Again, this conception is limited and not very useful for larger
numbers or operations on numbers.
Extraction is not a sufficiently powerful mechanism of abstraction even though there is a common belief that most, if not all,
abstract mathematical concepts are constructed by means of extraction. An important counter-example is the concept of mathe-
matical group. It is true that there is a vast number of examples of groups in Mathematics and other Sciences. This has led many
people to conclude that the concept arose from extracting the four properties in the modern Mathematical definition of group from
these examples. But this is not the case. In fact, Galois developed the concept of group, using only one category of examples- the set of
permutations of the roots of a polynomial (objects) and composition of two such permutations (binary operation). This strongly
suggests that there was something else present in addition to, indeed, before, extraction in the development of the concept of group.
For richer conceptions moving beyond extraction, Piaget considered two additional types of abstraction: pseudo-empirical abstrac-
tion and reflective abstraction. We will consider these in Section 2.2.
A second meaning ascribed by some to abstraction is decontextualization. Many authors feel that thinking about a concept
independently of any context is what makes abstraction difficult (Gravemeijer & Doorman, 1999). Therefore, these authors feel that
students should mainly be taught mathematical concepts in some “real-world” context. There are three things that are questionable
about such a replacement of decontextualization. First, what is “real-world” will be different for different individuals. You can’t use
stops on a subway to teach arithmetic to children who have never seen a subway, much less taken a ride on one. Second, there is a
real danger that the result of pedagogy focusing on a context will result in the students perhaps learning something about the context,
but little or nothing about the underlying Mathematics. Finally, there does not seem to be much, if any, research results that the use of
“real-world” or “realistic” contexts is helpful to students who are trying to learn Mathematics that is not based on “real-world”
contexts. We will discuss in the next section how Piaget’s concept of reflective abstraction includes decentralization and is an effective
alternative to extraction.
2.2. Piaget’s concept of reflective abstraction
An alternative to thinking of abstraction as extraction is to emphasize the essence of a concept (more information about our
notion of “essence” will be found in Section 4). Piaget considered that an individual’s understanding of a concept was constructed by
that individual (in a social context) in her or his mind. In many works, Piaget described what he believed was the mental mechanism
by which an individual can construct all mathematical concepts, at all levels, that is, make abstractions. He called this mechanism
reflective abstraction. According to Piaget (Beth & Piaget, 1966, p. 189):
“… reflective abstraction consists in deriving from a system of actions or operations at a lower level, certain characteristics whose
71
reflections (in the quasi-physical sense of the term) upon actions or operations at a higher level it guarantees; for it is only possible
to be conscious of the processes of an earlier construction through a reconstruction on a new plane…reflective abstraction
proceeds by reconstructions which transcend, while integrating previous constructions.”
In other words, reflective abstraction has two components: reflecting on operations on a lower level and reconstructing and
integrating them on a higher level. Piaget also studied a third form of abstraction which he called pseudo-empirical abstraction.
Pseudo-empirical abstraction will not be our concern in this study. So Piaget studied three kinds of abstraction, empirical, pseudo-
empirical and reflective, but he considered reflective abstraction to be the most important. Reflective abstraction is, in his view, the
highest form of human thinking and the basis for all development in mathematical thinking (Piaget, 1985).
Piaget gave many examples of reflective abstraction, mainly in the thinking of children. One example is the construction of the
concept that the number of objects in a collection is the same no matter the order in which you count them. The young child will
perform the relatively low level (in terms of thinking) action of counting the objects in a collection, mixing them up, and recounting
them. After repeating these actions several times and reflecting on them, the child will reconstruct them on the higher plane of
thought. That is, the child will no longer actually count the objects in the physical world, the world that is external to her or him, but
will perform the counting mentally, that is, in her or his internal world. The reconstructed actions are now operations which can be
integrated in the child’s thinking by noting that the result of the counting is always the same and this phenomenon becomes a part of
the child’s knowledge. Other examples of reflective abstraction both in the thinking of children and in the less elementary level of
advanced mathematical thinking can be found in Dubinsky (1991).
3. APOS theory and supporting pedagogy—computational thinking
In this section, we will focus on the profound connection between four aspects of learning mathematics: APOS Theory, the ACE
Teaching Cycle, Computational Thinking, and Reflective Abstraction. We will discuss the connection between APOS Theory and the
pedagogical strategy called the ACE Teaching Cycle and establish the connection between APOS Theory, the ACE Teaching Cycle and
Computational Thinking. We will provide some examples of how writing and running computer programs in the Mathematical
Programming Language (specifically ISETL) is used to foster the mental constructions of APOS Theory and we will give indications of
results from the literature about using this approach to help students understand and use mathematics.
3.1. Brief description of APOS theory
The key features of reflective abstraction, both in understanding how an individual’s understandings of mathematical concepts
develop and in devising pedagogical strategies for helping students in this development, are the mental constructions at a lower level
on which an individual reflects and their reconstructions and integration on a higher level. So far we have said nothing about the
nature of these mental constructions or their reconstructions and integration. The search for answers to these questions led Dubinsky
in the 1980s to develop APOS Theory and, together with colleagues, to continue its development until the present time.
The basic tenet of APOS, a constructivist theory based on Piaget’s notion of reflective abstraction, is that an individual's un-
derstanding of a mathematical topic develops through reflecting on problems and their solutions in a social context and constructing
or reconstructing certain mental structures and organizing these in schemas to use in dealing with these problem situations. These
mental structures are constructed by certain mental mechanisms which are essentially reflective abstractions. The mental structures
proposed by APOS Theory are: actions, processes, objects and schemas (and thus the acronym APOS). The structures are constructed
by means of certain mental mechanisms including interiorization, encapsulation, de-encapsulation, coordination, reversal, gen-
eralization and thematization. Fig. 1 shows the relationships among the mental mechanisms and some of the structures described by
APOS Theory.
According to APOS Theory, a mathematical concept is first understood as an action; a set of step-by-step instructions performed
explicitly to transform physical or mental objects. An individual needs external cues to apply actions to objects. Transformations are
perceived as external at this stage. The individual cannot imagine steps of the actions. For example, with the function concept, an
action would consist of plugging a value into an expression and calculating the result. The individual can form the composition of two
functions by substitution when explicit formulas for the functions are given. Nevertheless, he or she may not be able to think about
Fig. 1. Components of APOS Theory (Adapted from Arnon et al. (2013)).
72
the composition of functions in more complex situations, for example, when explicit formulas for the individual functions are not
given or when a function is defined in parts.
As an individual repeats and reflects on the action, it may be interiorized into a mental process. A process is a mental structure that
performs the same operation as the action but wholly in the mind of the individual. For the concept of function, this means that the
individual can imagine any element in the domain being transformed into an element of the range by an expression or any other
means. External cues are not needed at this stage. For example, trigonometric functions do not have explicit algebraic formulas, so the
individual limited to action conceptions of functions has difficulty in understanding such functions. The individual can, however,
make sense of trigonometric functions at the process stage. Interiorization is not the only way for the construction of a process. A
process can be constructed also by reversing a process or by coordinating two processes.
As the individual becomes aware of the total process, she/he realizes that transformations can act on it, and/or actually constructs
such transformations, the process is encapsulated into a mental object. A process is a transformation that one makes whereas an object
is a completed totality to which other actions or processes can be applied. With an object conception of function, an individual is able,
for instance, to perform arithmetic operations on functions to obtain new functions. An object can be de-encapsulated to get the
process back whenever necessary. For example, when an individual who has an object conception of functions deals with composing
two functions, he or she de-encapsulates function objects back to their original processes and then coordinates these processes to get
the process of composition which is then encapsulated to an object.
In developing an understanding of a mathematical topic, an individual may construct many actions, processes, and objects. When
these are organized and linked into a coherent framework, the individual has constructed a schema for the topic. The coherence of a
schema is what allows one to decide if it can be used in a particular mathematical situation. For example, the coherence of an
individual’s function schema might consist of an abstract definition of function: a domain set, a range set and a means of going from
an element of the domain to an element of the range. This would allow the individual to see functions in situations where “function”
is not explicitly mentioned, and use functions to solve problems.
A schema can be thematized into an object so that it can be utilized in other schemas. For example the quantification schema can
be thematized into an object and can be used in the limit schema to make sense of the formal definition of limit. A list of the mental
constructions involved in an individual’s construction of her or his understanding of a mathematical concept, together with the
mental mechanisms for constructing them and their relationships is called a genetic decomposition for the concept. For a more detailed
description of APOS Theory, see Arnon et al. (2013).
There are two ways in which APOS Theory can be and has been used. One way is to explain the difficulties students have with
developing their understanding of mathematical concepts. That is, experiments based on APOS Theory can tell us that a student who
has specific difficulties with a particular concept is not able to perform a certain action, or interiorize a certain action into a process,
or encapsulate a certain process into an object. Thus, the explanation offered by the theory for the student’s inability to learn a
specific concept is that the student has not succeeded in making the appropriate mental constructions. The second way in which
APOS Theory can be and has been used is to point to pedagogical strategies to help students overcome their difficulties in learning
various concepts in Mathematics. This is done mainly through the ACE Teaching Cycle which is discussed next in Section 3.2.
3.2. Pedagogy and the ACE teaching cycle
APOS-based instruction applies the basic tenet of APOS Theory. According to this hypothesis, students deal with mathematical
problem situations by constructing and reconstructing mental structures. The result of these constructions, according to APOS
Theory, is that with them, learning can then take place. Therefore, the first goal of APOS-based instruction for a particular concept is
to help students make the mental constructions called for in a genetic decomposition for that concept. If this happens, APOS Theory
hypothesizes, understanding the concept will not be difficult for the student and can be achieved through a number of pedagogical
strategies, either traditional or non-traditional. The ACE Teaching Cycle is a pedagogical strategy that follows very closely the APOS
view of learning mathematics by calling for activities designed to foster students making these mental constructions, classroom
discussion to help the students apply their newly developed structures to mathematical concepts, and exercises to firmly establish the
mathematics learned in the minds of students and to point to upcoming ideas.
The question then arises of what pedagogical strategies might help students perform actions, interiorize actions into processes,
encapsulate processes into objects and gather everything into coherent schemas. This is where computational thinking comes in. With
teachers applying APOS Theory in the classroom over the past 30 years, researchers have found that an effective pedagogy involves
students writing computer programs. For example, if students can express an action as a computer procedure, run it with various
inputs, and describe what the computer is doing to obtain outputs, then they will tend to interiorize this action into a process. If the
software used has the capability of treating such a procedure as data and performing operations on it, then using this feature to solve
problems helps students encapsulate the process underlying the computer procedure into an object. It has also been shown that it is
possible to foster encapsulation by having students use specially designed software to perform operations on processes, even infinite
processes (Weller et al., 2003). Working with appropriate software can help students make the mental constructions that lead to
learning mathematical concepts and provide a dynamic interactive environment for students to explore the properties of geometric
and other mathematical objects and their relationships.
The ACE Teaching Cycle consists of three components through which instruction cycles repeatedly. The first step in the cycle
consists of activities to foster the mental constructions which, according to an APOS-based analysis of the desired mathematical
content, the learner needs to make in order to develop her or his understanding of mathematical concepts. After doing the activities
for one or more concepts, the students meet in the class and work in groups. They are given paper and pencil tasks based on the
73
computer activities given in the laboratory. Although the role of the instructor is not direct teaching, he or she may provide defi-
nitions and explanations whenever necessary in the class. The instructor encourages the group discussions and guides the inter-group
discussion. Students get the opportunity to reflect on what they did in the laboratory activities in class discussions. They build on the
experience that they gained in the activities phase. Finally, students are assigned relatively traditional exercises. They again study in
groups and complete the exercises as homework. The main aim of the exercises is to reinforce what they constructed in activities and
classroom discussion phases. They have an opportunity to use mathematical notions that they learned. Occasionally they can be
required to think about notions that they will study later in the instructional sequence.
3.3. APOS theory, the ACE teaching cycle, and computational thinking
Exactly how do the activities in the ACE Cycle foster the mental constructions of APOS Theory? We recall from Section 1 the
definition of computational thinking that we are using: “the thought process involved in formulating problems and their solutions so
that the solutions are represented in a form that can be effectively carried out by an information-processing agent”. In our context, the
‘information-processing agent’ is a “mathematical programming language”. The phrase mathematical programming language refers
to a language that satisfies three properties:
1. The syntax is close to standard mathematical notation.

2. Certain mathematical features, such as functions, finite sequences, summation of the elements of a set or sequence, mod, re-
cursion, and their usual mathematical properties are supported.
3. Important data types, such as procedures and functions, are first-class objects so they can be operated on, called, modified and
returned by procedures and functions.
The first two properties tend to reduce the usual difficulties (syntax, bugs, etc.) that students have in writing programs. They also
ease the transition from mental structures to mathematical concepts in the minds of students. The third property is the most im-
portant feature of a mathematical programming language. Numerous research papers have been published providing data that
establish the fact that students writing code in a mathematical programming language can lead directly to their interiorizing an
action to a process and encapsulating a process to an object. We will discuss some examples of this use of a mathematical pro-
gramming language below, beginning in Section 3.4. A more complete discussion, examples and references can be found, for ex-
ample, in Weller et al. (2003), Dubinsky and Wilson (2013) and Arnon et al. (2013).
3.4. Examples of writing and using programs in a mathematical programming language
The ideas we have been discussing have been implemented in the mathematical programming language, ISETL. We will consider
the function concept. Consider the function, f, defined on the real numbers and represented by the expression, x3 − 7 ×2 + 4. In
many secondary schools throughout the world, functions represented by such an expression are the only ones that students encounter
and many students think that the presence of an expression is an essential feature of a function indeed, part of the definition. But this
leads to no more than an action conception of function and restricts the student to being able to do no more than replace the literal
with a number to obtain the value of the function for that number. Or, possibly, the student may learn to compose two functions by
substituting the first expression for the literal in the second expression and simplifying. Such a student does not see the power of the
generality of the function concept nor can he or she understand such related concepts as a function with split domain, the inverse of a
function, a function defined on a finite set, a function whose domain and/or range consists of objects other than numbers, a function
represented by a geometrical transformation, or the relation between points on the graph of a function and values of the function. In
addition, research (Schwarz, Dreyfus, & Bruckheimer, 1990) reports that students have great difficulty transferring from one re-
presentation for a function (e.g., set of ordered pairs) to another (e.g., graph). Almost all of these function concepts can be im-
plemented on the computer using ISETL so we can use programming in ISETL to help students develop process conceptions of
functions. We have found that having a process conception can help students overcome these difficulties (Dubinsky & Wilson, 2013).
According to Piaget’s notion of reflective abstraction, to construct a process conception, the individual must reflect on the external
action specified by the expression, which is the lower level, and reflect it onto a higher level which amounts to a reconstruction of the
external expression to the same action as that given by the expression, but now internal, residing in the mind of the individual. Our
research has shown (e.g., Breidenbach, Dubinsky, Hawks, & Nichols, 1992) that writing and a running a computer program such as
the following and thinking about what the computer is doing with this program, helps many students to interiorize their concept of
function to a process.
f: = func(x);
return x**3 − 7*x**2 + 4;
end;
Reflection begins when the student enters f(2); sees the result, −16, and is asked to explain what the computer did to get −16.
There are many topics in mathematics that require a conception of a function as an object. These include arithmetic operations on
functions, composition of two functions, the Fundamental Theorem of Calculus, and considering that the solution to a differential
equation is a function. The reason is that these topics all make use of one or more transformations of functions. Experience and
research strongly suggest that moving from a process to an object conception (and back, in many applications) is extremely difficult
and the function concept is no exception (Breidenbach et al., 1992). Fortunately, because of Property 3, a mathematical programming
74
language can be a powerful tool for helping students overcome such difficulties. Here is an example of what can be done with the
concept of the composition of two functions, using a mathematical programming language.
We help students develop the ability to write and run the following ISETL program. Here, the inputs, F and G are two variables,
each of whose value must be a procedure representing a function.
Comp: = func(F,G);
return func(x);
return F(G(x));
end;
end;
Now, if f and g are two specific procedures which represent functions, the student can enter the ISETL code Comp(f,g); or,
alternatively, f .Comp g; the return is a meaningless (to the student) short string of symbols which is the computer’s internal re-
presentation of a function. However, if the student enters the code, (f .Comp g)(5.37); then the computer returns the value of f(g
(5.37)). In this way, the functions f and g are treated as objects and transformed to another function, their composition. Of course,
tests can easily be inserted into the program Comp to deal with the issue of compatibility of the domains and ranges of the two
functions. What the mathematical programming language has done here is transform the abstract processes of two functions into
entities that can be acted upon, that is, into objects.
There are a rather large number of published reports on the results, in terms of learning, that have been obtained using the theory
and methods we have discussed. Following is a list of mathematical concepts that have been successfully treated by our approach of
using APOS Theory and the ACE Teaching Cycle, including having students write and run programs and explain them. For details
about these studies and their results, consult Arnon et al. (2013) and Weller et al. (2003).
1. In Pre-calculus, Calculus, Analysis: functions, derivative: chain rule; limit; definite integral; infinity; infinite iterative processes
and actual infinity; supremum; geometric transformations; functions of two-variables; The Fundamental Theorem of Calculus;
sequences; uncountable sets.
2. In Linear Algebra: vector spaces; linear transformations; systems of linear equations.
3. In Abstract Algebra: sets, subsets; the empty set; groups; subgroups; cosets; normality; quotient groups; equivalence structures on
sets; and isomorphisms of groups.
4. In other concepts in undergraduate mathematics: mean, standard deviation, and the Central Limit Theorem; permutations and
symmetries; mathematical induction; quantification; induction; and compactness.
5. In mathematics at the secondary level and teacher education: fractions; divisibility; place value and multi-digit structure.
4. Reflective abstraction in computational thinking
In this section, we will specifically address the role of reflective abstraction in computational thinking: we will address how
abstraction is defined in computational thinking literature and how we see it; we will complete the discussion of what abstraction
means in computational thinking from the APOS Theory perspective; we will provide recursion as an example of reflective abstraction
by using the comparison of what a computer does to execute a recursive program with what a recursive program “says”; and lastly we
will deal with the abstractions that might be done by an individual to learn recursion and loops concepts.
4.1. Abstraction in computational thinking
Abstraction has been a significant notion in computational thinking (Armoni, 2013; Dijkstra, 1972; Wing, 2006). Kramer (2007)
contended that successful computer science students are those who can deal with abstraction. Wing (2008) considered abstraction to
be a process underlying computational thinking and emphasized the need to think at multiple layers of abstraction. Frorer, Manes,
and Hazzan (1997) considered three faces of abstraction. The first, namely ignoring the details, refers to making two different things
equal by ignoring some of their properties while focusing on others. The second face, namely the relativity of the abstract, proposes
that abstract is not an absolute stance. What is abstract for one individual might be concrete for another. Moreover, an individual who
feels that a concept is abstract might feel that it is concrete after forming familiarity with the concept and achieving ease of use
related to it. The last face, namely properties, refers to considering properties of structures composed of objects and operations rather
than objects and operations itself.
Abstraction has been seen as a tool that requires working at different levels of thought. Wing (2006) suggested that “Thinking like
a computer scientist means more than being able to program a computer. It requires thinking at multiple levels of abstraction” (p.
34). Aharoni (2000) described three abstraction levels related to programming. The first (lowest) level is called programming-
language oriented. Students limited to this level need to use a specific programming language in constructing a solution to a problem.
The second level is called programming oriented. Students limited to this level do not use a specific programming language to solve a
problem, but their solutions are tied to programming languages. The last (highest) level is called programming-free. Students at this
abstraction level can construct solutions without giving any reference to programming languages.
Perrenet, Groote, and Kaasenbrood (2005) and Perrenet and Kaasenbrood (2006) studied students’ understanding of an algorithm
and constructed four levels of abstraction for the algorithm concept. The first level is called Execution in which students consider an
algorithm as a specific run on a specific concrete machine. The second level is called Program. Students interpret an algorithm as a
process in which a specific programming language is used. The third level is called Object in which an algorithm is not associated
75
with a programming language anymore. The fourth level is called Problem. At this level, students are capable of considering an
algorithm as a black box. Nevertheless, coping with abstraction can pose difficulties. Hazzan (2003) reported three different stra-
tegies students apply to reduce abstraction levels in learning computer science concepts. But we feel that rather than reduce ab-
straction it would be better to learn how to deal with it. One example of instruction to help students do this is the ACE Cycle,
described above in Section 3.2.
Although researchers have studied abstraction in computational thinking from different perspectives they converged on a defi-
nition of abstraction. They (Kramer, 2007; Liskov & Guttag, 1986; Wing, 2008) mainly focused on “ignoring the details” view of
abstraction. Wing (2008, p. 3718) expressed that “The abstraction process- deciding what details we need to highlight and what
details we can ignore- underlies computational thinking.” Ignoring the details is an example of empirical abstraction. It assumes that
an individual derives knowledge from the properties of an object (by choosing to ignore some of them while focusing on others).
Nevertheless, as suggested in Section 2, empirical abstraction is not a powerful mechanism of abstraction. For richer conceptions we
suggest reflective abstraction for concept construction. In reflective abstraction, knowledge is not drawn from the properties of the
object, rather it is drawn from the general coordinations of actions (Beth & Piaget, 1966). Therefore an alternative to thinking of
abstraction as ignoring details and focusing on similarities among objects in the details that remain, we suggest thinking of it as the
essence of a concept, and making mental constructions to form that essence. What we mean by this is a description of the concept that
is independent of any context and hence can apply to the concept in all situations in which it appears.
4.2. The essence
There are (at least) two ways in which the essence of a concept can be described. One is the formal definition of the concept and
the other is a genetic decomposition for the concept. Thus, for example, if you can apply what is known about functions in general to
a particular example, you can check if the example really is a function according to, say, the ordered pairs definition. Or, one can use
a genetic decomposition for the example. For the concept of function, a genetic decomposition is as simple as possible. The cognitive
development of the concept of function begins with an action conception of function, which consists of an explicit expression or
recipe for transforming elements of the domain to elements of the range. This is interiorized to a process conception which is the same
transformation as in the action conception of function except that it takes place in the mind of an individual whereas in the action
conception, the transformation is performed externally. Finally, the individual sees the process as a totality and encapsulates it into
an object which means that the individual can apply transformations to the process.
Each of these two formulations of essence has its advantages. For example, the ordered-pair version of essence is, in a sense, the
cleanest way to think about a function and can be very helpful in advanced thinking about functions. A domain set, a range set, and a
set of ordered pairs satisfying a single, easily checked condition is very definite and straightforward. A genetic decomposition, on the
other hand, can be more useful for investigation of properties of functions, including those that are standard, but troublesome for
many students. One example in the case of functions again is that something like the ordered-pair version is actually just one
representation of a function. In many problems, it is important to transfer to other representations, that is, the problem of, given a
function in one representation, find another representation for the same function. This is very useful if it turns out that a particular
investigation is more convenient to make using one representation than another. The literature tells us, however, that change of
representation can be very difficult for students (Bosse, Adu-Gyamfi, & Cheetham, 2011; Duval, 2006). One reason for the difficulties
students have here might be that they try to go directly from the given representation to the other (perhaps because this is what they
have been taught to do). A more effective approach is to go to the genetic decomposition and use the given representation to identify
the process of transforming a domain element to a range element and then express this process in the desired representation
(Dubinsky & Wilson, 2013). Fig. 2 is useful for helping students learn how to use this method to change from one representation of a
function to another.
Thus, if you have a function represented as, say, a set of ordered pairs, and wish to change it to a representation, as, say, an arrow
diagram, you don’t have to go directly across the bottom of the diagram, which may be difficult. Rather, you can go (on the diagram,
or mentally) to the set of ordered pairs node in the bottom of the figure, then move up to the top to see what are the domain, range
and process of the function. You can then use this information to construct an arrow diagram, that is, you move from the top down to
the arrow diagram node of the figure.
Fig. 2. Function representation transition (Adapted from Arnon et al. (2013)).
76
4.3. Recursion as an example for essence
Recursion is one of the important topics in computer science. Students have difficulties in learning the topic and teachers have
difficulty in teaching it (Benander & Benander, 2008; McCauley, Grissom, Fitzgerald, & Murphy, 2015). Due to its importance in
computer science and difficulties that students and instructors experience, researchers with different perspectives addressed the issue
in different ways in the literature. However, there is a tendency in the literature that emphasizes what a computer does to execute a
recursive program. Kahney (1989, p. 209) defined recursion “… as a process that is capable of triggering new instantiations of itself,
with control passing forward to successive instantiations and back from terminated ones.” He believed that experts have this notion
and referred it as the copies model of recursion. Kahney (1989) studied students’ mental models of the flow of the recursion. He
determined that students have copies, looping, magic/syntactic and odd mental models of recursion. George (2000) called the
forward flow in the copies model active flow and called the backward flow in copies model passive flow. Gotschi, Sanders, and Galpin
(2003) used the passive control, active control and base case to describe students’ mental models of recursion. They found that
students constructed copies, looping, magic/syntactic and odd mental models of recursion as defined by Kahney. Additionally, they
described new models – namely the active, step, return value and algebraic models- that students also constructed when learning the
recursion concept. Only the copies model is always viable in explaining the behavior of a recursive program. The active and looping
model might be viable in some cases where the solution of a recursive algorithm is built up during the active flow. Remaining models
are all non-viable. It is found that novices generally have non-viable mental models of recursion (George 2000; Sanders & Scholtz
2012). Researchers, in this approach, mainly attributed students’ difficulties in learning recursion to their construction of non-viable
mental models of recursion. Therefore they have studied how to develop innovative ways of teaching students the flow of control of
recursion (Benander & Benander, 2008; Zmuda & Hatch, 2007). Visualization has been used as the main tool to teach recursion (Dann,
Cooper, & Pausch, 2001; Stern & Naish, 2002).
The APOS theoretic view differentiates what a computer does to execute a recursive program (the flow of control of program)
from what a recursive program says. We hypothesize that what a computer does to execute a program is not related to the essence of
the recursion, rather it is related to details. We feel that the essence of recursion is related to what a recursive program says. We do
not suggest that the knowledge of what a computer does to execute a recursive program is totally useless. Computer scientists might
make use of the flow of a recursive program and trace such programs in programming, e.g. they can use it for debugging purposes.
Nevertheless, we suggest that instructors should emphasize what a program says while teaching recursion.
APOS Theory agrees with Roberts (2006, p.10) contending that “Paradoxically, even though recursion acts as a reductionist
process in the sense that it reduces each problem to a sum of its parts, writing recursive programs tends to require a holistic view of
the process. The big picture is what’s important not the details.” We believe that the essence of recursion is as follows:
Recursion is a method of solving a problem. In order to solve a problem using recursion, it is necessary that a recursive for-
mulation of the problem can be constructed. A recursive formulation of a problem consists of the following steps.
1. See the problem as that of performing an operation on an object that varies
2. Determine a base case for which the operation is immediate
3. Decompose the object into a pair that consists of a simple object for which the operation is immediate and a reduced object which
is either empty or is closer to the base case.
4. Do one of the following two steps
4a. If the reduced object is empty, perform the operation on the base case.
4b. If the reduced object is not empty, perform the operation on the reduced object using steps 3 and 4 and combine the result with
the result of performing the operation on the simple object.
We note that the fourth step refers to itself in that it requires the application of an action (or process) to an object which is not yet
fully formed. In our opinion, this is the main thing that distinguishes recursive programs from non-recursive programs and is the
major source of difficulty for students. We are investigating this in an ongoing study and it appears that the solution may lie in the
notion of totality (Weller, Arnon, & Dubinsky, 2013), which may be a new stage in APOS Theory.
In establishing a recursive formulation for a specific example, it is necessary to determine the base case in step 2, the decom-
position into a simple object and a reduced object in step 3 and the combination in step 4. The recursive formulation can be expressed
in informal language as: If the object is the base case, apply the operation and stop. Otherwise, remove a simple object and apply the
operation to the reduced object. Then combine that result with the result of applying the operation to the simple object.
It might clarify matters to see how the recursive formulation works in specific examples. We will consider two examples: compute
n! (the product of the integers 1,2,…, n) for a natural number, n; and given a finite sequence of objects, rewrite it in reverse order.
First we will apply the above recursive formulation of each of the two problems. Then we will develop preliminary genetic de-
compositions for factorial and reversal. After that, we will see how to apply this formulation in writing recursive computer programs
to solve the problem.
4.3.1. Examples of recursive formulations

4.3.1.1. n!. The object that varies is the sequence of natural numbers from 1 to n. The base case is 1 and the result of applying the
operation in this case is 1. The simple object is the largest integer in the sequence of integers and the reduced object is the sequence of
integers with the simple object removed. The combination is the product of the simple object and the result of applying the operation
to the reduced object.
4.3.1.2. Reverse the order of a sequence. The object that varies is a finite sequence of objects whose length varies. The base case is the
77
first element of the sequence and the result of applying the operation is this element. The simple object is the last element of the
sequence and the reduced object is the sequence with the simple object removed. The combination consists in adjoining the simple
object to the beginning of the sequence which is the result of applying the operation to the reduced object.
4.3.2. Genetic decompositions

4.3.2.1. Genetic decomposition for factorial. The following schemas must have been previously constructed: functions, sets, sequences.
The variable object is the sequence (1,2,…, n) of natural numbers which varies with the value of n. The base case is 1 and the
operation on this case transforms 1 to 1. The simple object is n and the reduced object is (1,2,…, n − 1) so the decomposition of (1,2,
…, n) is n, (1,2,…, n − 1).
If n = 1, the result of the operation is 1; if n > 1, then result of the operation is n times the result of the operation applied to (1,2,
…, n − 1).
The mental constructions in this genetic decomposition by constructing the variable object (1,2,…, n) as an action, which includes
doing some iterations with a few values of n. Repeating this action and reflecting on it leads to interiorizing it to a process.
Encapsulating that process into an object results from the action of multiplying n times the result of the operation applied to (1,2,…,
n − 1).
4.3.2.2. Genetic decomposition for reverse a list. The following schemas must have been previously constructed: functions, sets,
sequences. The variable object is a sequence (x1, x2, …, xn) of objects which varies with the value of n.
The base case is the sequence (x1) and the operation on this case transforms it to (x1). The simple object is the last term of the
sequence and the reduced sequence is the result of deleting the last term, so the decomposition is xn, (x1, x2,…, xn−1). If the sequence
is (x1), then the result of the operation is (x1); if the sequence is not (x1), then the operation is applied to the reduced object and the
simple object is inserted as the first element of the result.
The mental constructions in this genetic decomposition consist in constructing the variable object (x1, x2,…, xn) as an action,
interiorizing it to a process and encapsulating that process into an object. This is followed by the action of inserting the simple object
as the first element of the result of the operation applied to the reduced object.
4.3.3. Examples of recursive programs

We will use ISETL for our examples.
n!
Factorial: = func(n); $ n is a natural number
if n = 1 then return n; $ Base case
else return n*Factorial(n-1); $ Decomposition, combination
end; end;
Reverse a list
ReverseList: = func(L); $ L is a list of objects
if #L = 1 then return L; $ Base case
else return L(#L) + ReverseList(L(1.#L-1)); $ Decomposition, combination
end; end;
4.4. Loops in a computer program
Our discussion of loops in computer programs is based on that of Cetin (2015) where a more complete discussion of loops can be
found. The connection between loops and reflective abstraction lies in the genetic decomposition of loops which includes (i) pre-
action conception of loops, (ii) action conception of loops, (iii) process conception of loops and (iv) object conception of loops. In the
genetic decomposition it is assumed that students have necessary prerequisite knowledge, e.g. variable declaration and assignment,
expression, and conditional. Loops will be described in this section before giving the genetic decomposition. The description will be
considered in two headings: one-level loop and n-level loop.
4.4.1. One-level loop

A loop may or may not have one or more free variables. The free variables are assigned before the loop begins and act as constants
during the entire operation of the loop. A one-level loop begins with an initiation in which the values of one or more variables, called
control variables are set. The control variable(s) is (are) initialized before the first iteration of the loop begins and is (are) updated and
tested during each iteration of the loop. The test determines whether to do another iteration or to end the loop. The control variables
may also be used in the calculations within the loop. Next, a loop has a body of instructions. The instructions in the body are repeated
during each iteration of the loop. The body contains the instructions that change the value(s) of the control variable(s) and the bound
variables(s) and test the value of the control variable(s). It also produces any outputs that the loop may have. Finally, there may be an
afterward in which operations are performed using the value(s) of the bound variable(s). The afterward is not part of the loop.
The following eight lines of program that is written in pseudo-code is an example of a one-level loop. In this example N is a free
variable and its value is set in line 1 which is not part of the loop. S and i are bound variables and their values are set initially in lines
2, 3 which form the initiation. Lines 4–6 form the body of the loop. Here the values of the bound variable(s) are updated (lines 4, 5)
and external operations (line 6) may be performed. Line 7 is the test, which determines if there is to be another iteration of the body
78
of the loop or if control should pass to line 8 which forms the afterward of the loop.
1. N = 10;
2. S = 0;
3. i = 1;
4. S = S + i:
5. i = i + 1;
6. print S;
7. if i < = N then go to line 4;
8. print S;
4.4.2. N-level loop

A one-level loop can be seen as a single instruction and can be inserted in another one-level loop which gives a two-level loop. A
two level loop (together with other two-level or one-level loops) can be inserted in another one-level loop as a single command to get
a three-level loop. Individuals can continue doing this and construct loops of all levels. Hence an n-level loop is simply identical to a
one-level loop in which at least one of the instructions in the body of the loop is an (n − 1)-level loop where n > 1.
4.4.3. Pre-action conception of loops

Although individuals who have pre-action conception may be aware of the fact that the task will be repeated again and again,
they cannot express the syntax of a loop and they cannot use any kind of loop to construct a solution to a problem by using loops.
4.4.4. Action conception of loops

Individuals who are limited to an action conception can repeat the body of the loop until the test tells the control to pass to first
line after the loop. They need to explicitly perform each line of the body of the loop in each repetition. Instructions in the afterward
may be performed.
4.4.5. Process conception of loops

Individuals can interiorize the action of the previous stage into a process whereby the bound variables are initialized and the body
of the loop is performed repeatedly until the test tells the iteration to stop after which the instructions in the afterward are performed.
Nevertheless, they do not have to actually perform the instructions in the body of the loop, rather they can imagine the instructions in
the body being performed.
4.4.6. Object conception of loops

At this stage, individuals may consider the loop as a totality or a function having its input (a value of the free variable(s) is (are)
set), process (the loop is run until the test tells it to stop) and output (the afterward is performed). They can apply actions or processes
on the function, for example they can insert an (n − 1)-level loop into the body of a 1-level loop to obtain an n-level loop as
mentioned above.
5. Conclusion
Introductory computing courses have become increasingly popular in K-12 level education. Without any doubt abstraction is a
fundamental notion of computational thinking. We have used reflective abstraction to describe abstraction in computational
thinking. Using the concept of genetic decomposition we have made a start to applying APOS Theory to the study of student un-
derstanding of computational thinking. Application of the reflective abstraction notion, and hence APOS Theory, in computational
thinking will have consequences for both APOS Theory and computational thinking. APOS Theory was constructed in the context of
mathematics education in order to examine mental mechanisms and structures developed to learn a mathematical subject. Expanding
and applying the theory to computer science education significantly strengthens the theory as a general theory of learning. It was
used successfully to explore students’ understanding of loops (Cetin, 2015). Researchers can use APOS Theory to explore student
understandings of other computer science concepts, e.g. conditionals and recursion. As it is used for exploring students’ under-
standing of computer science concepts, APOS Theory will gain more validity and computer science education will gain insight into
students’ understanding.
Computer science education researchers will benefit from the application of APOS Theory in computational thinking. This theory
has been used successfully in mathematics education research for more than 30 years. It has produced important knowledge related
to students’ understanding and instructional design principles depending on this understanding. It has also been used as a research
framework. Computer science education research can utilize this knowledge and apply it to develop a new knowledge base related to
computational thinking. We have used APOS Theory to interpret the notion of abstraction in computational thinking and to give
meaning to students’ understanding of loops and recursion concepts. We are currently conducting a study related to recursion. We
expect to propose a new theoretical genetic decomposition for recursion which introduces the role of totality, test its validity, use it to
design instruction that aims to foster mental mechanisms that are described in the genetic decomposition, and test the effectiveness of
the instruction.
79
References
Aharoni, D. (2000). Cogito, Ergo sum! cognitive processes of students dealing with data structures. ACM SIGCSE Bulletin, 32(1), 26–30.
Armoni, M. (2013). On teaching abstraction in computer science to novices. Journal of Computers in Mathematics and Science Teaching, 32(3), 265–284.
Arnon, I., Cottrill, J., Dubinsky, E., Oktaç, A., Fuentes, S. R., Trigueros, M., et al. (2013). APOS Theory: a framework for research and curriculum development in
mathematics education. NY: Springer Science & Business Media.
Benander, A. C., & Benander, B. A. (2008). Student monks-teaching recursion in an IS or CS programming course using the Towers of Hanoi. Journal of Information
Systems Education, 19(4), 455–467.
Beth, E. W., & Piaget, J. (1966). Mathematical epistemology and psychology. Dordrecht, The Netherlands: Reidel.
Bosse, M. J., Adu-Gyamfi, K., & Cheetham, M. R. (2011). Assessing the difficulty of mathematical translations: Synthesizing the literature and novel findings.
International Electronic Journal of Mathematics Education, 6(3), 113–133.
Boticki, I., Barisic, A., Martin, S., & Drljevic, N. (2013). Teaching and learning computer science sorting algorithms with mobile devices: A case study. Computer
Applications in Engineering Education, 21(1), 41–50.
Breidenbach, D., Dubinsky, E., Hawks, J., & Nichols, D. (1992). Development of the process conception of function. Educational Studies in Mathematics, 23(3), 247–285.
Cetin, I. (2015). Students’ understanding of loops and nested loops in computer programming: An APOS theory perspective. Canadian Journal of Science, Mathematics
and Technology Education, 15(2), 155–170.
Dann, W., Cooper, S., & Pausch, R. (2001). Using visualization to teach novices recursion. ACM SIGCSE Bulletin, 33(3), 109–112.
Dijkstra, E. W. (1972). The humble programmer. Communications of the ACM, 15(10), 859–866.
Dubinsky, E., & Wilson, R. T. (2013). High school students’ understanding of the function concept. The Journal of Mathematical Behavior, 32(1), 83–101.
Dubinsky, E. (1991). Reflective abstraction in advanced mathematical thinking. In D. Tall (Ed.), Advanced mathematical thinking (pp. 95–123). Dordrecht, The
Netherlands: Kluwer.
Duval, R. (2006). A cognitive analysis of problems of comprehension in a learning of mathematics. Educational Studies in Mathematics, 61(1), 103–131.
Frorer, P., Manes, M., & Hazzan, O. (1997). Revealing the faces of abstraction. International Journal of Computers for Mathematical Learning, 2(3), 217–228.
George, C. E. (2000). EROSI-visualising recursion and discovering new errors. ACM SIGCSE Bulletin, 32(1), 305–309.
Gotschi, T., Sanders, I., & Galpin, V. (2003). Mental models of recursion. ACM SIGCSE Bulletin, 35(1), 346–350.
Gravemeijer, K., & Doorman, M. (1999). Context problems in realistic mathematics education: A calculus course as an example. Educational Studies in Mathematics,
39(1), 111–129.
Hazzan, O. (2003). How students attempt to reduce abstraction in the learning of mathematics and in the learning of computer science. Computer Science Education,
13(2), 95–122.
Kahney, H. (1989). What do novice programmers know about recursion? In E. Soloway, & J. C. Spohrer (Eds.), Studying the novice programmer (pp. 209–228). Hillsdale,
NJ: Lawrence Erlbaum.
Kramer, J. (2007). Is abstraction the key to computing? Communications of the ACM, 50(4), 36–42.
Liskov, B., & Guttag, J. (1986). Abstraction and specification in program development. Cambridge, MA: MIT Press.
McCauley, R., Grissom, S., Fitzgerald, S., & Murphy, L. (2015). Teaching and learning recursive programming: A review of the research literature. Computer Science
Education, 25(1), 37–66.
National Research Council (2010). Report of a workshop on the scope and nature of computational thinkingWashington, DC: National Academies Press.
Perrenet, J., & Kaasenbrood, E. (2006). Levels of abstraction in students' understanding of the concept of algorithm: The qualitative perspective. ACM SIGCSE Bulletin,
38(3), 270–274.
Perrenet, J., Groote, J. F., & Kaasenbrood, E. (2005). Exploring students' understanding of the concept of algorithm: Levels of abstraction. ACM SIGCSE Bulletin, 37(3),
64–68.
Piaget, J. (1985). The equilibration of cognitive structures. Cambridge, MA: Harvard University Press.
Roberts, E. (2006). Thinking recursively with Java. Hoboken, NJ: Wiley.
Robins, A., Rountree, J., & Rountree, N. (2003). Learning and teaching programming: A review and discussion. Computer Science Education, 13(2), 137–172.
Sanders, I., & Scholtz, T. (2012). First year students' understanding of the flow of control in recursive algorithms African Journal of Research in Mathematics. Science
and Technology Education, 16(3), 348–362.
Schwarz, B., Dreyfus, T., & Bruckheimer, M. (1990). A model of the function concept in a three-fold representation. Computers & Education, 14(3), 249–262.
Stern, L., & Naish, L. (2002). Visual representations for recursive algorithms. ACM SIGCSE Bulletin, 34(1), 196–200.
Weller, K., Clark, J., Dubinsky, E., Loch, S., McDonald, M., & Merkovsky, R. (2003). Student performance and attitudes in courses based on APOS Theory and the ACE
Teaching Cycle. In A. Selden, E. Dubinsky, G. Harel, & F. Hitt (Eds.), Research in collegiate mathematics education V (pp. 97–131). Providence, RI: American
Mathematical Society.
Weller, K., Arnon, A., & Dubinsky, E. (2013). Preservice teachers’ understanding of the relation between a fraction or integer and its decimal expansion: The case of and
1. Canadian Journal of Science, Mathematics and Technology, 13(3), 232–258.
Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35.
Wing, J. M. (2008). Computational thinking and thinking about computing. Philosophical Transactions of the Royal Society of London A, 366, 3717–3725.
Wing, J. M. (2011, February). Research notebook: Computational thinking- what and why? The Link Magazine, 20–23. Retrieved from https://www.scs.cmu.edu/link.
Zmuda, M., & Hatch, M. (2007). Scheduling topics for improved student comprehension of recursion. Computers & Education, 48(2), 318–328.
80
Computers and Education Open 2 (2021) 100043
Computers and Education Open

journal homepage: www.sciencedirect.com/journal/computers-and-education-open
Cognitive abilities and computational thinking at age 5: Evidence for

associations to sequencing and symbolic number comparison
Anaclara Gerosa a, *, Víctor Koleszar a, Gonzalo Tejera a, b, Leonel Gómez-Sena a, c,
Alejandra Carboni a, d
a
Centro Interdisciplinario en Cognición para la Enseñanza y el Aprendizaje, Universidad de la República, Uruguay
b
Facultad de Ingeniería, Universidad de la República, Uruguay
c
Laboratorio de Neurociencia, Facultad de Ciencias, Universidad de la República, Uruguay
d
Instituto de Fundamentos y Métodos, Facultad de Psicología, Universidad de la República, Uruguay
A R T I C L E I N F O A B S T R A C T
Keywords: Computational thinking (CT) is currently used as an umbrella term in education to describe the processes
21st century abilities associated with learning and applying computer science concepts and strategies for problem-solving. Despite a
Early years education growing body of literature on the subject within education and the development of a grand variety of tools
designed to enhance computational thinking abilities, the relations between CT and other cognitive abilities have
not been reported in young children. Understanding CT from a cognitive development perspective could be
useful for creating targeted interventions tailored for young children. In the present study, we compare a pre
viously reported computational thinking assessment with a battery of nine cognitive tests, which include fluid
intelligence, working memory, planning, sequencing, mental rotation, vocabulary, and early math precursors
such as numerical transcoding and symbolic magnitude comparison. Mixed-effects linear regressions were
implemented with CT as a dependent variable to explore the associations between our variables. A subsample of
children who participated in an educational robotics intervention were assessed in their robot programming
performance, and its outcomes were compared with their CT assessments. Results suggest temporal sequencing
ability and symbolic magnitude comparison are significant predictors of CT in kindergarten. Positive significant
associations between robot programming and CT performance are reported.
1. Introduction developmental psychology has defined several relevant skills involved

in children’s learning processes which might overlap or correlate to CT
The term “computational thinking” (CT) has gathered increasing skills, such as executive functioning, numerical skills or vocabulary.
attention from researchers, educators, and policymakers alike in the last Despite this, there is thus far little evidence on the associations between
fifteen years [10] to describe a general thinking skill related to the these cognitive abilities and CT, particularly at an early age. The present
application of computational concepts for problem solving. There is study aims to bridge this existing gap by exploring the associations be
consensus amongst researchers that learning CT early in life is a valuable tween CT and a variety of cognitive skills in a sample of kindergarten
contribution to children’s skills-set and development [32]. Despite its children.
origins dating back to the 1980s with Papert’s innovative work on robot
programming with LOGO [53] several authors point to Wing’s [76] 2. Background
seminal article titled “Computational thinking” as the starting point of
this renewed interest in the concept. Not only has academic literature on 2.1. Computational thinking
CT experienced significant growth over the last fifteen years [43],
several tools such as online resources, platforms, robots, or specific Wing contributed to the conceptualization of CT as a universal skill
curricula have been created in order to help children foster these abili (2006) broadening the scope of the CT construct into classrooms
ties. Meanwhile, the vast existing research in both cognitive and everywhere. Wing’s original definition of CT as a skill that involves
* Corresponding author.
E-mail address: agerosa@psico.edu.uy (A. Gerosa).
https://doi.org/10.1016/j.caeo.2021.100043
Received 23 February 2021; Received in revised form 28 June 2021; Accepted 9 August 2021
Available online 15 August 2021
2666-5573/© 2021 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
A. Gerosa et al. Computers and Education Open 2 (2021) 100043
“solving problems, designing systems, and understanding human on both an individual [[6], [74], [75]] and societal level [[35], [45],
behaviour, by drawing on the concepts fundamental to computer sci [61]]. During this stage in their development, children experience an
ence” [76] and later as the “thought processes involved in formulating exponential improvement in their executive function (EF) skills sup
problems and their solutions so that the solutions are represented in a ported structurally by the prefrontal cortex [[55], [81]]. EF refers to
form that can be effectively carried out by an information-processing several top-down neurocognitive processes needed for regulating
agent” Wing and collaborators [18] have been highly effective in pro thoughts, emotions, and goal-oriented behavior [[9], [23], [80]].
moting CT as a useful and attractive idea for education. CT has been Inhibitory control, working memory, and cognitive flexibility are the
embraced in educational settings to describe the thought processes three basic neurocognitive processes identified by Miyake and collabo
behind computer science and programming, gaining space within state rators [48] via factor analysis of adult performance as executive func
curricula in several countries [10]. Despite much interest, and even tions. Inhibitory control refers to the ability to effortfully control
though CT’s roots within early childhood education go back to the automatic responses and inhibit distracting stimuli in order to direct our
influential work of Papert in the 1980s under constructivist paradigms attention. Working memory is defined as our capacity to maintain in
[53], CT remains an evolving concept [69]. Despite the growing litera formation “online” for manipulation, while cognitive flexibility refers to
ture on CT, there has been ample debate among researchers as to what our ability to shift attention between tasks, attributes, or strategies [48],
does an operationalized definition of CT entail. Many of the proposed [80] and allows us to adjust our responses in the face of change. These
definitions include a very wide range of skills that encompass cognitive basic executive functions (cognitive flexibility developing a bit later
abilities (such as pattern recognition or abstraction) as well as than the former two) have been proposed by Diamond and Ling [24] as
socio-emotional dispositions (such as collaboration or persistence). the basis for other complex cognitive skills such as planning,
Lowe and Brophy [43] identified up to twenty-five related constructs problem-solving, and reasoning. Reasoning has been defined as the
while reviewing CT definitions, ultimately reducing their findings to process of concluding from premises, principles or evidence, using
nine main processes: abstraction, decomposition, pattern recognition previous information to infer or deduct a new conclusion [28].
and generalization, algorithms, data, parallels, iteration, simulation, Meanwhile, problem-solving is a broad term referring to the steps
and debugging. In a systematic literature review analyzing publication that allow moving from a given state to a desired outcome [5]. While
trends regarding the CT concept, Haseski et al., [34] found a spike in both concepts certainly overlap, problem-solving can be considered the
academic publications during the last two decades. However, their practical goal-oriented counterpart to reasoning’s abstraction. Both
findings suggest that while overall publications increased, the concept is problem-solving and reasoning have been equated with the concept of
still too broad to make significant contributions. Other authors have also fluid intelligence [23] as was theorized by Catell [14]. Problem-solving
expressed concern regarding the need for clear, operationalized defini literature is often characterized based on the presented task [73]. For
tions for research purposes, referring to the state of the field as a example, formal problem-solving deals with closed deterministic envi
“definitional confusion” [31]. The International Society for Technology ronments which are context-independent, while referring to informal
in Education (ISTE) and the Computer Science Teacher Association problem solving has been equated to open probabilistic problems that
(CSTA) proposed an operational definition that describes CT as a prob are context-dependent, thus having a higher ecological validity.
lem solving process that spans characteristics such as formulating Problem-solving often requires planning, that is, the process of estab
problems algorithmically, logically organizing data, achieving repre lishing a step-by-step guideline toward goal-oriented action [16]. The
sentation through abstraction, automatization, procuring time and act of planning emphasizes the fact that our problem-solving method
resource efficiency, and generalization. However, this definition ology must be not only effective, but also resource-efficient and often
explicitly proposes that while CT includes the characteristics mentioned requires keeping and manipulating online information (working
above, it is not limited to this process, leaving the concept open-ended. A memory).
recent review by Kalelioglu [39] found that most consensus seems to
link CT to problem solving, abstraction, algorithms, and pattern 2.3. Previous studies linking CT to other cognitive skills
recognition.
Moreover, most definitions have presented the defining elements of Previous evidence on the cognitive abilities underlying CT is scarce
CT as a homogeneous group, with few authors establishing hierarchical and there has not been, to our knowledge, a data-based characterization
relations amongst them. Few authors have aimed to simplify the concept of CT from a cognitive perspective for early stages in development.
by reducing it to one or a few of its most essential elements. Such is the Román-González and collaborators described the associations of CT and
case of Aho [1] who defines it as problem-solving using algorithms to other cognitive skills in adolescents [63] (by contrasting their Compu
represent its solutions, or Grover and Pea [31] who prioritize the process tational Thinking Test (CTT) to the cognitive skills measured by the
of abstraction as foundational to CT. Perhaps the most cited framework Primary Mental Abilities Battery (PMA), namely its verbal, spatial,
to understand CT is the one proposed by Brennan and Resnick [11], reasoning and numerical factors, as well as the RP30 problem-solving
which is based around three distinct components: computational con test. Their results show significant positive correlations of the CT with
cepts (such as sequencing, loops, or conditionals), computational prac problem-solving and the verbal, spatial, and reasoning subscales of the
tices (such as debugging or problem-solving) and computational PMA, but not with the numerical one. Using multiple linear regression,
perspectives, which describe aspects linked to cooperation and the authors created a model in which the studied cognitive skills
communication. In 2012, the Royal Society defined CT as “the process of explained 27% of the variance for teen’s performance in the CTT, with
recognizing aspects of computation in the world that surrounds us, and the spatial and reasoning subscale being significant predictors of CT.
applying tools and techniques from Computer Science to understand and Using data from 17-18 year olds, Guggemos [33] reported that
reason about both natural and artificial systems and processes” [66] reasoning and math skills were significant predictors of CT performance.
which defines CT as an approach through which individuals employ CS In a recent study by Tran [72], the CT abilities of 3rd graders (ages eight
elements towards reasoning. and nine) were explored using an ad-hoc paper-based assessment based
on Brennan and Resnick’s [11] description of computational concepts.
2.2. Cognitive development in early childhood This assessment was used to evaluate 183 children in their ability to
create sequences and algorithms, understanding loops, and debugging.
Early childhood has been identified as a window of opportunity for Recent evidence utilizing a cognitive development perspective can be
cognitive and socio-emotional development [12]. Environmental factors found in the field of educational robotics. Such is the work of Di Lieto
such as access to quality childcare and early childhood education during and collaborators, which found improvement in executive functions
the first five years of life have shown to have positive long-term impacts after a randomized control trial of an educational robotics intervention
2
[22]. Although its objective was not to enhance CT per se, some authors 3.3.2. Fluid intelligence
have proposed educational robotics programming as activities capable Tablet based version of Raven’s coloured progressive matrices [57].
of developing CT in young children [7]. Moreover, budding evidence This task asks children to identify the correct missing pattern from the
shows promising results regarding their capabilities for promoting stimuli in a six option multiple-choice format. The test implements
young children’s CT and cognitive skills. Kazakoff and collaborators different kinds of problems which include pattern continuation and
[40] showed a 1-week robotics intervention could improve kindergarten element abstraction. This instrument is a widely-implemented assess
children’s sequencing scores, while Bers et al., [8] concluded that chil ment of fluid intelligence for children, which has undergone validation
dren as young as three years old could grasp CT concepts via robotics. in a latin american population [54] and has shown stability across time
Studies with slightly older children [[30], [38], [52]] have reached and cultures [58].
similar conclusions.
Considering the evidence mentioned above, it is reasonable to 3.3.3. Working memory
conclude that there is scarce evidence for data-based characterizations Tablet-based Corsi Block Tapping Test [17]. Children are tasked with
of CT from a cognitive perspective, specifically in early childhood repeating an incremental sequence by following the order it’s been
development. Thus, we proposed the following research questions: initially presented. Higher working memory spans correspond with
• RQ1: Which cognitive abilities are associated with young children’s children’s ability to maintain online information for manipulation. Ev
outcome in a CT questionnaire? idence shows working memory increments throughout development
• RQ2: How does children’s CT assessed through a questionnaire until adolescence [27] and has been successfully adapted for tablets
correlate to their performance in an educational robotics intervention? [13].
3. Methodology 3.3.4. Planning

Tablet-based Tower of London task [68]. Two sets of coloured disks
3.1. Research model and procedure configurations are presented to children. One is a target setting, which
they must try to emulate in their own setting by moving the disks in the
We used a cross-sectional correlational design. We assessed children least amount of possible steps. The Tower of London task is a widely
for three weeks (half our sample was assessed in October 2018, while the used assessment of planning which has shown robust results in its val
rest were assessed in March 2019). Children participated in three eval idity and reliability [[19], [41]].
uation sessions of up to 25 minutes in order to prevent tiredness or
boredom. All children were assessed in the morning, between 9 and 11 3.3.5. Sequencing
am. Three researchers participated in the data collection. Tablet-based We assessed children’s ability to organize temporal sequences by
measurements were applied concurrently in groups, following a 4:1 using Langdon and Coltheart’s [42] subset of mechanical stimuli
ratio between children and adults. Paper-based assessments were (paper-based assessment). Children are tasked with organizing four
applied individually in a 1:1 ratio. picture sequences that denote temporal events.
3.2. Research context and sample 3.3.6. Vocabulary

Tablet based version of PPVT-III (Peabody Picture Vocabulary Test
102 children (male=52; overall mean age=5.6 years, SD=5.08 [25]). 36 item test which assesses children’s receptive vocabulary by
months) attending level 5 (kindergarten) at a public school in Mon tasking them with matching a heard word with its corresponding image.
tevideo, Uruguay participated in the study. Convenience sampling was Difficulty is incremental throughout the task. Normative data for latin
implemented. Sociocultural levels for our sample were characterized as american participants is available and suggests it is appropriate for use
middle-high according to the Uruguayan administration of public edu in this population [51].
cation. Inclusion criteria consisted of children aged 4-6 years with
typical development. 1 child was excluded from our sample due to 3.3.7. Symbolic magnitude comparison
having a diagnosed developmental disorder. Informed consent was ob Tablet based version of the classical task by Moyer and Landauer
tained from parents/caregivers, and the study was approved by the [50]. This task requires children to select the highest of two arabic
Research Ethics Committee of the School of Psychology at Universidad numbers presented on screen. Children are instructed to make the se
de la República, Uruguay . All methods were performed in accordance lection as fast as possible. This task quantifies children’s mental repre
with the Declaration of Helsinki [78]. sentation of numerical magnitudes. Previous studies show the symbolic
27 of the participating children (male=18, mean age=5.4 years, distance effect diminishes throughout development and is related to
SD=5.8 months) took part in educational robotics (ER) intervention later math achievement [36]. The reliability and validity of this task has
post-assessment. been studied [[44], [46]]. Lyons et al. [44] report a Cronbach’s alpha of
Video recorded sessions of ER activities were analysed. 5 minutes 0.82.
intervals of each session (starting point set to the time point in which
each task’s objective was first instructed to children) were used for data 3.3.8. Transcoding
analysis. Four trained observers participated in the coding of each ses This tablet based task evaluates children’s ability to identify and
sion. Inter-observer reliability was high, ranging from 83% to 100%. report a herd number-word onto arabic symbols [20]. A high reliability
has been reported by Moura and collaborators [49] (KR-20, r = .91)
3.3. Instruments
3.3.9. Mental rotation
3.3.1. CT assessment During this test (tablet-based), children are presented with either
The CT assessment implemented in this study was adapted from mirrored and rotated (incorrect) or rotated (correct) versions of a target
Yune Tran’s CT questionnaire for seven year old children [72]. This image and tasked with identifying the correct option from three
questionnaire assesses 5 CT related abilities, namely the ability to create (Quaiser-Pahl [56] reports for this test a retest-reliability of 0.72).
algorithms, loops, debugging, inferring from a conditional statement,
and sequencing. Children’s answers for each task were dummy coded for 3.3.10. Home use of technology
scoring (scoring range: 1-12). Scale reliability was acceptable (Cron Parents were asked to complete a brief questionnaire reporting their
bach’s alpha: 0.72). perceptions on technology and their children’s use of technology at
3
Table 1 Table 2
Summary of ER intervention Descriptive statistics. Age (in months) and fluid intelligence scores in our sample
Session Activity description Boys (N=52) Girls (N=49) Overall (N=101)
N◦
1 Session 1 consists of an introductory activity. We establish general rules Age (months)

of the workshop. Children get to know the robot in its colour-sensor Mean (SD) 67.4 (5.32) 69.1 (4.72) 68.3 (5.07)
modality and we explore their pre-existing notions on what a robot is. Median [Min, Max] 68.0 [54.0, 76.0] 69.0 [55.0, 76.0] 68.5 [54.0, 76.0]
Finally, we introduce RoboTito, name and recognize some of its parts Missing 5 (9.6%) 0 (0%) 5 (5.0%)
and functions. Fluid intelligence
2 Session 2 has the objective of establishing simple goals and introducing Mean (SD) 11.5 (6.20) 12.8 (5.44) 12.1 (5.86)
children to spatial concepts such as backwards, forward, left and right. Median [Min, Max] 11.0 [0, 26] 13.0 [3, 25] 12.0 [0, 26]
They program Missing 3 (5.8%) 5 (10.2%) 8 (7.9%)
short trajectories using colour sensors and learn how to use the cards to
program the robot so that it moves in a square loop
3 In session 3 children engage in simple planning and are prompted to use shows sample scripts which depict children’s dialogues extracted from a
the previously learnt rules to complete sequences in order to reach a
video recording of one of the activities. Each sample exemplifies each of
predetermined objective. The feedback purple card is introduced and
placed as a target for sequences.
the three possible scores. An “insufficient information” score was used if
4 Session 4 requires children to predict behaviors. Specifically, children behavioral cues were deemed insufficient in order for observers to make
are asked to observe a pre-established setting of the environment and a judgment of accomplishment of objectives. Objective fulfillment scores
explain the robot’s sequence given those conditions. They later propose were added up in order to create a final score.
alternatives to modify this trajectory based on their observations.
5 In session 5 children continue to plan sequential movements while being
prompted to focus on resource-efficiency. Children are asked to 3.3.12. Summary of our ER intervention
generate sequences towards a given objective using the least amount of The ER intervention took place in a spare classroom within the
colourd cards possible. school and was led by two researchers who took on the roles of group
6 Session 6 incorporates the inhibition of distracting objects placed on the coordinators. Children were divided into small groups of 5 to 7 mem
robot’s environment. Children are asked to repeat the previous task
incorporating task-irrelevant objects in the setting, which must be
bers, with each group receiving a robot for the activities. Up to two
avoided. groups worked simultaneously within the space. Mats of 1.20 square
7 Session 7 focuses on debugging: Children are presented with an meters were used in order to delimit an area for the robot’s movements.
erroneous setting and asked to correct the objects in order to achieve a A playful narrative in which our robot was a visitor from a far away
given objective
planet trying to return home connected our ER activities and was used as
8 Session 8 marks the switch towards the use of distance sensors. We
introduce this modality by identifying where the sensors are and a motivator for children.
explaining how the robot moves. Children are tasked to try to move the RoboTito [3] is equipped with two types of sensors through which it
robot using their hands, while paying attention to what happens when receives the information from the environment to program its move
we increase or decrease our distance. We explain the notion of range ments: colour and distance. Sessions 1-7 involved children working with
when dealing with distance sensors.
9 In session 9 we try to infer the rules of functioning of the distance
the robot using colour sensors by programming RoboTito through ar
sensors and create tests to try to establish them. Children are tasked to ranging coloured cards in the mat. Four different coloured cards (green,
try and guess the underlying functioning rule of the robot by observing yellow, blue and red) were used to programme the robot’s movement in
its movement several times and experimenting freely with different different directions (forward, backwards, left and right). A fifth coloured
settings of objects.
card (purple) was used to provide children positive feedback and
10 In session 10 children are explicitly introduced to the rules of the
distance sensors. We used an embodied approach to facilitate their motivation to complete the sequences. When this purple card is sensed,
learning of this rules by asking them to imagine they are the robots and the robot will turn on bright purple lights and “dance” by rotating in
perform the correct movements given a setting place. Sessions 8-11 involved children working with distance sensors.
11 Session 11 involves prediction with two rules and debugging: children The tangible materials used for programming in the distance sensor
are tasked to observe a given environment and predict the robot’s
movement. If the objective is not met, they are asked to create
mode were plastic cones 23 cm tall. In the distance sensor modality,
hypotheses regarding what actually happened and try to alter the RoboTito will sense surrounding objects (360◦ ) and then move in the
setting in order to obtain the desired results. direction of the object that is furthest away. Thus, children were
required to integrate two rules about the robot’s behavior: firstly, have a
notion of the sensor’s range, and secondly remember it will move to
home. A total of 83 parents (male=22) agreed to complete this ques
wards the farthest object. Table 1 provides a summary of the activity
tionnaire (84% of our sample). Parents reported on their use of tech
plan used for our ER intervention.
nology at home through a Spanish version of the Parental Perceptions of
Technology Scale [67] which explores parents’ self-efficacy and nega
3.4. Data analysis
tive beliefs regarding technology use. Additionally, parents completed
the Parent´s attitudes towards computer use scale [47], which asked
Statistical analysis was performed using R and R Studio software
them to report on their perceived advantages of computer use for their
[71]. Pearson correlations were performed in order to explore bivariate
children and technology-based activities at home.
correlations among our variables. Mixed-effects linear models (MLM)
were implemented with CT as our dependent variable to create a model
3.3.11. Educational robotics task performance
that accounts for the effects of our cognitive predictors on CT. Given the
After their CT and cognitive assessment, 27 children participated in
nature of the data collection process, it is important to clarify that no
an 11 session educational robotics (ER) intervention using RoboTito
causal relation is assumed for the present analysis. We first fitted an MLR
[[2], [3]], a robot designed to be programmed through tangible ele
model which included all of our measured variables. The model reduc
ments from its environment. ER sessions were implemented in groups of
tion was performed using backward stepwise deletion by the Akaike
5-7 children and lasted 25-30 minutes each [4]. Video-recordings of
Information Criteria (AIC). Models were contrasted using ANOVA.
each session were taken and analyzed. Children’s performance during
the task was coded in regards to their accomplishment of objectives. A
task-score of 2= totally accomplished objectives, 1=partially accom
plished objectives, or 0=did not accomplish objectives. Appendix B
4
Table 3 performed partial correlations controlling for children’s fluid intelli

Pearson correlations between CT and cognitive assessments, simple and gence scores. Our results show that findings regarding the association to
controlled by fluid intelligence sequencing ability, working memory, vocabulary, number transcoding,
Variable CT (Simple) CT (Controlled) and comparison remain, while the association with planning does not.
sequencing 0.51*** 0.57***
No significant associations were found between CT and children’s fre
working memory 0.21* 0.44* quency in technology use, parental technological self-efficacy, or
mental rotation 0.10 0.00 parental negative attitudes towards technology.
vocabulary 0.23* 0.64***
num. transcoding 0.39*** 0.51***
num. comparison 0.49*** 0.45*
4.3. Model fitting
planning 0.22* 0.33
age (months) 0.16 0.01
Data was fitted into a mixed effects linear regression with CT as our
dependent variable. Model 0 was built including every assessed variable.
4. Results The model reduction was performed using backward stepwise deletion
4.1. Descriptive statistics Table 4

Summary of our final linear regression model of cognitive skills onto CT results
Table 2 displays general descriptive statistics for our sample,
CT
including participant’s age and obtained scores in fluid intelligence.
Predictors Estimates CI p value

4.2. Pearson correlations between CT and cognitive outcomes (Intercept) 0.00 -0.29-0.29 0.998
Sequencing 0.40 0.23-0.57 <0.001
Num.comp 0.26 0.09-0.44 0.004
Results from bivariate Pearson correlations between the explored
variables are summarized in Table 3. We found moderate positive as Random effects
sociations between children’s CT outcomes and their sequencing ability, σ2 0.61
number comparison accuracy, numerical transcoding, and fluid intelli τ00 class 0.06
ICC 0.09
gence. Weak positive associations were also found with children’s vo N class 4
cabulary, working memory, and planning abilities. Associations Observations 101
between CT outcomes and children’s accuracy in our mental rotation Marginal R2 / Conditional R2 0.313 / 0.378
task and their age in months proved to be non significant. We then
Fig. 1. Graphical representation of the fixed effects in our multiple linear regression models. Our initial model (model 0) is presented on the left. Standardized
coefficients (β) are included for each variable. Our final model (model 1) is presented on the right. ANOVA results between both models were non significant (DF=10,
Chi Square=8.25, p=0.6)
5
Table 5 computational thinking conducted in older children showed varying

Pearson correlations between objective fulfilment in robotics tasks and cognitive results: for example, Román Gonzalez et al. [63], studied a sample of
assessments, simple and controlled by fluid intelligence 1251 Spanish children and teenagers (aged 10-15 years old) and found
Variable ER (Simple) ER (Controlled) associations between their computational thinking test and spatial,
reasoning and verbal factors of the Primary Mental Abilities test [26] but
CT 0.50* 0.55*** no association to the numerical factor. Multiple linear regression
sequencing 0.23 0.10 explained 27% of variance, with the spatial and reasoning factors being
working memory 0.58*** 0.56*** significant predictors of the CTT outcome. Meanwhile, Robertson and
mental rotation 0.30 0.23
collaborators [62] assessed the associations between executive functions
vocabulary 0.50* 0.49*
num. transcoding 0.44* 0.34 and CT in a sample of 25 children aged 11-12 years and found that
num. comparison 0.48* 0.43* children’s BRIEF2 scores [29] explained up to 60% of the variance in
planning 0.38 0.16 their DrScratch scores, a measure of their Scratch-built program’s
complexity proposed as a proxy for CT, finding positive associations
between children’s behavioral, emotional and cognitive regulation
by the Akaike Information Criteria (AIC). ANOVA results between our
abilities to their CT scores. The stark difference in these results despite
model 0 and our final model proved not to be statistically significant
working with similarly aged groups of children is probably linked to the
(DF=10, Chisquare=8.25, p=0.6), therefore, our simplified model was
difference in approaches towards assessing CT skills and evidences the
selected. A visual representation of the fixed-effect variables in both our
need for further studies in this area.
models is presented in Fig. 1. Our final model results are shown in
We complemented our baseline model by exploring the associations
Table 4. The normality of the regression model residuals was verified
between the explored cognitive variables and children’s objective
(W=0.97, p=0.08).
fulfillment in an educational robotics intervention in a subsample of
As shown in Table 4, our temporal sequencing and number com
children. Overall, we found the associations for our paper-based
parison tasks are significant predictors of CT. The standardized co
assessment (Table 3) were concordante with those of our ER assess
efficients of the model are β(Sequencing)=0.40 and β(Num.com)=0.26.
ment (Table 5). Exceptions to this were the lack of association between
Conditional R2 for our model was 0.37, meaning our resulting model
our ER assessment and sequencing ability, which was an unexpected
could explain 37% of our CT test scores’ variance. Model assumptions of
result, as sequencing was an integral part of the tasks in our ER inter
normality and variance homogeneity were assessed with satisfactory
vention. However, this difference could be due to our sequencing
results, (see appendix A). Possible multicollinearity was assessed via the
assessment relying on temporal sequencing and cause and effects re
variance inflation factor (VIF), obtaining scores below 1.2 for both in
lations (which is encompassed in the paper-based assessment by Tran
dependent variables, suggesting only low intercorrelation.
[72]) while our ER tasks mainly required spatial sequencing. We also
observed a lack of association between CT and mental rotation skills, an
4.4. Association with ER intervention performance association previously reported by Città and collaborators [15] for pri
mary school children. The seeming contradiction between these results
Additionally, we analyzed the scores of a subsample of children who might algo lie in assessment approaches: while the study conducted by
participated in an ER lab with video-recorded data and found a positive Città and collaborators [15]à, focused primarily on the embodied as
significant association between children’s CT scores at pre-test and their pects of a robotics intervention and targeted the assessment of spatial
ability to fulfill the required task objectives while working with the skills, our paper-based assessment of CT did not require the use of
robot, r(27) = 0.50, p<0.05. Further correlation analysis for children’s mental rotation. Moreover, while we might have expected to find a link
objective fulfillment in ER tasks and cognitive variables are shown in between mental rotation skills and our ER tasks, it is possible the nature
Table 5, both simple and controlling for children’s fluid intelligence. of the robot used (particularly, its capability to move towards any di
rection without rotating) might have diminished our ability to target this
5. Discussion particular skill during our intervention. Further studies on the impact of
different robotics tools’ attributes towards fostering specific cognitive
The present study showed the existing associations between the skills is thus needed.
outcomes of a reported CT assessment [72] and children’s performance The current study focused on the exploration of cognitive abilities
on several cognitive abilities. Our results showed that children’s CT linked to CT at an early age. It allowed us to shed light into the existing
performance was moderately and positively correlated to fluid intelli associations between domains that have so far not been linked through
gence, sequencing ability and early numerical skills such as symbolic empirical data during this stage in development. We implemented a
magnitude comparison and transcoding, with significant associations paper-based assessment with acceptable reliability that tapped into
with vocabulary and executive functioning measures such as planning concepts typically included in various definitions of CT such as creating
and working memory. So far, only a handful of studies [[22], [33], [62], an algorithm, identifying patterns, understanding conditional state
[63]] have tried to establish empirical links between CT and children’s ments and identifying errors in a sequence. Our results are congruent
performance in cognitive assessments at an early age. with Guggemon´s findings in adolescents [33] as well as with recent
The reason behind this is probably linked to the established struggle literature on the use of educational robotics during early childhood. For
amongst researchers and practitioners to create an operationalized example, Kazakoff and collaborators [40] reported increased
definition of CT [69]; and thus creating and implementing reliable sequencing abilities assessed through a picture sequencing test after an
assessment methods [[59], [65], [70], [79]]. The extent as to how intensive robotics workshop for a sample of kindergarteners. Our results
related is the concept of CT to activities such as programming which could be pointing out the existing similarities between early math skills
require an implementation through an information processing agent is a and CT at a young age. Wing [77] pointed out in one of her original
highly discussed topic [21]. As stated by Wing [77] CT is a thinking papers that CT is fundamentally linked to mathematics, with Shute and
ability related to problem solving that can be acquired by anyone. More collaborators [69] specifically pointing out the overlapping concepts
recent definitions of CT from the field of CS education, such as that of inherent to these skills. However, a developmental trajectory for the
ISTE [37] or the Royal Society [66], support this claim, reinforcing the diversification of CT and mathematical thinking has not been yet stated,
notion that it’s important to promote the cognitive processes underlying and it is currently unclear how the two relate at a young age. Our results
CS-related practices. However, the specific cognitive processes sup seem to suggest a close connection with similar basic processes sup
porting CT related tasks have seldom been explored. Similar studies on porting the two. Further exploration of CT in the context of early
6
Fig. A.1. Model diagnosis. A. Fixed-effects estimates. B. QQplot for model 1. C. Normality of residuals (Shapiro-Wilke test W = 0.98, p= 0.13). D. Homoscedasticity:
Levene test F=2.11, p=0.14
childhood education based on empirical associations with cognitive 6. Conclusion

development is needed in order to understand the underlying processes
supporting this newly-defined skill, and how this might change with Computational thinking is an emerging concept stemming from
children’s maturation. So far, CT has acted as a functional concept computer science that has been embraced by educators and policy
within education that has been successful in grouping several (yet makers and promoted by academia in order to transmit problem-solving
different) skills related to computer science identified as being of strategies similar to those used in these disciplines. As such, it is
pedagogical importance. Evidence-based research on the subject will imperative we acquire a better understanding of what this term means
allow practitioners to create tailored interventions that foster CT in and which specific skills are at play during these kinds of tasks. The
developmentally appropriate ways. Specifically, we recommend further present study provided evidence on the association between a previ
research to implement longitudinal designs in order to surpass one of the ously established CT questionnaire and various cognitive abilities in
limitations of the current cross-sectional correlational study, which is kindergarteners. Our results suggest computational thinking in early
the lack of assumed causation inherent to this approach. In this study, childhood was largely explained by participants’ temporal sequencing
we focused on the cognitive aspects of CT, but further research might skills and numerical abilities regarding their understanding of symbolic
incorporate the exploration of socio-emotional dispositions that support magnitude, further establishing the existing notions from previous evi
it, especially as these are often included in definitions, such as that of dence regarding CT’s associations with reasoning and early math skills.
ISTE and CSTA [60], which stresses the importance of collaboration and Additionally, we contrasted our results with children’s performance
perseverance. Some progress has been made recently in this regard for during an educational robotics intervention using a robot programmable
older children [64], with further variance being explained. Furthermore, through tangible objects, finding significant positive correlations be
further exploration on the impact of environmental factors is warranted, tween ER performance and children’s CT.
as findings from Guggemos [33] suggest participant’s parent’s The present study contributed to the characterization of the concept
socio-economic and cultural status, teen’s computer use and past com of CT in children attending kindergarten through empirical evidence,
puter science instruction were significant predictors of their CT. thus allowing for further research with children in this stage of devel
opment to focus on relevant subcomponents and create educational in
terventions targeted towards these skills. Additionally, it is relevant to
the study of CT in older children and adolescents, as it contributes
7
addressing the entire group)

Child A: Here’s one (grabs a different blue card and places the card on
position 3)
CG: Very well, let’s use the robot now. Remember, it starts from here
(points to position 1)
Child A: Manipulates the robot and carefully examines it while
placing. Once placed he makes sure its direction is correct. Program is
executed correctly.
Rationale for score: this child was able to correctly complete the
sequence and explain the robot’s movement. His answers showed un
derstanding of the robot’s underlying rules and ability to use those rules
to solve the task at hand.
Extract 2: Example of a child who obtained a score of 1 (partially
accomplished objective)
GC: - Ok, now let’s say we place the robot like this” (shows children the
mat, cards and robot configuration in Fig. B.1) Remember, our robot needs
Fig. B.1. Mat, cards and robot configuration for 1 activity in session 3.
to touch both cards. What will the robot do now, will it reach the target?”
Other child: No
towards establishing certain foundational skills which might lead to the
Child B: (we should use) Another yellow card
acquisition of more complex notions within CT later in life.
GC: Where would we put a yellow card?
Child B: points towards position 3
Declaration of competing interests Other children interject and complete the sequence. Child B listens to
them attentively and uses his hand to mimic the robot’s expected
The authors declare that they have no known competing interests movement.
that could influence the current work. GC: Will it reach the target now? (with correct cards)
Child B: Yes! it will
Funding Rationale for score: while child B initially chose an incorrect colour
for the card, he was able to correctly identify the position of the missing
The present study was supported by ANII (Agencia Nacional de card in the mat. He was also able to mimic the correct expected trajec
Investigación e Innovación, Uruguay) grant N◦ FSED_2_2017_1_138793 tory using his hands and was correct about the expected behaviour once
his group had selected the correct colour.
Appendix A Extract 3: Example of a child who obtained a score of 0 (did not
accomplish objective)
Model diagnostics Child C examined the robot but did not answer the group co
Fig. A.1 ordinator’s inquiries or prompts. He followed and enjoyed the robot’s
Appendix B. movement but wasn’t able to predict or try to make sense of its behav
Sample scripts for each scoring of objective fulfillment. iours. He showed excitement about the robot and story but was often
The three sample scripts are short extracts from a video recording distracted and his gaze wandered from the task.
assessed by observers to code children’s objective fulfillment. All of the Rationale for score: even though this child seemed to be motivated
scripts below were extracted from the same session (session 3) and by the robot, he did not exhibit an understanding of its functioning and
correspond to different children being presented with the same chal thus wasn’t able to manipulate the cards to program it.
lenge in a group session. In session 3, children were tasked with
completing the sequence in Fig. B.1 using 1 card. Fig. B.1 depicts the References
mat, cards (left) and robot (right) starting positions. The group coordi
nator (GC) would set the robot’s starting point (identified in the [1] Aho AV. Computation and computational thinking. The comput j 2012;55(7):
832–5.
Fig. with the number 1) and target (marked by the purple card) and [2] Bakała E, Visca J, Tejera G, Seré A, Amorin G, Gómez-Sena L. October). Designing
reminded children the robot must sense all the cards placed on the mat child-robot interaction with Robotito. IEEE; 2019. p. 1–6.
before reaching its target. Children were expected to predict and test the [3] Tejera G, Amorin G, Sere A, Capricho N, Margenat P, Visca J. December). Robotito:
programming robots from preschool to undergraduate school level. IEEE; 2019.
robot’s trajectory and select the correct card in order to reach the target. p. 296–301.
Given the starting configuration, the expected trajectory of the robot is: [4] Gerosa A, Koleszar V, Gómez-Sena L, Tejera G, Carboni A. Educational robotics and
1- sense blue - move forward until it reaches 2 - sense yellow, move right computational thinking development in preschool. In: 2019 XIV Latin American
Conference on Learning Technologies (LACLO). IEEE; 2019. p. 226–30.
indefinitely until exiting the mat. The correct trajectory after completion [5] Barbey AK, Barsalou LW. Reasoning and problem solving: models. Encyclopedia of
with the missing card (blue card in position 3) would be: 1- sense blue - neurosci 2009;8(2):35–43.
move forward until it reaches 2 - sense yellow move right until reaching [6] Belsky J. Social-contextual determinants of parenting. Encyclopedia on Early
Childhood Development; 2007. Topic Ed; Tremblay, RE, Bolvin, M., Peters, RDV,
3 - move forward until reaching target card. Eds.
Extract 1: Example of a child who obtained a score of 2 (totally [7] Bers MU. Coding as a playground: Programming and computational thinking in the
accomplished objective) early childhood classroom. Routledge 2020.
[8] Bers MU, González-González C, Armas Torres MB. Coding as a playground:
Group coordinator (GC): - Let’s imagine we’re placing the robot like
Promoting positive learning experiences in childhood classrooms. Computers and
this” (shows children the mat, cards and robot configuration in Fig. B.1) Education 2019;138:130–45.
Our robot needs to touch both cards, our blue card (points to the card) and [9] Blair C. Developmental science and executive function. Curr Dir Psychol Sci 2016;
our yellow card (points again). Which card do we need to get to the planet? 25(1):3–7.
[10] Bocconi, S., Chioccariello, A., Dettori, G., Ferrari, A., and Engelhardt, K. (2016).
(purple card, identified as the robot’s home planet in the narrative)” Developing computational thinking in compulsory education-Implications for
Child A: - A blue one! policy and practice (No. JRC104188). Joint Research Centre (Seville site).
GC: - A blue one? Ok, where should we place the blue card? [11] Brennan K, Resnick M. New frameworks for studying and assessing the
development of computational thinking. In: Proceedings of the 2012 annual
Child A: (Gets up and points to point 3 in Fig. B.1). Here meeting of the American educational research association1; 2012. p. 25.
GC: Ok, let’s try it. Who wants to place the blue card there? (GC is
8
[12] Britto PR, Lye SJ, Proulx K, Yousafzai AK, Matthews SG, Vaivada T, Lancet Early [48] Miyake A, Friedman NP, Emerson MJ, Witzki AH, Howerter A, Wager TD. The
Childhood Development Series Steering Committee. Nurturing care: promoting unity and diversity of executive functions and their contributions to complex
early childhood development. The Lancet, 2017;389(10064):91–102. “frontal lobe” tasks: A latent variable analysis. Cognit Psychol 2000;41(1):49–100.
[13] Brunetti R, Del Gatto C, Delogu F. eCorsi: implementation and testing of the Corsi [49] Moura R, Lopes-Silva JB, Vieira LR, Paiva GM, Prado ACDA, Wood G, Haase VG.
block-tapping task for digital tablets. Front psychol 2014;5:939. From “five” to 5 for 5 minutes: Arabic number transcoding as a short, specific, and
[14] Cattell RB. Theory of fluid and crystallized intelligence: A critical experiment. sensitive screening tool for mathematics learning difficulties. Arch Clin
J Educ Psychol 1963;54(1):1. Neuropsychol 2015;30(1):88–98.
[15] ... and Città G, Gentile M, Allegra M, Arrigo M, Conti D, Ottaviano S, Sciortino M. [50] Moyer RS, Landauer TK. Time required for judgements of numerical inequality.
The effects of mental rotation on computational thinking. Computers and Nature 1967;215(5109):1519–20.
Education 2019;141:103613. [51] ... and Olabarrieta-Landa L, Rivera D, Ibáñez-Alfonso JA, Albaladejo-Blázquez N,
[16] Cohen GN, Bronson MB, Casey MB. Planning as a factor in school achievement. Martín-Lobo P, Delgado-Mejía ID, Arango-Lasprilla JC. Peabody Picture
J appl developmental psychol 1995;16(3):405–28. Vocabulary Test-III: Normative data for Spanish-speaking pediatric population.
[17] Corsi P. Memory and the medial temporal region of the brain. Unpublished NeuroRehabilitation 2017;41(3):687–94.
doctoral dissertation). Montreal, QB: McGill University; 1972. [52] Papadakis S, Kalogiannakis M, Zaranis N. Developing fundamental programming
[18] Cuny, J., Snyder, L., and Wing, J. (2010). Computational thinking: a definition. concepts and computational thinking with ScratchJr in preschool education: a case
Unpublished manuscript. study. Int J Mobile Learning and Organisation 2016;10(3):187–202.
[19] Debelak R, Egle J, Köstering L, Kaller CP. Assessment of planning ability: [53] Papert, S. (1980). " Mindstorms" Children. Computers and powerful ideas.
Psychometric analyses on the unidimensionality and construct validity of the [54] Pasquali L, Wechsler S, Bensusan E. Raven’s colored progressive matrices for
Tower of London Task (TOL-F). Neuropsychology 2016;30(3):346. children: a validation study for Brazil. Avaliação Psicológica 2002;1(2):95–110.
[20] Deloche, G., and Seron, X. (1987). Numerical transcoding: A general production [55] Perone S, Almy B, Zelazo PD. Toward an understanding of the neural basis of
model. executive function development. The neurobiology of brain and behavioral
[21] Denning PJ, Tedre M. Computational thinking. MIT Press; 2019. development. Academic Press; 2018. p. 291–314.
[22] ... and Di Lieto MC, Pecini C, Castro E, Inguaggiato E, Cecchi F, Dario P, [56] Quaiser-Pohl C. The Mental Cutting Test" Schnitte" and the Picture Rotation Test-
Sgandurra G. Empowering Executive Functions in 5-and 6-Year-Old Typically two new measures to assess spatial ability. Int J Testing 2003;3(3):219–31.
Developing Children Through Educational Robotics: An RCT Study. Front psychol [57] Raven JC, Court JH. Raven’s progressive matrices and Raven’s coloured matrices.
2020;10:3084. London: HK Lewis; 1986.
[23] Diamond A. Executive functions. Annu Rev Psychol 2013;64:135–68. [58] Raven J. The Raven’s progressive matrices: change and stability over culture and
[24] Diamond A, Ling DS. Conclusions about interventions, programs, and approaches time. Cognit Psychol 2000;41(1):1–48.
for improving executive functions that appear justified and those that, despite [59] Relkin E, de Ruiter L, Bers MU. TechCheck: Development and validation of an
much hype, do not. Developmental cognitive neurosci 2016;18:34–48. unplugged assessment of computational thinking in early childhood education.
[25] Dunn LM, Dunn LM. Peabody Picture Vocabulary Test. American Guidance Service. J Sci Educ Technol 2020;29:482–98.
Circle Pines, MN; 1997. [60] ... and Resnick M, Maloney J, Monroy-Hernández A, Rusk N, Eastmond E,
[26] Ediciones, T., and de Estudios, S. (1997). PMA:(aptitudes mentales primarias): Brennan K, Kafai Y. Scratch: programming for all. Commun ACM 2009;52(11):
manual. Tea. 60–7.
[27] Farrell Pagulayan K, Busch RM, Medina KL, Bartok JA, Krikorian R. Developmental [61] Richter LM, Daelmans B, Lombardi J, Heymann J, Boo FL, Behrman JR. Investing
normative data for the Corsi Block-tapping task. J Clin Exp Neuropsychol 2006;28 in the foundation of sustainable development: pathways to scale up for early
(6):1043–52. childhood development. Lancet North Am Ed 2017;389(10064):103–18.
[28] Andrews G. Reasoning and problem solving. The Encyclopedia of Child and [62] Robertson J, Gray S, Toye M, Booth J. The relationship between Executive
Adolescent Development 2019:1–11. Functions and Computational Thinking. Int J Comput Sci Education in Schools
[29] Gioia GA, Isquith PK, Guy SC, Kenworthy L. BRIEF: Behavior rating inventory of 2020;3(4):35–49.
executive function. Lutz, FL: Psychological Assessment Resources; 2015. [63] Román-González M, Pérez-González JC, Jiménez-Fernández C. Which cognitive
[30] González YAC, Muñoz-Repiso AGV. A robotics-based approach to foster abilities underlie computational thinking? Criterion validity of the Computational
programming skills and computational thinking: Pilot experience in the classroom Thinking Test. Comput Hum Behav 2017;72:678–91.
of early childhood education. In: Proceedings of the Sixth International Conference [64] Román-González M, Pérez-González JC, Moreno-León J, Robles G. Extending the
on Technological Ecosystems for Enhancing Multiculturality; 2018. p. 41–5. nomological network of computational thinking with non-cognitive factors.
[31] Grover S, Pea R. Computational thinking in K–12: A review of the state of the field. Comput Hum Behav 2018;80:441–59.
Educational researcher 2013;42(1):38–43. [65] Román-González M, Moreno-León J, Robles G. Combining assessment tools for a
[32] Grover S, Pea R. Computational Thinking: A competency whose time has come. comprehensive evaluation of computational thinking interventions. Computational
Comput sci education: Perspectives on teaching and learning in school 2018;19. thinking education. Singapore: Springer; 2019. p. 79–98.
[33] Guggemos J. On the predictors of computational thinking and its growth at the [66] Royal Society (Great Britain). Shut down or restart?: The way forward for
high-school level. Computers and Education 2021;161:104060. computing in UK schools. Royal Society; 2012.
[34] Haseski Hİ, Ilic U, Tugtekin U. Defining a New 21st Century Skill-Computational [67] Sanders W, Parent J, Forehand R, Sullivan AD, Jones DJ. Parental perceptions of
Thinking: Concepts and Trends. Int Education Stud 2018;11(4):29–42. technology and technology-focused parenting: Associations with youth screen
[35] Heckman JJ, Masterov DV. The productivity argument for investing in young time. J appl developmental psychol 2016;44:28–38.
children. Appl Econ Perspectives and Pol 2007;29(3):446–93. [68] Shallice T. Specific impairments of planning. Philosophical Transactions of the
[36] Holloway ID, Ansari D. Mapping numerical magnitudes onto symbols: The Royal Society of London. B, Biol Sci 1982;298(1089):199–209.
numerical distance effect and individual differences in children’s mathematics [69] Shute VJ, Sun C, Asbell-Clarke J. Demystifying computational thinking.
achievement. J Exp Child Psychol 2009;103(1):17–29. Educational Res Rev 2017;22:142–58.
[37] ISTE I, CSTA C. Operational definition of computational thinking for K–12 [70] Tang X, Yin Y, Lin Q, Hadad R, Zhai X. Assessing computational thinking: A
education. Nat Sci Foundation 2011. systematic review of empirical studies. Computers and Education 2020;148:
[38] Jung SE, Won ES. Systematic review of research trends in robotics education for 103798.
young children. Sustainability 2018;10(4):905. [71] Team RC. In: R: A Language and Environment for Statistical Computing (Version
[39] Kalelioğlu F. A new way of teaching programming skills to K-12 students: Code. 3.5. 2, R Foundation for Statistical Computing2018; 2019.
org. Comput Hum Behav 2015;52:200–10. [72] Tran Y. Computational thinking equity in elementary classrooms: What third-grade
[40] Kazakoff ER, Sullivan A, Bers MU. The effect of a classroom-based intensive students know and can do. J Educational Comput Res 2019;57(1):3–31.
robotics and programming workshop on sequencing ability in early childhood. [73] van de Vijver FJ, Willemsen ME. Abstract thinking. Advances in psychology, 103.
Early Childhood Education J 2013;41(4):245–55. North-Holland; 1993. p. 317–42.
[41] Köstering L, Nitschke K, Schumacher FK, Weiller C, Kaller CP. Test–retest [74] Vandell DL, Belsky J, Burchinal M, Steinberg L, Vandergrift N. Do effects of early
reliability of the Tower of London Planning Task (TOL-F). Psychol Assess 2015;27 child care extend to age 15 years? Results from the NICHD study of early child care
(3):925. and youth development. Child development, 2010;81(3):737–56.
[42] Langdon R, Coltheart M. Mentalising, schizotypy, and schizophrenia. Cognition [75] Vandell DL, Burchinal M, Pierce KM. Early child care and adolescent functioning at
1999;71(1):43–71. the end of high school: Results from the NICHD Study of Early Child Care and
[43] Lowe T, Brophy S. An operationalized model for defining computational thinking. Youth Development. Dev Psychol 2016;52(10):1634.
In: 2017 IEEE Frontiers in Education Conference (FIE). IEEE; 2017. p. 1–8. [76] Wing JM. Computational thinking. Commun ACM 2006;49(3):33–5.
[44] Lyons IM, Bugden S, Zheng S, De Jesus S, Ansari D. Symbolic number skills predict [77] Wing JM. Computational thinking and thinking about computing. Philos Trans R
growth in nonsymbolic number skills in kindergarteners. Dev Psychol 2018;54(3): Soc, A 2008;366(1881):3717–25.
440. [78] Association World Medical. World Medical Association Declaration of Helsinki.
[45] Magnuson K, Duncan GJ. Can early childhood interventions decrease inequality of Ethical principles for medical research involving human subjects. Bull World
economic opportunity? RSF: The Russell Sage Foundation J Soc Sci 2016;2(2): Health Organ 2001;79(4):373.
123–41. [79] Zapata-Cáceres M, Martín-Barroso E, Román-González M. Computational thinking
[46] Maloney EA, Risko EF, Preston F, Ansari D, Fugelsang J. Challenging the reliability test for beginners: Design and content validation. In: 2020 IEEE Global Engineering
and validity of cognitive measures: The case of the numerical distance effect. Acta Education Conference (EDUCON). IEEE; 2020. p. 1905–14.
Psychol (Amst) 2010;134(2):154–61. [80] Zelazo PD, Blair CB, Willoughby MT. Executive Function: Implications for
[47] Mikelic Preradovic N, Lešin G, Šagud M. Investigating Parents’ Attitudes towards Education. NCER 2017-2000. National Center for Education Research; 2016.
Digital Technology Use in Early Childhood: A Case Study from Croatia. Informatics [81] Zelazo, P. D., Carlson, S. M., and Kesek, A. (2008). The development of executive
in education 2016;15(1):127–46. function in childhood.
9
Science Education International
31(2), 159-163
https://doi.org/10.33828/sei.v31.i2.4
ORIGINAL ARTICLE
Level of Computational Thinking Skills among Secondary

Science Student: Variation across Gender and Mathematics
Achievement
Samri Chongo1*, Kamisah Osman1, Nazrul Anuar Nayan2
1
Faculty of Education, National University of Malaysia, 2Faculty of Engineering and Built Environment, National University of Malaysia
*Corresponding Author: samrichongo@gmail.com
ABSTRACT
Computational thinking (CT) is often associated with computer science and mathematics in general. However, the involvement of
computer science according to gender is seen to be imbalanced. Therefore, this study aimed to identify the level of students’ CT skills
by gender and their relationship to achievement in mathematics. The study employed survey research design and purposive sampling
that involved 128 participants from four science stream students in one of the states in Malaysia (n = 128). The measurement of CT
skills was done using a modified CT test adapted from the previous studies. The results of the study were analyzed using descriptive
and inferential statistics. Spearman correlational analysis was applied to examine the relationship between CT skill and mathematical
achievement and students’ t-test analysis was used to determine the difference in CT skills across gender. The study found that the
students’ CT skills were at the honors level. The relationship between CT skills and mathematics achievement was statistically significant,
whereas there exist no significant gender differences in CT skills. This study suggests that the teaching of mathematical logic needs to
be considered to improve students’ CT skills.
KEY WORDS: computational thinking; science students; gender differences; mathematics achievement; higher-order thinking skills
INTRODUCTION technology, engineering, and mathematics (STEM) education

include shortages of learning materials, a declining number of
I
n the 21st century, we would argue that every country
students pursuing STEM in secondary or higher education,
wants to create a society that can solve problems by
and a deterioration in student achievement at home and in
thinking critically, creatively, and innovatively. As a result,
international benchmarking studies, such as the Trends in
many countries have focused on higher-order thinking skills
International Mathematics and Science Study and the PISA
(HOTS) in the education system to compete in the global digital
economy. However, based on the results of the Program of (Wan Nor Fadzilah et al., 2017). This issue was discussed by
International Student Assessment (PISA), Malaysian students Kamisah and Rohaida (2014) who evaluated STEM issues
exhibit a moderate level of problem-solving skills. As a result, based on teaching and learning (T and L) approaches that do
in 2017, the integrated curriculum secondary school (KBSM) not relate to real-life situation and the integration of STEM
curriculum changed to the standard curriculum secondary into science subjects. Therefore, the use of technology, such
school (KSSM), and computer science subjects were introduced as computers, should be emphasized by teachers in the T and
at the secondary level to equip students with programming L process (Salihuddin et al., 2016).
skills and an understanding of algorithms to produce creative, The use of computer-based learning is a T and L method that
innovative, and dynamic students (KPM, 2016a). can enhance HOTS (Salihuddin et al., 2016); however, thinking
Science stream students in Malaysia are those taking classes skills alone are inadequate. For example, computer use in
in physics, chemistry, and biology. These streams have an chemistry T and L requires a problem-solving approach and
admission requirement dependent on science and mathematics a tool for solving problems through computational thinking
achievements based on the Form Three Assessment (PT3) (CT). One objective of computer science education includes
conducted by the Ministry of Education (KPM). Malaysia the application of algorithms using logic to solve complex
needs 493,830 scientists and engineers (Wan Nor Fadzilah et problems through CT (KPM, 2016a). The goals of computer
al., 2017); however, there were only 97,095 science stream science include motivating students to go beyond the screen
students that sat for the Malaysian Certificate of Education and investigate how computers work and solutions for various
(Sijil Pelajaran Malaysia; SPM) in 2017 (KPM, 2018) and problems (Sysło and Kwiatkowska, 2015). CT is a cognitive
314,095 science students (science and technical) in 2016 process involving logical thinking to solve problems and better
(KPM, 2016b). In Malaysia, issues associated with science, understand procedures and systems (Csizmadia et al., 2015).
Science Education International ¦ Volume 31 ¦ Issue 2 159

31(2), 159-163
Chongo, et al.: Gender issues and mathematics in computational thinking
Olabe et al. (2014) found that new pedagogical approaches, The involvement of male and female students in STEM
such as simple programming using Scratch, showed an ability fields needs to be balanced to avoid gender discrimination
to solve real-life problems. CT skills need to be integrated (Sexton, 2017). In terms of CT skills, there is little difference
into pure science subjects to allow their application to solve between men and women. The previous studies report no
problems more systematically and improve the number and difference in CT skills between male and female students aged
abilities of science stream students. 1518 years (Atmatzidou and Demetriadis, 2016; Korkmaz and
Oluk, 2016); however, gender differences are inconsistent
Each student exhibits different levels of CT skills based on
(Atmatzidou and Demetriadis, 2016). In most cases, female
area, gender, and achievement in mathematics. CT skills are
students require more time in their training sessions to achieve
among the elements that teachers need to consider before
the same CT skills as males (Atmatzidou and Demetriadis,
integrating CT into any subject, especially those involving
2016), although such skills encourage students, especially
science students (e.g. chemistry, physics, and biology). CT
women, to pursue STEM (Repenning et al., 2015) and
skills need to be identified at an early stage to plan more
especially computer science. In addition, differences in CT
effectively CT-based activities. Acquisition of CT skills will
skills between men and women can be influenced by several
be important in future (Saritepeci, 2019), with CT ability
factors, with Grover et al. (2015) showing that female students
often associated with mathematical logic and gender favoring
spend more time online after school demonstrating better CT
males. Therefore, the purpose of this study was to identify
skills than males; however, there remain few female students
CT levels, relationships between CT skills and mathematical
in STEM fields, especially at the university level (Shute et al.,
achievement, and differences in CT skills by gender. The three
2017).
research questions were: (1) What are students CT skill level,
(2) is there a relationship between CT skills and mathematical
achievement, and (3) are there differences in CT skills based METHODOLGY
on gender? The study developed two null hypotheses: Research Design
H01: There is no significant relationship between CT skills This study used a quantitative approach to perform a survey
and mathematical achievement. to evaluate CT skill level, its relationship to mathematics
H02: There is no significant difference in CT skills by achievement (PT3), and differences in CT skills by gender. The
gender. survey design was used to obtain CT skill level information
from 128 samples of form four science students studying at
LITERATURE REVIEW public secondary school.
CT skills involve problem-solving, system design, and Population and Samples
understanding human behavior by illustrating basic concepts in This study was conducted in a district of Tawau in the state
computer science (Wing, 2006). In 2010, Wing introduced a new of Malaysia, with the study population comprising science
definition of CT describing it as a thinking process involved in stream students in secondary schools (~16 years old) taking
formulating problems and representing solutions in a form that chemistry. The study cohort included 432 students comprising
can be effectively implemented by an information-processing 137 boys and 295 girls and adhered to certain criteria with an
agent. According to Chao (2016), CT involves a visual problem- aim to generalize the findings based on the characteristics of
solving environment for allowing acquisition of programming people or events in the population (Mcmillan, 1996). The study
skills with technological support of computational problem- samples included 128 students used methods of purposive
solving. These definitions focus on the use of technology, sampling, and sampling was applied to selected individuals
specifically computer science, for solving problems. Although based on their experience level (Creswell, 2009).
there is no consensus definition of CT (Román-González et al.,
Instrument
2018), it can generally be defined as a process of thinking and a
The CT skills test was adapted from the UK Bebras
tool for solving problems using computer concepts either with
Computational Challenge (Blokhuis et al., 2015; 2016;
a computer (plugged-in) or without one (unplugged).
Blokhuis et al., 2017). There are five CT skills exhibited by
Computer science combines mathematical and engineering students: Abstraction, algorithmic thinking, decomposition,
concepts and relies heavily on a mathematical foundation evaluation, and generalization (Csizmadia et al., 2015; Selby
(Wing, 2006). CT is the ability to understand and apply the and Woollard, 2013). There are 15 objective questions with
basic principles of computer science (KPM, 2016a) and allows difficulty levels according to a specified ratio and comprising
operation of complex systems to address real-world problems four difficult questions (26.7%), seven moderate questions
(Wing, 2008). STEM students are particularly well suited (46.7%), and four simple questions (26.7%). Student
for CT and its future applications (León et al., 2014). CT mathematics achievement is based on PT3 results from the
skills potential offer a higher level of problem-solving skills Ministry of Education in 2018 and includes five grades:
(Sridaran and Shailaja, 2015). Wing (2006) states that CT is Excellent (A), credit (B), good (C), satisfactory (D), and
not only applied by computer scientists but also by anyone reached minimum level (E). Student CT levels were determined
with basic problem-solving skills. based on scores from the modified CT test according to the
160 Science Education International ¦ Volume 31 ¦ Issue 2

31(2), 159-163
SPM grades, which can be categorized into four levels based

Table 1: CT skills level
on the percentage of marks obtained: Excellence (70–100),
credit (50–70), pass (40–49), and fail (0–39) (Table 1). The SPM grades Marks Details Marks CT skills level
difficulty and percentage of questions concerning CT skills A+ 90–100 Supreme excellence 70–100 Excellence
are shown in Table 2. A 80–89 High excellence
A− 70–79 Excellence
Data Analysis B+ 65–69 Supreme credit 50–70 Credit
In a CT test comprising 15 objective questions, an analytical B 60–64 High credit
scoring method was used, where each correct answer received C+ 55–59 Upper credit
a score that was then converted to a percentage. Student C 50–54 Credit
achievement in PT3 mathematics and gender was derived D 45–49 Upper pass 40–49 Pass
from the profiles of student taking the CT test. Data analyses E 40–44 Pass
involved descriptive statistics and inferential statistics, with G 0–39 Fail 0–39 Fail
descriptive statistics used to identify student CT levels, and CT: Computational thinking
inference statistics using Spearman’s correlation applied to
determine relationships between CT skills and mathematics
achievement. A Student’s t-test was used to determine Table 2: Element of CT skills, questions difficulty level,
significant differences in CT skills by gender. and percentage (%)
Question Element of CT skills Difficulty level Percentage (%)
FINDINGS AND DISCUSSION 1 Algorithmic thinking Easy 26.7
2 Abstraction
The results were analyzed using descriptive statistics to 3 Algorithmic thinking
evaluate levels of student CT skills in terms of frequency (f), 4 Decomposition
percentage (%), and mean and standard deviation, whereas 5 Abstraction Moderate 46.7
inferential statistics was used to examine the relationship of 6 Evaluation
CT skill level with mathematical achievement and gender 7 Generalization
differences according to correlation analysis and t-tests after 8 Decomposition
normality testing. 9 Algorithmic thinking
10 Abstraction
Normality Tests 11 Generalization
Normality tests were performed to determine the parametric 12 Decomposition Hard 26.7
or non-parametric nature of the data. Here, we used the 13 Evaluation
Kolmogorov–Smirnov test and histograms to determine that 14 Generalization
the data were normally distributed and that differences in CT 15 Evaluation
skills were not significant, ρ > 0.05 (KS = 0.093, df = 128, Total 100
ρ = 0.08). The data distribution of student CT skill levels is
shown in Figure 1.
Table 3: Level of students’ CT skills
Levels of Student CT Skills
The level of CT-based knowledge and the percentage of CT CT skills Frequency Percentage Valid Cumulative
levels percentage percentage
skills among science stream students by category are shown in
Fail 22 17.2 17.2 17.2
Table 3. Overall, respondent CT skill levels were moderate or
Pass 37 28.9 28.9 46.1
credit based on CT skills level (M = 52.42, SD = 16.995), with
Credit 50 39.1 39.1 85.2
<15% of respondents achieving excellent and 17.2% failing.
Excellence 19 14.8 14.8 100.0
The level of CT skills could be categorized into four stages:
Total 128 100.0 100.0
Fail, pass, credit, and excellence.
Identifying a Relationship between CT Skills and hypothesis. Spearman’s correlation suggested a moderate
Mathematic Achievement (PT3) and significantly positive relationship between CT skill and
The null hypothesis (Ho1) states that there is no relationship mathematics achievement (r = 0.322, n = 128; ρ < 0.05).
between CT skills and mathematics achievement. Table 4
These results show that achievement in mathematics can
shows the level of student achievement in mathematics (PT3)
promote CT skills through the application of logical thinking.
from public examinations when students in form three. The
Randles and Overton (2015) explained that expert problem
data show that 46.9% of students achieved distinction (“A”),
solvers develop a strategy and apply a logical and scientific
and only 0.8% achieved the minimum (“E”).
approach to solving problems. Moreover, the previous studies
Table 5 shows a significant correlation between CT skill and report a close relationship between CT and mathematics
mathematics achievement, thereby rejecting the first null ability (Weintrop et al., 2016). Wing (2006) indicated that

31(2), 159-163
Table 4: Students’ achievement in mathematics (PT3)

Mathematics achievement (PT3) Grade Frequency Percentage Valid percentage Cumulative percentage
Reached minimum level E 1 0.8 0.8 0.8
Satisfying D 5 3.9 3.9 4.7
Good C 23 18.0 18.0 22.7
Credit B 39 30.5 30.5 53.1
Distinction A 60 46.9 46.9 100.0
Total 128 100.0 100.0
Table 5: Relationship between CT skills and mathematics Identifying Gender-Based Differences in CT Skills
achievement (PT3) Testing of the second null hypothesis (Ho2) revealed no
significant difference between males and females in terms
CT levels
of CT skills (t = 0.055; df = 126, ρ >.05). The t-test analysis
Spearman’s rho Mathematic Correlation coefficient 0.322**
results shown in Table 6, and a relationship between gender
achievement Sig. (two tailed) 0.000
CT skills was weak. These results suggest that gender does not
n 128
play a significant role in student CT skills. This result agrees
**Correlation is significant at the 0.01 level (two tailed). CT: Computational
thinking with Korkmaz and Oluk (2016) who previously reported
no significant differences in CT skills according to gender.
Conversely, Atmatzidou and Demetriadis (2016) showed that
Table 6: The t-test differences between female and male females require more time to obtain similar levels of CT skills
in their skills of CT as males. However, the present study indicated that integrating
Gender Number of Mean (x̄) Std. t df Sig. CT skills are not influenced by gender.
participants (N) deviation
Male 57 52.51 17.29 0.055 126 0.956 CONCLUSION
Female 71 52.35 16.88
The level of CT proficiency and the elements that influence
the outcomes needs to be identified before integrating it into
a curriculum. Therefore, identifying levels of student CT
skills are important for planning student activities to allow
effective application of CT skills. These findings showed
that gender factors do not influence CT skills, whereas
mathematic achievement is related to CT skills. Therefore,
basic mathematical knowledge, including computational areas
involving algorithms, should be emphasized early in T and L
activities to enhance critical thinking skills. Improvements
in critical thinking skills will stimulate students to solve
problems, thereby enhancing CT skills and student engagement
in STEM fields.
REFERENCES
Atmatzidou, S., & Demetriadis, S. (2016). Advancing students’ computational
thinking skills through educational robotics: A study on age and gender
Figure 1: Normality test for computational thinking (CT) skills test relevant differences. Robotics and Autonomous Systems, 75, 661-670.
Bell, S.E. & Sexton, S.S. (2018). Science education professional
development for primary/elementary teachers: A tale of two systems.
CT is not only relevant to science and mathematics but also Science Education International, 29(2), 117-123.
for all subjects, including the humanities. Activities should Blokhuis, D., Csizmadia, A., Millican, P., Roffey, C., Schijvers, E., &
be designed to be easily adopted and integrated into existing Sentence, S. (2017). UK Bebras Computational Thinking Challenge.
mathematics and science curricula (Weintrop et al., 2016). UK Bebras. Available from: http://www.bebras.uk/answer-booklets.
html.
A study from New Zealand reported early integration of Blokhuis, D., Millican, P., Roffey, C., Schrijvers, E., & Sentance, S. (2015).
mathematics and language studies as effective for teachers (Bell UK Bebras Computational Thinking Challenge 2015. UK Bebras.
And Sexton, 2018). Applications of CT skills for instruction of Available from: http://www.bebras.uk/answer-booklets.html.
reading, writing, and arithmetic can enable children to think Blokhuis, D., Millican, P., Roffey, C., Schrijvers, E., & Sentance, S. (2016).
UK Bebras Computational Thinking Challenge. UK Bebras. Available
analytically while solving problems (Wing, 2006). To ensure from: http://www.bebras.uk/answer-booklets.html.
students understand such concepts, CT-based activities should Chao, P.Y. (2016). Exploring students’ computational practice, design
be relevant to real-world problems (Bell and Sexton, 2018). and performance of problem-solving through a visual programming
162 Science Education International ¦ Volume 31 ¦ Issue 2

31(2), 159-163
environment. Computers and Education, 95, 202-215. & Repening, N. (2015). Scalable game design : A strategy to bring
Creswell, J.W. (2009). Research Design: Qualitative, Quantitative and systemic computer science education to schools through game design
Mixed Methods Approaches. 3rd ed. New York, United States: Sage and simulation creation. ACM Transactions on Computing Education,
Publication, Inc. 15(2), 1-31.
Csizmadia, A., Curzon, P., Humphreys, S., Ng, T., Selby, C., & Woollard, J. Román-González, M., Pérez-González, J.C., Moreno-León, J., & Robles, G.
(2015). Computational Thinking: A Guide for Teachers. California, (2018). Extending the nomological network of computational thinking
United States: Computing At School. with non-cognitive factors. Computers in Human Behavior, 80, 441-459.
García, P., José, F., & Mendes, A.J. (2017). Exploring the computational Salihuddin, M.S., Hasnah, M., Zaleha, A., Norasykin, M.Z., Baharuddin, A.,
thinking effects in pre-university education. Computers in Human & Mageswaran, S. (2016). Enhancing student’s higher order thinking
Behavior, 80, 407-411. skills (HOTS) through the socratic method approach with technology.
Grover, S., Pea, R., & Cooper, S. (2015). Designing for deeper learning in a In: 1St ICRIL-International Conference on Innovation in Science and
blended computer science course for middle school students. Computer Technology. Kuala: Universiti Teknologi Malaysia.
Science Education, 25(2), 199-237. Saritepeci, M. (2019). Developing computational thinking skills of high
Kamisah, O., & Rohaida, M.S. (2014). Editorial: Science, technology, school students : Design-based learning activities and programming
engineering and mathematics (STEM) education in Malaysia. EURASIA tasks. The Asia Pacific Education Researcher, 29, 35-54.
Journal of Mathematics Science and Technology Education, 10(3), 153- Selby, C.C., & Woollard, J. (2013). Computational Thinking: The
154. Developing Definition. ACM. Available from: https://www.eprints.
Korkmaz, Ö., & Oluk, A. (2016). Comparing students’ scratch skills with soton.ac.uk/356481.
their computational thinking skills in terms of different variables how Sexton, S. (2017). Meaningful intersections of social justice and
to teach computational thinking view project instructional technology contemporary cultural competencies in a New Zealand master’s
view project comparing students’ scratch skills with their computational. level initial teacher education programme. Journal of Contemporary
Modern Education and Computer Science, 11, 1-7. Educational Research, 1(1), 33-42.
KPM. (2016a). Basics Computer Science: Standard Curriculum and Shute, V.J., Sun, C., & Clarke, J.A. (2017). Demystifying computational
Assessment Document. Putrajaya: KPM. thinking. Educational Research Review, 22, 142-158.
KPM. (2016b). Malaysia Educational Statistics. Report No. Putrajaya: Shailaja, J., & Sridaran, R. (2014). Computational thinking, the Intellectual
KPM. Available from: http://www.moe.gov.my. thinking for the 21st Century. Available from: https://www.researchgate.
KPM. (2018). Announcement of 2017 Malaysian Education Certificates net/publication/273776374_Computational_Thinking_The_
(SPM) Result Analysis. Report No. Putrajaya. Available Intellectual_Thinking_for_the_21st_century.
from: http://www.lp.moe.gov.my/images/bahan/spm/2018/ Sysło, M.M., & Kwiatkowska, A.B. (2015). Introducing a new computer
PengumumanAnalisisKeputusanSPM2017.pdf. science curriculum for all school levels in Poland. In: Brodnik, A., &
León, J., Núñez, J.L., & Liew, J. (2014). Self-determination and STEM Vahrenhold, J., (Eds.), ISEEP 2015. LNCS 9378. Ljubljana: Springer.
education: Effects of autonomy, motivation, and self-regulated learning pp. 141-154.
on high school math achievement. Learning and Individual Differences, Wan Nor Fadzilah, W.H., Suhaiza, M.S., & Lilia, H. (2017). The Inculcation
43, 156-163. of STEM Culture Outside the Classroom. Malaysia: UKM.
Mcmillan, J.H. (1996). Educational Research : Fundamentals For The Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L., &
Consumer. 2nd ed. New York, United States: Haper Collins Publisher Wilensky, U. (2016). Defining computational thinking for mathematics
Inc. and science classrooms. Journal of Science Education and Technology,
Olabe, J.C., Basogain, X., Olabe, M.Á., Maíz, I., & Castaño, C. (2014). 25, 127-147.
Solving math and science problems in the real world with a computational Wing, J.M. (2006). Computational Thinking. Communications of the ACM,
mind. New Approaches in Education Research, 3(2), 75-82. 49(3), 33-35.
Randles, C.A., & Overton, T.L. (2015). Expert vs novice : Approaches used Wing, J.M. (2008). Computational thinking and thinking about computing.
by chemists when solving open-ended problems. Chemistry Education Philosophical Transactions of the Royal Society A, 366(1881), 3717-
Research and Practice, 16(4), 811-823. 3725.
Repenning, A., Webb, D.C., Koh, K., Nickerson, H., Miller, S.B., Brand, C., Wing, J.M. (2010). Computational Thinking: What and Why? Available from:
Horses, I., Basawapatna, A., Gluck, F., Grover, R., Gutierez, K., https://www.cs.cmu.edu/~CompThink/resources/TheLinkWing.pdf.

Research in Science & Technological Education
ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/crst20
Improving early childhood pre-service teachers’

computational thinking teaching self-efficacy
beliefs in a STEM course
Ayşe Çiftçi & Mustafa Sami Topçu
To cite this article: Ayşe Çiftçi & Mustafa Sami Topçu (2022): Improving early childhood pre-
service teachers’ computational thinking teaching self-efficacy beliefs in a STEM course, Research
in Science & Technological Education, DOI: 10.1080/02635143.2022.2036117
To link to this article: https://doi.org/10.1080/02635143.2022.2036117
Published online: 09 Feb 2022.
Submit your article to this journal
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=crst20
RESEARCH IN SCIENCE & TECHNOLOGICAL EDUCATION
https://doi.org/10.1080/02635143.2022.2036117
RESEARCH ARTICLE
Improving early childhood pre-service teachers’

computational thinking teaching self-efficacy beliefs in
a STEM course
a b
Ayşe Çiftçi and Mustafa Sami Topçu
a
Department of Mathematics and Science Education, Muş Alparslan University, Muş, Turkey; bDepartment of
Mathematics and Science Education, Yıldız Technical University, Istanbul, Turkey
ABSTRACT KEYWORDS
Background: In order for computational thinking (CT) to be suc Computational thinking
cessfully implemented in educational environments, teachers’ CT teaching; computational
teaching self-efficacy beliefs should be developed. The existing thinking integrated STEM
education; self-efficacy
research on this topic is generally focused on training in-service
beliefs; STEM education;
teachers on CT to develop their self-efficacy. Conversely, the current early childhood pre-service
study aims to contribute to the development of early childhood teachers
pre-service teachers’ CT teaching self-efficacy beliefs through CT-
integrated STEM education.
Purpose: In the current study, the effect of CT-integrated STEM
education on CT teaching self-efficacy beliefs of early childhood
pre-service teachers was explored.
Sample: The study group was composed of 68 second-year
early childhood pre-service teachers. All were attending
a state university in the city of İstanbul during the 2020–2021
academic year.
Design and Methods: The study employed a quasi-experimental
design, i.e. the control group was given only STEM education, while
the experimental group was given CT-integrated STEM education.
The data collection tool used in the current study was ‘the scale of
self-efficacy beliefs towards teaching computational thinking’. For
analysis, an independent samples t-test and a paired samples t-test
were used.
Results: The findings have revealed that there is a significant dif
ference in the post-test scores on the ‘scale of self-efficacy beliefs
towards CT teaching’ in favor of the experimental group. The effect
size calculated for the difference between the mean pre-test and
post-test scores of the experimental group students was found to
be large. These results show that CT-integrated STEM education
contributed to the development of the pre-service teachers’ CT
teaching self-efficacy beliefs.
Conclusion: The current study contributes to the literature by
showing that CT-integrated STEM education fosters pre-service
teachers’ professional development and their self-efficacy beliefs
about CT teaching.
CONTACT Ayşe Çiftçi a.ciftci@alparslan.edu.tr Department of Mathematics and Science Education, Muş
Alparslan University, Muş, Turkey
© 2022 Informa UK Limited, trading as Taylor & Francis Group
2 A. ÇIFTÇI AND M. S. TOPÇU
Introduction
‘Computational thinking (CT) is taking an approach to solving problems, designing
systems and understanding human behavior that draws on concepts fundamental to
computing’ (Wing 2008, 3717). In the 21st century, CT is considered a basic skill – in line
with reading, writing, and arithmetic – to be acquired by every child (Barr, Harrison, and
Conery 2011; Wing 2006). In this context, policy makers in many countries such as the
United States, have included applications for CT in their early childhood education
curricula (Bers and Sullivan 2019). The CT approach has recently been integrated into
early childhood education to help children to gain CT skills easily as learning and devel
opment occur very quickly in early childhood (Çiftçi, Topçu, and Foulk 2020).
Another growing area of interest in early childhood education in the 21st century is
STEM education. STEM is an approach based on the integration of science, technology,
engineering, and mathematics disciplines into education. Many studies have emphasized
that it is appropriate to integrate CT and STEM, both of which are considered important
during the early childhood period (Jona et al. 2014; Sun et al. 2020; Weintrop et al. 2014,
2016; Zhang and Biswas 2019). There is a symbiotic, complementary relationship between
CT and STEM, and CT tools support effective STEM learning (Grover and Pea 2018).
Moreover, STEM enriches CT and causes it to be used more widely (Jona et al. 2014;
Weintrop et al. 2016).
Integration of CT with STEM is helpful in the enrichment, dissemination, and teaching
of CT in early childhood education (Jona et al. 2014; Weintrop et al. 2016); teachers play an
important role in this integration and in the teaching of CT. Barr and Stephenson (2011)
emphasize that in-service and pre-service teacher trainings are important factors in
successfully integrating CT into learning environments. The existing CT literature gener
ally focuses on the professional development of in-service teachers – there is a limited
interest in how pre-service teachers will teach CT in their future classes (Yadav et al.
2017b). Furthermore, the literature review revealed that pre-service elementary teachers
are generally given training about CT (Kaya et al. 2019, 2020; Mason and Rich 2019) but
the number of studies conducted with early childhood pre-service teachers (e.g.
Vasconcelos, Arslan-Ari, and Ari 2020) is very limited. Correspondingly, further CT research
with the participation of early childhood pre-service teachers is needed.
Institutions such as Next Generation Science Standards (NGSS), Computer Science
Teachers Association (CSTA), and International Society for Technology in Education
(ISTE) have stated that CT is an important component of curricula (Kaya et al. 2019).
However, teachers’ self-confidence and self-efficacy towards CT teaching are low (Kaya
et al. 2019, 2020; Mason and Rich 2019). For CT to be successfully implemented in
educational environments, teachers’ CT teaching self-efficacy beliefs should be
developed.
Research has shown that there is a strong relationship between teachers’ self-efficacy
and students’ academic success, and that teachers’ high self-efficacy positively affects
students’ academic achievement (e.g. Gulistan and Hussain 2017; Shahzad and Naureen
2017). Moreover, when teachers’ self-efficacy beliefs are high, students’ deep learning
approaches are positively affected; however, when teachers have low self-efficacy,
students engage in superficial learning (Mazlum, Cheraghi, and Dasta 2015).
Accordingly, it is important to develop teachers’ CT self-efficacy so that students may
RESEARCH IN SCIENCE & TECHNOLOGICAL EDUCATION 3
perform highly in CT, engage with deep learning approaches towards CT, and develop
their CT skills. In order for teachers to develop their self-efficacy, they need to receive
training on the subject. The existing research is generally focused on giving training to
in-service teachers on CT to develop their self-efficacy (Zhao et al. 2020). Conversely, the
current study aims to contribute to the development of pre-service teachers’ CT teaching
self-efficacy beliefs through CT-integrated STEM education. Moreover, this study con
tributes to the literature because it was conducted with early childhood pre-service
teachers, unlike the studies mentioned above (Kaya et al. 2019, 2020; Mason and Rich
2019; Zhao et al. 2020).
Teaching CT as disconnected from daily life or without being associated with any
discipline causes students to have problems in learning CT concepts and negatively
affects their motivation to learn (Goode, Estrella, and Margolis 2006). Thus, in the current
study, CT teaching self-efficacy beliefs of pre-service teachers were examined by con
sidering CT concepts in the context of STEM. The STEM disciplines offer easy and
effective daily life contexts for teaching CT (Grover and Pea 2018). Therefore, through
STEM, students can learn CT concepts in a more meaningful and interesting way by
associating them with their own lives. Similarly, if pre-service teachers encounter CT in
the context of STEM, rather than on its own, they may find the concept more meaningful
and interesting. For example, consider teaching algorithms, which are a sub-concept of
CT. The water cycle in science or even the life of a person can be presented as
sequential, algorithmic events. In this way, the algorithm could be taught to children
through real-life examples. By combining CT with the more approachable areas of STEM,
pre-service teachers would be able to learn how to teach children the mechanisms
behind science subjects by using CT concepts, such as algorithms. We hypothesized that
this will lead to an increase in pre-service teachers’ self-efficacy towards teaching CT and
its sub-concepts.
The impetus for working with pre-service teachers in the current study is that pre-
service education is seen as one of the important factors affecting teachers’ self-efficacy
(Kraut, Chandler, and Hertenstein 2016). The field of early childhood education was
chosen due to the necessity of developing CT skills from an early age; this is necessary in
order to train creative and innovative individuals who produce technology rather than
consuming technology (Hoić-Božić, Mezak, and Tomljenović 2019). If early childhood
pre-service teachers’ CT teaching self-efficacy beliefs develop, they may tend to conduct
more activities to develop children’s CT skills in early childhood learning environments
as teachers. In line with the reasons stated above, the research questions were as
follows:
● Is there a statistically significant difference between the effects of computational

thinking–integrated STEM education and STEM education alone on early childhood
pre-service teachers’ computational thinking teaching self-efficacy beliefs?
● Is there a significant effect of computational thinking–integrated STEM education on
early childhood pre-service teachers’ computational thinking teaching self-efficacy
beliefs?
● Is there a significant effect of STEM education on early childhood pre-service tea
chers’ computational thinking teaching self-efficacy beliefs?
Research background
Computational thinking (CT) and sub-concepts
The concept of computational thinking (CT) was first used in the context of K-12 educa
tion in 1980 by Seymour Papert to teach children mathematical concepts with the LOGO
programming language (Czerkawski and Lyman 2015). Then, CT gained popularity with
the study ‘Computational Thinking’, published by Wing in Communications of the ACM in
2006. Wing (2006) explained CT as a universal skill set that every child, not just computer
scientists, should learn. Later, in 2010, Cuny, Snyder, and Wing described CT as follows:
Computational thinking is a means of thinking required when posing a problem and
finding a solution to it, which can be performed effectively by a computer or a human
(Wing 2011).
In the literature (e.g. Weintrop et al. 2016), the necessity of dividing CT into well-
defined sub-concepts is strongly emphasized. In the current study, CT applications were
carried out according to these sub-concepts: data collection, data analysis, data repre
sentation, problem decomposition, abstraction, algorithms and procedures, paralleliza
tion, debugging, and pattern recognition. The sub-concepts were used as an instructional
framework and integrated into the activities developed within the context of the study.
These concepts are also included in the ‘scale of self-efficacy beliefs towards teaching
computational thinking’, which was used in the current study. The definitions of the CT
sub-concepts used in this study are given below and an example for each concept is
given:
● Data collection is the process of finding a suitable data source (e.g. observation,
measurement) and collecting information about a problem (Barr and Stephenson
2011; ISTE 2011). For example, making different parachute designs that can safely
land all the eggs placed inside, and noting in a table the number of eggs that are
safely grounded in each parachute design.
● Data analysis means making the data meaningful and drawing conclusions by
making some statistical calculations about the data (Barr and Stephenson 2011;
ISTE 2011). For example, students may compare the number of eggs that different
parachute designs have safely landed on the ground.
● Data representation is the process of organizing and displaying data through tools
such as tables, graphs, and images (ISTE 2011). For example, creating a graph
showing how the number of eggs that are safely grounded changes as the air-
contacting surface of the parachute changes.
● Problem decomposition means breaking down complex problems into manageable
small problems (Yadav, Hong, and Stephenson 2016). For example, determining the
roles and responsibilities of group members and the materials needed to complete
the parachute design.
● Abstraction is deciding which details are important and which can be ignored,
focusing on the main idea (Wing 2008). For example, focusing on the variable that
will increase the number of eggs that can be safely grounded in the parachute
design, while ignoring other variables. Abstraction also includes representations,
such as models (Gautam, Bortz, and Tatar 2020).
● Algorithms and procedures refers to the steps that should be used to solve the
problem (Yadav, Hong, and Stephenson 2016). For example, following the steps
required for the parachute design, in order.
● Parallelization means dividing tasks in such a way as to be done simultaneously (Barr
and Stephenson 2011). For example, while one group in the classroom is designing
the canopy of the parachute, another group simultaneously makes the container
where the eggs will be placed.
● Debugging includes applications that identify and fix problems (Kafai et al. 2019). For
example, identifying and fixing design problems when parachutes fail to ground
without breaking the eggs.
● Pattern recognition means recognizing that objects are organized according to a rule
or rules, and categorizing objects according to certain characteristics and similarities
(Rowe et al. 2017). For example, identifying the similarities of different parachute
designs made by students and categorizing the designs according to their
similarities.
STEM education and CT

STEM education is an interdisciplinary approach relating concepts in science, technology,
engineering, and mathematics to daily life (Tippett and Milford 2018). STEM is an educa
tional approach that can be applied at all levels of schooling, from pre-school to post-
doctoral levels, both in the classroom and in after-school programs (Gonzalez and Kuenzi
2012). Ramaley defines STEM as an innovative approach that allows students to solve real-
world problems through context-based learning (Watson and Watson 2013).
STEM education in early childhood offers outcomes such as improving children’s
creativity (Üret and Ceylan 2021), fostering school readiness and concept acquisition
(Toran, Aydin, and Etgüer 2020), and supporting children’s future social and academic
success (Tippett and Milford 2018). Furthermore, Çiftçi, Topçu, and Foulk (2020) con
cluded that STEM education given in early childhood education contributes to the
development of 21st century skills, personal-social development and professional devel
opment, development of positive attitudes towards STEM disciplines, love for nature and
living things, and to the growth of a country. All these outcomes show the importance
and necessity of STEM education in early childhood and therefore the training of pre-
service teachers on this approach.
In recent years, importance has been given in the literature to the benefits of the
integration of STEM fields with CT (Jona et al. 2014; Lavigne, Lewis-Presser, and Rosenfeld
2020; Swaid 2015; Zhang and Biswas 2019). CT is seen as an important supporter of STEM
education (Dede, Mishra, and Voogt 2013). There are many studies in which CT is
integrated with science (e.g. Yin et al. 2019), mathematics (e.g. Lavigne, Lewis-Presser,
and Rosenfeld 2020), engineering (e.g. Ehsan, Rehmat, and Cardella 2021; Yin et al. 2019)
and technology (e.g. Hacker 2018). For example, to integrate mathematics with CT
(Lavigne, Lewis-Presser, and Rosenfeld 2020), the teacher asked preschool children to
indicate in order the steps that would lead a monkey character to the picnic area in
a digital application. The children placed numbers to determine which steps the monkey
should follow. For example, within the scope of ordinal numbers, they put ‘1’ in a house
with a blue roof, ‘2’ in a house with a red roof, and ‘3’ in the picnic area. In this way, they
integrated mathematics and the CT sub-concept of algorithmic thinking.
In another study, Yin et al. (2019) integrated CT with physics and engineering. They
used abstraction, one of the CT sub-concepts, by showing breadboard circuit designs as
a diagram. Using decomposition, secondary school students divided the circuit construc
tion task into sub-tasks and the students made each branch of the circuit. Algorithmic
thinking was used to ensure that the steps used to construct the circuits were specified in
order. At the end of the study, the researchers concluded that STEM provides a good
context for improving students’ physics and engineering knowledge, CT skills, and CT
dispositions (Yin et al. 2019). These studies indicated that CT has the potential to be
integrated with mathematics, physics, and engineering at the pre-school and secondary
school level and that this integration could positively affect learning outcomes (physics
and engineering knowledge, CT skills, and CT dispositions). However, each of these
studies focused on student learning outcomes in their discussions, not addressing teacher
training to achieve these outcomes.
Although it has been determined in the literature that the teaching/use of CT in the
context of STEM has a positive effect on many learning outcomes of preschool and
secondary school students, none of the studies have focused on pre-service teacher
training, especially early childhood pre-service teachers. So far, students’ perspective on
STEM and CT integration has been explored; in the current study, we would like to
contribute to the current literature by adding the teacher perspective on STEM and CT
integration. Gulistan and Hussain found that teaching self-efficacy is a major factor in
improving student outcomes (2017), so by addressing teacher confidence we can take
a multi-faceted approach to integrating CT more fully in the STEM curriculum. This study
aimed to improve the CT-teaching self-efficacy beliefs of early childhood pre-service
teachers by using a STEM context and integrating CT. Therefore, the current study
makes an important contribution to the literature by addressing the integration of CT
and STEM at the early childhood pre-service teacher training.
Computational thinking teaching self-efficacy beliefs

Albert Bandura defines self-efficacy, which plays a central role in the structure of social
cognitive theory (Bandura 2001), as one’s beliefs in his/her ability to organize and execute
action plans to manage possible situations (Bandura 1995). According to Bandura’s (1977)
self-efficacy theory, teachers’ behaviors and performances are motivated by perceived
self-efficacy beliefs and outcome expectancy. Perceived self-efficacy is defined as
a person’s belief that he/she can attain different performance levels (Pastorelli et al.
2001). Outcome expectancy refers to the belief that certain outcomes can be obtained
as a result of a specific behaviour (Bandura 1977).
There are varieties of perceived self-efficacy, one of which is personal efficacy.
Bandura (1997) argues that what motivates individuals’ behaviour is personal effi
cacy – their belief that they can effectively perform a task. Bandura’s (1977) self-
efficacy theory guides ‘the scale of self-efficacy beliefs towards teaching computa
tional thinking’ used in the current study to explore early childhood pre-service
teachers’ CT teaching self-efficacy. When the items in the scale are examined,
expressions including ‘can do’ that reflect early childhood pre-service teachers’ CT
teaching self-efficacy can be easily seen (sample items: ‘I can teach how to express
the solution of a problem using an algorithm’, ‘I can break down the problem
statement into parts and explain which parts will be important in the solution
process’.). As seen in these sample items, the use of the different versions of “can
do“ (such as “can teach” or ”can break down”) reflects Bandura’s self-efficacy theory
in which individuals present beliefs in his/her ability to organize and execute action
plans to manage possible situations (Bandura 1995). For this study, the possible
situation is “CT teaching”. Early childhood pre-service teachers in the current study
presented beliefs in his/her ability to organize and execute plans to manage CT
teaching situations.
Teachers’ personal efficacy is their belief in their capacity to affect student performance
(Berman et al. 1977). For example, within the context of CT, if a teacher has high self-
efficacy regarding CT teaching, this can positively affect students’ performance in CT.
Teachers’ high self-efficacy in CT teaching positively affects their CT teaching performance
and motivation (Kaya et al. 2020; Zhao et al. 2020). Teachers’ self-efficacy beliefs affect
their choices, the level of effort they will invest, their coping with difficulties, and the
amount of stress and depression they will experience (Bandura 1990). Teachers with low
self-efficacy cannot effectively manage situations, despite having the necessary knowl
edge and skills (Bandura 1990). Therefore, teachers should be given the necessary training
and support to improve their self-efficacy in CT teaching.
According to Bandura, context plays an important role in the development of self-
efficacy; he argues that the self-efficacy of a person may vary from one context to
another, even for the same task (Bandura 1997; as cited in Psycharis et al. 2018). In
this regard, in the current study, implementations were conducted in two different
contexts (STEM and CT-integrated STEM) in an attempt to determine which context
resulted in further improvement of early childhood pre-service teachers’ self-efficacy
beliefs in CT teaching.
Method
Research design
In order to investigate the effect of CT-integrated STEM education on pre-service teachers’
CT teaching self-efficacy beliefs, a quasi-experimental design was used in this study
(Fraenkel, Wallen, and Hyun 2012). After the pre-test, CT-integrated STEM education
activities were carried out in the experimental group and STEM education activities
were given in the control group. The post-test data were collected from both groups at
the same time.
Study group
The participants of the current study were 68 second-year students studying in the
department of Early Childhood Teaching at a state university in İstanbul. The participants
had not previously received any training on STEM education or CT, and volunteer to
participate in the current study. All participants have provided informed consent. The pre-
service teachers were between the ages of 18 and 33; of them, 7 were male and 61 female.
The experimental group had 2 male and 33 female participants and the control group had
5 male and 28 female.
Implementation
The intervention was conducted by the researchers within the ‘Science Education in
Early Childhood’ course. CT-integrated STEM education was implemented in the
experimental group and STEM education alone was implemented in the control
group. Unlike the pre-service teachers in the control group, the pre-service teachers
in the experimental group learned what CT and its sub-concepts are and observed
sample CT activities carried out by the researchers. Then, they themselves planned CT-
integrated STEM activities and applied the activities they had developed to their peers
in the class. During this time, pre-service teachers in the control group also developed
STEM activities to share with the class. In summary, the pre-service teachers in the
experimental group planned to integrate CT sub-concepts into the activities and
taught CT through CT-integrated STEM activities.
The following stages were followed in the implementation process for both groups:
implementation of the pre-test, instructor information about the approach to be used,
demonstration and implementation of sample activities suitable for the approach,
development of activities by the pre-service teachers, feedbacks given by the research
ers about the developed activities, implementation of the activities by the pre-service
teachers, and administration of the post-test (Figure 2, Figure 5). As seen in Figure 2 and
Figure 5, the same stages were followed in both groups, with additional CT approaches
used in the experimental group. The template/structure used in the activities was the
same for both groups and the pre-service teachers developed their activities according
to this template. The template used by the control group is shown in Appendix 1 and
the template used by the experimental group is shown inAppendix 2. The only differ
ence seen in the templates (Appendix 1 and Appendix 2) is that unlike the control
group, the pre-service teachers in the experimental group integrated CT sub-concepts
into their activity plans in the context of STEM. In order to create a more inquiry and
learner-centered environment, the determination of the sample problem situation,
topics and content of the activities was left to the discretion of the pre-service teachers
in the experimental and control groups. Thus, both groups had the same autonomy in
determining the topics of the activities.
Each activity in both the experimental and control groups followed the stages of the
engineering design process (Figure 1, EiE engineering design process stages). During the
implementation of the developed activities, EiE’s (Engineering is Elementary) engineering
design process stages (ask, imagine, plan, create, improve) were followed (Engineering is
Elementary [EiE] 2021.). In each activity, participants focused on identifying the problem
and its limitations in the ask phase, generating ideas for solving the problem through
brainstorming and choosing the best solution in the imagine phase, and drawing
a diagram and making a list of the materials needed in the plan phase (Hester and
Cunningham 2007). The product was produced in line with the plan during the create
Ask Imagine
Improve Plan
Create
Figure 1. Engineering design process (adapted from EiE (2021)).
phase, and in the improve phase, the working and non-working parts of the product were
tested and how the product could be improved were studied (Hester and Cunningham
2007).
In both groups, 8 activities were implemented. The implementation process lasted for
8 weeks. The activities were carried out in groups, synchronously and live through
distance education (Zoom platform) in groups of 4 to 5 participants. Intervention is
discussed in more detail below.
Implementation of STEM education to the control group

Before starting activities with the control group, the ‘scale of self-efficacy beliefs
towards teaching computational thinking’ was administered as a pre-test. After the
pre-test, the instructor informed the pre-service teachers about the definition of
STEM education, its historical development and features, STEM education applica
tions in Turkey and the world, its effect on the development of 21st century skills,
and the applicability of STEM to early childhood education. Then, a STEM-based
lesson (‘I’m designing a space shuttle’) was delivered by the researchers to demon
strate how to conduct a STEM-based activity. In the next lesson, the pre-service
teachers were asked to design a STEM-based activity of their own. The participants
developed their STEM activities in groups of 4 to 5 persons. Then, they sent the STEM
activities to the researchers by e-mail and received feedback.
In the control group, a total of 8 groups developed the following activities: ‘I am
making an elevator for the farm’, ‘I am helping ants find their homes’, ‘We are
preparing a shelter for Kermit the Frog and Cookie Monster’, ‘Little masters of
Mars’, ‘I am making a parachute for Luna and her friends’, ‘We are cleaning our
forest’, ‘A boat in the forest’, and ‘We’re helping our grandfather the miller’. Sample
pictures of the activities conducted in the control group are shown in Figure 3. For
example, in the Kermit the Frog and Cookie Monster activity developed by Group 3,
the goal was to design a shelter that is resistant to rain. For this purpose, the pre-
service teachers in Group 3 suggested some materials for shelter design. These
materials included plastic plates, a pack of tongue depressors, refrigerator bag,
cardboard, toothpicks, aluminum foil, chenille, ruler, stretch, scissors, fabric, needle,
rope, wooden skewers, a shoebox, glue, and tape. The designs were evaluated by
checking whether the shelter was waterproof when it rained. In this activity, the
focus was on the concept ‘weather events, seasons and balance’ within the scope of
science objectives and the concepts of ‘estimating the measurement result and
sharing the measurement result, explaining what the standard measurement tools
are’ within the scope of mathematics objectives. Regarding the engineering objectives,
the engineering design process stages set forth by the EiE were followed.
The groups implemented each activity to their classmates in about 45 minutes. During
the implementation of the activities, a different group member provided the materials for
each group each week and designed the products related to the activities. Thus, all the
members of the group were enabled to take an active role in the design. Group members
held meetings to exchange ideas while designing. After the completion of the activities,
‘the scale of self-efficacy beliefs towards teaching computational thinking’ was adminis
tered as a post-test.
Implementation of
Introducing STEM
Pre-test sample STEM
Education
activities
Implementation of Developing of STEM

Receiving feedback
developed activities activities by early
from researchers on
by early childhood childhood pre-
developed activities
pre-service teachers service teachers
Post-test
Figure 2. Implementation process and content of the training in the control group.
Figure 3. Pictures showing some activities conducted in the control group.
Implementation of computational thinking–integrated STEM education to the

experimental group
The intervention conducted in the experimental group also lasted for 8 weeks. In the
first stage, the ‘scale of self-efficacy beliefs towards teaching computational thinking’
was administered as a pre-test. In the second stage, the instructor gave information
about the definition, historical development, and features of STEM education; STEM
education applications in Turkey and the world; the effect of this approach on the
development of 21st century skills; and the applicability of STEM to early childhood
education. The instructor also covered the definition, historical development, and
features of computational thinking; CT applications in daily life; computer-based CT
applications (code.org, ScratchJr); and importance of coding in STEM education. The
definitions of the CT sub-concepts (data collection, data analysis, data representation,
problem decomposition, abstraction, algorithms and procedures, parallelization,
debugging, and pattern recognition) were explained to the pre-service teachers by

the researchers, and examples were given for each sub-concept. Next, the research
ers conducted the ‘Designing Paper Baskets’ activity. This activity was used with
permission from Tank et al. (2016). It is featured in the PictureSTEM website (http://
picturestem.org/picturestem-units/kindergarten-baskets/) as a sample activity for pre-
service teachers to experience a CT-integrated STEM-based activity. Within the scope
of this activity, the pre-service teachers were asked to design a durable paper basket
that could carry 30 dry stones and 30 wet stones at different times. In this way, the
pre-service teachers were enabled to discover the properties and durability of
different paper types. Applications related to abstraction, pattern recognition, and
debugging, which are sub-concepts of CT, were carried out. Repeated patterns are
shown using letters (abstraction). Within the scope of pattern recognition, the focus
was on determining the pattern to be used in the design of the basket, and within
the scope of debugging, participants worked on identifying and eliminating the
problems in the pattern used in the basket design. After the demonstration and
implementation of the activity, the pre-service teachers were asked to design their
own CT-integrated STEM-based activity in groups of 4 to 5 persons.
In the experimental group, a total of 8 activities were developed and implemented by
pre-service teachers. The activities developed by the pre-service teachers were as follows:
‘I’m designing water and food bowls’, ‘Water-well and rope’, ‘We’re protected from
viruses’, ‘We’re designing a catapult’, ‘We’re building a cat house’, ‘I’m discovering the
wind’, ‘We’re designing a sailboat’, and ‘We’re designing scales’. Sample pictures of the
activities conducted in the experimental group are shown in Figure 6. For example, Group
3, in their ‘We’re protected from viruses’ activity (Çiftçi and Topçu 2021), aimed to design
a fun mask suitable for children’s faces, which does not hurt the ears and which allows for
comfortable breathing. To this end, the pre-service teachers in Group 3 suggested the use
of the following materials for mask design: 2 pieces of fabric, baby elastic (60 cm), 1 mask
wire, needle, rope, scissors, and measuring tools (ruler, measuring tape, etc.). When the
activity will be carried out with pre-school children, the needle and scissors should be
used with the help of the teacher.
In this activity, importance was given to the subjects of ‘protecting, raising aware
ness and taking precautions against the Covid-19 virus’ within the context of science
objectives and to the subjects of ‘measuring, sorting out objects/entities according to
their properties, matching’ within the context of mathematics objectives. Within the
Figure 4. Pattern recognition in the activity we’re protected from viruses (Çiftçi & Topçu 2021).
scope of CT, one of the concepts focused on is algorithms and procedures. Here, the
pre-service teachers were asked to create an algorithm with arrows for the child to
reach the mask in the maze. In this activity, another sub-concept focused on is pattern
recognition. Within the scope of pattern recognition, it was requested to specify which
colour mask would be added in the boxes with question marks according to the
pattern (see Figure 4). In relation to engineering objectives, the engineering design
process stages set forth by the EiE were followed.
The groups implemented each activity in about 45 minutes. During the implementa
tion of the activities, a different group member provided the materials for each group
each week and designed the products related to the activities. Thus, all the members of
the group were enabled to take an active role in the design. The group members held
meetings to exchange ideas while designing. After the completion of the activities, ‘the
scale of self-efficacy beliefs towards teaching computational thinking’ was administered
as a post-test.
Data collection
In the current study, the ‘scale of self-efficacy beliefs towards teaching computational
thinking’ developed by Özçınar and Öztürk (2018) was used to measure pre-service
teachers’ self-efficacy beliefs towards CT teaching. The scale consists of 31 items within
4 sub-dimensions: teaching to build a problem (15 items), algorithmic thinking (6 items),
teaching assessment (3 items), and planning and teaching CT (7 items).
Introducing STEM
Pre‐test Introducing CT
Education
Developing of CT
Giving feedback by integrated STEM Implementation of
researchers on activities by early sample CT integrated
developed activities childhood pre‐service STEM activities
teachers
Implementation of
developed activities by
Post‐test
early childhood pre‐
service teachers
Figure 5. Implementation process and content of the training in the experimental group.
Figure 6. Pictures showing some activities conducted in the experimental group (Sample picture for
mask design was taken from Çiftçi & Topçu(2021)).
The items of the scale include many CT sub-concepts such as algorithms, problem
decomposition, abstraction, data collection, data analysis, data representation, and pat
tern recognition. For example, in the item ‘I can tell which parts will be important in the
solution process by breaking the problem statement into parts’, the concept of problem
decomposition is emphasized. In the item ‘I can teach you how to express the solution of
a problem using an algorithm’, there is content related to algorithms. The item ‘I can teach
you how to visualize data’ includes data representation.
Using the scale, the self-efficacy beliefs of the pre-service teachers regarding the
teaching of the CT concepts are measured. The items of the scale are related to personal
efficacy, which is a type of perceived self-efficacy in Bandura’s (1977) theory of self-
efficacy (Fernandez-Ballesteros et al. 2002). The ‘Scale of self-efficacy beliefs towards
teaching computational thinking’ was prepared in the light of Bandura’s theory and was
developed to determine self-efficacy for CT teaching. As stated by Bandura (2006), the

expression ‘can do’ was used instead of ‘will do’ in the items of this scale in accordance
with the goal of evaluating perceived self-efficacy.
In the following sample items from the scale used in the current study, self-
efficacy beliefs about CT teaching came to the forefront with the expression ‘can
do’: ‘I can teach you how to find repetitive patterns in a dataset’, ‘I can teach how
to express the solution of a problem in sequential steps’, and ‘I can design activities
that will improve students’ awareness with the abstractions they use while solving
problems’. As seen in the examples, ‘perceived self-efficacy’ in Bandura’s self-
efficacy theory was reflected in this scale items in the context of ‘CT teaching self-
efficacy’.
The language of the scale is Turkish. Some items were translated into English by the
authors and are given in Table 1 to show sample items for each sub-dimension.
Table 1. Sample items for the sub-dimensions of the scale.

Dimensions Sample items
Teaching to build a problem I can break down the problem statement into parts and explain which parts will be
important in the solution process.
Algorithmic thinking I can teach how to express the solution of a problem using an algorithm.
Teaching assessment I can teach how to test the effectiveness of different solutions to a problem.
Planning and teaching I feel competent in terms of motivating students to learn CT.
computational thinking
The scale is a 10-point scale ranging from ‘I feel completely inadequate’ to ‘I feel
completely adequate’. The Cronbach Alpha value was calculated to be .92 for the whole
scale and ranged from .94 to .97 for the sub-dimensions developed by Özçınar and Öztürk
(2018). The experimental and control groups were both administered the scale twice, as
pre-test and post-test. The data were collected via Google Forms. The completion of the
scale by the pre-service teachers took about 10 to 15 minutes.
Data analysis
The data obtained from the pre-test were first analyzed using the independent samples
t-test to compare the data of the experimental and control groups. Then, the post-test
scores of the pre-service teachers were compared with the independent samples t-test.
Afterwards, the paired samples t-test was applied separately to both groups in order to
compare the pre-test and post-test scores. In addition, the effect size value was also
calculated to show the magnitude of the difference.
In the current study, the independent samples t-test and paired samples t-test were
applied because the data regarding the pre-test, post-test and difference scores from the
experimental and control groups showed normal distribution. The paired samples t-test,
the independent samples t-test, and calculation of the effect size were carried out in
Jamovi 1.6.15 program (R Core Team 2020; The Jamovi Project 2021). The results were
interpreted by considering the p value of .05.
Results
In the current study, the effect of CT-integrated STEM education on early childhood pre-
service teachers’ CT teaching self-efficacy beliefs was examined. In this connection, firstly,
findings obtained from the comparison of the pre-test scores of the groups are presented.
Then, the findings are presented for each research question.
Pre-test scores of experimental and control groups

Findings obtained from the comparison of the pre-test scores taken from the whole scale
and its sub-dimensions regarding the CT teaching self-efficacy beliefs of the experimental
and control groups are presented in Table 2.
Table 2. The independent samples t-test results regarding the pre-test scores of the experimental and
control groups.
Pre-test scores
Dimensions Group N x SD t p
Algorithmic thinking Experimental 35 21.8 12.57 −1.166 .248
Control 33 25.5 14.10
Planning and teaching computational thinking Experimental 35 30.6 17.80 −.771 .443
Control 33 33.7 15.81
Teaching to build a problem Experimental 35 91.3 35.01 −.599 .552
Control 33 96.0 29.99
Teaching assessment Experimental 35 17.8 7.19 −.398 .692
Control 33 18.5 6.32
Total score Experimental 35 161.4 62.33 −.846 .401
Control 33 173.8 57.55
p < .05
As seen in Table 2, there is no significant difference between the total pre-test scores
(t (66) = −.846, p > .05) and pre-test scores for the sub-dimensions (Algorithmic Thinking
(t (66) = −1.166, p > .05), Planning and Teaching Computational Thinking ((t (66) = −.771,
p > .05), Teaching to Build a Problem (t (66) = −.599, p > .05), Teaching Assessment
(t (66) = −.398, p > .05)). Both groups were similar in terms of ‘CT teaching self-efficacy
beliefs’ before the intervention.
Findings for the first research question

Is there a statistically significant difference between the effects of computational thinking–
integrated STEM education and STEM education alone on early childhood pre-service tea
chers’ computational thinking teaching self-efficacy beliefs?
The results of the independent samples t-test for the comparison of the post-test
scores of the experimental and control groups regarding CT teaching self-efficacy beliefs
are given in Table 3.
Table 3. The independent samples t-test results regarding the post-test scores of the experimental
and control groups.
Post-test scores
Dimensions Group N x SD t p
Algorithmic thinking Experimental 35 42.3 11.82 4.395 .000
Control 33 28.0 14.93
Planning and teaching computational thinking Experimental 35 53.9 12.11 4.164 .000
Control 33 39.0 17.03
Teaching to build a problem Experimental 35 125.1 23.46 3.475 .001
Control 33 101.4 32.24
Teaching assessment Experimental 35 25.8 4.30 3.326 .001
Control 33 21.2 6.86
Total score Experimental 35 247.0 48.08 4.207 .000
Control 33 189.6 63.80
p < .05
As can be seen in Table 3, when the total post-test scores (t (66) = 4.207, p < .05) and
post-test scores taken from the sub-dimensions (Algorithmic Thinking (t (66) = 4.395,
p < .05), Planning and Teaching Computational Thinking (t (66) = 4.164, p < .05), Teaching
to Build a Problem (t (66) = 3.475, p < .05), Teaching Assessment (t (66) = 3.326, p < .05))
were compared, statistically significant differences were seen in favor of the experimental
group.
Findings for the second research question

Is there a significant effect of computational thinking integrated STEM education on early
childhood pre-service teachers’ computational thinking teaching self-efficacy beliefs?
Paired samples t-test results regarding the comparison of pre – and post-test scores of
the experimental group are given in Table 4.
Table 4. The paired samples t-test results regarding the pre-test and post-test scores of the experi
mental group.
Dimensions X N df t p
Algorithmic thinking Pre-test 21.8 35 34
Post-test 42.3 35 34 −7.27 .000
Planning and teaching computational thinking Pre-test 30.6 35 34
Post-test 53.9 35 34 −6.84 .000
Teaching to build a problem Pre-test 91.3 35 34
Post-test 125.1 35 34 −5.52 .000
Teaching assessment Pre-test 17.8 35 34 −6.96 .000
Post-test 25.8 35 34
Total score Pre-test 161.4 35 34
Post-test 247.0 35 34 −7.24 .000
p < .05
As seen in Table 4, the experimental group students’ pre-test and post-test scores
taken from the whole scale (t (34) = −7.24, p < .05) and the sub-dimensions (Algorithmic
Thinking (t (34) = −7.27, p < .05), Planning and Teaching Computational Thinking
((t (34) = −6.84, p < .05)), Teaching to Build a Problem ((t (34) = −5.52, p < .05)),
Teaching Assessment ((t (34) = −6.96, p < .05)) showed significant improvement in favor
of the post-test. Effect size of this difference was calculated and Cohen’s d was found to be
1.22. When the values given by Cohen (1988) (.01 = small effect, .06 = moderate effect,
.14 = large effect) are considered, this effect size is said to be large. This shows that CT
integrated STEM education is highly influential on the development of pre-service tea
chers’ CT teaching self-efficacy beliefs.
Findings for the third research question

Is there a significant effect of STEM education on early childhood pre-service teachers’
computational thinking teaching self-efficacy beliefs?
Paired samples t-test results regarding the comparison of the pre-test and post-test
scores of the control group are given in Table 5.
Table 5. The paired samples t-test results regarding pre-test and post-test scores of the control group.
Dimensions X N df t p
Algorithmic thinking Pre-test 25.5 33 32
Post-test 28.0 33 32 −1.03 .311
Planning and teaching computational thinking Pre-test 33.7 33 32
Post-test 39.0 33 32 −2.01 .053
Teaching to build a problem Pre-test 96.0 33 32
Post-test 101.4 33 32 −1.25 .222
Teaching assessment Pre-test 18.5 33 32 −2.25 .031
Post-test 21.2 33 32
Total score Pre-test 173.8 33 32
Post-test 189.6 33 32 −1.90 .066
p < .05
As seen in Table 5, a difference was found between the pre-test and post-test scores of
the control group in favor of the post-test. However, this difference is not statistically
significant (t (32) = −1.90, p > .05). Moreover, no significant difference was found for the
sub-dimensions of ‘Algorithmic Thinking’ (t (32) = −1.03, p > .05), ‘Planning and Teaching
Computational Thinking’ (t (32) = −2.01, p > .05), ‘Teaching to Build a Problem’
(t (32) = −1.25, p > .05); yet a significant difference was found for the sub-dimension of
‘Teaching Assessment’ in favor of the post-test (t (32) = −2.25, p < .05).
Discussion and conclusion

The current study investigated the effectiveness of STEM education and CT-integrated
STEM education in developing early childhood pre-service teachers’ CT teaching self-
efficacy beliefs. In this context, we concluded that the self-efficacy beliefs of pre-service
teachers who received CT-integrated STEM education developed more than those of the
pre-service teachers who received STEM education alone. These results show that it would
be more effective to use CT-integrated STEM in teacher training programs – instead of
using STEM alone – to improve pre-service teachers’ CT teaching self-efficacy.
The results of this study align with Kaya et al. (2019), Kaya et al. (2020), and Mason and
Rich (2019), who indicated that CT applications are effective in developing pre-service
and/or in-service elementary teachers’ CT teaching self-efficacy beliefs. This study expands
the literature in two ways: through the use of CT-integrated STEM education and by the
inclusion of early childhood pre-service teachers. In the current study, significant devel
opment occurred in the pre-service teachers’ CT teaching self-efficacy beliefs, which may
positively affect their CT teaching motivation and performance in their future early
childhood education classes (Bandura 1993; Kaya et al. 2020; Zhao et al. 2020). In future
studies, CT-integrated STEM implementations could be conducted with in-service early
childhood teachers who feel inadequate to implement CT integrated STEM education in
their classrooms and who have not received any CT-integrated STEM-based education in
their undergraduate education.
The significant improvement between the pre-test and post-test scores of the
experimental group students is notable; the effect size of this difference was calcu
lated to be large. These results show that CT-integrated STEM education is not just
effective, but highly effective in developing early childhood pre-service teachers’ CT
teaching self-efficacy beliefs. The pre-service teachers may have developed self-
efficacy beliefs due to their experiences teaching CT-integrated STEM (CT-
integrated STEM-based activity plan development and applications). Therefore, early
childhood pre-service teachers/teachers should be encouraged to plan and imple
ment activities based on the integration of CT and STEM for the development of
their CT teaching self-efficacy beliefs. At this point we can suggest that much more
emphasis could be put on the integration of STEM and CT in early childhood teacher
education programs.
When comparing experimental group students’ pre-test and post-test scores in terms
of each sub-dimension (algorithmic thinking, planning and teaching computational
thinking, teaching to build a problem, teaching assessment), the results showed
a significant difference in favor of the post-test. Each of the sub-dimensions had
a specific tie to the training the participants received.
The development of the pre-service teachers’ self-efficacy beliefs toward algorithmic
thinking can be explained by the fact that they included implementations related to
algorithmic thinking in their activities (e.g. the creation of an algorithm using arrows to
enable a child to reach the mask at the end of the maze in the activity titled ‘We’re
protected from viruses’). Thus, early childhood pre-service teachers should be engaged in
activities for algorithm design and implementation in order to aid in the development of
their self-efficacy beliefs towards algorithmic thinking. Thus, we can infer that early
childhood pre-service teachers trained in CT will be able to better teach children CT
concepts such as algorithmic thinking when they become teachers.
The development of the experimental group students’ self-efficacy beliefs in the sub-
dimension of ‘Planning and teaching computational thinking’ can also be attributed to
the development and implementation of activity plans for CT-integrated STEM. Unlike the
pre-service teachers in the control group, the pre-service teachers in the experimental
group learned what CT and its sub-concepts are, observed the sample CT activities carried
out by the researchers, planned the CT-integrated STEM-based activities, and applied the
activities they developed to their peers in the classroom. This process might have con
tributed to the development of their self-efficacy beliefs in the sub-dimension of ‘Planning
and teaching computational thinking’. Based on these findings, we suggest that pre-
service teachers should have knowledge about the concept of CT and its components and
experience sample activities involving CT in order to increase their self-efficacy about CT
planning and teaching.
The development of pre-service teachers’ self-efficacy beliefs towards ‘teaching assess

ment’ can be attributed to their experience gained regarding this dimension. For exam
ple, in the ‘water and food bowl design’ activity (Çiftçi and Topçu 2021), the pre-service
teachers in Group 1 asked their peers questions and observed them in order to evaluate
whether they learned pattern recognition, problem decomposition, and debugging. In
another activity, ‘water-well and rope’, the pre-service teachers in Group 2 evaluated their
classmates’ learning level of the sub-concepts of abstraction, algorithm and procedures,
data collection, data analysis, and data representation. As the pre-service teachers in the
experimental group checked whether their classmates learned certain CT concepts during
the implementation of activities, their self-efficacy beliefs in the sub-dimension of ‘teach
ing assessment’ might have developed. At this point, it can be suggested that pre-service
teachers should be subjected to activities allowing them to evaluate the extent to which
the concept of CT and its components have been learned.
The development of self-efficacy beliefs towards ‘teaching to build a problem’ can
be explained by the fact that STEM education and CT both include applications for real
world problems (Stohlmann, Moore, and Roehrig 2012; Weintrop et al. 2016; Yadav
et al. 2017a). In the current study, the researchers pointed out to the pre-service
teachers that it is important to identify problems in CT-integrated STEM activities.
Problem identification was also included in the first step (ask) of the EiE engineering
design process cycle, which was drawn on during the activity implementation process.
In addition, the pre-service teachers understood how to pose and formulate a problem
based on an example activity shown by the researchers. They also experienced the
process of problem construction by developing CT-integrated STEM activities and
applying them to their classmates.
The pre-service teachers in the experimental group were engaged in the problem-
posing process in the context of CT-integrated STEM, instead of only creating a problem in
the context of STEM as the control group students. This might have increased their
engagement and ability to design a problem that covers more fields (CT and STEM) and
these fields to be integrated. This integration succeeds because both CT and STEM
education involve the active participation of individuals in problem identification and
solving in the context of daily life (Psycharis and Kotzampasaki 2019; Sen, Ay, and Kiray
2021; Yadav et al., 2017a). The joint learning of two separate but related fields may have
further improved their self-efficacy beliefs in the ‘teaching to build a problem’ sub-
dimension. In future studies, activities related to CT-integrated STEM-based problem crea
tion can be applied to pre-service teachers/teachers in order to develop their self-efficacy
beliefs towards ‘teaching to build a problem’, as CT and STEM each focus on this topic in
slightly different ways.
In the current study, the pre-service teachers in the control group showed improve
ment regarding their CT teaching self-efficacy beliefs after intervention, but this difference
was not statistically significant. This result shows that STEM education alone is not enough
to significantly improve CT teaching self-efficacy beliefs. Thus, the context of STEM alone
may not be sufficient to develop pre-service teachers or teachers’ CT-teaching self-efficacy
beliefs. For this reason, the results indicate that CT should be integrated into the STEM
context in teacher training programs.
The results obtained in the current study may be useful in providing implications
for curriculum developers and policy makers concerned with the training of early
childhood educators. For example, policy makers and curriculum developers can
update early childhood education program objectives, content, and practices in the
context of CT-integrated STEM education. By adding CT, which is an important 21st
century skill, to teacher education curriculums, teachers’ self-efficacy about CT
teaching can be developed, and therefore their students’ CT skills can be improved
as well.
In summary, the results of this study present a strong example of the education and
improved self-efficacy of early childhood pre-service teachers in the context of CT-
integrated STEM. Results obtained in the current study can be useful for early childhood
educators, pre-service teachers and researchers by providing example CT-integrated
STEM teaching content and processes. In this context, CT-integrated STEM-based activ
ities mentioned in the ‘Implementation’ section (for example, we’re protected from
viruses and designing paper baskets) can be useful as sample activities for in-service
teachers/practitioners and pre-service teachers. Furthermore, future studies could use
a similar format to investigate CT pedagogical content knowledge growth or teaching
outcomes using a similar format.
The current study provides researchers with important empirical data on the effective
ness of the integration of CT and STEM on CT teaching self-efficacy beliefs on the basis of
a quasi-experimental design. One limitation is the lack of qualitative data. In future
studies, the reflections of CT-integrated STEM on early childhood pre-service teachers,
teachers and children can be examined with qualitative and quantitative studies to
provide more insight into teacher experiences and best practices for integrated CT
education in teacher training programs.
Acknowledgments
A part of this research is going to bepresented at National Association of Research in Science
Teaching (NARST) Conference, Vancouver, BC, Canada, 2022, March, 27-30.
Disclosure statement
No potential conflict of interest was reported by the author(s).
ORCID
Ayşe Çiftçi http://orcid.org/0000-0001-9005-4333
Mustafa Sami Topçu http://orcid.org/0000-0001-5068-8796
References
Bandura, A. 1977. “Self-efficacy: Toward A Unifying Theory of Behavioral Change.” Psychological
Review 84 (2): 191–215. doi:10.1037/0033-295X.84.2.191.
Bandura, A. 1990. “Perceived Self-efficacy in the Exercise of Control over AIDS Infection.” Evaluation
and Program Planning 13 (1): 9–17. doi:10.1016/0149-7189(90)90004-G.
Bandura, A. 1993. “Perceived Self-efficacy in Cognitive Development and Functioning.” Educational

Psychologist 28 (2): 117–148. doi:10.1207/s15326985ep2802_3.
Bandura, A. 1995. Self-efficacy in Changing Societies. New York: Cambridge University Press.
Bandura, A. 1997. Self-efficacy: The Exercise of Control. New York: W. H. Freeman & .
Bandura, A. 2001. “Social Cognitive Theory: An Agentic Perspective.” Annual Review of Psychology
52 (1): 1–26. doi:10.1146/annurev.psych.52.1.1.
Bandura, A. 2006. “Guide for Constructing Self-Efficacy Scales.” Self-efficacy Beliefs of Adolescents
5 (1): 307–337.
Barr, D., J. Harrison, and L. Conery. 2011. “Computational Thinking: A Digital Age Skill for Everyone.”
Learning & Leading with Technology 38 (6): 20–23.
Barr, V., and C. Stephenson. 2011. “Bringing Computational Thinking to K-12: What Is Involved and
What Is the Role of the Computer Science Education Community?” ACM Inroads 2 (1): 48–54.
doi:10.1145/1929887.1929905.
Berman, P., M. McLaughlin, G. Bass, E. Pauly, and G. Zellman. 1977. Federal Programs Supporting
Educational Change: Vol. VII. Factors Affecting Implementation and Continuation (Rep. No. R-1589/
7-HEW). Santa Monica, CA: RAND.
Bers, M., and A. Sullivan. 2019. “Computer Science Education in Early Childhood: The Case of
ScratchJr.” Journal of Information Technology Education: Innovations in Practice 18 (1):
113–138.
Çiftçi, A., and M.S. Topçu. 2021. “Erken Çocukluk Döneminde STEM Eğitimi ve Bilgi İşlemsel Düşünme
[STEM Education and Computational Thinking in Early Childhood.” In Erken Çocukluk Eğitimi
Kuramdan Uygulamaya [Early Childhood Education: Theory to Practice], edited by F. Yaşar-Ekici
and K. A. Kırkıç, 219–250. Nobel Tıp Kitabevleri: İstanbul.
Çiftçi, A., M.S. Topçu, and J. A. Foulk. 2020. “Pre-service Early Childhood Teachers’ Views on STEM
Education and Their STEM Teaching Practices.” Research in Science & Technological Education
1–27. doi:10.1080/02635143.2020.1784125.
Cohen, J. 1988. Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ: Erlbaum.
Czerkawski, B. C., and E. W. Lyman. 2015. “Exploring Issues about Computational Thinking in Higher
Education.” TechTrends 59 (2): 57–65. doi:10.1007/s11528-015-0840-3.
Dede, C., P. Mishra, and J. Voogt. 2013. “Working Group 6: Advancing Computational Thinking in
21st Century Learning.” 1–6. Accessed 21 April 2021. https://ris.utwente.nl/ws/files/6168377/
Advancing_computational_thinking_in_21st_century_learning.pdf
Ehsan, H., A. P. Rehmat, and M. E. Cardella. 2021. “Computational Thinking Embedded in
Engineering Design: Capturing Computational Thinking of Children in an Informal Engineering
Design Activity.” International Journal of Technology and Design Education 31: 441–464.
Engineering is Elementary [EiE]. 2021. Accessed 10 May 2021. Retrieved from https://www.eie.org/
why-eie#:~:text=The%20Engineering%20Design%20Process%20is,just%20engineering%20and
%20computer%20science!&text=As%20children%20try%20different%20solutions,communica
tion%2C%20and%20critical%20thinking%20skills .
Fernandez-Ballesteros, R., J. Diez-Nicolas, G. V. Caprara, C. Barbaranelli, and A. Bandura. 2002.
“Determinants and Structural Relation of Personal Efficacy to Collective Efficacy.” Applied
Psychology (Print) 51 (1): 107–125. doi:10.1111/1464-0597.00081.
Fraenkel, J. R., N. E. Wallen, and H. H. Hyun. 2012. How to Design and Evaluate Research in Education.
New York, NY: McGraw-Hill.
Gautam, A., W. Bortz, and D. Tatar. 2020. “Abstraction through Multiple Representations in an
Integrated Computational Thinking Environment”. In Proceedings of the 51st ACM Technical
Symposium on Computer Science Education, 11-14 March 2020, Portland, OR, USA. (pp. 393–399).
Gonzalez, H. B., and J.J. Kuenzi. 2012. “Science, Technology, Engineering, and Mathematics (STEM)
Education: A Primer”. Washington, DC: Congressional Research Service, Library of Congress.
Goode, J., R. Estrella, and J. Margolis. 2006. “Lost in Translation: Gender and High School Computer
Science.” In Women and Information Technology: Research on Underrepresentation, edited by
J. M. Cohoon and W. Aspray, 89–114. Cambridge, MA: MIT Press.
Grover, S., and R. Pea. 2018. “Computational Thinking: A Competency Whose Time Has Come.”
Computer Science Education: Perspectives on Teaching and Learning in School 19: 1257–1258.
Gulistan, M., and A. Hussain. 2017. “Relationship between Mathematics Teachers’ Self Efficacy and
Students’ Academic Achievement at Secondary Level.” Bulletin of Education and Research 39 (3):
171–182.
Hacker, M. 2018. “Integrating Computational Thinking into Technology and Engineering Education.”
Technology and Engineering Teacher 77 (4): 8–14.
Hester, K., and C. Cunningham. 2007. “Engineering Is Elementary: An Engineering and Technology
Curriculum for Children.” In 2007 Annual Conference & Exposition 24-27 June 2007, Honolulu,
Hawaii. (pp. 12–639).
Hoić-Božić, N., J. Mezak, and K. Tomljenović. 2019. “Enhancing Teachers’ Computational Thinking
Skills through Game Based Learning.” Paper presented at the Proceedings of SLET-2019 –
International Scientic Conference Innovative Approaches to the Application of Digital
Technologies in Education and Research, Stavropol – Dombay, Russia, May 20-23.
ISTE. 2011. “Computational Thinking in K–12 Education Leadership Toolkit.“ Accessed 2 May 2021.
Retrieved from https://www.iste.org/explore/computational-thinking/computational-thinking-all
The Jamovi Project. 2021. “Jamovi. (Version 1.6) [Computer Software].“ Accessed 13 April 2021.
Retrieved from https://www.jamovi.org .
Jona, K., U. Wilensky, L. Trouille, M. S. Horn, K. Orton, D. Weintrop, and E. Beheshti. 2014. “Embedding
Computational Thinking in Science, Technology, Engineering, and Math (CT-STEM).” In Future
Directions in Computer Science Education Summit Meeting, Orlando, FL.
Kafai, Y. B., D. DeLiema, D. A. Fields, G. Lewandowski, and C. Lewis. 2019. “Rethinking Debugging as
Productive Failure for CS Education.” In Proceedings of the 50th ACM Technical Symposium on
Computer Science Education, 27 February-2 March 2019, Minneapolis, Minnesota, USA. (pp.
169–170).
Kaya, E., A. Newley, E. Yesilyurt, and H. Deniz. 2020. “Measuring Computational Thinking Teaching
Efficacy Beliefs of Preservice Elementary Teachers.” Journal of College Science Teaching 49 (6):
55–64.
Kaya, E., E. Yesilyurt, A. Newley, and H. Deniz. 2019. “Examining the Impact of a Computational
Thinking Intervention on Pre-service Elementary Science Teachers’ Computational Thinking
Teaching Efficacy Beliefs, Interest and Confidence.” Journal of Computers in Mathematics and
Science Teaching 38 (4): 385–392.
Kraut, R., T. Chandler, and K. Hertenstein. 2016. “The Interplay of Teacher Training, Access to
Resources, Years of Experience and Professional Development in Tertiary ESL Reading Teachers’
Perceived Self-efficacy.” Gist: Education and Learning Research Journal 12 (12): 132–151.
doi:10.26817/16925777.248.
Lavigne, H. J., A. Lewis-Presser, and D. Rosenfeld. 2020. “An Exploratory Approach for Investigating
the Integration of Computational Thinking and Mathematics for Preschool Children.” Journal of
Digital Learning in Teacher Education 36 (1): 63–77. doi:10.1080/21532974.2019.1693940.
Mason, S. L., and P. J. Rich. 2019. “Preparing Elementary School Teachers to Teach Computing,
Coding, and Computational Thinking.” Contemporary Issues in Technology and Teacher Education
19 (4): 790–824.
Mazlum, F., F. Cheraghi, and M. Dasta. 2015. “English Teachers’ Self-Efficacy Beliefs and Students
Learning Approaches.” International Journal of Educational Psychology: IJEP 4 (3): 305–328.
doi:10.17583/ijep.2015.1137.
Özçınar, H., and E. Öztürk. 2018. “The Scale of Self-Efficacy Perception Towards Teaching
Computational Thinking: A Validity and Reliability Study.” Pamukkale University Journal of Social
Sciences Institute 30 (30): 173–195. doi:10.5505/pausbed.2018.82574.
Pastorelli, C., G.V. Caprara, C. Barbaranelli, J. Rola, S. Rozsa, and A. Bandura. 2001. “The Structure of
Children’s Perceived Self-Efficacy: A Cross-National Study.” European Journal of Psychological
Assessment 17 (2): 87–97. doi:10.1027//1015-5759.17.2.87.
Psycharis, S., and E. Kotzampasaki. 2019. “The Impact of a STEM Inquiry Game Learning Scenario on
Computational Thinking and Computer Self-Confidence.” Eurasia Journal of Mathematics, Science
and Technology Education 15 (4): em1689. doi:10.29333/ejmste/103071.
Psycharis, S., D. Mastorodimos, K. Kalovrektis, P. Papazoglou, L. Stergioulas, and M. Abbasi. 2018.

“Algorithm Visualization and Its Impact on Self-Efficacy, Metacognition and Computational
Thinking Concepts Using the Computational Pedagogy Model in STEM Content Epistemology.”
International Journal of Physics & Chemistry Education 10 (4): 71–84.
R Core Team. 2020. “R: A Language and Environment for Statistical Computing. (Version 4.0)
[Computer Software].” Accessed 13 April 2021. https://cran.r-project.org .
Rowe, E., J. Asbell-Clarke, S. Gasca, and K. Cunningham. 2017. “Assessing Implicit Computational
Thinking in Zoombinis Gameplay.” In Proceedings of the 12th International Conference on the
Foundations of Digital Games. 14-17 August 2017, Hyannis, MA, USA. (pp. 1–4).
Sen, C., Z. S. Ay, and S. A. Kiray. 2021. “Computational Thinking Skills of Gifted and Talented Students
in Integrated STEM Activities Based on the Engineering Design Process: The Case of Robotics and
3D Robot Modeling.” Thinking Skills and Creativity 42: 100931. doi:10.1016/j.tsc.2021.100931.
Shahzad, K., and S. Naureen. 2017. “Impact of Teacher Self-Efficacy on Secondary School Students’
Academic Achievement.” Journal of Education and Educational Development 4 (1): 48–72.
doi:10.22555/joeed.v4i1.1050.
Stohlmann, M., T. J. Moore, and G. H. Roehrig. 2012. “Considerations for Teaching Integrated STEM
Education.” Journal of Pre-College Engineering Education Research (J-PEER) 2 (1): 28–34.
doi:10.5703/1288284314653.
Sun, L., L. Hu, W. Yang, D. Zhou, and X. Wang. 2020. “STEM Learning Attitude Predicts Computational
Thinking Skills among Primary School Students.” Journal of Computer Assisted Learning 37 (2):
346–358. doi:10.1111/jcal.12493.
Swaid, S. I. 2015. “Bringing Computational Thinking to STEM Education.” Procedia Manufacturing 3:
3657–3662. doi:10.1016/j.promfg.2015.07.761.
Tank, K. M., T.J. Moore, E. Gajdzik, M.T. Sanger, A.M. Rynearson, B.L. Dorie, and E. Mann. 2016.
“Designing Paper Baskets: A Picturestem Curriculum Module (P12 Resource Exchange)”. Paper
presented at the 2016 ASEE Annual Conference & Exposition, New Orleans, June 26- 29.
Tippett, C., and T. Milford. 2018. “Making the Case for STEM in Early Childhood Education”. Accessed
19 August 2021. Retrieved from https://dspace.library.uvic.ca/handle/1828/13131
Toran, M., E. Aydin, and D. Etgüer. 2020. “Investigating the Effects of STEM Enriched
Implementations on School Readiness and Concept Acquisition of Children.” Ilkogretim Online
19 (1): 299–309. doi:10.17051/ilkonline.2020.656873.
Üret, A., and R. Ceylan. 2021. “Exploring the Effectiveness of STEM Education on the Creativity of
5-Year-Old Kindergarten Children.” European Early Childhood Education Research Journal 29 (6):
842–855.
Vasconcelos, L., I. Arslan-Ari, and F. Ari. 2020. “Early Childhood Preservice Teachers’ Debugging
Block-Based Programs: An Eye Tracking Study.” Journal of Childhood, Education & Society 1 (1):
63–77. doi:10.37291/2717638X.20201132.
Watson, A. D., and G.H. Watson. 2013. “Transitioning STEM to STEAM: Reformation of Engineering
Education.” Journal for Quality and Participation 36 (3): 1–5.
Weintrop, D., E. Beheshti, M. Horn, K. Orton, K. Jona, K. L. Trouille, and U. Wilensky. 2016. “Defining
Computational Thinking for Mathematics and Science Classrooms.” Journal of Science Education
and Technology 25 (1): 127–147. doi:10.1007/s10956-015-9581-5.
Weintrop, D., E. Beheshti, M.S. Horn, K. Orton, K. L. Trouille, K. Jona, and U. Wilensky. 2014.
“Interactive Assessment Tools for Computational Thinking in High School STEM Classrooms.” In
International Conference on Intelligent Technologies for Interactive Entertainment (pp. 22–25).
Cham: Springer.
Wing, J. 2006. “Computational Thinking.” Communications of the ACM 49 (3): 33–36. doi:10.1145/
1118178.1118215.
Wing, J. M. 2008. “Computational Thinking and Thinking about Computing.” Philosophical
Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences
366 (1881): 3717–3725. doi:10.1098/rsta.2008.0118.
Wing, J. 2011. “Research Notebook: Computational Thinking—What and Why? the Link Magazine,
Spring.“ Accessed 23 April 2021.
Yadav, A., J. Good, J. Voogt, and P. Fisser. 2017a. “Computational Thinking as an Emerging
Competence Domain.” ed. Mulder, M. In Competence-Based Vocational and Professional
Education, 1051–1067. Cham: Springer.
Yadav, A., S. Gretter, J. Good, and T. McLean. 2017b. “Computational Thinking in Teacher Education.”
Rich , P. J., and Hodges, C. B. eds. In Emerging Research, Practice, And Policy On Computational
Thinking, 205–220. Cham: Springer.
Yadav, A., H. Hong, and C. Stephenson. 2016. “Computational Thinking for All: Pedagogical
Approaches to Embedding 21st Century Problem Solving in K-12 Classrooms.” TechTrends
60 (6): 565–568. doi:10.1007/s11528-016-0087-7.
Yin, Y., R. Hadad, X. Tang, and Q. Lin. 2019. “Improving and Assessing Computational Thinking in
Maker Activities: The Integration with Physics and Engineering Learning.” Journal of Science
Education and Technology 29 (2): 189–214.
Zhang, N., and G. Biswas. 2019. “Defining and Assessing Students’ Computational Thinking in
a Learning by Modeling Environment.” Kong , Siu-Cheung, and Abelson, Harold eds. In
Computational Thinking Education, 203–221. Singapore: Springer.
Zhao, M., M. Zhao, X. H. Wang, and H.L. Ma. 2020. “Promoting Teaching Self-efficacy in
Computational Thinking of School Teachers”. In 2020 Ninth International Conference of
Educational Innovation through Technology (EITT). 13-17 December 2020, Porto, Portugal. (pp.
235–239). IEEE.
Appendix 1. Template for STEM activity conducted with the control group
Unit/Theme:
Topic:
Concepts:
Content of the Activity:
Duration:
Age:
Learning Outcomes:
(1) Science learning outcomes:
(2) Mathematics learning outcomes:
(3) Engineering learning outcomes:
(4) Technology learning outcomes:
Tools and Equipment:
Safety Measures:
Warm-up Activities:
Scenario-Story:
Your Task:
Before designing your model, you must answer the following questions.
In the scenario:
(1) Who (for which target audience) is the design/model being prepared for?
(2) What is the problem? Can you define it?
(3) What are the criteria to be successful?
(4) What are the limitations?
Draw your design (model) below.
Evaluation Criteria/s:
Evaluation Questions:
Possible Designs:
References:
Appendix 2. Template for CT-integrated STEM activity conducted with the

experimental group
Unit/Theme:
Topic:
Concepts:
CT concepts:
Content of the Activity:
Duration:
Age:
Learning Outcomes:
(1) Science learning outcomes:
(2) Mathematics learning outcomes:
(3) Computational thinking learning outcomes:
(4) Engineering learning outcomes:
(5) Technology learning outcomes:
Tools and Equipment:
Safety Measures:
Warm-up Activities:
Scenario-Story:
Your Task:
Before designing your model, you must answer the following questions.
In the scenario:
(1) Who (for which target audience) is the design/model being prepared for?
(2) What is the problem? Can you define it?
(3) What are the criteria to be successful?
(4) What are the limitations?
Draw your design (model) below.
Evaluation Criteria/s:
Evaluation Questions:
Possible Designs:
References:
Liao et al. Int J Educ Technol High Educ (2022) 19:43
https://doi.org/10.1186/s41239-022-00347-5
RESEARCH ARTICLE Open Access
Exploring the relationship
between computational thinking and learning
satisfaction for non‑STEM college students
Chien Hsiang Liao1, Chang‑Tang Chiang2* , I‑Chuan Chen2 and Kevin R. Parker3
*Correspondence:
arthurctchiang@gmail.com Abstract
2
Department of Information While various studies have focused on the significance of computational thinking (CT)
Management, Chinese
Culture University, for the future career paths of individuals in science, technology, engineering, and math‑
55, Hwa‑Kang Rd., ematics (STEM), few studies have focused on computational thinking for non-STEM
Yang‑Ming‑Shan, college students. This study explores the relationship between computational thinking
Taipei 11114, Taiwan
Full list of author information and learning satisfaction for non-STEM-major college students. A conceptual model is
is available at the end of the proposed to examine the structural relationships among computational thinking, self-
article efficacy, self-exploration, enjoyment and learning satisfaction in an AppInventor-based
liberal education course. Collecting data from 190 undergraduate students from Tai‑
wan and analyzing the data by using partial least squares (PLS) methods, the research
framework confirms the six proposed hypotheses. These results show that both
computational thinking and enjoyment play significant roles in both self-exploration
and digital self-efficacy. Moreover, digital self-efficacy and self-exploration also have a
significant positive influence on learning satisfaction. These findings have implications
for influencing the learning outcomes of non-STEM-major college students, computa‑
tional thinking course instructors, and computational thinking relevant policies.
Keywords: Computational thinking, Digital self-efficacy, Self-exploration, Enjoyment,
Learning satisfaction, Non-STEM college students
Highlights
• This study develops a model for computational thinking and learning satisfaction.
• Computational thinking positively influences on digital self-efficacy and self-explo-
ration.
• Enjoyment positively influences on digital self-efficacy and self-exploration.
• Digital self-efficacy and self-exploration positively influence on learning satisfaction.
© The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits
use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original
author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third
party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the mate‑
rial. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or
exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://
creativecommons.org/licenses/by/4.0/.
Liao et al. Int J Educ Technol High Educ (2022) 19:43 Page 2 of 21
Introduction
Computational thinking is regarded as a thinking process that enables the understand-
ing of problems and the formulation of creative solutions to these problems through the
iteration of abstraction and algorithmic thinking (Chen et al., 2017a, 2017b; Romero
et al., 2017). Computational thinking has been defined as “solving problems, designing
systems, and understanding human behavior, by drawing on the concepts fundamental
to computer science” (Wing, 2006, p. 33). Computational thinking, an essential part of
science, technology, engineering, and mathematics (STEM) education, addresses the
essential concepts of STEM and involves the integration of multiple disciplines and
cross-domain knowledge during the problem-solving process (Lu et al., 2021). That is, by
drawing on principles and practices central to computer science, computational think-
ing is a capacity and skill set that individuals should possess and acquire at a basic level
to solve ambiguous, complex and open-ended problems for the future world in various
contexts (ISTE, 2022). It is therefore considered one of the essential twenty-first century
skill that not only STEM workers but also everyone from different disciplines should
learn (Güven & Gulbahar, 2020; Tekdal, 2021).
Computational thinking has been a trending topic in recent decades in learning
research and educational practice. Scholars argue that children as young as 4 to 6 years
old can build and program robots and learn computational thinking skills (Bers et al.,
2014). Previous studies have also suggested that learning computer science and STEM
in early childhood assists students in developing certain competences and thinking
archetypes, and the early experiences of children are likely to allow them to substan-
tially develop positive attitudes toward perseverance in future career development
(Chen et al., 2017a, 2017b; Israel et al., 2015). However, the majority of these studies
have focused on the STEM field with no mention of computational thinking. Accord-
ing to a computational thinking review article from Hsu and her colleagues, the existing
computational thinking studies are largely aimed at biology, program coding, computer
science, and robot design classes in terms of the subject and at K12 students in terms
of age (Hsu et al., 2018; Tikva & Tambouris, 2021). Computational thinking should not
be a privilege of STEM-related majors, but rather it is imperative to learn for students
in other domains to learn to solve problems relevant to all disciplines (Czerkawski &
Lyman, 2015; Wing, 2008). How to foster the learning performance of computational
thinking for non-STEM college students remains unknown.
A very recent bibliometric analysis of computational thinking education research
identified and selected 321 articles published from 2008 to 2020 in 36 journals in which
the authors included the term computational thinking to investigate (Tekdal, 2021). The
review indicates that the current computational thinking studies are largely conducted
with students from elementary education to high school and focuses on the integration
of computational thinking into STEM education. The author also recommended that
future studies be expanded to cover higher education.
In a similar vein, Czerkawski and Lyman (2015) called for applying computational
thinking methods to address some of the most challenging problems facing society. The
purpose of social studies is to promote civic competence to confront large and com-
plex problems and to provide information about diverse disciplines such as geography,
history, law, and civics (Güven & Gulbahar, 2020). College students studying social
studies can use the core concepts of computational thinking—abstraction and algorith-
mic thinking—to reason about and solve complex problems, design systems, and under-
stand human behaviors (Kules, 2016). Other than these few conceptual studies that tout
the application of computational thinking to social studies, there is limited empirical
research relating this topic to college students in non-STEM disciplines. The gap in this
area explicitly indicates that there is a need for critical research on the learning effective-
ness of computational thinking in non-STEM disciplines among college students.
The study, consequently, addresses this gap in the educational literature and aims to
examine how the learning effectiveness of computational thinking can be enhanced.
Specifically, the purpose of this study is to investigate the structural relationships
among computational thinking, self-efficacy, self-exploration, enjoyment, and learning
satisfaction in an AppInventor-based liberal education class for non-STEM-major col-
lege students. The results contribute significant information for non-STEM-major col-
lege students that can help to improve their critical thinking skills while encouraging
a more innovative and forward-thinking mindset to discover computational solutions
(Chong et al., 2018; Kules, 2016). Moreover, the findings also aid computational thinking
instructors in designing more accommodating computational thinking courses to reduce
the digital divide that results from socioeconomic and cultural backgrounds (Czerkawski
& Lyman, 2015).
Conceptual framework and hypothesis development

Computational thinking
The definitions of computational thinking vary by scholar. Wing (2006) introduced the
concept of computational thinking and characterized it as a recursive process that uses
the skill of abstraction and decomposition to confront a large complex task or design a
large complex system. Israel et. al. (2015, p. 246) developed the definition and charac-
terized computational thinking as “students using computers to model their ideas and
develop programs that enhance those programs.” Barr and Stephenson (2011, p. 115)
explained the concept of computational thinking in the K-12 context as “an approach to
solving problems in a way that can be implemented with a computer.”
Although the definitions of computational thinking vary and a consensus is lacking,
the various accounts have several implications. It can be accepted that fostering com-
putational thinking-relevant skills can enhance problem solving and abstract reasoning
capability. Computational thinking is a type of analytical thinking for problem solving
that comprises the scoping of problems with a suitable expressive format or media,
interpreting these topics through abstraction, and finally formulating computerized
solutions to the problems (Gong et al., 2020). In addition, some computational thinking
involves learning the competencies and know-how required to engage in programming
skills. Finally, computational thinking essentially encompasses the concepts of abstrac-
tion, problem solving, analysis, decomposition, integration, algorithmic thinking, gener-
alization, coding, and debugging (Yılmaz et al., 2018).
Enjoyment
Enjoyment is defined as a positive affective reaction that mirrors general feelings such
as liking, delight, and fun derived from an activity in which an individual is engaged
(Raedeke, 2007). Applying this notion to the learning field, enjoyment can be described
as the extent to which a learner gains a sense of pleasure, joy, and fun and as a factor that
contributes to a positive, all-inclusive experiences from a class (Moorthy et al., 2019).
Enjoyment is an intrinsic motivation with an innate tendency to discover novelty and is
experienced when an individual faces challenges in expanding and exercising his or her
abilities to learn and explore (Gomez et al., 2010; Teo & Noyes, 2011). A previous study
has shown that learners’ motivations are positively related to learning performance
(Gomez et al., 2010). Students experience psychological pleasure in game-based classes.
That is, game-oriented teaching can promote learners’ motivations and interests, and
learners enjoy these learning activities (Hsu et al., 2018).
Digital self‑efficacy
Rooted in social cognitive theory, self-efficacy refers to beliefs about one’s capabilities
to learn or perform behaviors at designated levels (Bandura, 2006, 2010). The concept
postulates that an individual’s achievement relies on interactions among one’s behaviors,
personal factors (e.g., thoughts, beliefs), and environmental conditions (Bandura, 2006;
Bandura & Schunk, 1981). Self-efficacy accounts for individual accomplishments and
human well-being. Specifically, individuals with a higher sense of self-efficacy believe
that they possess greater capabilities to accomplish challenging and difficult tasks. In a
digital learning environment, digital self-efficacy, also called computer efficacy or inter-
net efficacy, is usually used to measure individual self-efficacy in the digital domain
(Mun & Hwang, 2003; Wei et al., 2020). Venkatesh and Davis defined digital self-efficacy
as a self-assessment of one’s ability to use information technology (IT) or one’s belief
that people can use computer or internet-related technologies well (Venkatesh & Davis,
1996). Based on this definition, this study modifies the definition to fit the research set-
ting and defines digital self-efficacy as the learner’s assessment of his or her ability to
learn in an IT-mediated environment or his or her belief that he or she can use internet-
related technologies to learn.
The relationship between computational thinking and digital self‑efficacy

Expectancy-value theory posits that individuals’ expectations for success and the value
of them succeeding are important factors in their motivation to accomplish various tasks
(Eccles, 1983). That is, if a learner accepts that a class that he or she takes contributes
to his or her capabilities or future career development, then the perceived value will
increase and motivate the learner to engage more fully in the class. Applying this ration-
ale to a computational thinking class, a student’s motivation to learn may be aroused if
the student can learn analytical skills, problem-solving skills, and thinking skills and thus
improve his or her capabilities. Therefore, this study proposes the following hypothesis:
H1 Computational thinking has a positive influence on digital self-efficacy.
The relationship between enjoyment and digital self‑efficacy

Enjoyment is an intrinsic and affective motivation that can result in behavior changes
and active learning (Goh & Yang, 2021). Self-efficacy reflects one’s internal motivation
based on ability, while emotional arousal is a critical source of the development of self-
efficacy (Bandura, 2006, 2010). When an individual faces threatening situations or dif-
ficult tasks, negative emotions, such as anxiety, may emerge and thus restrain one’s
capabilities to cope. In contrast, when an individual perceives the tasks to be interesting
or joyful, he or she might have more confidence in his or her ability to accomplish these
tasks. A substantial body of evidence has shown positive associations between aspects
of enjoyment and self-efficacy. For example, Chen et. al. (2017a, 2017b) found that
enjoyment is an antecedent of self-efficacy in a physical activity environment. Mun and
Hwang (2003) also concluded that learners with higher perceived levels of enjoyment
exhibit higher digital self-efficacy when using web-based information systems. Based on
such findings, this study proposes the following hypothesis:
H2 Enjoyment has a positive influence on digital self-efficacy.
Self‑exploration
Self-exploration is an individual’s conscious internal or external behavior of analyz-
ing information and pursuing knowledge related to his or her career (Flum & Kaplan,
2006). Furthermore, information analysis results in the formation of self-meaning and
has a systemic influence on fostering self-development. Educational psychologists argue
that self-exploration is one of the core factors that explains identity formation because
it assists individuals in examining their identifications by facing them in terms of their
underexamined viewpoints, advancing beliefs about alternatives, and directing them to
explore novel and unfamiliar knowledge areas (Kaplan & Madjar, 2017). Self-exploration
is integral to career development and vocational choices (Flum & Kaplan, 2006).
The relationship between computational thinking and self‑exploration

One goal of education is to cultivate in students the competencies and capabilities to lev-
erage current technologies to solve undiscovered problems (Durak & Saritepeci, 2018).
The International Society for Technology in Education (ISTE, 2022) also emphasizes
that the purpose of computational thinking is to educate young students as computa-
tional thinkers who can solve tomorrow’s problems by using today’s technologies. Com-
putational thinking is essentially a set of transferable skills that enable people to foster
technological literacy for the twenty-first century and to succeed in a wider range of jobs
and tasks beyond their own disciplines (Nägele & Stalder, 2017). Since computational
thinking is cross-disciplinary, a non-STEM college student who takes a computational
thinking class might think about how these skills can be used in his or her major domain
and explore the feasibility of applying these skills to his or her future career. When a
class is more highly valued, the possibility is higher that the student seeks ideas for his or
her own sake and links to his or her innate values and beliefs. Therefore, this study pro-
poses the following hypothesis:
H3 Computational thinking has a positive influence on self-exploration.

The relationship between enjoyment and self‑exploration

Enjoyment is an intrinsic affective motivation that drives individuals to do something
that they enjoy (Gomez et al., 2010; Moorthy et al., 2019; Raedeke, 2007; Teo & Noyes,
2011). Self-exploration is associated with the framing of an individual’s identity in terms
of intrinsic cognitive motivation (Kaplan & Madjar, 2017). Similar to enjoyment and
self-efficacy, which are both forms of intrinsic motivation and are correlated with one
another (Meyer et al., 2019), we assume enjoyment and self-exploration to be positively
related. On the basis of self-determination theory, interesting or joyful classes might
increase the engagement of students because students are eager to know and make dis-
coveries about these classes’ influences and outcomes (Ryan & Deci, 2017). Enjoyment
might foster an individual’s psychological reflection and self-exploration. Therefore, this
study hypothesizes that students with higher perceptions of enjoyment have stronger
self-exploration motives.
H4 Enjoyment has a positive influence on self-exploration.
Learning satisfaction
Learning satisfaction and learning achievement are two typical indicators of learning
effectiveness, which is an ultimate learning outcome (Hu & Hui, 2012). Learning satis-
faction is the overall level of fulfillment of a learner’s expectations that pertains to a class
experience (Cidral et al., 2018). Because the research setting of the present study involves
a technology-based learning environment (i.e., AppInventor) that enjoys the advantages
of increased learning opportunities, the satisfaction of students’ requirements, the sup-
port of blended learning (both online and offline workshops), and exceedingly diversi-
fied learning, an assessment of learning performance should consider in the learning
environment both the actual effectiveness in learning content and students’ attitudes
and expectations (Bostrom et al., 1990). Additionally, previous literature on information
systems and computer-assisted learning has indicated that learner or user satisfaction is
an important measure of learning performance and the effectiveness of online learning
system implementation (Ke & Kwak, 2013). Consequently, this study uses learning satis-
faction as the dependent variable in the proposed model.
The relationship between digital self‑efficacy and learning satisfaction

Higher self-efficacy contributes to higher academic performance and satisfaction
(Karadag et al., 2021). Learners with higher efficacy set higher goals and believe that
they can achieve these goals, even when facing difficulties (Bandura, 2006, 2010). That
is, students with higher self-efficacy may reflect and engage in their learning experiences
and reshape their learning behaviors to achieve better learning performance. Self-effi-
cacy has been regarded as one of the critical determinants that accounts for a learner’s
achievement in an educational setting. Zysberg and Schwabsky (2020), for example, it
has been recently confirmed that self-efficacy acts as a mediator that links school climate
and academic achievement (i.e., math, English and the relative rank) for middle and
high school students. Similarly, Vayre and Vonthron (2019) suggest that both academic
self-efficacy and learning engagement are antecedents that predict successfully passing
exams for students enrolled in online university courses. Therefore, this study hypoth-
esizes the following:
H5 Higher digital self-efficacy leads to higher learning satisfaction.
The relationship between self‑exploration and learning satisfaction

A university education is an important stage for employment preparation and career
development. Before career development, individuals usually engage in self-explora-
tion to understand the relationship between their interests, values, and needs and the
external environment through information gathering and processing, experience accu-
mulation, and self-evaluation (Cai et al., 2015). As self-exploration is a factor of career
development, it is reasonable to assume that when individuals understand themselves
more, their tendency is higher to engage in tasks with the purpose of enhancing their
capabilities, namely, to learn, to understand and to develop skills. Therefore, a positive
association between self-exploration and learning satisfaction can be expected.
H6 Higher self-exploration leads to higher learning satisfaction.
Drawing on the arguments above, the research model is shown in Fig. 1 for computa-
tional thinking, enjoyment, self-efficacy, self-exploration, and learning satisfaction and
their hypotheses. The hypotheses are repeated below for ease of reference.
H1 Computational thinking has a positive influence on digital self-efficacy.
H2 Enjoyment has a positive influence on digital self-efficacy.
H3 Computational thinking has a positive influence on self-exploration.
H4 Enjoyment has a positive influence on self-exploration.
H5 Higher digital self-efficacy leads to higher learning satisfaction.
H6 Higher self-exploration leads to higher learning satisfaction.
Fig. 1 Research model

Method
Measures
The proposed model with five variables and six hypotheses was assessed by using a
quantitative survey. All constructs were measured with multiple items, and seven-point
Likert-type scales that ranged from “strongly agree” to “strongly disagree” were used.
The five variables were adopted from the relevant literature and modified to fit the cur-
rent research context (Cho et al., 2017). That is, learning satisfaction was measured by
using five items adopted from Hu and Hui (2012), computational thinking was measured
with five items modified from Durak and Saritepeci (2018), enjoyment (EJ) was meas-
ured by using four items from Kong et. al. (2018), digital self-efficacy was measured with
four items from Kim and Jang (2015), and self-exploration was measured by using four
items from Afzal et. al. (2010). Because the survey participants were Chinese students,
the original instruments were translated into Chinese by a professional translator. After
the authors ensured that the intended meaning was conveyed by the instruments, a
reverse translation was performed to ensure accurate interpretation (Cidral et al., 2018).
These items and sources are listed in Appendix 1. The questionnaire is divided into five
sections, each of which consists of questions designed to elicit information about one of
the constructs, specifically, computational thinking, enjoyment, self-efficacy, self-explo-
ration, or learning satisfaction.
The context, participants, and procedures

Data were collected from a university in northern Taiwan. The university opened a lib-
eral education course worth two credits. The course was a mobile device programming
class that was available only to undergraduate students whose major was not STEM. This
course aimed to teach students how to solve problems by identifying the problem and
understanding the modeling, abstracting, and designing of an algorithm and by teach-
ing basic computer programming concepts to students who were without programming
skills. AppInventor (https://www.appinventor.mit.edu/), which was originally invented
by Google and transferred to the Massachusetts Institute of Technology (MIT) for oper-
ation and management in 2012, served as the main instrument for teaching computa-
tional thinking concepts in this course, while other problem-solving techniques, such
as flowcharts and case studies, were discussed in the class. To gain hands-on compu-
tational thinking experiences and inspire learning interest, students who took the class
practiced how to solve daily problems with AppInventor. Specifically, the instructor used
online restaurant ordering, body mass index (BMI), and a ninety-nine multiplication
table as the course materials to deliver the core concepts of computational thinking. The
students could build their own apps after observing the instructor’s initial demonstra-
tion. Some of the student works are shown in Appendix 2. Moreover, the students were
assigned a term project to propose how to leverage their skills and application of compu-
tational thinking in their daily life or profession.
At the end of the school term, the instructor communicated the objectives of the sur-
vey to all students who took the class and emphasized that there were no definite right
or wrong answers or perspectives and that answers to the survey would not impact their
class performance. Later, a follow-up email with a link to a GoogleDoc survey was sent
to all students to encourage them to participate. A small amount of extra credit was
offered to encourage students to complete the survey (Shapiro et al., 2017). The students
voluntarily completed the survey, and they could choose to omit their name if they had
any concerns. Of the 276 enrolled students from six classes in five semesters, 190 stu-
dents participated in the survey for a 71.7% participation rate. The gender distribution
was 68.9% female and 31.1% male. Of the sample, 36.8% were 1st-year students, 49.5%
were 2nd-year students, 3.7% were 3rd-year students, and 10.0% were 4th-year students.
Results
Because the proposed model contains five constructs, which are also referred to as latent
variables, and the purpose of this study is to explore the relationship among these con-
structs, structural equation modeling (SEM) is suitable for assessing the results (Hair
et al., 2017a). SEM is a statistical method that examines and tests causal relationships
with a combination of statistical data and theoretical causal assumptions (Sarstedt et al.,
2017). There are two types of SEM techniques, namely, covariance-based SEM (CB-
SEM) and partial least squares SEM (PLS-SEM). CB-SEM is a parametric statistical
method that is primarily used for the confirmation of an established theory. Accordingly,
statistical significance is a standard output of this technique; in contrast, PLS-SEM is
considered to be more relevant for analyzing complex models and exploratory research
(Lee & Jung, 2021).
To assess the proposed model and test the study hypotheses, the present study applied
PLS-SEM to examine the relationships among the constructs in complex models. There
are several advantages to using this technique. First, PLS-SEM comprises a measure-
ment model and a structural model evaluation, which makes it superior to the one-step
evaluations of SEM (Hair et al., 2010; Molinillo et al., 2018). The structural model exam-
ines the relationships among constructs in structural models, whereas the measurement
model assesses the reliability and validity of the constructs (Hair et al., 2017a, 2017b).
Reliability refers to the consistency of the scale items (Hair et al., 2017b). The measure-
ment indicators include individual item reliability and internal consistency. Factor load-
ing is usually used to assess individual item reliability. In contrast, composition reliability
(CR) and the Cronbach’s alpha are two metrics that test the internal consistency of the
latent variable. The recommended threshold value must exceed 0.7 (Hair et al., 2017a,
2017b).
Validity refers to the correctness of the scale items, and the measurement metrics usu-
ally include convergent validity and discriminant validity. Convergent validity measures
the correlation between items within the same construct. Discriminant validity measures
the correlation between items with different constructs. The average variance extraction
(AVE) is a suitable indicator to detect discriminant validity, providing the recommended
threshold value exceeds 0.5 (Hair et al., 2017a, 2017b). If the square root value of the
diagonal AVE is greater than the correlation coefficient value of the horizontal or verti-
cal column, then this represents discriminant validity (Hair et al., 2017b).
Second, PLS-SEM is particularly suitable for research in fields such as education and
requires only a limited sample size (Goh & Yang, 2021; Hair et al., 2017a). Otherwise, the
data do not follow a normal distribution (Molinillo et al., 2018). Moreover, as in the pre-
sent research, predicting capability and evaluating the relationships of latent variables
Table 1 Reliability and convergent validity

AVE Composite reliability Cronbach’s alpha
Computational thinking 0.835 0.962 0.950

Enjoyment 0.907 0.975 0.966
Learning satisfaction 0.876 0.972 0.964
Digital self-efficacy 0.790 0.938 0.911
Self-exploration 0.899 0.973 0.962
Table 2 Correlation among the construct scores

Computational Enjoyment Learning Digital Self-exploration
thinking satisfaction self-
efficacy
Computational thinking 0.914

Enjoyment 0.782 0.952
Learning satisfaction 0.788 0.791 0.936
Digital self-efficacy 0.730 0.769 0.800 0.889
Self-exploration 0.793 0.803 0.827 0.823 0.948
The boldface figures on the diagonal represent the square root of the AVE figures
(unobservable constructs) enable PLS-SEM to develop exploratory analysis models

(Hair et al., 2017a). Unlike traditional statistical techniques such as a t-test and analysis
of variance (ANOVA) that examine whether the group means differ from one another
or whether one group’s mean differs in different times, PLS-SEM enables the education
studies to explore the structural relationships among a group of latent variables. Addi-
tionally, PLS-SEM is more suitable for exploratory studies to understand the specific
path coefficients and variance of the dependent variable as explained by the independ-
ent variables in the proposed framework instead of examining the goodness of model fit
(Hair et al., 2017a, 2017b; Lee & Jung, 2021).
Assessment of the measurement model

The measurement model estimates construct validity and reliability, including conver-
gent and discriminant validity. The Cronbach’s alpha, which represents construct reli-
ability, is used to test the reliability of all constructs in the proposed framework. The
Cronbach’s alphas fell between 0.911 and 0.966 for all constructs in this research model.
These values far exceed the threshold of 0.7 (Nunnally & Bernstein, 1994). In addition,
the composite reliability (CR) was beyond 0.938, which far exceeds the cutoff value of
0.5 (Chin & Gopal, 1995). These results confirm the internal reliability of each construct.
Moreover, estimated pairwise correlations between factors did not exceed the 0.85 limit
(Kline, 2015). The average variance extracted value (AVE) was applied to examine con-
vergent validity. The outcome AVE value fell between 0.790 and 0.907, which exceeded
the cutoff value of 0.5 (Hair et al., 2010). Fornell–Larcker criteria are commonly used to
test discriminant validity and require that the square root of the AVE be greater than all
correlations between each pair of constructs (Fornell & Larcker, 1981). The reliability
and convergent validity results of the proposed constructs are summarized in Tables 1
Fig. 2 Results of the research model
Table 3 Summary of the hypothesis results

β t-statistic p-value Result
H1: Computational thinking → digital self-efficacy 0.329 3.976 *** H1 supported

H2: Enjoyment → digital self-efficacy 0.512 6.253 *** H2 supported
H3: Computational thinking → self-exploration 0.425 5.549 *** H3 supported
H4: Enjoyment → self-exploration 0.470 6.586 *** H4 supported
H5: Digital self-efficacy → learning satisfaction 0.371 4.671 *** H5 supported
H6: Self-exploration → learning satisfaction 0.522 6.592 *** H5 supported
and 2. Additionally, the factor loadings of items fell between 0.872 and 0.968, which
exceeded the cutoff value of 0.5 (Lee & Jung, 2021), as displayed in Appendix 1.
Assessment of the structural model

PLS uses bootstrapping with a substantial resampling (3000 times in this study) to
compute the beta (β), R2, and respective t-statistics of the structure model (Hair et al.,
2017a). The beta (β) represents the regression path coefficient. The t-statistics test the
statistical significance of both the outer and inner models and are also provided with
t-value ≥ 1.96, t-value ≥ 2.58, and t-value ≥ 3.29, which denote p < 0.05, p < 0.01, and
p < 0.001, respectively (Field, 2009). The R2 value indicates the explained variance of the
endogenous latent variables for the structural model. A graphic illustration of the struc-
tural model for this study is given in Fig. 2, while a summary of the hypotheses is pre-
sented in Table 3.
Based on Fig. 2 and Table 3, all proposed hypotheses in the research model were
supported.
• Computational thinking had a statistically significant positive impact on digital self-

efficacy (β = 0.329, t-value = 3.976, p < 0.001), which supports H1.
• Enjoyment had a statistically significant positive impact on digital self-efficacy
(β = 0.512, t-value = 6.253, p < 0.001), which supports H2.
• Computational thinking had a statistically significant positive impact on self-explora-
tion (β = 0.425, t-value = 5.549, p < 0.001), which supports H3.
• Enjoyment had a statistically significant positive impact on self-exploration

• Digital self-efficacy had a statistically significant positive impact on learning satisfac-
tion (β = 0.371, t-value = 4.671, p < 0.001), which supports H5.
• Self-exploration had a statistically significant positive impact on learning satisfaction
Moreover, the interpretation of variation (R2) from computational thinking and

enjoyment to digital self-efficacy was 0.634. The R2 from computational thinking and
enjoyment to self-exploration was 0.714, and that from digital self-efficacy and self-
exploration to learning satisfaction was 0.728.
Discussion
The findings show that computational thinking has a positive impact on both digital
self-efficacy (β = 0.329, p < 0.001) and self-exploration (β = 0.425, p < 0.001), and enjoy-
ment also has a positive impact on both digital self-efficacy (β = 0.512, p < 0.001) and
self-exploration (β = 0.470, p < 0.001). Furthermore, both digital self-efficacy (β = 0.371,
p < 0.001) and self-exploration (β = 0.522, p < 0.001) have a positive impact on learn-
ing satisfaction. These findings have several implications. First, computational think-
ing courses are not proprietary to STEM curricula or precollege students. Empirically
confirming a previous study (Czerkawski & Lyman, 2015), this study empirically sug-
gests that non-STEM college students can enjoy the learning benefits of computational
thinking and take an interest in computational thinking classes. That is, computational
thinking teaches a set of transferable and marketable skills that are appropriate for any
domain.
Next, computational thinking fosters digital self-efficacy (β = 0.329, p < 0.001), even in
students in non-STEM majors. The results indicate that computational thinking classes
can stimulate students to think about how to understand themselves and explore new
ideas. This implies that students might think about how to use these skills to solve prob-
lems in their majors or related areas.
In addition, this study shows that enjoyment is a factor of digital self-efficacy
(β = 0.512, p < 0.001). Previous studies have shown that enjoyment is positively associ-
ated with digital self-efficacy (Mun & Hwang, 2003; Wei et al., 2020) and learning sat-
isfaction in online learning settings. This study shows that interesting or game-based
classes can promote student digital self-efficacy. Interesting classes can remove or reduce
learning barriers and encourage students to challenge themselves and can enhance stu-
dent growth and development.
Furthermore, this study creates a computational thinking class with a suitable
design by adding an interesting element and can promote learners’ digital self-efficacy
(β = 0.512, p < 0.001) for non-STEM-major students, which further enhances their learn-
ing satisfaction.
Finally, this study contributes to the literature on computational thinking learning in
that self-exploration is an antecedent of learning satisfaction (β = 0.522, p < 0.001). Self-
exploration helps prepare students for career development and vocational choices (Flum
& Kaplan, 2006). In this study, students with a higher degree of self-exploration skills
showed greater readiness for career preparation. Accordingly, classes that can increase
student confidence in career preparation will lead to higher learning satisfaction.
This study has several implications for computational thinking instructors. AppInven-
tor is an effective instrument for learning computational thinking and for the promo-
tion of computational thinking education. In addition to developing problem-analysis,
thinking, and problem-solving skills, computational thinking learning assists students
in understanding themselves. Furthermore, instructors can encourage students to use
computational thinking to solve problems in their domains or daily lives to enhance
their learning satisfaction. Although AppInventor contains entertaining ingredients,
an instructor might add more entertaining factors to his or her classes. This study, for
example, used restaurant ordering as a lab exercise for learning content. A practical but
interesting daily problem might help or persuade a student to overcome challenges and
increase learning satisfaction. Correspondingly, this study provides educational policy
makers with a venue for computational thinking education for non-STEM college stu-
dents. Computational thinking educational resource allocation for non-STEM college
students will enhance overall social thinking skills. Additionally, as enjoyment can effec-
tively lower the learning barrier, computational thinking educational resource configura-
tion can be linked to joyful or playful learning material.
Limitations
One limitation of this study is that all participants were students at a college located
in northern Taiwan. Future studies might investigate the effect of cultural, social, or
economic factors on computational thinking learning performance for non-STEM
college students.
Another limitation of this study is that AppInventor limits the learning material con-
sidered. In addition to AppInventor, other relevant learning tools, such as Turtle Art,
Scratch, Code.org, and Scalable Game Design (Hsu et al., 2018), could be considered for
similar studies. Future studies should examine the learning outcomes of other tools. It
would be interesting to explore the relationships between information and communica-
tions technology (ICT) learning tools and non-ICT tools such as flow charts, fishbone
diagrams, brain storming, and mind maps. Understanding the role of the relationships
between these tools will help learners obtain comprehensive problem-solving skills and
transferable knowledge.
Finally, because the course was not mandatory but elective, it might also be a limita-
tion of this study that non-STEM students had the choice of enrolling or not enrolling in
this particular course, and those who enrolled already may have had a positive attitude
toward technology, programming and computational thinking. Future studies might
investigate a comparison of the results of this class property (mandatory vs. elective) or
other factors that affect attitudes toward computational thinking.
Conclusion
This study affirms that non-STEM college students can effectively learn computational
thinking with the support of proper learning instruments. As society becomes increas-
ingly digitalized, it is virtually certain that non-STEM knowledge workers will be
required to interact with information technology professionals in a variety of domains.
Learning computational thinking does not make it possible for a student who does not
understand how to program to become a qualified programmer, but it does cultivate the
thinking and logic competencies for problem solving and aids the student in interacting
across domains.
These findings also emphasize the importance of the computational thinking cur-
riculum in self-exploration and how effective computational thinking learning can be
achieved. The findings of this study help make the argument that computational think-
ing provides benefits across the curriculum. Furthermore, these findings encourage pol-
icy makers to allocate resources to expand the use of this beneficial educational tool in
the classroom.
Appendix 1. Questionnaire items and sources
Construct Item Factor loading Source
Computational thinking CT1. I can learn analytical skills 0.874 Durak and Saritepeci (2018)
CT2. I can learn problem-solving skills 0.926
CT3. This class can train my thinking 0.920
skills
CT4. This class can strengthen my think‑ 0.901
ing skills
CT5. This class can improve my capabili‑ 0.945
ties
Enjoyment EJ1. Programming is interesting 0.950 Kong et. al. (2018)
EJ2. I am curious about the content of 0.936
programming
EJ3. I think the content of program‑ 0.955
ming is fun
EJ4. I am very attracted to computer 0.968
programming activities
Self-efficacy SE1. Compared with other students in 0.895 Kim and Jang (2015)
this class, I expect to do well
SE2. I am certain I can understand the 0.872
ideas taught in this course
SE3. I expect to do very well in this class 0.910
SE4. Compared with others in this class, 0.877
I think I am a good student
Self-exploration SX1. I want to understand myself better 0.944 Afzal et. al. (2010)
SX2. I want to explore new ideas 0.940
SX3. I want to challenge myself 0.953
SX4. I want to enhance my personal 0.955
growth and development
Learning satisfaction LS1. I like the idea of learning AppIn‑ 0.921 Hu and Hui (2012)
ventor in a lab like this
LS2. Learning AppInventor by taking a 0.944
lab like this is a good idea
LS3. My learning experience in this lab 0.953
is positive
LS4. Overall, I am satisfied with this lab 0.909
LS5. As a whole, the lab is effective for 0.952
my learning AppInventor
Appendix 2. Some student works

(A) Restaurant Ordering App

(B) BMI Calculator

(C) Ninety-nine Multiplication Table

Acknowledgements
The authors would like to thank the students for their consensus to share their work and the anonymous reviewers for
their positive comments concerning this paper.
Author contributions
CHL conceptualized the research mode. CTC undertook the research, analyzed the data, and prepared the initial manu‑
script. ICC contributed part of the data. KP revised and finalized the manuscript. All authors read and approved the final
manuscript.
Funding
Not applicable.
Availability of data and materials

The datasets are available from the corresponding author on reasonable request.
Declarations
Competing interests
The authors declare that they have no competing interests.
Author details
1
Department of Information Management, Fu Jen Catholic University, No. 510, Zhongzheng Rd., Xinzhuang Dist., New
Taipei City 24205, Taiwan. 2 Department of Information Management, Chinese Culture University, 55, Hwa‑Kang Rd.,
Yang‑Ming‑Shan, Taipei 11114, Taiwan. 3 Department of Informatics, Idaho State University, 921 S. 8th Ave., Stop 8020,
Pocatello, ID 83209‑8020, USA.
Received: 24 January 2022 Accepted: 7 April 2022
References
Afzal, H., Ali, I., Khan, M. A., & Hamid, K. (2010). A study of university students’ motivation and its relationship with their
academic performance. International Journal of Business and Management, 5(4), 80–89.
Bandura, A. (2006). Guide for constructing self-efficacy scales. Self-Efficacy Beliefs of Adolescents, 5(1), 307–337.
Bandura, A. (2010). Self‐efficacy. The Corsini encyclopedia of psychology, 1–3.
Bandura, A., & Schunk, D. H. (1981). Cultivating competence, self-efficacy, and intrinsic interest through proximal self-
motivation. Journal of Personality and Social Psychology, 41(3), 586.
Barr, V., & Stephenson, C. (2011). Bringing computational thinking to K-12: What is involved and what is the role of the
computer science education community? Inroads, 2(1), 48–54.
Bers, M. U., Flannery, L., Kazakoff, E. R., & Sullivan, A. (2014). Computational thinking and tinkering: Exploration of an early
childhood robotics curriculum. Computers & Education, 72, 145–157.
Bostrom, R. P., Olfman, L., & Sein, M. K. (1990). The importance of learning style in end-user training. MIS Quarterly, 14,
101–119.
Cai, Z., Guan, Y., Li, H., Shi, W., Guo, K., Liu, Y., Li, Q., Han, X., Jiang, P., Fang, Z., & Hua, H. (2015). Self-esteem and proactive
personality as predictors of future work self and career adaptability: An examination of mediating and moderating
processes. Journal of Vocational Behavior, 86, 86–94.
Chen, G., Shen, J., Barth-Cohen, L., Jiang, S., Huang, X., & Eltoukhy, M. (2017a). Assessing elementary students’ computa‑
tional thinking in everyday reasoning and robotics programming. Computers & Education, 109, 162–175.
Chen, H., Sun, H., & Dai, J. (2017b). Peer support and adolescents’ physical activity: The mediating roles of self-efficacy and
enjoyment. Journal of Pediatric Psychology, 42(5), 569–577.
Chin, W. W., & Gopal, A. (1995). Adoption intention in GSS: Relative importance of beliefs. ACM SIGMIS Database: The DATA-
BASE for Advances in Information Systems, 26(2–3), 42–64.
Cho, M. H., Kim, Y., & Choi, D. (2017). The effect of self-regulated learning on college students’ perceptions of community
of inquiry and affective outcomes in online learning. The Internet and Higher Education, 34, 10–17.
Chong, S., Pan, G. T., Chin, J., Show, P. L., Yang, T. C. K., & Huang, C. M. (2018). Integration of 3D printing and industry 4.0
into engineering teaching. Sustainability, 10(11), 3960.
Cidral, W. A., Oliveira, T., Di Felice, M., & Aparicio, M. (2018). E-learning success determinants: Brazilian empirical study.
Computers & Education, 122, 273–290.
Czerkawski, B. C., & Lyman, E. W. (2015). Exploring issues about computational thinking in higher education. TechTrends,
59(2), 57–65.
Durak, H. Y., & Saritepeci, M. (2018). Analysis of the relation between computational thinking skills and various variables
with the structural equation model. Computers & Education, 116, 191–202.
Eccles, J. (1983). Expectancies, values and academic behaviors. Achievement and Achievement Motives.
Field, A. (2009). Discovering statistics using SPSS. Sage Publications.
Flum, H., & Kaplan, A. (2006). Exploratory orientation as an educational goal. Educational Psychologist, 41(2), 99–110.
Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement
error. Journal of Marketing Research, 18(1), 39–50.
Goh, T. T., & Yang, B. (2021). The role of e-engagement and flow on the continuance with a learning management system
in a blended learning environment. International Journal of Educational Technology in Higher Education, 18(1), 1–23.
Gomez, E. A., Wu, D., & Passerini, K. (2010). Computer-supported team-based learning: The impact of motivation, enjoy‑
ment and team contributions on learning outcomes. Computers & Education, 55(1), 378–390.
Gong, D., Yang, H. H., & Cai, J. (2020). Exploring the key influencing factors on college students’ computational thinking
skills through flipped-classroom instruction. International Journal of Educational Technology in Higher Education, 17,
1–13.
Güven, I., & Gulbahar, Y. (2020). Integrating computational thinking into social studies. The Social Studies, 111(5), 234–248.
Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis (7th ed.). Pearson.
Hair, J., Hollingsworth, C. L., Randolph, A. B., & Chong, A. Y. L. (2017a). An updated and expanded assessment of PLS-SEM
in information systems research. Industrial Management & Data Systems, 117(3), 442–458.
Hair, J. F., Hult, G. T. M., Ringle, C., & Sarstedt, M. (2017b). A primer on partial least squares structural equation modeling (PLS-
SEM) (2nd ed.). SAGE.
Hsu, T. C., Chang, S. C., & Hung, Y. T. (2018). How to learn and how to teach computational thinking: Suggestions based on
a review of the literature. Computers & Education, 126, 296–310.
Hu, P. J. H., & Hui, W. (2012). Examining the role of learning engagement in technology-mediated learning and its effects
on learning effectiveness and satisfaction. Decision Support Systems, 53(4), 782–792.
Israel, M., Pearson, J. N., Tapia, T., Wherfel, Q. M., & Reese, G. (2015). Supporting all learners in school-wide computational
thinking: A cross-case qualitative analysis. Computers & Education, 82, 263–279.
ISTE, C. (2022). Computational thinking in K-12 education leadership toolkits. Retrieved January 24, 2022, from https://
www.iste.org/standards/computational-thinking?_ga=2.183977958.1744591423.1564987683-1256986531.15649
87683
Kaplan, H., & Madjar, N. (2017). The motivational outcomes of psychological need support among pre-service teachers:
Multicultural and self-determination theory perspectives. Frontiers in Education, 2, 1–14.
Karadag, E., Su, A., & Ergin-Kocaturk, H. (2021). Multi-level analyses of distance education capacity, faculty members’ adap‑
tation, and indicators of student satisfaction in higher education during COVID-19 pandemic. International Journal
of Educational Technology in Higher Education, 18(1), 1–20.
Ke, F., & Kwak, D. (2013). Online learning across ethnicity and age: A study on learning interaction participation, percep‑
tion, and learning satisfaction. Computers & Education, 61, 43–51.
Kim, H. J., & Jang, H. Y. (2015). Factors influencing students’ beliefs about the future in the context of tablet-based interac‑
tive classrooms. Computers & Education, 89, 1–15.
Kline, R. B. (2015). Principles and practice of structural equation modeling. The Guilford Press.
Kong, S. C., Chiu, M. M., & Lai, M. (2018). A study of primary school students’ interest, collaboration attitude, and program‑
ming empowerment in computational thinking education. Computers & Education, 127, 178–189.
Kules, B. (2016). Computational thinking is critical thinking: Connecting to university discourse, goals, and learning out‑
comes. Proceedings of the Association for Information Science and Technology, 53(1), 1–6.
Lee, J., & Jung, I. (2021). Instructional changes instigated by university faculty during the COVID-19 pandemic: The effect
of individual, course and institutional factors. International Journal of Educational Technology in Higher Education,
18(1), 1–19.
Lu, K., Yang, H. H., Shi, Y., & Wang, X. (2021). Examining the key influencing factors on college students’ higher-order think‑
ing skills in the smart classroom environment. International Journal of Educational Technology in Higher Education,
18(1), 1–13.
Meyer, O. A., Omdahl, M. K., & Makransky, G. (2019). Investigating the effect of pre-training when learning through immer‑
sive virtual reality and video: A media and methods experiment. Computers & Education, 140, 103603.
Molinillo, S., Anaya-Sánchez, R., Aguilar-Illescas, R., & Vallespín-Arán, M. (2018). Social media-based collaborative learning:
Exploring antecedents of attitude. Internet and Higher Education, 38(1), 18–27.
Moorthy, K., T’ing, L. C., Wei, K. M., Mei, P. T. Z., Yee, C. Y., Wern, K. L. J., & Xin, Y. M. (2019). Is facebook useful for learning? A
study in private universities in Malaysia. Computers & Education, 130, 94–104.
Mun, Y. Y., & Hwang, Y. (2003). Predicting the use of web-based information systems: Self-efficacy, enjoyment, learning
goal orientation, and the technology acceptance model. International Journal of Human-Computer Studies, 59(4),
431–449.
Nägele, C., & Stalder, B. E. (2017). Competence-based vocational and professional education. Springer.
Nunnally, J., & Bernstein, I. H. (1994). Psychometric theory. McGraw-Hill.
Raedeke, T. D. (2007). The relationship between enjoyment and affective responses to exercise. Journal of Applied Sport
Psychology, 19(1), 105–115.
Romero, M., Lepage, A., & Lille, B. (2017). Computational thinking development through creative programming in higher
education. International Journal of Educational Technology in Higher Education, 14(1), 1–15.
Ryan, R. M., & Deci, E. L. (2017). Self-determination theory: Basic psychological needs in motivation, development, and well-
ness. Guilford Publications.
Sarstedt, M., Ringle, C. M., & Hair, J. F. (2017). Partial least squares structural equation modeling. Handbook of Market
Research, 26(1), 1–40.
Shapiro, A. M., Sims-Knight, J., O’Rielly, G. V., Capaldo, P., Pedlow, T., Gordon, L., & Monteiro, K. (2017). Clickers can promote
fact retention but impede conceptual understanding: The effect of the interaction between clicker use and peda‑
gogy on learning. Computers & Education, 111, 44.
Tekdal, M. (2021). Trends and development in research on computational thinking. Education and Information Technolo-
gies, 26(5), 6499–6529.
Teo, T., & Noyes, J. (2011). An assessment of the influence of perceived enjoyment and attitude on the intention to use
technology among pre-service teachers: A structural equation modeling approach. Computers & Education, 57(2),
1645–1653.
Tikva, C., & Tambouris, E. (2021). Mapping computational thinking through programming in K-12 education: A concep‑
tual model based on a systematic literature review. Computers & Education, 162, 104083.
Vayre, E., & Vonthron, A. M. (2019). Relational and psychological factors affecting exam participation and student achieve‑
ment in online college courses. The Internet and Higher Education, 43, 100671.
Venkatesh, V., & Davis, F. D. (1996). A model of the antecedents of perceived ease of use: Development and test. Decision
Sciences, 27(3), 451–481.
Wei, X., Lin, L., Meng, N., Tan, W., & Kong, S. C. (2020). The effectiveness of partial pair programming on elementary school
students’ computational thinking skills and self-efficacy. Computers & Education, 160, 104023.
Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35.
Wing, J. M. (2008). Computational thinking and thinking about computing. Philosophical Transactions of the Royal Society
A: Mathematical, Physical and Engineering Sciences, 366(1881), 3717–3725.
Yılmaz, F. G. K., Yılmaz, R., & Durak, H. Y. (2018). Teaching computational thinking in primary education. IGI Global.
Zysberg, L., & Schwabsky, N. (2021). School climate, academic self-efficacy and student achievement. Educational Psychol-
ogy, 41(4), 467–482.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Tugas Review Artikel Kelompok 1

Diunggah oleh

Informasi Dokumen

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Tugas Review Artikel Kelompok 1

Diunggah oleh

Hak Cipta:

Format Tersedia

REVIEW ARTIKEL INTERNASIONAL

Sebagai Salah Satu Persyaratan untuk Memenuhi Tugas

Dosen Pengampuh : Dr. Hasan S. Panigoro, S.Pd., M.Si

Mamat Abas (703522001)

A Study of the Relationship between Secondary School Students'

Article in Turkish Online Journal of Educational Technology · October 2021

Muhammed Paf Beste Dinçer

SEE PROFILE SEE PROFILE

Computational thinking and creative problem solving skills View project

The user has requested enhancement of the downloaded file.

A Study of the Relationship between Secondary School Students' Computational

Copyright © The Turkish Online Journal of Educational Technology

Copyright © The Turkish Online Journal of Educational Technology

the scope of the investigation.

POPULATION AND SAMPLE

Table 1. Demographic Features of the Students

5 th Class 269 24.5

Status of Following Technological Yes 697 63.5

DATA COLLECTION TOOLS

COMPUTATIONAL THINKING SKILLS SCALE

Copyright © The Turkish Online Journal of Educational Technology

CREATIVE PROBLEM-SOLVING INVENTORY

Creativity 1098 4.07 .72

Copyright © The Turkish Online Journal of Educational Technology

Problem solvin 1098 3.53 .95

THE DIFFERENCE IN COMPUTATIONAL THINKING SKILL LEVELS AMONG

Female 570 4.18 .63

Female 570 3.74 .76

Female 570 4.22 .80

Male 570 3.73 .87

Male 570 3.68 .95

Computational Male 570 3.89 .56

DIFFERENTIATION OF SECONDARY SCHOOL STUDENTS' COMPUTATIONAL

Copyright © The Turkish Online Journal of Educational Technology

Copyright © The Turkish Online Journal of Educational Technology

DIFFERENTIATION OF SECONDARY SCHOOL STUDENTS' COMPUTATIONAL

Yes 697 4.12 .68

Yes 697 3.76 .80

Yes 697 4.12 .84

Yes 697 3.73 .89

Yes 697 3.57 .97

Computational Yes 697 3.83 .57

FINDINGS ON SECONDARY SCHOOL STUDENTS' CREATIVE PROBLEM-SOLVING

Table 6. Descriptive Statistics of Students' Creative Problem-Solving Skill Levels

Divergent Thinking 1098 3.70 .78

Copyright © The Turkish Online Journal of Educational Technology

Motivation 1098 3.75 .81

GENDER DIFFERENCE IN SECONDARY SCHOOL STUDENTS' CREATIVE PROBLEM-

Female 570 3.72 .78

Female 570 3.93 .71

Female 570 3.79 .81

Female 570 4.09 .76

Female 570 3.53 .84

Creative Problem Female 570 3.86 .61

Copyright © The Turkish Online Journal of Educational Technology

DIFFERENCE IN CREATIVE PROBLEM-SOLVING SKILL LEVELS AMONG SECONDARY

Between Groups .229 3 .076 .145 .933

Between Groups .919 3 .306 .462 .709

Between Groups .516 3 .172 .285 .837

General Between Groups 3.059 3 1.020 1.394 .243

Creative Between Groups .353 3 .118 .301 .825

DIFFERENTIATION OF SECONDARY SCHOOL STUDENTS' CREATIVE PROBLEM-

Yes 697 3.81 .76

Yes 697 3.92 .69