Success, activity and drop-outs in MOOCs An exploratory study on the UNED COMA courses

Jose Luis Santos, Joris Klerkx, Erik Duval

Computer Science department, KU Leuven Celestijnenlaan 200 A Leuven, Belgium

David Gago, Luis Rodrguez

Centro Superior para la Enseanza Virtual (CSEV) Avda. de Esparta s/n. Edicio UNED Las Rozas de Madrid, Spain

{joseluis.santos, joris.klerkx, erik.duval} at cs.kuleuven.be ABSTRACT

This paper presents an exploratory study about two language learning MOOCs deployed in the UNED COMA platform. The study identies three research questions: a) How does activity evolve in these MOOCs? b) Are all learning activities relevant?, and c) Does the use of the target language inuence?. We conclude that the MOOC activity drops not only due to the drop-outs. When students skips around 10% of the proposed activities, the percentage of passing the course decrease in a 25%. Forum activity is a useful indicator for success, however the participation in active threads is not. Finally, the use of the target language course is not an indicator to predict success.

{dgago, lrodriguez}@csev.org

Categories and Subject Descriptors

K.3.2 [Computers and Education]: Computer Science Education; J.1 [Administrative Data Processing]: Education

drop-outs, students who are not interested in passing the course but check periodically the learning resources and, nally, students who pass the course [6, 7]. Drop-outs are one of the main concerns in MOOC studies [1]. This paper selected three of the research questions where the analysis of MOOC log data in the UNED platform provided useful insights. Other research questions were discarded because was necessary to track more user interactions. The remainder of this text is structured as follows: next section presents how the courses are structured in order to provide a better understanding of the dataset. Then, we present the results of the analysis. The paper ends with a summary of the conclusions and future work.

2.

UNED COMA

General Terms
Management, Measurement, Performance, Design, Experimentation, Human Factors.

Keywords
Learning analytics, participation, MOOCs, Visualisation

1.

INTRODUCTION

Massive Open Online Courses (MOOCs) give us the opportunity to collect unprecedented volumes of data on student interaction with learning systems as well as to study how the use of social media impacts on the learner [4]. However, student participation in MOOCs is varied. The participation has been categorised in: people who do not even start the course, students who did not nish the course or

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation on the rst page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specic permission and/or a fee. LAK14 March 24 - 28 2014, Indianapolis, IN, USA Copyright 2014 ACM 978-1-4503-2664-3/14/03 $15.00. http://dx.doi.org/10.1145/2567574.2567627 ...$15.00.

UNED COMA is the OpenMooc [2] platform of the Spanish National Distance Education University (UNED). Participants in UNED COMA have access to learning materials in the form of short videos recorded by teachers, selfassessment activities to check progress, and a forum where they can ask questions. Badges are automatically awarded to participants, recognising their progress and contributions to the learning community. In addition to badges, two dierent types of accreditation are available, namely: a) credential: validation of having successfully nished the course and having passed an online test and b) UNED COMA certicate: in collaboration with UNED (after formal f2f test in a UNED centre). The second week of October 2013, there were more than 141,000 unique registrations in the 21 courses available in the platform. So far, more than 5,000 UNED COMA certicates have been awarded. In order to analyse the data in some detail, we focus on two of the language courses: Fundamentals of German for Spanish speakers and Professional English. The German and English language courses had 23065 and 33811 students, respectively. The German language course is for beginners and relied on peer review and questionnaires, and the English language course is for advanced students with student progress assessed through questionnaires only. Around 5% of the students are likely to pass the language courses. These percentages are similar to other non language courses in the same platform.

3.

RESEARCH QUESTIONS

Drop-out rates and social interactions are two elements often discussed in MOOC research [1, 3]. Below, we report on our analysis of these two aspects with the UNED COMA platform. In this section, we discuss three research questions. Every research question contains hypotheses that we have explored through the use of visualisations. Section 3.1 focuses on low completion ratios of MOOCs[1, 3], from two dierent points of view. We compare students who only access activities with students that also take assessments. We also compare the discussion forum activity of students who passed the course with the total students. Section 3.1 results drive our research towards the exploration of how this activity is linked to the learning results. We know from previous studies that the lack use of fora has a relation with failure [9, 10], therefore Section 3.2 explores how activity within the platform may relate with those students who passed the course. Moreover, we also know that the use of the target language in language courses [8] can be relevant. Section 3.3 explores how the use of the language in the fora related to the students.

study of a blended learning course [11]. Hypothesis 3: The percentage of students who pass the course when they follow the course till the end is signicantly higher. Around 25% (3763 students) and 9% (2489 students) started the second activity of the last unit of the German and English course (we exclude the rst activity of the unit, because it breaks the patterns as reported before). So we may assume that these people remain in the course till the end. Considering only these students, the percentage of students who pass the course is up to 40% and 41%. Whilst these percentages are more acceptable than the initial ratios of 3% and 6%.

3.2

RQ2: Are all learning activities relevant?

3.1

RQ1: how does activity evolve?

MOOC studies point out that the activity in forums drops considerably as the course progresses [1]. We know from previous experiments [11] that there also is a drop in activity after the rst week in blended learning courses. In order to analyse this research question, we have created two visualisations per course (see Figure 1) and dened three hypotheses. The rst chart visualises student access to the activities of each unit (blue bar chart) and student access to the assessments (red line chart). The second chart visualises the fora activity per day: in that diagram, the blue line represents the total student activity per day and the red line represents the activity in the fora considering only those students who eventually passed the course. Hypothesis 1: Drops-outs occur at the beginning of the course. Figure 1 shows that both courses share a characteristic: 75% of the students drop-out in the rst three units of the course, so before the students reach the half of the course. Roughly, each unit loses 25% of students until unit 4, when the number of students becomes more or less stable. Moreover, Figure 1 shows that the rst activity of each unit is more visited than the rest of the activities of the same unit and more than the last activities of the previous units. Access to the activities decreases, but the rst activity of each unit breaks this pattern. This can be due to many students accessing only the rst activity of each unit, in order to explore course content. Maybe, here lies a hidden opportunity to reduce the number of drop-outs: MOOCs should pay attention to these rst activities in each unit, in order to motivate and engage students. Hypothesis 2: The drop activity in the fora is not due only to drop-outs. As we reported at the beginning of this section, some studies point out that the reduction of fora activity on fora or social media is due to drop-outs leaving the course [1]. Figure 1 shows that for those students who passed the course (red line), the activity also decreases, following a pattern similar the blue line which includes the drop-outs. This activity drop is not so unusual, we already reported it in a case

Section 3.1 reports that among those students who remain active in the course untill the end, the success rate is around 40%. In this section, we analyse the data in order to nd relevant metrics or common characteristics among those students who passed the course. For example, a study reported that higher social network activity does not imply better performance, however usually lower activity does imply worse performance [9]. Our hypotheses are: Hypothesis 1: There is a threshold on activities done. Above this threshold, students are not more likely to pass the course. In order to explore this hypothesis, we have visualised the percentage of students who passed the course based on the number of the learning activities that they did (see Figure 2).The rst bin in the x-axis includes those students who did less than the total number of activities minus 50. The total number of activities are 130 activities for the English course and 110 activities for the German course. We chose total minus 50 activities as a low threshold because nobody in that bin passed any of the courses. The y-axis in Figure 2 represents the percentages of students who pass the course per bin represented in the x-axis. Consecutive bins increase by 10 activities, until the maximum number of activities in the course. Figure 2 shows that students who did more course activities had higher chances to pass the course. Therefore, we do not nd the threshold hypothesised in hypothesis 1. The more learning activities a student does, the more likely the student will complete the course successfully. This evidence is also supported by statistical analysis when we perform the Spearmans rank correlation test to the individual activity per student. The test result is a signicant positive correlation with p-value<0.001 and =0.32. Indeed, skipping only 10 of the course activities decreases the percentage drastically decreased by 25 points. Therefore, doing all learning activities was a key factor to pass the course. Hypothesis 2: There is a threshold on forum participation. Above this threshold, students are not more likely to pass the course. In order to validate hypothesis 2, we have visualised the percentage of students who passed the course based on their fora activity (see Figure 3). We grouped the students in bins for students who have between 0 and 10 posts, between 10 and 20 posts, between 20 and 30 posts, or more than 30 posts (see x-axis in Figure 3). We then calculated the percentages of students who passed the course in each bin (see y-axis in Figure 3).

Figure 1: Absolute activity and ltered activity visualized along the course

Figure 4: Percentages of passed students per thread bins (English and German Course)

Figure 2: Relation between learning activities and success

English course and p-value<0.001 and = 0.09 for the German course. However, we see in Figure 4 that the last bins contain an unusual high percentage of students who passed the course compared with the previous bins. Therefore, we analysed manually the content of these threads. The last bin in the English course contains only a thread that students used to post solutions to the assignments. In a more fair play way, the last bin in the German course contained a thread which students posted external educational resources. Therefore, neither Figure 4 nor the slightly negative correlation shows that active threads are a relevant metric to explore in our courses.

3.3
Figure 3: Relation between fora activity and success

RQ3: Does the use of the target language in fora inuence learning outcomes?

Figure 3 shows that more activity in the fora correlates with higher likelihood of success. This evidence is also supported by statistical analysis. The Spearmans rank correlation coecient is signicant with p-value<0.001 and = 0.20. However, the increased success ratio attens out quickly which conrms hypothesis 2. Hypothesis 3: Active threads are key threads. If fora activity is a relevant metric for students who passed the course, we considered the possibility that more active threads can have more impact on those students who passed the course. In order to explore this hypothesis, we have visualised in Figure 4 the percentages of passed students per bin. These bins are based on the number of posts that each thread contains, therefore we build up bins from 0 to 10 posts, 10 to 20 post till the 140-150 bin (see x-axis in Figure 4). We then calculated the percentages of students who passed the course in each bin (see y-axis in Figure 4) Figure 4 shows that most of the threads in the German course have a success rate around 50% and in the case of the English course around 30%. However, Figure 4 shows that there is no apparent correlation between active threads and percentages of students who passed the courses. This evidence is also supported by statistical analysis. The Spearmans rank correlation coefcients shows a slightly negative correlation signicant for both courses, with p-value<0.001 and = 0.04 for the

The use of social network tools such forum, wiki and blogs has been widely researched in language courses [10]. The use of the course target language seems to have positive eects [8]. Hypothesis 1: The students who passed the course use more frequently the target language in the fora. In order to explore this hypothesis, we have classied all the messages with the probability that they are written in the target language or Spanish using UClassify [5]. The xaxis in Figure 5 shows the probability of writing in the target language. The y-axis contains the normalised number of students. The red line represents the students who fail and the blue line is the students who pass the course. We can discard the German chart in Figure 5 because a very low percentage of students use German in the fora. The blue line decreases just behind of the red line. Considering the raw data of the German course, only 1% of the users wrote in German with higher probability than 10%. 93% of users wrote in Spanish with higher probability than 90% . Almost, all the students used Spanish as the communication language in the fora. Figure 5 shows that the target language was not used very often in the communications. We expected this in the German course because it is a course that does not require previous knowledge. However it looks unusual in a Professional English course. The evolution of the English course graph in Figure 5 is very similar for the students who pass the courses and those who fail. Therefore, we can say that the use of the target language in media is not correlated with the probability of passing the course.

5.

ACKNOWLEDGMENTS

The research leading to these results has received funding from the European Communitys Seventh Framework Programme (FP7/2007-2013) under grant agreement No 318499 - weSPOT project.

6.

REFERENCES

Figure 5: Comparing success and failure in the English and German language course when students use the target language

Considering the raw data of the English course, only 10% of the students wrote in English with higher probability than 90%, while 67% were likely to use Spanish with a probability higher than 90%. When we perform the Spearmans rank correlation test, we see that the coecients are not strictly signicant (p-value<0.05). When we focus on absolute values, the results do not change much. Only 13% of the students who used english with a higher possibility than 60% passed the course.

4.

CONCLUSIONS

This paper describes an exploratory analysis of two MOOCs in the UNED COMA platform. We have presented that the percentage of students who passed the course excluding drop-outs is around 40%. Regarding the drop-outs, the study points out that 50% of the students drop out during the rst two units. Furthermore, whilst drop-outs are one of the causes of the drop in the fora activity, this pattern is also present when we consider only those students who passed the course as well as in other non MOOCs experiments. Participating in the learning activities and fora correlates with the probabilities of a students passing the course. Around 70% of the students that perform all the activities pass the course. This percentage decreases close to 25% when they only skip 10 learning activities. Moreover, our study shows that students who are above a threshold in fora activity do not gain any additional benecial inuence from extra activities Finally, we have to highlight our surprise that our analysis points out that the use of the target languages does not inuence the probability of passing these courses.

[1] C. Alario-Hoyos, M. P erez-Sanagust n, C. Delgado-Kloos, H. Parada G., M. Mu noz Organero, and A. Rodr guez-de-las Heras. Analysing the Impact of Built-In and External Social Tools in a MOOC on Educational Technologies. In ECTEL13, volume 8095 of Lecture Notes in Computer Science, pages 518. Springer, 2013. [2] M. Castro, S. Martin, D. Gago, and D. Torres. Building communities of practice & learning through moocs exploration. In Proceedings of the EMOOCs 2014, the Second MOOC European Stakeholders Summit. Accepted, 2014. [3] D. Clow. MOOCs and the funnel of participation. In Proceedings of the Third International Conference on Learning Analytics and Knowledge, LAK 13, pages 185189. ACM, 2013. [4] S. Cooper and M. Sahami. Reections on Stanfords MOOCs. Commun. ACM, 56(2):2830, Feb. 2013. [5] W. H. Fletcher. Corpus analysis of the world wide web. The Encyclopedia of Applied Linguistics, 2012. [6] P. Hill. Emerging Student Patterns in MOOCs: A Graphical View, 2013. [7] R. F. Kizilcec, C. Piech, and E. Schneider. Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. In Proceedings of the Third International Conference on Learning Analytics and Knowledge, LAK 13, pages 170179, New York, NY, USA, 2013. ACM. [8] G. S. Levine. Student and Instructor Beliefs and Attitudes about Target Language Use, First Language Use, and Anxiety: Report of a Questionnaire Study. The Modern Language Journal, 87(3):343364, 2003. [9] N. Michinov, S. Brunot, O. Le Bohec, J. Juhel, and M. Delaval. Procrastination, participation, and performance in online learning environments. Comput. Educ., 56(1):243252, Jan. 2011. [10] T. Miyazoe and T. Anderson. Learning outcomes and students perceptions of online writing: Simultaneous implementation of a forum, blog, and wiki in an e blended learning setting. System, 38(2):185 199, 2010. [11] J. Santos, S. Charleer, G. Parra, J. Klerkx, E. Duval, and K. Verbert. Evaluating the use of open badges in an open learning environment. In ECTEL13, volume 8095 of Lecture Notes in Computer Science, pages 314327. Springer, 2013.