Evaluation framework for m-learning systems: Current

situation and proposal

Christian Xavier Navarro Cota

Ana Isabel Molina Daz

Faculty of Engineering, Architecture

and Design, Autonomous University of
Baja California. Ensenada, Mxico
+52 (646) 174 4333

Department of Technologies and

Information System, University of
Castilla-La Mancha. Ciudad Real,
Spain
+34 (926) 295 300

Department of Technologies and

Information System, University of
Castilla-La Mancha. Ciudad Real,
Spain
+34 (926) 295 300

AnaIsabel.Molina@uclm.es

Miguel.Redondo@uclm.es

cnavarro@uabc.edu.mx

ABSTRACT
This paper presents the analysis of recent research on mobile
learning and usability areas, applying a systematic mapping study.
It also presents the main focus adopted by each publication, the
types of mobile device and operative systems. The aim is also to
understand the tendencies and needs in the fields of design and
evaluation of the m-learning systems. Results demonstrate that
research in the area has had a large growth since the 2012, and
due to the recent boom of mobile devices in education, we believe
that the number of studies will continue to grow in the following
years. Also we identify a necessity when we see that not all the mlearning applications have used usability tests, we did not find
guidelines or frameworks to evaluate them either. With this results
and tendencies, we propose the creation of a model that evaluates
the m-learning applications during the development stages,
considering educational factors, usability and student experience,
to improve the quality in use and to improve the student
experience when using mobile learning.

Categories and Subject Descriptors

K.3 [Computers and Education]: Computer Uses in Education
Distance learning.
C.5 [Computer System Implementation]: Microcomputers
Portable devices.

General Terms
Design, Measurement, Documentation, Human Factors.

Keywords
m-learning, aprendizaje mvil, usabilidad, mapeo sistemtico.

1. INTRODUCTION
Nowadays mobile devices are part of our culture and lifestyle.
These devices have achieved an historical access to
communication
and
information.
The
International
Telecommunications Union has published in their report The
World in 2013: ICT Facts and Figures a prediction that affirms
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
Interaccion14, Sep 1012, 2014, Puerto de la Cruz, Tenerife, Spain.
Copyright 2014 ACM 978-1-4503-2880-7/12/10$15.00.
DOI: http://dx.doi.org/10.1145/2662253.2662265

Miguel ngel Redondo Duque

that soon there will be as many subscriptions to mobile

technology as the number of people in the world, with a total of
6.8 billion subscribers at the time, and with a projection that this
year there will be more than 7 billion users [3].
These devices are easy, flexible and constantly improving, which
allows us to believe that there are so many new possibilities for
users to take benefit from, and one of these areas is education.
Mobile devices are also useful tools that go with us everywere;
this could be perfect for any student in a learning process. The
information is available at any time and context. Even, UNESCO
has considerate mobile devices as a platform to allow equal
opportunities in education among different countries. In the other
hand, it is registered that the publication of papers related to mlearning, has increased during the last years.
However, mobile devices present a challenge due to the size of
their screens and the amount of information that can be displayed;
this shows us a challenge in the area, related with the usability,
because of the difficulties that could be generated during the
learning process. For these reasons, this paper presents the recent
situation of research in the area of usability on m-learning. We
identify the different approaches of research papers in the area of
m-learning, as well as the most used mobile devices and operative
systems. The main goal is to understand the tendencies and
opportunity areas within m-learning, through a systematic
mapping study. Also, we present a proposal to create an
evaluation model for m-learning applications, to be used on the
development stages, considering educational factors, usability and
student experience, to improve the quality in use of the mlearning applications.

2. SYSTEMATIC MAPPING STUDY:

USABILITY AND M-LEARNING
Considering the facts stated in this document, we can see that
usability is important, when we talk about mobile computing and
in general, as well as when we approach it in an education context
like m-learning. For this reason, part of this work focuses on
making a research on the existing literature on this topic, using a
systematic mapping study. This methodology will allow us to
evaluate, and interpret all the papers relevant to mobile usability
and m-learning [2], making an emphasis in mobile devices like
smartphones and tablets.
Because our main interest is to know the current status of research
on m-learning and mobile usability areas, we have outlined the
following research questions:

1.

What is the current state of m-learning and mobile usability

research? (Considering only papers using mobile devices like
smartphones and tablets). Is there an increase in the number
of publications during the last years?

2.

What are the different approaches for publications in the

areas of m-learning and usability? What kind of mobile
devices and operative systems are the most used according to
these publications?

In order to represent a full description of the classification of

articles within this research, we present a scheme with the
different categories of the analyzed papers, up to the point where
we conclude that there arent guidelines, frameworks or tools that
evaluate the educational factors and usability in the m-learning
applications (Figure 1, classification 4).
With this information we can see that there is a lack of evaluation
tools for m-learning applications, in the following section we
propose a model to develop and evaluate m-learning applications,
considering usability, learning aspects and the student experience.

Clasification 1
(different
approaches)

M-learning
applications
(30)

Analyzed
papers(68)

Clasification
2
(Types of
Guidelines
and
Frameworks)

Development
Guidelines
and
Frameworks
(16)

- M- learning pps (5)

- M-learning
environments (4)
- MLMS (3)
- M-learning contents
(2)
- Mobile devices
contents (1)
- Mobile environments
(1)

Evaluation
Guidelines
and
Frameworks
(9)

- Mobile applications (3)

- M-learning pplications
(2)
- Comercial m-learning
applications(1)
- Usability contents (1)
- M-learning devices(1)
- Mobile environments
(1)

Guidelines and
Frameworks
(25)
Specific aspects
of m-learning
(9)
Analysis and
tendencies of
m-learning (4)

Clasification 3
(different areas of
development and
evaluation)

Clasification 4
(papers that consider
both m-learning and
usability)

- M- learning
environments (4)
- M-learning
applications(3)
- MLMS (3)
- M-learning
contents(2)

being so recent, like MOOCs, tablets, game based learning, and

learning analytics. In fact, all of these interact with the mobile
devices, or it is probable that they will be doing it soon.
Therefore, the usability becomes an important matter to consider
when we talk about m-learning, due to the limitations associated
with the mobile devices.
This state of the art analysis has allowed us to have a general
vision about m-learning, the tendencies and inclusion in
education. The results of the systematic mapping study shows the
number of papers in the area have increased significantly since
2012, and due to the boom of mobile devices in education, we
believe that the publications on the theme will continue increasing
in the following years. We can also see that only 57% of the
analyzed applications used usability test, and only 35% are based
in heuristic evaluations or revised by experts. When we
performed this study we also found that there are neither
guidelines nor frameworks of evaluation for these types of
systems.
Based on these results, we propose a model that can evaluate the
m-learning applications during the different phases of
development, considering learning factors, usability and the
student experience, to improve the quality in use, and to improve
the learning experience in the m-learning environments. We know
that m-learning applications are up to date available options to
assist the students learning process, therefore our model will allow
the evaluation of the quality of the applications to improve the
learning experience of the students.

- Mobile devices
evaluation(1)

Figure 1. Sequence of classification of all the papers

considered for this study

3. PROPOSAL OF AN EVALUATION
MODEL FOR M-LEARNING SYSTEMS
The proposed model is based in the methodology developed by
Granollers [1], because it considers users through the software
development stages, their concerns, needs and limitations. The
generation of prototypes and evaluation are two key phases that
are applied until the users fulfills their expectations (Figure 2).

4. CONCLUSION
For the scholars in the area, m-learning is a concept in evolution.
This definition has suffered a few changes, until reaching the most
recent, more centered in the students and their learning process.
Also m-learning is part of the new learning tendencies, together
with other technologies that have been widely accepted despite

Figure 2. M learning system evaluation model proposal

5. REFERENCES
[1]

Granollers, T. User Centred Design Process Model.

Integration of Usability Engineering and Software
Engineering. INTERACT 2003 (Doctoral Consortium),
Zurich (Switzerland).

[2] Kitchenham, B. Guidelines for performing Systematic

Literature Reviews in Software Engineering. EBSE
Technical Report EBSE-2007-01, 2007.

[3] Unin Internacional de Telecomunicaciones (ITU). The

World in 2013: ICT Facts and Figure