This article was downloaded by: [Escuela Sup Pol del Ejercito ]

On: 02 September 2013, At: 06:40

Publisher: Taylor & Francis
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,
37-41 Mortimer Street, London W1T 3JH, UK

International Journal of Human-Computer Interaction

Publication details, including instructions for authors and subscription information:
http://www.tandfonline.com/loi/hihc20

Usability Principles for Augmented Reality Applications

in a Smartphone Environment
a

Sang Min Ko , Won Suk Chang & Yong Gu Ji

Yonsei University , Seoul , Korea

Published online: 10 Jun 2013.

To cite this article: Sang Min Ko , Won Suk Chang & Yong Gu Ji (2013) Usability Principles for Augmented Reality
Applications in a Smartphone Environment, International Journal of Human-Computer Interaction, 29:8, 501-515, DOI:
10.1080/10447318.2012.722466
To link to this article: http://dx.doi.org/10.1080/10447318.2012.722466

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the Content) contained
in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no
representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the
Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and
are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and
should be independently verified with primary sources of information. Taylor and Francis shall not be liable for
any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever
or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of
the Content.
This article may be used for research, teaching, and private study purposes. Any substantial or systematic
reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any
form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://
www.tandfonline.com/page/terms-and-conditions

Intl. Journal of HumanComputer Interaction, 29: 501515, 2013

Copyright Taylor & Francis Group, LLC
ISSN: 1044-7318 print / 1532-7590 online
DOI: 10.1080/10447318.2012.722466

Usability Principles for Augmented Reality Applications

in a Smartphone Environment
Sang Min Ko, Won Suk Chang, and Yong Gu Ji

Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

Yonsei University, Seoul, Korea

Through the rapid spread of smartphones, users have access to

many types of applications similar to those on desktop computer
systems. Smartphone applications using augmented reality (AR)
technology make use of users location information. As AR applications will require new evaluation methods, improved usability
and user convenience should be developed. The purpose of the current study is to develop usability principles for the development
and evaluation of smartphone applications using AR technology.
We develop usability principles for smartphone AR applications
by analyzing existing research about heuristic evaluation methods,
design principles for AR systems, guidelines for handheld mobile
device interfaces, and usability principles for the tangible user
interface. We conducted a heuristic evaluation for three popularly
used smartphone AR applications to identify usability problems.
We suggested new design guidelines to solve the identified problems. Then, we developed an improved AR application prototype
of an Android-based smartphone, which later was conducted a
usability testing to validate the effects of usability principles.

1. INTRODUCTION
Augmented reality (AR) is a view of a real-world environment that is modified by a computer. It is a subset of
virtual reality (VR) but differs from VR in that it offers a
greater sense of realism to its users. The difference between
AR and VR is that AR overlaps virtual information with reality,
whereas VR substitutes reality. This technology offers information delivery, acquisition, and application by observing objects.
Early AR-related technologies were developed and studied for
use with industry, military readiness, surgery training, computer games, and computer-supported collaboration. Research
on AR-related technology is ongoing as the spread of small
handheld devices and smartphones increases (Azuma et al.,
A preliminary version of this article was published in Korean in
the Journal of Korean Institute of CALS/EC, 16, 3547, 2011. This
work was supported by Basic Science Research Program through the
National Research Foundation of Korea funded by the Ministry of
Education, Science and Technology (2012R1A1A2006949).
Address correspondence to Yong Gu Ji, Department of Information
and Industrial Engineering, Yonsei University, 262 Seongsanno,
Seodaemun-gu, Seoul, 120-749, Korea. E-mail: yongguji@
yonsei.ac.kr

2001; Billinghurst, Belcher, Gupta, & Kiyokawa, 2003; Bruns,

Brombach, & Bimber, 2008; Livingston et al., 2002; Nassar &
Meawad, 2010; Thomas et al., 2002; Wiedenmaier, Oehme,
Schmidt, & Luczak, 2003).
The latest high-end smartphones offer global positioning
systems (GPS), which allow the user to track their location,
as well as geo-magnetic sensors, G-sensors, and digital compasses. They also include high-performance processors and
high-resolution digital cameras and displays. Moreover, these
phones can retrieve information regarding a users current location. These properties of smartphones satisfy the requirements
for an AR system, which needs to recognize objects using a
camera by detecting object properties, provide related content
by obtaining information related to the users location information, and handle many types of data at the same time. Unlike
the traditional personal computer environment, the AR environment requires new considerations for user. AR is usually
used while the user is mobile and subject to many distractions,
unlike the traditional personal computer (Chincholle, Goldstein,
Nyberg, & Eriksson, 2002). When using AR on mobile phones,
users can be distracted by disordered displays, complicated
execution, inappropriate functions, and inconsistency between
manipulation methods. However, most of these distractions are
manageable (Shneiderman, 2010).
Studies regarding AR usually include precision devices and
software technologies for special situations. However, because
these studies are led by technologies, usability researchers point
out that they lack of consideration of user convenience and
usability (Dnser, Grasset, & Billinghurst, 2008; Livingston,
2005; Nilsson & Johansson, 2006). Existing studies have usually evaluated the aspects of cognition and performance (Dnser
et al., 2008) and have not considered the special contexts of AR
environments (Ullmer & Ishii, 2000).
In this research, we studied usability principles for
smartphone AR applications based on user location information. We then conducted heuristic evaluations using these
principles with people in Korea using smartphone-augmented
applications. Through this evaluation, we developed improved
smartphone AR application prototypes and validated results
from the heuristic evaluation by conducting usability testing.

501

502

S. M. KO ET AL.

Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

2. LITERATURE REVIEW
2.1. Augmented Reality
Concepts and Characteristics of Augmented Reality
AR overlaps virtual information with reality, whereas VR
provides three-dimensional (3D) images based on computer
information. Thus, AR provides a greater sense of realism
than VR can. Studies on AR began after Ivan Sutherland published his 1968 article A Head-Mounted Three Dimensional
Display. The term augmented reality is reputed to have been
used for the first time in the early 1990s by Tom Caudell at
Boeing in his system development research of wire assembly
work for flight manufacturers (Caudell & Mizell, 1992).
When comparing the concept of VR to the concept of
AR, AR is closer to reality because VR can only offer virtual information to users. There have been many arguments
for and against the efficiency of AR. In 1994, Paul Milgram
differentiated reality from virtuality. He divided AR from augmented virtuality (AV) between virtuality and reality. Milgram
and Kishino (1994) distinguished virtuality, reality, and
in the middle of virtuality and reality by using the term the
realityvirtuality continuum (Milgram & Colquhoun, 1999;
Milgram & Kishino, 1994; see Figure 1). Since then, AR has
been recognized as an independent field of study.According to
the concept of the realityvirtuality continuum, users recognize
less and less information in reality as they get closer to virtuality. Mixed reality (MR), which is between virtuality and reality,
can be roughly divided into two categories: AR and AV. AV is
distinguished from AR by the fact that insertion and background
in AV is reflected by the computer-generated environment.
In 1997, Ronald Azuma published a survey on AR and he
defined AR in 2001 as follows: An AR system supplements
the real world with virtual (computer-generated) objects that
appear to coexist in the same space as the real world (p. 34).
He also defined three representative properties of AR: the necessity of combination between reality and virtuality, the necessity
of interaction in real time, and the necessity of observing the
real world in 3D. To date, the definition of AR by Milgram
and Azuma is accepted by many researchers and is the most
important standard for AR.
The concept of AR can be applied to many fields, but most
research has focused on human eyesight. Through AR systems,

FIG. 1.

users are provided with additional real-time 3D graphics with

an original view of reality. That is, the system provides rendered virtual images to users while maintaining spatial relation
between reality objects and virtual objects followed by the
user view (Henrysson, 2007). Until the mid-2000s, AR studies remained limited to R&D and trial applications. However,
the appearance of smartphones with highly efficient cameras,
graphic-processing capabilities, wireless communication function, and GPS elevated the importance of AR studies (Fckler,
Zeidler, Brombach, Bruns, & Bimber, 2005; Miyashita et al.,
2008; Morrison et al., 2009; Nassar & Meawad, 2010).

Augmented Reality in a Mobile Environment

There is no commonly used formal definition for mobile AR.
In earlier studies, an AR system referred to a system constructed
on a laptop-based system that provided mobility to users. Later,
the definition expanded to a system constructed on a handheld
device. Nowadays the definition of mobile AR has expanded
to include other mobile devices (e.g., smartphones, tablets, and
other devices) that have AR systems (Jeon & Lee, 2011).
AR services provided in the mobile environment can be
classified based on their object-tracking methods as similar,
marker-based, and vision-based AR methods (Shin, Oh, Suh, &
Woo, 2010). The similar AR method identifies augmented locations and visualizes relevant contents by using the sensors of
a mobile device such as GPS or a digital compass. This is
the most common method, because smartphones are usually
equipped with digital sensors. The origin of mobile similar
AR is MARS (Mobile Augmented Reality System). MARS is
a mobile tour guide AR service system that provides a variety of information based on the users location and direction
through a Head-Mounted Display (Hllerer, Feiner, Terauchi,
Rashid, & Hallaway, 1999). Marker-based AR methods augment digital information by identifying objects and locations
using markers associated with the objects to be recognized by
the camera in a mobile device. A disadvantage of marker-based
AR is that the marker cannot be recognized by other sensors
such as digital compasses or GPS. Invisible-marker-based AR
such as that using infrared marker has been studied as a supplement to current marker-based AR problems (Park & Park,
2010). Vision-based AR methods are ways to augment digital

RealityVirtuality Continuum (modified from Milgram & Kishino, 1994; 1999) (color figure available online).

Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

USABILITY FOR AUGMENTED REALITY APPLICATIONS

information by identifying the properties of objects and tracking them after identification. These methods have the advantage
of being able to track objects by recognizing them without a
marker but have the disadvantage of being difficult to implement because they require real-time image processing and
augmentation.
Smartphone AR applications are used in providing locationbased services (LBS), which make use of the location information of the mobile device. Virrantaus et al. (2001) defined
LBS as services accessible with mobile devices through the
mobile network and utilizing the ability to make use of the
location of the terminals (p. 66). These characteristics of
smartphone AR applications have some limitations. In the
marker-based AR method, the marker should be installed additionally. In the vision-based AR method, there are some technical limitations such as real-time video processing and object
recognition.
One of the currently best-known applications is called Layer.
This application provides LBS through smartphones and is
used worldwide in iPhones and Android phones. This application augments digital contents related to camera images using
location and direction information (Shin et al., 2010; VaughanNichols, 2009). Each layer refers to a function such as allowing
users to search for a coffee shop nearby. It also allows users to
create custom layers and share them with other users.

2.2. Usability of Augmented Reality Applications

Related Studies
Swan and Gabbard (2005) compiled usability research on
AR and found that only 14.3% (38 papers out of 266) mentioned
user-centered design in the perspective on humancomputer
interaction (HCI) and only 7.9% (21 papers out of 266) had
conducted experiments in terms of general-use users (Swan &
Gabbard, 2005). Swan and Gabbard categorized user-centered
experiments related to AR into three categories. The first category included experiments of which the main purpose was
to understand the recognition and perception of users AR
operations. The second category of experiments was those measuring the task performance of users during use of specific
AR applications. The last category of experiments included
those seeking to interact and communicate with AR users.
They pointed out that there is a lack of research regarding
user interface and interaction from a user-centeric point of
view. They also pointed out the necessity for additional studies
regarding AR.
Dnser expanded on Swan and Gabbards work even further. He examined the AR-related studies that had been done
up until 2007 (Dnser et al., 2008). Only 10% of 161 studies had conducted user-performed AR experiments. This review
also found that there were only 41 studies on actual usability,
excluding studies on recognition and perception point of view
and performance. The authors pointed out that the lack of
studies regarding the AR user experience has led to there being

503

insufficient training regarding how to evaluate AR experiences,

how to design experiments, how to select and apply proper
methods, and how to analyze the results from experiments.

Usability Features of Augmented Reality Applications

We studied existing research on AR applications and categorized existing problems into four different categories. The
studies reviewed involved smartphone AR applications, which
provide services based on users location information and the
usability problems of these applications.
Augmented reality interacting with a small display device.
There have been many studies regarding improving usability
while reading and interacting with small-display devices (Jones,
2006). Gabbard and Swan Ii (2008) performed user tests on
information expression for implementing AR in a mobile environment. They observed problems due to the small size of the
displays on mobile devices. They found that in the expression
of information over reality, a small display gives too much
information compared with the small size (Gabbard & Swan
Ii, 2008). To implement AR, actual reality and the augmented
digital contents should be expressed on the display. There exists
the possibility that AR may be too complex for current mobile
devices because the interface might have too many contents and
menus to manipulate.
Multimodal interface. AR is closely related to tangible user
interfaces (TUIs) while still maintaining properties of a graphic
user interface (GUI) for the purpose of information searching
and transmission (Nilsson & Johansson, 2006). For users to use
AR applications on their smartphones, they will need to acquire
reality information and track objects with their smartphone
cameras to obtain images. Therefore, consideration of various
user interfaces, including GUI and TUI, will be necessary.
Limited manipulation. According to Henrysson (2007), the
typically studied AR methods have distinguishing features.
Virtual information in these AR systems could be manipulated with both hands with the use of a head-mounted display.
However, in a smartphone environment, one hand is required for
holding the device and the other hand is used for manipulating
the display (Henrysson, 2007). Therefore, it will be necessary to
develop new interaction methods for mobile devices that make
use of one-handed manipulation.
User on the go. In the process of conducting tasks such as
searching for directions, users recognition and attention could
be dispersed because of the properties of location-based services described by Chincholle et al. (2002) as user on the
go properties. They also studied how smartphones can provide location-based services, and pointed out that mobile users
do not have sufficient patience to learn new operating systems
(Chincholle et al., 2002). Location-based services should not
obstruct the ongoing tasks (e.g., driving, walking, etc.) of users.
During these activities, time is an extremely important factor to
the user. Therefore, the manipulation of location-based services
should be simple.

Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

504

S. M. KO ET AL.

3. DEVELOPMENT OF USABILITY PRINCIPLES FOR

AUGMENTED REALITY APPLICATION
In this study we developed usability principles for
smartphone AR application and conducted a heuristic evaluation to identify usability problems. Then, we developed a
prototype and conducted a usability testing to validate the
effects of usability principles. The research framework of this
study is illustrated in Figure 2. The research framework can be
divided into two phases.
In Phase 1, we collected usability principles through the literature review and conducted an expert meeting to discuss the
collected usability principles. Next, we classified usability principles through the principle component analysis. In Phase 2, we
conducted heuristic evaluation of smartphone AR applications
using the classified usability principles developed in Phase 1.
Then, we implemented an improved prototype of a smartphone
AR application, which later was conducted a usability testing.

3.1. Usability Principles for Augmented Reality

Application
In this study, we present principles for new forms of usability
evaluation by considering the properties of smartphone AR

FIG. 2.

applications as discussed in section 2.2. To do so, we collected

usability principles by reviewing and analyzing existing studies
in order to develop suitable usability principles.
The purpose of this study is to develop suitable usability
principles for smartphone AR application services based on
users location information. We collected existing usability
principles from the studies of Atkinson, Bennett, Bahr, and
Nelson (2007); Dnser, Grasset, Seichter, and Billinghurst
(2007); Gong and Tarasewich (2004); and Kim et al. (2007).
See Table 1. Atkinson et al. (2007) reorganized the heuristic
evaluation methods of four authoritative researchers (Nielsen,
Shneiderman, Tognazzini, and Tufte) and developed a multiple heuristics evaluation table. They identified overlap and
combined conceptually related concepts in order to simplify a
comprehensive set of heuristics. We considered this study as
an adequate preliminary study of heuristic evaluation involving smartphone AR and usability principles. In our study, we
selected 12 usability principles, including softwareuser interaction, learnability, and cognition facilitation (Atkinson et al.,
2007).
Dnser et al. (2007) investigated general HCI principles
related to AR application design. They found that AR-related
studies have been primarily technology driven and may focus

Research framework. Note. AR = augmented reality.

TABLE 1
Collected Usability Principles
References
Atkinson et al. (2007)

Dnser et al. (2007)

Gong and Tarasewich
(2004)

Kim et al. (2007)

Usability Principles
Cognition facilitation, Consistency, Defaults, Error management, Graphic design, Help and
documentation, Learnability, Navigation and exiting, Software-user interaction, System-real
world match, System-software interaction, User control and software flexibility
Affordance, Error tolerance, Flexibility in use, Learnability, Low physical effort, Reducing
cognitive overhead, Responsiveness and feedback, User satisfaction
Allow for personalization, Consistency, Design dialogs to yield closure, Design for top-down
interaction, Design for enjoyment, Design for limited and split attention, Design for multimodal
interfaces, Design for multiple and dynamic contexts, Design for speed and recovery, Enable
frequent users to use shortcuts, Error prevention and simple error handling, Offer informative
feedback, Reduce short-term memory load, Reversal of actions, Support internal locus of control
Accuracy, Arrangement, Attractiveness, Consistency, Controllability, Direct manipulation,
Durability, Ease, Effectiveness, Efficiency, Error indication, Familiarity, Feedback, Graspability,
Learnability, Match between system and real world, Pleasantness, Predictability, Prevention,
Safety, Simplicity, Size, Subject satisfaction, Tolerance principle, User control

Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

USABILITY FOR AUGMENTED REALITY APPLICATIONS

excessively on hardware and software issues. They suggest that,

to develop successful AR systems, studies need to be properly
focused on user-centered design principles as well as technical
issues. They searched various fields for HCI design principles
and guidelines to improve AR usability and chose appropriate design principles for AR systems based on these findings.
They then validated their studies by providing existing studies
and examples that explain the relationship between the design
principles and properties of AR systems. For these reasons, the
Dnser et al. (2007) study is suitable to present adequate preliminary data for researchers wanting to study the usability of
AR applications. In this study, we selected eight usability principles, including affordability, reducing cognitive overhead, low
physical effort, and so on (Dnser et al., 2007).
Gong and Tarasewich (2004) discussed the features and limitations of mobile device interfaces and compared them to a
desktop environment. They presented guidelines for a handheld mobile device interface based on desktop user interfaces
and research on mobile user interfaces and usability. These
design guidelines are helpful to HCI researchers and practitioners who are working on mobile usability and interface design.
Their study is a precursor to our study on the usability of
smartphone AR applications in a mobile environment. For our
study, we selected 15 usability goals, including enable frequent users to use shortcuts, offer informative feedback,
and design dialogs to yield closure (Gong & Tarasewich,
2004).
TUIs imitate basic human senses and behavior, including
touch and feel (Ishii, 2008). AR connects digital information
to objects between actual space and virtual space and controls digital information through the use of real objects. When
considering these properties, TUI is an interface adequate for
AR. The gesture, which expands or reduces objects on a touch
screen, is a representative example of interaction with a TUI.
Kim et al. (2008) suggested usability principles and an evaluation framework for TUIs to evaluate usability and identify
design issues of TUIs. In their study, the researchers reviewed
existing usability-related studies and suggested TUI-adequate
principles through retrospective analysis. Their study is suitable to present suitable preliminary data for considering the
properties of TUIs for use with smartphone AR. In our study,
we selected 25 usability principles, including user control,
controllability, and feedback (Kim, Kim, Chio, & Ji, 2008).
We conducted an expert meeting to discuss standards for
61 usability principles from Atkinson et al. (2007), Dnser
et al. (2007), Gong and Tarasewich (2004), and Kim et al.
(2008). We selected 61 usability principles that directly impact
the properties and performance of smartphone AR applications. We applied the following standards to these 61 selected
usability principles to decide on the final usability principles.
We selected and integrated these usability principles according
to deletion standards, which depended on whether the principles
were objective or subjective and whether they were repetitive (see Table 2). Finally, we selected 22 usability principles

505

TABLE 2
Deleted and Integrated Usability Principles
Availability, Consistency, Context-based, Defaults, Direct
manipulation, Enjoyment, Error management, Exiting,
Familiarity, Feedback, Help and documentation, Hierarchy,
Learnability, Low physical effort, Multimodality,
Navigation, Personalization, Predictability, Recognition,
Responsiveness, User control, Visibility

by reviewing the relationships among these principles. In the

example of error management, users could easily perform
supported tasks if predictable errors were prevented. Error management directly affects work performance and should be part
of selection standards. Next, we developed a classification system for usability principles. First, we developed an interrelation
matrix to prove relationships between each usability principle.
If there was a relationship between a pair of usability principles,
it was indicated with a 2. If the relationship was ambiguous, this
was indicated with a 1. A 0 indicated that no relationship existed
between a pair of usability principles. Ten experts who each had
at least 2 years of experience in UI and UX fields participated.
The result of this process is shown in Table 3.
In this study, each factor included principles with factor
loading of at least 0.6. The low physical effort (factor loading
<0.28) principle was accepted by a majority of the participants. It was agreed that this factor it is essential and should be
included due to the properties of AR applications, which must
be controlled by smartphone users.As a result of the analysis
of the main factors, usability principles were classified into five
different groups: user-information, user-cognitive, user-support,
user-interaction, and user-usage (see Figure 3). The definitions
of these groups are as follows.
1. User-information: The user-information group consists of
principles related to providing information to users such as
visual information, classified menu structure, and familiarity
with expression.
2. User-cognitive: The user-cognitive group consists of principles related to cognitive aspects required for users to minimize memory loads, react as expected, and learn applications
easily.
3. User-support: The user-support group consists of principles related to user support: providing useful information,
reducing errors, handling, and personalizing.
4. User-interaction: The user-interaction group consists of
principles related to interaction between users and applications, such as providing feedback with minimum
manipulation.
5. User-usage: The user-usage group consists of principles
related to actual usage, which include reactions appropriate
for surrounding situations and methods to use or stop the
applications easily.

506

S. M. KO ET AL.

TABLE 3
Results Obtained From a Principal Component Analysis With
Varimax Rotation
Factors

Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

Principles
Multimodality
Enjoyment
Familiarity
Visibility
Hierarchy
Defaults
Recognition
Predictability
Learnability
Consistency
Error management
Help and
documentation
User control
Personalization
Feedback
Direct manipulation
Responsiveness
Low physical effort
Context based
Exiting
Navigation
Availability

.884
.819
.817
.789
.681
.619
.905
.886
.880
.853
.870
.853
.744
.652

FIG. 3.

.925
.707
.701
.282
.822
.816
.603
.603

Remaining principles after adjustment.

The usability principles classification system is shown in

Table 4.
3.2. Validation for the Developed Usability Principles
From the obtained usability principles from this research, we
investigated the usability problem of smartphone AR application based on heuristic evaluation. Moreover, we proposed a
prototype by applying improvements from usability problem
obtained by heuristic evaluation. Then user-based evaluation
was conducted to validate the research. The validation of this
research was done based on previous research methodology
which mostly conducted heuristic evaluation and user-based
evaluation of the improvements obtained from the evaluation
(Allen, Currie, Bakken, Patel, & Cimino, 2006; Rau & Liang,
2003).
Heuristic Evaluation With Developed Usability Principles
Heuristic evaluation is a usability inspection method that
helps to identify usability problems. This is an informal method
that involves usability specialists judge where each dialogue

Structured usability principles.

element follows established usability principles (Nielsen &

Mack, 1994). There are some studies that validate effectiveness
of heuristic evaluation, organize usability principle, and conduct a heuristic evaluation with experts to suggest applicability
on the result of the study (Desurvire, Caplan, & Toth, 2004;
Hvannberg, Law, & Lrusdttir, 2007; Kostaras & Xenos, 2007;
Kurniawan & Zaphiris, 2005; Sutcliffe & Gault, 2004; Tan,
Liu, & Bishu, 2009; Tang, Johnson, Tindall, & Zhang, 2006).
We developed usability principles specialized for the evaluation of smartphone AR applications. As Law and Hvannberg
(2004) pointed out, expertise is required to avoid omitting
problems or having failed deliveries when the usability evaluation is completed by someone unfamiliar with the process
(Law & Hvannberg, 2004). Therefore, it is necessary to classify
heuristic evaluations using general-use heuristic evaluations.
In this study, we defined evaluations that were found by experts
to be heuristic evaluations. Usability principles refer to principles developed for the evaluation of smartphone AR applications. We also conducted heuristic evaluations of smartphone
AR applications using the 22 usability principles developed in
the previous section.
For the heuristic evaluation, we selected three applications
that provide services based on users location information.
These three applications use AR technology now in use in
Korea. The applications have been developed for both Android
and iOS. They are all recently launched, popular applications
in Korea that provide various functions based on users location
information. These three applications are free downloads from
Android Market (Google Play) or iTunes. They provide search
services, share information, and provide AR functions based on
search information. Some applications provide SNS functions
such as finding and recommending related people through

USABILITY FOR AUGMENTED REALITY APPLICATIONS

507

TABLE 4
Usability Principles Developed for the Evaluation and for the Development of AR Application
Principles
User-Information

Definition
Defaults

Enjoyment
Familiarity
Hierarchy

Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

Multi-modality

User-Cognitive

Visibility
Consistency
Learnability
Predictability
Recognition

User-Support

Error management
Help and documentation
Personalization
User control

User-Interaction

Direct manipulation
Feedback
Low physical effort

User-Usage

Responsiveness
Availability
Context-based
Exiting
Navigation

The initial establishment should be operated easily by the users. Also, the
frame designating the input space and the instances related to the form of
input should be provided.
An aesthetic design including colors should be used to provide exciting
experiences to users.
Not only familiar metaphors and icons but also user-centered languages
should be used.
If the quantity of information is large, the information should provide a
phased design to the users to make it easier to utilize.
A modality such as sound as well as a visual screen should be provided
when information is provided.
The graphic factors should be designed properly.
The generally used terms and interfaces should be maintained consistently
to prevent confusion.
The functions and the features of the application should be effective and
easy to learn for users.
The functions and the features of the application should be effective and
easy to learn for users.
Necessary information should be provided properly so users dont have to
use short-term memory.
The errors that occur while using applications should be supported by the
method of prevention and solution.
Appropriate Help should be provided to support user applications.
The interface should be easy enough to be modified by the users tastes and
specialties.
It should make the users feel that they are controlling the system and the
system is responding to their actions.
When the users control the device, the information appearing on the screen
and the action of the users should be matched intuitively.
The sequence of the process and the state of system should be consistently
provided to the users.
It should minimize the effort of operating the application and the tiredness
of the users.
It should react quickly to the action of the users.
The application should have rapid initial operating speed and the previous
working state and options should remain the same after reoperation.
The user-interface should be designed considering various kinds of
environments and designed to correspond to the using environments.
To stop working or to go back to previous working parts should be easy.
The users should be allowed to navigate freely on the application when they
are performing their work.

pattern analysis. They also provide linkage to smartphones

and social commerce. According to Android Market, these
three applications have been downloaded at least 1 million
times and possibly up to 10 million times. This study focused
on conducting a heuristic evaluation of the functions and

properties of popularly used AR applications. We selected three

applications for Android and iOS (OVJET, ScanSearch, and
Qook Town).
According to previous studies three to five evaluators were
sufficient to find the majority of usability problems (Faulkner,

508

S. M. KO ET AL.

Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

2003; Nielsen & Landauer, 1993). With this in mind, we

choose seven participants (five male, two female; average age =
29.1) who have used smartphones and had at least 2 years
of work experience in the field of UI design or usability
evaluation.
We applied usability principles specialized for smartphone
AR environments (domain specific). Experts conducted
heuristic evaluation using an evaluation form that included
the seven attributes suggested by Cockton and Woolrych in
2001. Prior to the evaluation, we carried out an orientation so
the experts would understand each usability principle and the
evaluation form (see Table 5).
TABLE 5
Forms for Organizing the Problems
1
2
3
4
5
6
7

Divisions of the problem numbers

Simple explanation of the problems
Difficulties expected for the users
Specific problem situations (context)
Problem factors
Usability principles which should be applied
Seriousness of problems (serious, normal, trivial)

As a result of heuristic evaluation, we drew 208 problems from the three applications. After exclusion of duplicated
problems, 96 problem lists remained. The importance of each
problem was estimated by scoring its seriousness: 3 (serious), 2
(normal) and 1 (trivial). The top problems of the applications in
order of importance are as shown in Table 6.
The three applications had six common problems out of
the top ten problems. The six common problems are shown in
Table 7.
In this study, we presented design guidelines to improve
the problems analyzed from the experts problem lists. These
design guidelines were made based on the cause of the problems along with context and related usability principles. The
completed guidelines are as in Table 8.
The purpose of this study was to develop usability principles
for smartphone AR application properties based on previous
usability-related studies. To accomplish this, we reviewed previous HCI usability studies and smartphone environments. Our
study also referred to studies that applied previous usability
principles to mobile devices. To understand the TUI properties
of AR, we reviewed studies of TUI usability and smartphone
GUIs. We developed usability principles that will take on two
roles in the application development process. One role will
be to determine the direction of smartphone AR application

TABLE 6
Final Usability Problem Lists and Obtained Measurement of Importance Drawn From Heuristic Evaluation
Applications
No.
1
2
3
4
5
6
7
8

9
10

Problem Lists

Difficulties of selection and expression on icons and text occur when presenting the result of
searching positional information because the application shows duplicate results on the screen.
Only limited essential information is provided to the users when they see the result of searching
positional information.
Confusion can result due to lack of explanation of icons which are unfamiliar to the users.
When the users search for positional information, they are not provided with functions to
expand their search range. They are also provided with insufficient options.
Sometimes explanation about a menu is located in other place like options and sometimes there
is a lack of explanation about the menu. These problems make beginner users uncomfortable.
When the application is expected to be operated by one hand (especially the left hand), menus
are located on one side (e.g., left side) and icons are too small to operate.
Complicated technical terms make the users frustrated and cause misunderstanding when they
are trying to set the options.
When the users want to obtain additional information by adjusting their smartphones to the
special object, their smartphones dont react as the users intended or they require a deliberate
process to obtain additional information.
A method of going back to the previous screen without selecting any themes in the theme menu
does not exist.
When selecting the category for search location, items which could be plurally selected could be
mixed up with items which could not be plurally selected. This problem can lead to a
cancellation of the users operation and cause confusion to the users.

15

17

11

10

9
7

9
6

5
8

10

10

509

It is difficult for users to

confirm or tab the
hidden information.

Expected Difficulties

The Reason for the

Problem

Developers used
unfamiliar icons or
didnt consider the
touching environment
which doesnt have the
mouse-over function.

When there are more than Providing information

two results in the same
without considering the
direction.
complexity of
information
When the users require
Insufficient options or
additional information
functions for indicating
from the search result
additional information
(e.g., distance)
(e.g., distance, grade)

Problem Context

Only limited name

Tabbing is required to
information is provided
access the essential
when the users see the
information and
result of searching for
difficulties are expected
positional information.
when moving.
Absence of explanation
The users should operate When the users want to
and familiarity of icons.
the application step by
operate the menus or
step because they dont
functions (such as
understand the meaning
selecting search
of icons.
category)

Shows the duplicated

results on the screen.

Description

Context-based, Error
management, Hierarchy,
Low physical effort,
Visibility
Context-based,
Personalization,
Visibility

Related Usability
Principles

Consistency, Error
management,
Familiarity, Help and
documentation,
Learnability,
Navigation,
Recognition, Visibility
Problem 4
The problem of
It is difficult for the users When expanding the
The application is not
Availability,
Expanding the
expression and function
to expand the searching
search range because
provided with adequate
Context-based,
range of searching supporting the
range, find a function, or the users are not
icons, functions,
Familiarity,
expansion of the
operate the function.
satisfied with the result.
expression methods or
Learnability,
searching range.
function for expanding
Personalization,
searching range
Recognition,
Responsiveness,
Visibility,
Problem 5 Help
Misallocation of Help
The users have trouble
When the users are
Contents are insufficient
Error management, Help
menu
menu and problem of
with finding the location
curious about or want to
or Help menu is located
and documentation,
content.
of Help menu and
try a specific function.
in the option menu.
Hierarchy, Learnability,
comprehending the
contents of it.
When user wants to
The function of moving or Context-based, Defaults,
Problem 6
Absence of functions and Menus are tilted on the
options for one-handed
one side. That makes it
operate with only one
magnifying the menu is
Low physical effort,
One-handed
operation.
difficult to operate the
hand (especially the left
not provided.
Personalization,
operation
menus and makes the
hand) in mobile
Visibility
icons too small to
environment.
operate.

Problem 3
Unfamiliar icon
expression

Problem 1
Duplicated
expression of
information
Problem 2
Providing limited
information

Problem

TABLE 7
Detailed Information: Common Factors in the Top 10 Problems

Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

510

S. M. KO ET AL.

TABLE 8
Guidelines for Improving the Top 10 Problems
No.
1
2
3

Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

Problem

Guideline

Duplicated expression
of information
Providing limited
information
Unfamiliar icon
expression
Expanding the range of
searching

Use an active way of expressing information to obtain hidden information by only

operating the camera.
Provide additional information about what users want from the result of positional search.
Provide it according to the number of search results or according to the user option.
Design the icons with familiarity and consistency based on the analysis of widely used
icons. Support the users with a simple explanation about the icons.
The function of expanding searching range on the initial screen should be provided
intuitively and should react promptly to user actions. It also has to provide option
storage function about what users want on the range search.
Considering the unfamiliarity of augmented reality, Help menu should be provided with
1 level menu and should be divided into many steps for the users to remember easily.
Location and size of the menus and icons should make one-handed operation possible.
Initial menus and icons should be placed where they are back-camera reachable while
adjusting the location and size of the menus and icons.

Help menu

One-handed operation

development. The other role will be to evaluate the design and

usability of smartphone AR applications.
We conducted heuristic evaluations of the three most-used
smartphone AR applications in Korea by applying the developed usability principles. As a result we found problems corresponding with the properties of the AR applications. The
problems we found usually involved operation of the device
with one hand or displaying too much information on a small
screen. We concluded that a lack of systemic and flexible
information and options for users brought out these problems.
We also found a lack of explanation in help menus and a lack of
cognitive support for users. We concluded that these problems
occurred because the AR applications were designed based on
previous application development design principles.

Improving AR Application Prototype With

Heuristic Evaluation Results
We then attempted to validate our smartphone AR application usability principles. We developed an improved AR
application prototype of an Android-based smartphone based on
the developed design guidelines in order to conduct a usability
testing. Twenty participants were recruited for this test using
both iOS and Android applications.
Some limitations existed with regard to developing a new
prototype including cost, time, and the proficiency level of
developers with Android SDK. Specialists drew two representative problems from the heuristic evaluation and developed
an improved prototype to accept the realistic problems previously shown and to reflect the result of the heuristic evaluation.
Among the common problems of smartphone AR applications, Problem 1 (duplicated expression of information) and
Problem 2 (providing limited information) have one common

aspect: how they express information to users. This problem occurs when providing too much information on a small
smartphone screen. For the usability testing, we developed a
user interface improved prototype. This prototype was based on
guidelines designed to solve the problem of duplicate information (Problem 1) and to solve the problem of providing limited
information (Problem 2).
To reduce duplicate information (Problem 1) we designed
the prototype to not require any additional tapping operations. We provided bar-shaped duplicate search results as the
camera spontaneously moves around the center of the screen.
To improve the problem of providing limited information
(Problem 2), we made a distance indicator on the bar as shown
in Figure 4. In addition, we provided a distance-indication
setting on the basic search results screen. We provided a manual distance indicator to users so that they can use prototypes

FIG. 4. Distance display function of the prototype (color figure available

online).

USABILITY FOR AUGMENTED REALITY APPLICATIONS

511

TABLE 9
Applications for the Usability Testing
Operating Screen
App

Providing Information About the

Results

Providing Detailed Information

Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

Prototype

whenever they want and provide distance information when the

search result is not sufficient (Figure 4).
As shown in Table 9, we selected iOS application A, Android
application B, and the developed prototype in the environment
of Android OS for use in this research.
Usability Testing With AR Application Prototype
There were two types of task scenarios for the participants
(see Table 10). Each was designed for Problems 1 and 2, which
were drawn from the heuristic evaluation. Each scenario design
considered frequently used tasks when using the place-based

TABLE 10
Tasks for the Usability Testing
Task 1
Task 2

Search the nearest () from here. Then input text of

the name and distance.
Find () and make a phone call after confirming the
location on the map by using the detailed
information.

service AR application in a mobile environment. To control for

the ordering and learning effects, we designated different search
objects for each application. We also designed the number of
search results to be similar to make the evaluation fair for both
applications.
The questionnaire for the usability testing was divided into
two phases. The first phase of this questionnaire was administered before the experiment took place. We developed a
pretask questionnaire regarding participants personal backgrounds with regard to smartphones. The second phase of the
questionnaire was designed to be asked after processing each
of two tasks. In this phase, the definitions of usability principles were established. These usability principles were related
to Problems 1 and 2 and included visibility, personalization,
context-based, error management, hierarchy, and low physical
effort. We also developed an additional questionnaire composed
of eight questions. This questionnaire included two questions
from Lewiss After Scenario Questionnaire, which should be
processed after finishing the usability testing scenario (Lewis,
1991).
The usability testing was conducted by providing a scenariobased task to the participant. After providing the scenario-based

512

S. M. KO ET AL.

TABLE 11
Results Obtained From the Usability Testing
M of Rating Score (SD)

Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

Dimension

App A

App B

Prototype

Usability Principles

Context-based
Error management
Hierarchy
Low physical effort
Personalization
Visibility
Total

3.05 (1.19)
3.10 (0.97)a,b
3.20 (1.28)a,b
3.25 (1.29)a
2.55 (1.23)a
3.25 (1.37)a
3.07 (0.90)a

3.00 (0.97)
2.75 (1.20)a
2.90 (0.97)a
2.95 (1.05)a
2.40 (0.88)a
2.80 (0.89)a
2.80 (0.61)a

3.65 (1.35)
3.90 (1.33)b
4.10 (1.71)b
4.80 (1.20)b
4.45 (0.81)b
4.70 (1.08)b
4.25 (0.99)b

ASQ

Amount of time
Ease of completing

2.70 (1.22)a
3.50 (1.32)a

2.95 (1.15)a
3.20 (1.20)a

5.05 (1.23)b
5.00 (1.52)b

Note. ASQ = after-scenario questionnaire.

a
Mean scores are significantly different at the a = 0.01( ) and a = 0.001( ). b Mean scores are
significantly different at the a = 0.05 when the superscripts (a, b) are different.

task to the participant, the test was conducted by observation of and the questionnaire from the participant. Twenty
people participated in this test. The participants were undergraduate students, graduate students, and company employees
(10 male, 10 female; average age = 27.05). These participants
included 12 who had previously used AR applications on their
smartphone.
Each participant was asked to fill out the pretask questionnaire on their personal smartphone background after listening to
the outline and the method of the experiment. The participants
were then asked to answer six questions based on the usability
principles related to Problems 1 and 2. They were then asked to
answer eight questions including two questions from the After
Scenario Questionnaire. While conducting this experiment, the
observer was asked to record the elapsed time and any errors
(Lewis, 1991).
The explanation of the experiment required about 10 min
on average. Progression and response took 5 min each. It took
about 40 min to finish the entire experiment. Because this was
a comparison experiment, we randomly chose the sequence of
the task progression order of application.
Results of Usability Testing
We conducted a repeated analysis of variance to analyze the
results of the questionnaire. The Greenhouse-Geisser method
was applied to revise the assumption violation of sphericity.
As a result of applying a one-way analysis of variance by using
application as an independent variable, there was a meaningful
difference in visibility, F(2, 38) = 24.51, p < .001; personalization, F(2, 38) = 28.35, p < .001; error management, F(1.53,
29.323) = 6.496, p = .007; hierarchy, F(2, 38) = 6.58, p = .004;
low physical effort, F(2, 38) = 18.17, p < .001, of usability

principles and total sum, F(2, 38) = 24.93, p < .001, that
depended on application. No difference was found in the case
of context based. There were significant differences both on the
satisfaction about the usefulness of the task, F(1.56, 29.540) =
15.777, p < .001, and satisfaction about the elapsed time to
finish the task, F(2, 38) = 27.91, p < .001. After analyzing
the effects of sex and possession of smartphones, no meaningful main effect was found. The average, standard deviation and
the result of analysis through the Tukey examination of each
question item are presented in Table 11.
As a result of postexamination using the Tukey test, error
management and hierarchy appeared to have meaningful differences between application B and the prototype using average
scores. However, there was no significant difference between
application A and application B or between application A
and the prototype. This result can be interpreted to mean
that the composition of application B, which is based on an
Android environment, is lower than application A or the prototype, which are both based on the iOS environment. In the
case of context based, there was no significant difference
between application A and application B. In terms of task
time, satisfaction with application B appeared to be higher
than with application A. This result was expected because
there is no function to indicate distance on the search result in
application A.
For all the question items except satisfaction about the
elapsed time to complete the task, it appeared that users were
most highly satisfied with the applications in the order of
prototype, application A, and application B. There were significant differences between prototype and applications A and B
except from context based, which showed no significant difference between the three aspects. In addition, difference between
application A and B was not encounter.

Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

USABILITY FOR AUGMENTED REALITY APPLICATIONS

FIG. 5.

The average time to complete task (color figure available online).

Figure 5 shows the result of the quantitative evaluation of

the time elapsed while working on tasks. On the same task,
the improved prototype showed shorter elapsed time when
compared to application A and application B. We also found
a statistically meaningful difference in the average elapsed time
between the elapsed task time of the prototype and the elapsed
task time of applications A and B.
As seen in the analysis of the usability testing, the evaluation result of the prototype was better than the evaluation result
of the existing applications A and B. We applied improved
guidelines drawn from the heuristic evaluation to the development of the prototype. In this research, we also conducted a
usability testing based on the developed smartphone AR application usability principles. As a result, we found that existing
problems were solved and user satisfaction was improved.

4. DISCUSSION AND CONCLUSIONS

In this study, 22 usability principles were developed for
user-centered design and evaluation of AR applications on
smartphones. Heuristic evaluation was also conducted based
on these principles. We also gathered previously published
user principles and integrated and selected principles suitable for AR applications on smartphones. Expert evaluation showed that previous design principles were inadequate
for newly developed technologies. Significant improvement
occurred through usability testing of an Android-based prototype using updated design guidelines. It is expected that the
usability principles suggested in this study could be used for
developing and evaluating AR applications in the future
In this research, consideration of the exterior environment
of the smartphone was excluded because the usability testing was conducted only in a laboratory environment. As a
result, research on the principle of context based was limited
because the user environment wasnt relevant to the principle.

513

Improvement was shown for all items except context based

compared to the available applications. Through this result, we
confirmed that the usability principles developed in this study
and the methods of heuristic evaluation are appropriate for AR
application evaluation. It is expected that the usability principles
that were developed in this research will be used in the process
and evaluation of development of AR applications.
This study has some limitations. First, we drew problems
from three popularly used smartphone AR applications which
are currently used in Korea. We selected the applications by
considering their download times. A limitation existed in that
the applications are only popularly used in Korea and dont offer
English versions. Future work could conduct a heuristic evaluation and usability testing on various and globally used applications. Second, we developed a prototype based on Android
circumstance and conducted a usability testing in this study.
Future work could include usability testing and heuristic evaluation of various kinds of smartphone device OS such as iOS
and Windows Mobile. Third, this usability testing was conducted on a fixed display of smartphone. This can be a limitation
of this study because smart device circumstance is changing
rapidly. The next study should reflect various display sizes and
device types. Fourth, we only studied AR application that uses
basic information of the user location information. However,
the future work should be conducted on various types of AR
applications.

REFERENCES
Allen, M., Currie, L. M., Bakken, S., Patel, V. L., & Cimino, J. J.
(2006). Heuristic evaluation of paper-based Web pages: A simplified
inspection usability methodology. Journal of Biomedical Informatics, 39,
412423.
Atkinson, B. F. W., Bennett, T. O., Bahr, G. S., & Nelson, M. M. W. (2007).
Development of a multiple heuristics evaluation table (MHET) to support software development and usability analysis. Paper presented at the
Proceedings of the 4th International conference on Universal Access in
HumanComputer Interaction: Coping With Diversity (pp. 563572). July
2227, 2007, Beijing, China.
Azuma, R. T. (1997). A survey of augmented reality. Presence-Teleoperators
and Virtual Environments, 6, 355385.
Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., & MacIntyre, B.
(2001). Recent advances in augmented reality. Computer Graphics and
Applications, IEEE, 21, 3447.
Billinghurst, M., Belcher, D., Gupta, A., & Kiyokawa, K. (2003).
Communication behaviors in colocated collaborative AR interfaces.
International Journal of HumanComputer Interaction, 16, 395423.
Bruns, E., Brombach, B., & Bimber, O. (2008). Mobile phone-enabled
museum guidance with adaptive classification. IEEE Computer Graphics
and Applications, 28, 98102.
Caudell, T. P., & Mizell, D. W. (1992). Augmented reality: An application of heads-up display technology to manual manufacturing processes.
Proceedings of the Twenty-Fifth Hawaii International Conference on the
System Sciences, 1992, 2, 659669.
Chincholle, D., Goldstein, M., Nyberg, M., & Eriksson, M. (2002). Lost
or found? A usability evaluation of a mobile navigation and locationbased service. Paper presented at the Proceedings of the 4th International
Symposium on Mobile HumanComputer Interaction (pp. 211224).
September 1820, 2002, Pisa, Italy.

Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

514

S. M. KO ET AL.

Cockton, G., & Woolrych, A. (2001). Understanding inspection methods:

Lessons from an assessment of heuristic evaluation. People and Computers,
15, 171192.
Desurvire, H., Caplan, M., & Toth, J. A. (2004). Using heuristics to evaluate
the playability of games. Paper presented at the CHI04 extended abstracts
on Human Factors in Computing Systems (pp. 15091512). April 2429,
2004, Vienna, Austria.
Dnser, A., Grasset, R., & Billinghurst, M. (2008). A survey of evaluation
techniques used in augmented reality studies. Paper presented at the ACM
SIGGRAPH ASIA 2008 courses, Singapore.
Dnser, A., Grasset, R., Seichter, H., Billinghurst, M. (2007). Applying HCI
principles to AR systems design. Proceedings of the 2nd International
workshop on Mixed Reality User Interfaces: Specification, Authoring,
Adaptation (MRUI 2007) (pp. 3742). March 11, 2007, Charlotte, NC,
USA.
Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased
sample sizes in usability testing. Behavior Research Methods, 35, 379383.
Fckler, P., Zeidler, T., Brombach, B., Bruns, E., & Bimber, O. (2005).
PhoneGuide: Museum guidance supported by on-device object recognition on mobile phones. Proceedings of the 4th international conference
on Mobile and Ubiquitous Multimedia (pp. 310). December 810, 2005,
Christchurch, New Zealand.
Gabbard, J. L., & Swan Ii, J. E. (2008). Usability engineering for augmented
reality: Employing user-based studies to inform design. IEEE Transactions
on Visualization and Computer Graphics, 14, 513525.
Gong, J., & Tarasewich, P. (2004). Guidelines for handheld mobile device interface design. Proceedings of the 2004 DSI Annual Meeting (pp. 37513756).
November 2023, 2004, Boston, Massachusetts, USA.
Henrysson, A. (2007). Bringing augmented reality to mobile phones. Linkping,
Sweden: Linkping University.
Hllerer, T., Feiner, S., Terauchi, T., Rashid, G., & Hallaway, D. (1999).
Exploring MARS: Developing indoor and outdoor user interfaces to a
mobile augmented reality system. Computers and Graphics (Pergamon), 23,
779785.
Hvannberg, E. T., Law, E. L.-C., & Lrusdttir, M. K. (2007). Heuristic
evaluation: Comparing ways of finding and reporting usability problems.
Interacting with Computers, 19, 225240.
Ishii, H. (2008). The tangible user interface and its evolution. Communications
of the ACM, 51(6), 3236.
Jeon, J. H., & Lee, S. Y. (2011). Standardizations for Mobile Augmented Reality
Technology Electronics and telecommunications trends. Electronics and
Telecommunications Research Institute, 26, 6174.
Jones, M. (2006). Mobile interaction design. Hoboken, NJ: Wiley & Sons.
Kim, H. J., Kim, M. H., Chio, J. K., & Ji, Y. G. (2008). A study of evaluation
framework for tangible user interface. Proceedings of the 2nd International
Conference on Applied Human Factors and Ergonomics AHFE2008. July
1417, 2008, Las Vegas, USA.
Kostaras, N., & Xenos, M. (2007). Assessing educational web-site usability
using heuristic evaluation rules. Paper presented at the 11th Panhellenic
Conference on Informatics with International Participation, Patras, Greece.
Kurniawan, S., & Zaphiris, P. (2005). Research-derived web design guidelines
for older people. Proceedings of the 7th International ACM SIGACCESS
conference on Computers and Accessibility, 129135.
Law, E. L.-C., & Hvannberg, E. T. (2004). Analysis of strategies for improving
and estimating the effectiveness of heuristic evaluation. Proceedings of the
Third Nordic Conference on HumanComputer Interaction, 241250.
Lewis, J. R. (1991). Psychometric evaluation of an after-scenario questionnaire
for computer usability studies: the ASQ. SIGCHI Bulletin, 23, 7881.
Livingston, M. A. (2005). Evaluating human factors in augmented reality
systems. Computer Graphics and Applications, IEEE, 25(6), 69.
Livingston, M. A., Rosenblum, L. J., Julier, S. J., Brown, D., Baillot, Y., II,
J. E. S., . . . Hix, D. (2002). An augmented reality system for military
operations in urban terrain. Proceedings of Interservice/Industry Training,
Simulation & Education Conference, 89.
Milgram, P., & Colquhoun, H. (1999). A taxonomy of real and virtual world
display integration. Mixed Reality: Merging Real and Virtual Worlds, pp.
530.

Milgram, P., & Kishino, F. (1994). Taxonomy of mixed reality visual displays.
IEICE Transactions on Information and Systems, E77D, 13211329.
Miyashita, T., Meier, P., Tachikawa, T., Orlic, S., Eble, T., Scholz,
V., . . . Lieberknecht, S. (2008). An Augmented Reality museum guide.
Proceedings of the 7th IEEE/ACM International Symposium on Mixed and
Augmented Reality, 103106.
Morrison, A., Oulasvirta, A., Peltonen, P., Lemmel, S., Jacucci, G., Reitmayr,
G., . . . Juustila, A. (2009). Like bees around the hive: A comparative study
of a mobile augmented reality map. Proceedings of the 27th international
conference on Human Factors in Computing Systems, 18891898.
Nassar, M. A., & Meawad, F. (2010). An Augmented Reality exhibition guide
for the iPhone. Proceedings of the 2010 International Conference on User
Science and Engineering (iUSEr), 157162.
Nielsen, J., & Landauer, T. K. (1993). A mathematical model of the finding of usability problems. Proceedings of the INTERACT  93 and CHI
 93 conference on Human Factors in Computing Systems, 206213.
Nielsen, J., & Mack, R. L. (1994). Usability inspection methods (pp. 2564).
New York, NY: Wiley.
Nilsson, S., & Johansson, B. (2006). User experience and acceptance of a mixed
reality system in a naturalistic settingA case study. Proceedings of the 5th
IEEE and ACM International Symposium on Mixed and Augmented Reality,
247248.
Park, H., & Park, J.-I. (2010). Invisible markerbased augmented reality.
International Journal of HumanComputer Interaction, 26, 829848.
Rau, P.-L. P., & Liang, S.-F. M. (2003). Internationalization and localization: evaluating and testing a Website for Asian users. Ergonomics, 46,
255270.
Shin, C., Oh, Y., Suh, Y., & Woo, H. Y. (2010). A survey on mobile augmented
reality services and an outlook toward a sustainable content ecosystem.
Korea Information Science Society Review, 28, 4350.
Shneiderman, B. (2010). Designing the user interface: Strategies for effective
humancomputer interaction (5th ed.). Boston, MA: Addison-Wesley.
Sutcliffe, A., & Gault, B. (2004). Heuristic evaluation of virtual reality applications. Interacting with Computers, 16, 831849.
Sutherland, I. E. (1968). A head-mounted three dimensional display.
Proceedings of the December 911, 1968, fall joint computer conference,
Part I, 757764.
Swan, J. E., & Gabbard, J. L. (2005). Survey of user-based experimentation
in augmented reality. Proceedings 1st International Conference on Virtual
Reality (pp. 19). July 2227, Las Vegas, Nevada, USA.
Tan, W.-s., Liu, D., & Bishu, R. (2009). Web evaluation: Heuristic evaluation vs.
user testing. International Journal of Industrial Ergonomics, 39, 621627.
Tang, Z., Johnson, T. R., Tindall, R. D., & Zhang, J. (2006). Applying heuristic
evaluation to improve the usability of a telemedicine system. Telemedicine
Journal & E-Health, 12, 2434.
Thomas, B., Close, B., Donoghue, J., Squires, J., Bondi, P. D., & Piekarski,
W. (2002). First person indoor/outdoor augmented reality application:
ARQuake. Personal Ubiquitous Computing, 6, 7586.
Ullmer, B., & Ishii, H. (2000). Emerging frameworks for tangible user interfaces. IBM Systems Journal, 39, 915931.
Vaughan-Nichols, S. J. (2009). Augmented Reality: No longer a novelty?
Computer, 42, 1922.
Virrantaus, K., Markkula, J., Garmash, A., Terziyan, V., Veijalainen,
J., Katanosov, A., & Tirri, H. (2001). Developing GIS-supported
location-based services. Proceedings of the Second International
Conference on Web Information Systems Engineering, 2001, 6675.
Wiedenmaier, S., Oehme, O., Schmidt, L., & Luczak, H. (2003). Augmented
Reality (AR) for assembly processes design and experimental evaluation. International Journal of HumanComputer Interaction, 16,
497514.

ABOUT THE AUTHORS

Sang Min Ko is a Ph.D. student in the Department of
Information and Industrial Engineering at Yonsei University,
Korea. His research interests include Haptic Interface,

USABILITY FOR AUGMENTED REALITY APPLICATIONS

Downloaded by [Escuela Sup Pol del Ejercito ] at 06:40 02 September 2013

Multisensory Interaction, Eye-Tracking, and Augmented

Reality.
Won Suk Chang received a masters degree from the
Department of Information and Industrial Engineering at
Yonsei University, Korea, in 2011. His research interests
include Augmented Reality and Mobile Interaction Design.

515

Yong Gu Ji is an Associate Professor in the Department

of Information and Industrial Engineering at Yonsei University,
where he directs the Interaction Design Laboratory. He received
his Ph.D. in Industrial Engineering from Purdue University.
His research interests include Usability, Emotional Design,
Accessibility, and Elderly in HCI.