Usability

Evaluation
Mengapa evaluasi diperlukan?
Evaluasi Usability
USABILITY - ISO 9241
The effectiveness, efficiency and satisfaction
with which specified users achieve specified
goals in particular environments.
Jenis Evaluasi

Dua macam evaluasi:

1. Expert analysis → Heuristic Evaluation

evaluasi oleh ahli usability
2. User Participation → Usability Testing
evaluasi dengan mengujikan sistem kepada
user
 Evaluation Goal
(Alan Dix dalam buku HCI)

1. Menilai jangkauan dan aksesibilitas dari

fungsi sistem.
Sistem harus mudah digunakan, sesuai dengan kebutuhan dan
harapan users

2. Menilai pengalaman pengguna (users

experience)
Apakah sistem mudah dipelajari, apakah user senang dan puas
saat berinteraksi dgn sistem

3. Menemukan masalah-masalah tertentu

terkait sistem.
 WHEN
When usability evaluation needs to be done?
• test early and often
• always be testing.
◦ every system should be evolving based on the user behavior
◦ changing along with user shifting upon the age, experience
and interest of a user.
WHEN
 METHODS
Usability Inspection Usability Testing
Reviewer Expert User (end-user)
Findings Supplement Primary
Data Mostly qualitative Qualitative or quantitative
or combination of qual. & quant.
Stage Throughout design life-cycle Tangible product provided
Methods • Cognitive Walkthrough • Usability Testing
• Heuristic Evaluation • Benchmark Testing
• Competitive Usability Testing
• Summative Usability Testing
• Remote Evaluation
• Think Aloud Testing
• Wizard of Oz
Heuristic Evaluation (HE)
Overview
Heuristic Evaluation :
• A usability evaluation method in which one
or more reviewers, preferably experts,
compare a software, documentation, or
hardware product to a list of design
principles (commonly referred to as
heuristics) and identify where the product
does not follow those principles.
 Kelebihan Evaluasi Heuristik
• Lebih murah dibandingkan teknik evaluasi usability
lainnya
• Lebih intuitif dan mudah untuk memotivasi evaluator
• Tidak memerlukan persiapan yang kompleks
• Evaluator tidak harus pernah mengenyam pelatihan
formal tentang evaluasi usability (Nielsen dan Molich).
• Dapat dilakukan di awal siklus bahkan di seluruh siklus
pengembangan desain
• Lebih cepat dibandingkan studi evaluasi dalam
laboratorium
 Kekurangan Evaluasi Heuristik (HE)
• Permasalahan usability yang ditemukan melalui HE
mungkin tidak sebanyak teknik evaluasi lain, misalnya
usability testing (ke user).
• HE dimungkinkan mengevaluasi lebih banyak
permasalahan minor dan lebih sedikit permasalahan
mayor
• Dimungkinkan terjadi bias karena perbedaan persepsi
antar evaluator.
• Permasalahan yang ditemukan melalui HE bisa saja
berupa false alarms artinya masalah yang ditemukan
belum tentu merupakan permasalahan aktual usability.
HE vs User Testing
• HE is much faster
1-2 hours each evaluator vs. days-weeks

• HE doesn’t require interpreting user’s actions

• User testing is far more accurate
◦ Takes into account actual users and tasks
◦ HE may miss problems & find “false positives”

• Good to alternate between HE & user-based

testing
◦ Find different problems
◦ Don’t waste participants
Procedure
1. Establish an appropriate list of heuristics
2. Select your evaluators
3. Brief your evaluators
4. First evaluation phase
5. Second evaluation phase
6. Record problems
7. Debriefing session
Procedure HE (1)
1. Establish an appropriate list of
heuristics
Use Nielsen 10 heuristics as inspiration
and steppingstone. The list could be
combined with other relevant design
guidelines and market research.
 10 Usability Heuristics
1. Visibility of system status
The system should always keep users informed about
what is going on, through appropriate feedback within
reasonable time.
 10 Usability Heuristics
2. Match between system and the real
world

The system should speak

the users' language, with
words, phrases and
concepts familiar to the
user, rather than system-
oriented terms. Follow real-
world conventions, making
information appear in a
natural and logical order.
 10 Usability Heuristics
3. User control and freedom
Users often choose system functions by mistake and
will need a clearly marked "emergency exit" to leave
the unwanted state without having to go through an
extended dialogue. Support undo and redo and a
clear way to navigate.
 10 Usability Heuristics
4. Consistency and standards

Users should not have to

wonder whether different
words, situations, or actions
mean the same thing.
Follow platform conventions.
 10 Usability Heuristics
5. Error prevention

Even better than good error

messages is a careful
design which prevents a
problem from occurring in
the first place.
Either eliminate error-prone
conditions or check for them
and present users with a
confirmation option before
they commit to the action.
 10 Usability Heuristics
6. Recognition rather than recall
Minimize the user's memory load by making objects,
actions, and options visible.
 10 Usability Heuristics
7. Flexibility and Efficiency of use

The Interface should

be flexible
transforming itself
between a novice
user and an
advanced user.
 10 Usability Heuristics
8. Aesthetic and minimalist design
Dialogues should not contain information which is
irrelevant or rarely needed.
 10 Usability Heuristics
9. Help users recognize, diagnose, and recover
from errors
Error messages should be expressed in plain language (no
codes), precisely indicate the problem, and constructively suggest
a solution.
 10 Usability Heuristics
10.Help and documentation
It may be necessary to provide help and
documentation. Any such information should
be easy to search, focused on the user's task,
list concrete steps to be carried out, and not be
too large.
 Procedure HE (2)
2. Select your evaluators
• Your evaluators should not be your end users.
They should typically be usability experts and
preferably with domain expertise in the industry
type that your product is in.
• 3-5 evaluator are suggested. Single evaluators found
only 35% of the usability problems. The exact
number of evaluators to use would depend on a
cost-benefit analysis.
 Rule of Thumb

https://measuringu.com/novice-expert-issues/
 Procedure HE (3)
3. Brief your evaluators
• The evaluator should know exactly what they are
meant to do and cover during their evaluation.
• The briefing session should be standardized to
ensure the evaluators receive the same
instructions; otherwise, you may bias their
evaluation.
 Procedure HE (4)
4. First evaluation phase
• The first evaluation generally takes around two
hours, depending on the nature and complexity of
your product. The evaluators will use the product
freely to gain a feel for the methods of interaction
and the scope.
• They will then identify specific elements that they
want to evaluate.
 Procedure HE (5)
5. Second evaluation phase
• In the second evaluation phase, the evaluators will
carry out another run-through, whilst applying the
chosen heuristics to the elements identified
during the first phase.
• The evaluators would focus on individual elements
and look at how well they fit in the overall design.
 Procedure HE (6)
6. Record problems
• The evaluators must either record problems
themselves or you should record them as
they carry out their various tasks to track any
problems they encounter.
• Be sure to ask the evaluators to be as
detailed and specific as possible when
recording problems.
 Procedure HE (6)
6. Record problems
• Establishes a ranking between problems
(severity level):
◦ 0 - don’t agree that this is a usability problem
◦ 1 - cosmetic problem
◦ 2 - minor usability problem
◦ 3 - major usability problem; important to fix
◦ 4 - usability catastrophe; imperative to fix

• First rate individually, then as a group

Procedure HE (7)
7. Debriefing session
• The debriefing session involves collaboration
between the different evaluators to collate
their findings and establish a complete list
of problems.
• They should then be encouraged to suggest
potential solutions for these problems on
the basis of the heuristics.
Usability Testing
 Perencanaan Evaluasi

1. Apa yg akan dievaluasi? 6. Partisipan (expert/user)

2. Tujuan 7. Tahapan evaluasi
3. Waktu dan tempat 8. Pengukuran
4. Sesi pengujian 9. Peran masing-masing
5. Alat orangdalam tim
 Memilih Partisipan

1. Tentukan karakteristik partisipan

Usia, gender, latar belakang pendidikan, pengalaman/pengetahuan
tertentu (bisa disesuaikan dengan karakteristik user)

2. Tentukan jumlah partisipan

Rule of Thumb: gunakan kalkulator Usability Sample Size
Misalnya: https://assets.blinkux.com/usability-sample-size/
 Memilih Partisipan

Test 5 users in a usability study

This lets you find almost as many usability problems as
you'd find using many more test participants.

Exceptions:
Quantitative studies (aiming at statistics, not insights):
Test at least 20 users to get statistically significant numbers; tight
confidence intervals require even more users.
Usability Problem vs Test Users
 Menentukan Pengukuran

Penetuan metode pengukuran berdasarkan definisi

usability
 USABILITY - ISO 9241

• Effectiveness: the accuracy and completeness

with which specified users can achieve specified
goals in particular environments
• Efficiency: the resources expended in relation to
the accuracy and completeness of goals achieved
• Satisfaction: the comfort and acceptability of the
work system to its users and other people affected
by its use
 Menentukan Pengukuran

Effectiveness
• Accuracy → Error rate
• Completeness → Successful task completeness
Efficiency (resources expended)
• Time → Time on task
Satisfaction (comfort and acceptability)
• System Usability Scale (SUS)
Resume
• Usability Evaluation dilakukan dan bisa
dilakukan di setiap fase pengembangan
desain.
• Usability Inspection (misalnya Heuristic
Evaluation)
• Usability Testing dengan melibatkan user.
• Teknik evaluasi dipilih berdasarkan Evaluation
Goal yang telah ditetapkan peneliti/researcher.
References
• http://www.usabilitybok.org
• https://blog.prototypr.io/10-usability-heuristics-with-
examples-4a81ada920c
• https://www.usability.gov/how-to-and-
tools/methods/planning-usability-testing.html
• https://www.nngroup.com/articles/how-many-test-
users/
• https://www.interaction-
design.org/literature/article/heuristic-evaluation-how-to-
conduct-a-heuristic-evaluation
Thank You