User Interface Evaluation

Usability Testing Methods

Usability Testing Methods

Caring out experiments to find out specific information about a design and/or product. Basis comes from experimental psychology. Uses statistical data methods

Quantitative and Qualitative

Usability Testing Methods

During usability testing, users work on specific tasks using the interface/product and evaluators use the results to evaluate and modify the interface/product.

Widely used in practice, but not appropriately used.

Often abused by developers that consider themselves to be usability experts. Can be very expensive and time consuming.

Usability Testing Methods

Performance Measurement
Thinking-aloud Protocol Question-asking Protocol

Coaching Method

Usability Testing Methods

Co-discovery Learning
Teaching Method Retrospective Testing

Remote Testing

Performance Measurement

Applicable Stages:

Design, Code, Test & Deployment

Personnel

Usability Experts, approximately 1. Developers, 0. Users, 6.

Performance Measurement

Usability Issues Covered

Effectiveness: Yes Efficiency: Yes Satisfaction: No

Quantitative Data is collected.

Can NOT be conducted remotely.

Can be used on any system.

Performance Measurement

What is it? Used to collect quantitative data. Typically, you will be looking for benchmark data. Objectives MUST be quantifiable

75% of users shall be able to complete the basic task in less than 30 minutes.

Performance Measurement

How can I do it? Define the goals that you expect users to perform

Performance Measurement

How can I do it?

Quantify the goals

The time users take to complete a specific task. The Ratio between successful interactions and errors. The time spent recovering from errors. The number of user errors. The number of commands or other features that were never used by the user. The number of system features the user can remember during a debriefing after the test. The proportion of users who say that they would prefer using the system over some specified competitor.

Performance Measurement

How can I do it? Get participants for the experiments Conduct very controlled experiments

All variables must remain consistent across users

Performance Measurement

Problems With Performance Measurement No qualitative data.

Thinking-aloud Protocol

Applicable Stages:

Design, Code, Test & Deployment

Personnel

Usability Experts, approximately 1. Developers, 0. Users, 4.

Thinking-aloud Protocol

Usability Issues Covered

Effectiveness: Yes Efficiency: No Satisfaction: Yes

Quantitative Data is NOT collected.

Can NOT be conducted remotely.

Can be used on any system.

Thinking-aloud Protocol

What is it? Technique where the participant is asked to vocalize his or her thoughts, feelings, and opinions while interacting with the product. .

Thinking-aloud Protocol

How can I do it? Select the participants, who will be involved? Select the tasks and design scenarios. Ask the participant to perform a task using the software.

Thinking-aloud Protocol

How can I do it? During the task, ask the user to vocalize

Thoughts, opinions, feelings, etc.

http://www.cs.umd.edu/~zzj/ThinkAlo.htm

Thinking-aloud Protocol

Problem With Thinking-Aloud Protocol Cognitive Overload

Can you walk & chew gum at the same time? Asking the participants to do too much.

Question-asking Protocol

Applicable Stages:

Design, Code, Test & Deployment

Personnel

Usability Experts, approximately 1. Developers, 0. Users, 4.

Question-asking Protocol

Usability Issues Covered

Effectiveness: Yes Efficiency: No Satisfaction: Yes

Quantitative Data is NOT collected.

Can NOT be conducted remotely.

Can be used on any system.

Question-asking Protocol

What is it? Similar to Thinking-aloud protocol. Instead of participant saying what they are thinking, the evaluator prompts the participant with questions while using the system.

Question-asking Protocol

How can I do it? Select the participants, who will be involved? Select the tasks and design scenarios. Ask the participant to perform a task using the software.

Question-asking Protocol

How can I do it? During the task, ask the user to questions about the product

Thoughts, opinions, feelings, etc.

http://www.cs.umd.edu/~zzj/Question.htm

Question-asking Protocol

Problem With Thinking-Aloud Protocol Cognitive Overload++

Can you walk, chew gum & talk at the same time? Asking the participants to do too much. Added pressure when the evaluator asks questions. Can be frustrating on novice users.

Coaching Method

Applicable Stages:

Design, Code, Test & Deployment

Personnel

Usability Experts, approximately 1. Developers, 0. Users, 4.

Coaching Method

Usability Issues Covered

Effectiveness: Yes Efficiency: No Satisfaction: Yes

Quantitative Data is NOT collected.

Can NOT be conducted remotely.

Can be used on any system.

Coaching Method

What is it? A system expert sits with the participant and acts as a coach. Expert answers the participants questions. The evaluator observes their interaction.

Coaching Method

How can I do it? Select the participants, who will be involved? Select the tasks and design scenarios. Ask the participant to perform a task using the software in the presence of a coach/expert.

Coaching Method

How can I do it? During the task, the user will ask the expert questions about the product

http://www.cs.umd.edu/~zzj/Coaching.htm

Coaching Method

Problem With Coaching Method In reality, there will not be a coach present. This is good for creating a coaching system, but not for evaluating the interface.

Co-Discovery Learning

Applicable Stages:

Design, Code, Test & Deployment

Personnel

Usability Experts, approximately 1. Developers, 0. Users, 6.

Co-Discovery Learning

Usability Issues Covered

Effectiveness: Yes Efficiency: No Satisfaction: Yes

Quantitative Data is NOT collected.

Can NOT be conducted remotely.

Can be used on any system.

Co-Discovery Learning

What is it?

Two test users attempt to perform tasks together while being observed. They are to help each other in the same manner as they would if they were working together to accomplish a common goal using the product.
They are encouraged to explain what they are thinking about while working on the tasks.

Thinking Aloud, but more natural because of partner.

Co-Discovery Learning

How can I do it? Select the participants, who will be involved? Select the tasks and design scenarios. Ask the participants to perform a task using the software.

Co-Discovery Learning

How can I do it? During the task, the users will help each other and voice their thoughts by talking to each other.

http://www.cs.umd.edu/~zzj/Codiscov.htm

Co-Discovery Learning

Problem With Co-Discovery Learning Neither is an expert

The blind leading the blind.

Teaching Method

Applicable Stages:

Design, Code, Test & Deployment

Personnel

Usability Experts, approximately 1. Developers, 0. Users, 4.

Teaching Method

Usability Issues Covered

Effectiveness: Yes Efficiency: No Satisfaction: Yes

Quantitative Data is NOT collected.

Can NOT be conducted remotely.

Can be used on any system.

Teaching Method

What is it? You have 1 participant use the system. Ask the participant to teach a new novice participant how to use the system.

Teaching Method

How can I do it? Select the participants, who will be involved? Select the tasks and design scenarios.

Ask the 1st participant to perform a task using the software. Ask the 1st participant to teach a new participant.

Teaching Method

How can I do it? Observe their interactions.

http://www.cs.umd.edu/~zzj/Teaching.htm

Teaching Method

Problem With Teaching Method Neither is an expert

The blind leading the blind. Possible to discover some interesting things about the learn-ability of your interfaces.

Retrospective Testing

Applicable Stages:

Design, Code, Test & Deployment

Personnel

Usability Experts, approximately 1. Developers, 0. Users, 4.

Retrospective Testing

Usability Issues Covered

Effectiveness: Yes Efficiency: Yes Satisfaction: Yes

Quantitative Data can be collected.

Can NOT be conducted remotely.

Can be used on any system.

Retrospective Testing

What is it? A videotape of the session is observed by the usability expert and the participants.

Retrospective Testing

How can I do it? Select the participants, who will be involved?

Select the tasks and design scenarios.

Use one of the usability testing methods that we have discussed. Videotape the session.

Retrospective Testing

How can I do it? Review the videotape with the users.

http://www.cs.umd.edu/~zzj/Retrospe.htm

Retrospective Testing

Problem With Retrospective Testing Extremely time consuming!

Remote Testing

Applicable Stages:

Design, Code, Test & Deployment

Personnel

Usability Experts, approximately 1. Developers, 0. Users, 5.

Remote Testing

Usability Issues Covered

Effectiveness: Yes Efficiency: Yes Satisfaction: Yes

Quantitative Data can be collected.

Can be conducted remotely.

Can be used on any system.

Remote Testing

What is it? The participants are separated from the evaluators. No formal observation. No usability lab.

Remote Testing

How can I do it? Give the product/software to participants.

Collect information about how they use your software/product. Methods

Same-Time Different Place Different-Time Different Place

Remote Testing

How can I do it? Lotus Video Cam, Look@Me, SnagIt

Usability Logger

http://www.usabletools.com/

Journaled Sessions

http://www.cs.umd.edu/~zzj/Retrospe.htm

Remote Testing

Problem With Remote Testing

The evaluator is not there. Cant observe facial expressions.

Great for Web based systems.

Usability Testing Methods

Select Select

the method that works best for you.

the method that fits your implementation. thorough during your experiments. more data, the better.

Be

The

Usability Testing Methods

Hawthorne
The

Effect

tendency for people to improve their performance after any change when they know their performance is being studied.