Running head: VIRTUAL IMMERSION STUDY SUPPORT

Usability Test
Virtual Immersion Study Support
Juliana Shajari
California State University, Monterey Bay

IST 622: Assessment and Evaluation

Dr. Bude Su
Summer 2016

VIRTUAL IMMERSION STUDY SUPPORT

Table of Contents
Abstract..........................................................................
......................3
Objectives.4
Prototype Description.4
Tryout Conditions...5
Executive Summary....5
Methodology....5
Expected Outcome..5
Assessment Instrument..6
Method of Gathering Data.6
Participants (Who were Tested)....
6
Training...7
Procedure8
Data Collected.....8
Usability Tasks and Data Graphs (Pre and Post-Test).....
..9
T-test....12
Completion Rate.....12
Subjective evaluation....
.....12

VIRTUAL IMMERSION STUDY SUPPORT

Findings and Recommendations....
...13
Summary.....13
Appendices..........14
Appendix A.14
Appendix B.....15

Abstract
This paper defines a test plan for conducting a usability test during the
development of Virtual Immersion, Study Support Course. The goals of
usability testing include creating a model of user performance, creating and
validating user performance procedures, and address the potential design
concerns to improve the productivity, production, and end-user satisfaction.
In this study, six participants first took the pre-test questioner. The
participants received the detailed instructions of how to perform the test.
Then the result of the pre-test was inserted in the worksheet and marked
Pre-test. After participants finish the related module
https://virtualsupport.moodlecloud.com/login/index.php
User ID: student
Password: UPFstudayhall

VIRTUAL IMMERSION STUDY SUPPORT

The post-test questioner was presented to them. The final questioner
distributed and the results of both were added to the worksheet under the
corresponding labels "Pretest and Post-test.

VIRTUAL IMMERSION STUDY SUPPORT

The Objectives:

To determine design variations and usability difficulties inside the user

interface and content ranges.

Potential causes of error may consist of:

a) Course-plotting mistakes failure to locate tasks, unnecessary
keystrokes to complete a purpose, inability to follow possible screen
movement.
b) Presentation errors failure to establish and appropriately entertain
the desired information on display, or errors due to labeling
uncertainties.
c) Control usage complications improper toolbar or access field

procedure.
Exercise the application or website under controlled test settings with
typical users. Data will be used to advocate whether usability areas
regarding an operative, efficient, and well-received and user boundary

have been achieved.

The user routine and user approval levels of the user edge for future
usability will be tested.

Prototype Description
The LMS (Study Support) consists of three Semesters: Each Semester
then branches out to two Reading and Listening Modules. In each module,
there are ten categories; Culture, Geography, Grammar, Role play, Writing,
Newsroom, Children Books, Movies, Games, and Military. The primary goal of
this LMS is motivating the students, and performing as a supportive tool in

VIRTUAL IMMERSION STUDY SUPPORT

learning Persian-Farsi. Through Virtual Support, students have access to a
tailored open learning world of materials. The students are free to choose the
subject of their interest, yet learn the topics and subjects directing them
towards our goal at DLI (Defense Language Institute). The target proficiency
and performance level of 3/3/2 in the final Test (DLPT) are the foundations of
this LMS.
Tryout Conditions
The user group that will be participating in the usability test are seven
students, two students, each, from 3 different semesters, and one class
leader. They will contribute to the usability test and later they will indicate
whether they took the test in the barracks or home.
Executive Summary
In this usability test, participants will complete a set of tasks and
questions in Target Language(Persian_Farsi), in a typical office environment
or barracks. The facilitator will brief the attendants, and explains that the
goal is an evaluation of the application, not the participants. The participants
will complete a pretest questioner and go through the course. Then they will
complete the post-test questioner. Subsequent the review of this usability
test plan for Study Support module, we anticipate that the results show
Study Support is a great help to increase the student's pedagogy level to
2+/2+/2 and consequently to 3/3/2 in DLPT test.

VIRTUAL IMMERSION STUDY SUPPORT

Methodology
There are a variety of user experience methods for a usability test. In
the usability test of the Study Support Course, theUn Moderated Usability
Testing is being used. In Unmoderated Usability Testing, the fundamental
technique is used by usability professionals to get the feedback from users.
The expected outcomes
Upon completion of the modules in the semester I and II, learners will
be expected to be able to know the level I and II knowledge in Culture,
Geography, Grammar, Role play, Writing, Newsroom, Children Books, Movies,
Games, and Military of the Target Country. We designed the questions, to
build the foundation of the information for knowing what we want to achieve.
In order to measure learning effectiveness; two tests were used, pre and post
test. It is hypothesized that the participants would exhibit a satisfactory
significant increase in mean scores of the post-test.
Assessment Instruments. The instrument contains 12 multiple
choice questions. The questions include partial Farsi. The pre and post test
was created in google form and was emailed to the participants. First, the
pre-test was emailed. Post-test was not available to take untile the learning
module was completed.
The method for Gathering Data. Because the learning modules
allowed for free navigation between the contents and semesters, the
participants have the maximum flexibility to refer to the available materials

VIRTUAL IMMERSION STUDY SUPPORT

in different modules. The participants had the possibility of referring to the
learning contents by multiple attempts.

Who were Tested

Seven participants, having the following characteristics, evaluated the
Study Support Course. They are in the 7th week of training and have limited,
but sufficient knowledge of Persian-Farsi to be able to read questions which
are partially in Farsi. Some items require referring to more than one module
to get the correct answer. This way we also measure their ability to
remember previous training. One aspect of their success is their ability to
recall the protocols and the final learning objectives and get mastery in the
Target Language.

Audience Type
In Service
Military
Members

Computer Usage
7

Daily 6 to 10
hours

VIRTUAL IMMERSION STUDY SUPPORT

Age
18-25
26-39
TOTAL
(participants)

Gender
5
2
7

Women
Men
TOTAL
(participants)

3
4
7

The participants completed a pretest, and posttest to measure the

usability and adequacy of the program. During the debriefing, the
participants were asked to provide an honest opinion about the usability of
the program.
Participants are in-service military members currently studying PersianFarsi. A minimum of 7 weeks of Persian-Farsi language training is a
prerequisite for taking the course and participating in the usability test.
Participants are required to take semester I and part of semester II modules
of the program.
Training
Participants received an overview of the usability test and procedure.
They were informed that a part of the program is presented in Persian-Farsi
and seven weeks of basic training is a prerequisite for taking this course.
Participants were told that they have the choice of selecting the subject of
their interest. The course is designed to cover 47 weeks of training, and it is
divided into three semesters. Based on the amount of Farsi language
knowledge, participants are free to perform the usability test for the related
semester.

VIRTUAL IMMERSION STUDY SUPPORT

Procedure
Participants will participate in the usability test outside the teaching
hours, in their barracks, or home. The username and password for the course
are provided to them, and they will need internet access to sign into the
course. The Google forms for pre and post test along with the questioner
were emailed to their Edu email address. The participants interaction with
the program was not personally monitored with the facilitator. A zoom link
was provided to the participants in the event they would need assistance or
clarification.
Participants were briefed that they are evaluating the application, not
themselves being evaluated. They all consent and acknowledged that
participating in the usability test is voluntary, and they can cease it at any
time. The process of the test could not be videotaped, due to the security
reasons as the participants were the active military members.
For the same reason, members were not required to complete a pretest
demographic and background information questioner. The process and the
duration of the pre-test and post-test were logged by the participants and
reported to the facilitator.

What Data We Collected

In the pretest data, the participants answered to a set of questions in
Target Language. A variety of Vocabulary, Grammar, Culture, Geography,

VIRTUAL IMMERSION STUDY SUPPORT

Economy and Military knowledge of the participants were tested. Then they
logged on to the Virtual support site to take the course.
https://virtualsupport.moodlecloud.com/login/index.php
User ID: student
Password: UPFstudayhall
After taking the course, the post-test was forwarded to them, and they took
the posttest. The results are as follow;

Usability Tasks and Data

Pre and Post Test Results and Graphs

PreTest

Post Test

VIRTUAL IMMERSION STUDY SUPPORT

VIRTUAL IMMERSION STUDY SUPPORT

VIRTUAL IMMERSION STUDY SUPPORT

t-Test: Paired Two Sample for Means

Variable 1
Mean
Variance
Observations
Pearson Correlation
Hypothesized Mean
Difference
df
t Stat
P(T<=t) one-tail
t Critical one-tail
P(T<=t) two-tail
t Critical two-tail

Reject the null

because the

57.14285714
32.14285714
7
-0.245326691

Variable 2
6.428571429
30.95238095
7

0
6
15.13725086
2.62129E-06
1.943180281
5.24257E-06
2.446911851

T-Stat Is bigger than T Critical, so we support

the research

Because we are comparing

same students taking pre
and posttest, we check tTest Pairs Two Sample for
Means

Completion Rate
Participants completed the pre and post test and answered the
questioner without critical errors.
Subjective Measures

VIRTUAL IMMERSION STUDY SUPPORT

The independent opinions about specific tasks, time to perform each
task, features, and functionality was surveyed. Participants rated their
satisfaction with the overall system.

Significant Findings and Recommendations

Subsequent the review of this usability test plan for Study Support
Course, the pre and post test results were compared and entered in the
chart. The mean and median, as well as the frequency of correct answers,
were registered in the table. Of 7 applicants all were responded to the
questions in the pre-test and logged on to the course and later responded to
the post-test. The pre and post test question were identical, and the test was
a paired two-sample of the mean. So a paired-sample-t-test was performed
on the pre and post-test mean scores to verify the hypothesis that the
learning module would result in a satisfactory significant increase in the
instrument rating. Based on the t-test statistics and the df =6, the critical
value of (1.943180281) is less than the t state (15.13725086), so when the T-Stat is bigger
than T Critical, we support the research. The results and data indicate that
the program is well-managed, and is successful. The students will benefit
from the course and could regulate their study habits. Overall, the product of
the usability test reveals that Study Support Course is a great help to
promote the students performance in target language learning.
Summary

VIRTUAL IMMERSION STUDY SUPPORT

In this usability test, participants completed pre and post-test for the
usability and acceptability of the interface, in their barracks. This
effectiveness pre and post test measured the differences in mean scores
before and after the participants completed the learning modules. The
hypothesis was that after taking the course there would be an increase of
knowledge in the level of their familiarity with the Target Language and the
participants could score better in the post-test. The pre and post-test data
showed that the post-test results were significantly higher than pretest
results.

Appendices

Appendix A. Prototype screenshot.

VIRTUAL IMMERSION STUDY SUPPORT

Appendix B. Study Support Pretest/Posttest

What is the highest rank in Iranian Army? *

VIRTUAL IMMERSION STUDY SUPPORT

5 points
a.
b.
c.
d.

What is the present stem of verb * ?

5 points
a.
b.
c.
d.

is needed for Iranians to *

5 points
Work in the United States
Work in a factory
Work on their school project
Work for Iranian government
What is one of the famous in Isfahan? *
5 points
a.
b.
c.
d.

What is past participle of verb * ?

5 points
a.
b.
c.
d.
What do the Iranian do on * ?

VIRTUAL IMMERSION STUDY SUPPORT

5 points
a. They do the spring cleaning
b. They go out to the parks
c. They stay home with the family
d. They make a special dinner

Where is * ?
5 points
a. In the South East region of Iran
b. In the Northern region of Iran
c. In the Mountain region of Iran
d. In the South West region of Iran
Where is * ?
5 points
Shiraz
Mashhad
Kashan
Tehran
What is the third month of summer in Iranian calendar? *
5 points
a.
b.
c.
d.

What is the concept of * ?

5 points
a. Started his business
b. Started to work with his hands
c. Started to paint
d. Started to do the job

VIRTUAL IMMERSION STUDY SUPPORT

Where is located? *
5 points
a. In the south of Iran
b. In the north of Afghanistan
c. In the border of Iran and Russia
d. In the south of Indian Ocean

What is Cyrus Cylinder? *

5 points
a. The cylinder shape lake in the Fars region
b. The first human rights declaration
c. The name of Cyrus the greats palace
d. None of the above