MNSU Technical Communication Website Test

Bernadette L. Cash

November 28, 2016

1
 Table of Contents

Introduction 3

Executive Summary 3

Methodology 4
Sessions 4
Participants 5

Results 6
Session 1: Card Sorting 6
Session 2: Interview 7
Overall Metrics 9

Recommendations 10

Conclusion 11

Appendices 12
Appendix A: Card Sorting Topics 12
Appendix B: List of Phone Interview Questions 13
Appendix C: Emails to Study Participants 15

2
Introduction
The website at english.mnsu.edu/techcomm represents a subdiscipline of the
English department at Minnesota State University, Mankato. This link provides
information to current and prospective students about requirements,
coursework, faculty, and resources for the schools technical communication
program.

A study was conducted to determine the usability issues that students have
with the website. Testing was carried out remotely with a small group of
university students. These users were asked to organize content into
categories and then provide feedback about the website. Results from the
sessions were noted and analyzed to identify suggested areas for
improvement to the website.

The first remote testing session was administered using the OptimalSort
feature from OptimalWorkshop.com. Sixteen topics from the current MNSU
tech comm website served as online cards which users sorted into groups.
The team leader recorded participants categories and their results.

The next session consisted of phone interviews between the test

administrator and participants. By emailing instructions and hyperlinks
beforehand, more information was obtained from participants regarding their
card sorting results and about the tech comm website. The test administrator
remotely interviewed participants while they viewed material on their own
computer. Participants remarks, ratings, and responses were noted during
the testing. (Throughout this document, the different roles of team leader,
test administrator, and project leader all refer to the same person, Bernadette
Cash.).
Executive Summary
This large usability study was conducted in two sessions. Card sorting
occurred from November 9 through November 13, and interviews were held
on November 16th and November 18. Based in central Illinois, the project
leader conducted the testing online, over the phone or by using Skype to
connect with participants located in Minnesota and Maryland. The purpose of
the study was to assess the usability of the tech comm websites information
architecture and interface.

Four students participated in the testing. They were informed in advance that
the time estimates for the card sorting and interviews would each be
approximately 15 minutes long.

In general, most participants found that the MNSU tech comm website
contains valuable information they needed, and that it is a source of reliable
content. Participants who are tech comm majors consulted the website more
frequently (as much as 8 times a month) than the minors (as little as 23
times total).

3
 The usability study identified several issues including:
A need for organizing sidebar menu items into appropriate categories.
A need to check that all links function correctly.
A need to maintain consistency throughout the website.
A lack of modifications to accommodate students with disabilities.
A need to increase the font size and use high-contrast colors.
A possible consideration of updating the current vertical format to be
concise and effective in its structure of content.
A suggestion to provide an overview of what students will learn
from tech comm coursework.

This document contains participant feedback, agreement ratings, task

completion percentages, and recommendations for improvements. A copy of
materials used in the testing session are included in the Attachments section.
Methodology
Sessions
Dr. Dawn Armfield, the ENG 466/566 professor, specifically arranged the
team leader and participants for this usability study. Therefore, no pre-test
questionnaires were given to recruit eligible participants.

The project leader sent emails to participants informing them of the studys
purpose and testing logistics, as well as requesting their timely participation.
After they completed the card sorting, participants were invited to sign up for
an interview during a time slot that was convenient for them.

Card sorting
Participants completed the card sorting in sessions that lasted anywhere from
8 to 19 minutes. The OptimalSort testing instructed participants to sort 16
topics (see Appendix A) into groups that made sense to them, and then they
were to give each category a name.

Interview
Several days later participants were to call or use Skype for their interviews,
which lasted an average of 20 minutes. The test administrator emailed each
participant beforehand. The message contained specific information that
participants would refer to during the testing. Participants were also asked to
spend time looking at the tech comm home page and browse through the
website.

The interview began with getting preliminary information, followed by a

discussion (post-test) about participants card sorting results.
Format preference
In the next portion of the interview, participants were shown several
examples of navigational menus from other schools tech comm websites.
They offered their opinions on the different menu layouts and chose the
format they preferred most for a potential tech comm website update.
Evaluation Tasks/Scenarios
These scenarios were created based on the project leaders prior history of
unsuccessful attempts in accessing information from specific parts of the tech

4
 comm website. Consequently, the next part of the interview was to determine
if participants achieved similar results after completing the same tasks.

The test administrator first asked participants to identify how they knew if
text on the home page was a hyperlink leading to additional information.
Then they remarked on what they observed after doing the following tasks
(see #7ac in Appendix B):
Click on the Technical Communication text in the slideshow photos.
Click on the Spring 2014 Techniques newsletter link.
Click on the News & Events link.
Questionnaire
The test administrator then asked participants to rank the interface on a
5-point Likert Scale ranging from Strongly Disagree to Strong Agree (see #9
in Appendix B). The following measures were based on Peter Morvilles User
Experience Honeycomb:
a. This website has information that I need.
b. I think this website is easy to use.
c. I like how the design of this website looks.
d. I can find information quickly on this website.
e. People with disabilities would have no problems using this website.
f. This website is a dependable source of information.
Finally, the test administrator asked participants the following questions:
What did you like most about the website?
What did you like least about the website?
What recommendations do you have to improve the website?

See Appendix B for the full list of interview questions.

Participants
All participants were either senior or 4th-year undergraduate students in the
Fall 2016 semester of ENG 466 Usability with Dr. Armfield at Minnesota State
University, Mankato. The participants completed both testing sessions over a
9-day period from November 9th to November 18h. All four participants
completed the card sorting and all four were interviewed by the test
administrator. Of the four participants, three were male and one was female.

Involvement
As part of the preliminary information, participants described their connection
to technical communication by mentioning their declared fields of study (see
Table 1). How often participants used the website depended on their level of
involvement with tech comm: those who are majors tend to visit the website
more often than those who are minors.

Table 1: Undergraduate involvement

tech comm tech comm tech comm
major minor certificate

2 2 0

5
Results
Session 1: Card Sorting
All participants sorted the 16 topics into their own individual groups.
However, nearly every participant preferred how the topics as they are
currently presented on the tech comm website due to familiarity. Only one
participant chose an individual preference over the arrangement currently
displayed on the website. Also, no two participants results were exactly alike
in how they were sorted, including the labels chosen to name the groups.

Three participants said that categorizing topics in accordance with their

personal preference had limitations. First, they had difficulty with sorting
leftover items that didnt fit neatly into the groups they created. Secondly,
three participants expressed that creating groups which made sense for their
own individual situation meant a system that would not be applicable for all
students.

Sorting patterns
Regardless of participants lack of confidence about individually creating an
effective grouping system, there were some patterns from the results that
may offer additional clues on how to categorize the topics.

The following items all had different group names and were sorted like this:

Bachelors degree Masters degree

Undergraduate minor Publications
Undergraduate certificate Graduate certificate

Financial aid Masters degree
Admission requirements Graduate certificate
Bachelors degree Capstone projects
Undergraduate certificate
Undergraduate minor Admission requirements
Courses Masters degree
Graduate certificate

Considering these specific lists, it makes sense to have at least two groups
that differentiate appropriate topics into groups called Undergraduate and
Graduate. In a similar manner, participants suggested labels of Help to
Succeed and Helpful College Resources could simply be known as Resources.

Faculty was seen twice as an item in a category by itself with labels like
Faculty/Staff Information and Faculty Information. Interestingly, most
participants put Class Schedule and Courses in the same group, just as they
currently are on the tech comm website. Participants agreed that Class
Schedule was the one website item they used most frequently.

While there were no identical matches among the participants varied

responses, there were a few potentially groups in common: Undergraduate,
Graduate, Resources, Faculty, and Courses & Class Schedule.

6
Session 2: Interview

Format preference
Most participants picked a vertical navigational menu as their first choice (see
Table 2). Even though the MSU Mankatos menu is also vertical, participants
liked Michigan Techs menu better. They appreciated its clean and highly
organized format: the use of main headings, subheadings, and subheading
menu items. Also, compared to Michigan Tech, the items listed in the MSU
Mankato menu are not general category headings, but, as one participant
found, resemble a list of too many headings on the side (see Figure 1). It
was suggested that some of these be grouped together and given category
labels.

Figure 1: Menus, Michigan Tech vs. MSU Mankato

The group of tiles format was a hesitant second choice. Participants thought
the photos with descriptive captions were helpful as visual category icons.
However, they also critiqued it as having a jumbled appearance and looking
clustered and scattered. Every participant gave the horizontal format the
lowest rating for having too few categories that were also too broad in
scope and looking very blank, causing a user to inefficiently search through
all the drop-down menus.

Table 2: Participants format preferences

Website 1st choice 2nd choice 3 rd choice

Vertical
3 1 0
(Michigan Tech)
Horizontal
0 0 4
(Utah State Univ)
Group of tiles
1 3 0
(Univ of Minnesota)

Evaluation Tasks/Scenarios
All participants said they recognized hyperlinks on the home page because
such text are typically underlined; one participant also added that a hyperlink
is highlighted (changes a different color) when a users mouse hovers over

7
 it. Using these identifying characteristics, the phrases indicated in all three
tasks are technically supposed to be hyperlinks.

A task was deemed successful if clicking on the specified link led to another
page containing information (see column labeled Successful hyperlink in
Table 3). Otherwise, the three other column labels describe what happened
after a participant clicked on the text. There appears to be a high failure rate
in working hyperlinks.

Only a single task was successful for one person. That participant could
access information about the Spring 2014 Techniques newsletter; however,
for everyone else, clicking on the hyperlink resulted in a session that
eventually timed out.

Table 3: Summary of task responses

Task Not a Timed Blank Successful Percent
link out page hyperlink success

A 4 0 0%
Technical Communication
B 3 1 25%
Techniques newsletter
C 4 0 0%
News & Events

Questionnaire
Using a 5-point Likert scale ranging from 1 to 5 (where 1 is Strongly
Disagree, 3 is Neutral, and 5 is Strongly Agree), participants ranked MSU
Mankatos tech comm website interface. The statements used by participants
to evaluate the website (see #9 in Appendix B) correspond to the factors
from Morvilles User Experience Honeycomb.
Useful
Three-quarters of participants agreed that the tech comm website offered
information that they needed. In fact, this represents the highest mean rating
of all the statements in the questionnaire. Several participants mentioned that
the Current schedule feature prompted them to use the website regularly.
Credible
Three-quarters of participants agreed they could trust the information
presented on the tech comm website. Credibility ranked as the second highest
mean rating in the questionnaire.
Desirable
Half the participants liked the tech comm websites design. Two people
remarked positively on the menus basic vertical format; one indifferent
participant suggested that MSU Mankato keep the same basic vertical format
but suggested an update like Michigan Techs layout.
Usable, Findable
Only one-quarter of participants found the tech comm website easy to use,
making it the second lowest mean rating in the questionnaire. Regarding
usability, one participant felt the nonfunctioning hyperlinks contributed to the
websites overall image of not being entirely usable. Similarly, only
one-quarter of participants thought that information could be found quickly on
the website. In fact, one critic resented having to spend so much time
clicking to find things.

8
 Accessible
Unfortunately, the issues participants experienced with usability and
findability resulted in no participants agreeing that the website is particularly
suitable for people with disabilities.

Overall Metrics

User experience
During the previous portion, participants rated six aspects of overall user
experience (see Table 4), including measures of the websites:
Usefulness
Usability
Desirability
Findability
Accessibility
Credibility

Seventy-five percent of participants agreed (i.e., agree or strongly agree) that

the website has information they need; the same percentage also considered
the information to be believable. With an average agreement rating of 2.5 or
less, only 25% agreed that they could find information quickly and that the
website was easy to use. No one (0%) felt that the website was accessible.

Table 4: Interview Question #9, parts af

Neither
Strongly Strongly Mean Percent
Disagree Agree or Agree
Disagree Agree Rating Agree
Disagree
The website has
information I 1 2 1 4.0 75%
need.
I think this
website is easy to 1 1 1 1 2.5 25%
use.
I like how the
design of this 1 1 2 3.3 50%
website looks.
I can find
information
1 2 1 2.8 25%
quickly on this
website.
People with
disabilities would
1 1 2 2.3 0%
have no problems
using this website.
This website is a
dependable source 1 3 3.8 75%
of information.
Percent Agree (%) = Agree & Strongly Agree Responses combined

Likes, Dislikes, Participant Recommendations

After browsing the website on their own before the interview and completing
the tasks and questionnaire, participants then provided feedback on what
they liked most and least about the site, and recommendations for improving
it.
Liked Most
The following items are what participants said they liked most:
slideshow of photos

9
 the vertical aspect; color scheme matches the Universitys; home
page at least tries to look professional
information that students need (e.g., current schedule) is centrally
located and conveniently accessed from the home page
Liked Least
The following comments capture what the participants liked the least:
finding things on the website is like a maze
One participant had to do a Google search for the schools technical
communication minor to find pertinent information.
information on website needs to be more professional: text should
be direct in what its trying to say
purple hyperlinks need to stand out more from the gray background
(of the three columns on the home page)
text looks too wordy; lines of text look cramped
Recommendations for Improvement
check all links for accuracy
use a bigger font for the lines of text
use white space more effectively
consolidate text; make text concise and connected to tech comm
majors
have fewer headings in the left sidebar; headings need to be grouped
into categories
explain what students will learn from taking tech comm classes
Recommendations
A website should be accessible
Participants were asked to rank the website on accessibility.
Change Justification Severity

Enlarge the font size
Use a higher contrast for
better readability
None of the participants agreed that the website High
was accessible. The website earned the lowest
mean rating of 2.3 (out of 5) due to not being
suitable for those with disabilities.

A website is easy to use

Participants were asked to rank the website on ease of use.
Change Justification Severity

Make headings and content Only 25% of participants agreed that the website High
more concise was easy to use. For this aspect, the website
Use white space more received its second lowest mean rating of 2.5
(out of 5).
effectively
Participants said comments such as: text looked
cramped; overall information is too wordy.

10
 Information on a website can be found quickly
Participants were asked to rank how quickly information could be found on the website.
Change Justification Severity

Items in left sidebar should
be grouped into categories
Check links and content for
accuracy
Maintain consistency
throughout the website
Just 25% of participants (half were neutral) Moderate
agreed that it was quick to find information on
the website. In this regard, the website received
its the third lowest mean rating of 2.8 (out of
5).
Nearly every participant was unsuccessful in
using (what they expected to be) links that led to
additional information.

A website is appealing to its users

Participants were asked to rank how they liked the design of the website.
Change Justification Severity

Consider modifying the
current vertical menu format
to have more organization
Provide an overview about
the tech comm field
Half of the participants liked the design elements, Moderate
although 75% felt the current vertical format
could be tweaked to be more appealing and
arranged in a systematic way.
The participants who were tech comm minors
thought it would be helpful to know what they
would learn from tech comm classes.

Conclusion
Most participants felt that MSU Mankatos tech comm website contains
necessary information, and that its content is generally trustworthy. Even
though the current website is serviceable for students needs, all participants
agreed that there are areas of improvement. Revising the site to have
improved organization, enhanced readability, accuracy, and a visually
pleasing layout would do much to boost the overall user experience for a
variety of students.

11
 Appendices
Appendix A: Card Sorting Topics
Bachelors Degree Courses Internships
Undergraduate Minor Faculty Organizations
Undergraduate News & Events Publications
Certificate Admission Class Schedule
Masters Degree Requirements Techniques
Graduate Certificate Capstone Projects Financial Aid

12
Appendix B: List of Phone Interview Questions

Preliminary
1. Tell me what year of school you are in.

2. What is your major? Are you minoring in anything?

3. Why are you taking ENG 466 Usability?

Card Sorting
4. If you recall, the purpose of this usability study is to improve our schools tech comm
website. For the card sorting task, you sorted 16 topics taken from the website and
arranged them into groups that made sense to you.

Here are your results:

[Screenshot of participants results inserted here.]
Walk me through your reasoning about why you grouped items the way you did.

5. Before this interview, you were asked to look at the MSU Mankato tech comm website,
http://english.mnsu.edu/techcomm/. Considering the way how topics are organized on
the website vs. how you arranged the topics during the card sorting, which way do you
prefer? Why?

Format preference
6. These are layout examples from other schools tech comm websites. If the tech comm
website were to undergo a change, which navigation menu do you like the best:
vertical, horizontal, or a group of tiles?

Vertical (same type as current site)

http://www.mtu.edu/humanities/undergraduate/stc/
Horizontal http://techcomm.usu.edu/
Group of tiles http://cla.umn.edu/writing-studies/undergraduate

Tasks
7. Lets go to the MSU Mankato tech comm website at the following link
http://english.mnsu.edu/techcomm/. How can you tell if any specific text on this page
is a hyperlink that leads to more information?

a. The words Technical Communication in the slideshow image are underlined. Tell me
what happens when you click on that text.
b. Techniques is the name of the newsletter put out by the grad students. Tell me
more about the Spring 2014 edition when you click on the link.
c. Tell me more about what you find out when you click on News & Events.

Questionnaire
8. Before this usability study, had you ever been to the tech comm website before? (If so,
how often? If not, why not?)

9. For each of the following statements, please rate how you feel about it using a Likert
scale of 1 to 5: Strongly Disagree, Disagree, Neither Agree or Disagree, Agree, or
Strongly Agree.

a. This website has information that I need.

b. I think this website is easy to use.
c. I like how the design of this website looks.

13
 d. I can find information quickly on this website.
e. People with disabilities would have no problems using this website.
f. This website is a dependable source of information.

10. What about the website do you like the most?

11. What about the website do you like the least?

12. What suggestions do you have for improving the website?

14
Appendix C: Emails to Study Participants
Group Recruiting email
Nov 8, 2016

Hi, everyone!

This is Bernadette Cash, and Dr. Armfield has assigned you to be a participant in my usability study. I
am looking for feedback that will help improve the technical communication website of Minnesota State
University, Mankato. This is our schools own tech comm page at http://english.mnsu.edu/techcomm/.

What will you be doing in the usability study?

You will first be asked to do a card sorting task using topics from the tech comm website. In a later
session, I will conduct an interview to learn more about your card sorting responses, as well as get your
opinions, preferences, and suggestions about organizing content on the tech comm website.

How long will the testing take? About 15 minutes for card sorting, then 15 minutes for an interview.

When and where?

Want to enjoy your Thanksgiving break? Me, too. Then lets finish testing by Saturday, Nov 19! Here are
my suggested time frames for getting things done:
Nov 813, TueSun: Complete and submit the online card sorting task.
Nov 1419, MonSat: Meet with me for a phone interview.

I'll soon be emailing you a link for the first task, card sorting. After I receive everyone's responses, Ill
ask you to sign up for a phone interview with me next week. At that time, it would be helpful for you to
have access to a computer when we talk.

Give your best effort

As much as I would appreciate a quick turnaround, I also want this study to reflect accurate feedback.
Take time to provide thoughtful responses. The results of this study will be valuable to share with the tech
comm faculty.

Any questions? Feel free to email me at bernadette.cash@mnsu.edu.

Thanks so much for your help!

Bernadette Cash
Project Leader

Testing Session 1 email

Nov 8, 2016

Everyone, ready? Here is a link to the open card sorting session:

https://hdy78ys1.optimalworkshop.com/optimalsort/ajh8wyf7

Have fun!

Bernadette

15
 Reminder and Intro for Testing Session 2 email
Nov 10, 2016

Just a friendly reminder to finish card sorting before 11:59pm THIS Sunday, Nov 13. Here is that link
again: https://hdy78ys1.optimalworkshop.com/optimalsort/ajh8wyf7

Next step for the group: Check your schedule and select a time for a phone interview next week. I will
give you access to the sign-up sheet soon.

Keep the following in mind:

Interviews are on the hour and half-hour but should take about 15 minutes.
Green = unavailable time blocks, so write your name in a white or gray blank.
I will wait on you to call me for your appt. The # to call is 309 XXX-XXXX, Central Time zone.
Have internet access on a computer when we talk.
I will ask questions about you and your responses to the card sorting task.

I will also ask for your feedback about our school's tech comm website, so be sure to look at
http://english.mnsu.edu/techcomm/ BEFORE the interview.

Thanks, and I look forward to talking with each of you soon.

Bernadette

Link for Testing Session 2 email

Nov 10, 2016

Everyone,

In case you didn't get the invitation to edit, here is the link to the interview sign-up sheet:

https://docs.google.com/spreadsheets/d/1HLBrAQ-
ZykkJC38El8f4odrMfb97QSknxBZjuZ7kkhY/edit?usp=sharing

Bernadette

16
 Individual Testing Session 2 Reminder email (sample)
Nov 15, 2016

Dear ____________,

This is a reminder and instructions for Wednesday, Nov 16 at 12:00pm Central Time.

Call 309 XXX-XXXX. If you would prefer, I can call you instead but you'll need to give me your
number.
Please have internet access on a computer ready.
Take time to look through our school's tech comm website before we meet:
http://english.mnsu.edu/techcomm/

I will ask 12 questions. Below is an overview; I didn't list everything but this is what to expect. Have this
email message open when we talk.

Looking forward to our interview! Thanks again for your help with this usability study.

Bernadette

Preliminary
#1 Your year in school
#2 Your major/minor
#3 Your reason for taking ENG 466

Card sorting
#4 Your results:

<Insert screenshot of participants card sorting results here.>

#5 MSU Mankato tech comm website http://english.mnsu.edu/techcomm/

Format preference
#6 Which of the 3 do you like best? Layout examples from other schools' tech comm websites:
Vertical http://www.mtu.edu/humanities/undergraduate/stc/
Horizontal http://techcomm.usu.edu/
Group of tiles http://cla.umn.edu/writing-studies/undergraduate

Tasks
#7 MSU Mankato tech comm website http://english.mnsu.edu/techcomm/
a.
b.
c.

Questionnaire
#8 How often you've been to our school's tech comm website
#9 For the next 6 statements, use a Likert scale of 1 to 5:
1 Strongly Disagree
2 Disagree
3 Neither Agree or Disagree
4 Agree
5 Strongly Agree
a)
b)
c)
d)
e)
f)

#10 What do you like most about the website?

#11 What do you like the least about the website?
#12 Suggestions for improvement?

17