Evaluation

Project Final Report

Online Module for Student Staff at ASU University Housing




















May 2, 2016
Prepared for University Housing at Arizona State University
by Takako Kobayashi

Abstract

This report describes an evaluation of the newly designed online module Community Mentor
Workplace Communication for Event Planning (CM Event Planning) for University Housing at
Arizona State University. The evaluation was conducted during the spring of 2016 with the
purpose of assessing the effectiveness of the online training module and ways to improve it
before the actual implementation.
The CM Event Planning online module was designed during the spring 2016 semester by Takako
Kobayashi who has been working as a Community Mentor for the Sophomore Year and Beyond
(SYB) community at Cholla Apartments. CM Event Planning consists of a scenario-based learning
content and assessment portion (i.e., quiz items). The scenario-based e-learning training was
designed to help CMs understand the entire process of organizing community events and
acquire procedural skills for the specific paperwork for item purchasing. The training time
required for the module is estimated to be 10 15 minutes.
To assess the effectiveness of this online training module, data were not only collected from
the current CMs, but also from internal stakeholders such as Community Directors who are
supervising CMs at their housing communities and Administrative Assistants who usually deal
with the paperwork. The following data were collected: an online survey for CMs, CD interviews,
and Administrative Assistant interview.
The results of the evaluation project indicate that the overarching experiences with the online
module were positive. Despite different opinions in ways to improve, the overall concerns are
associated with its interface instead of the learning content. Furthermore, the ways in which
the online module can be used among the participants are similar (i.e. as a supplemental
resource for CMs). The main difference, however, is that the CDs and Administrative Assistant
are aware of the additional process needed for the official deployment.
It is likely that not best time to conduct the evaluation project. It is difficult to measure the
learning effectiveness without newly hired CMs who have not received the training previously.
It is therefore recommended that the revised online module will be implemented at the
beginning of next year, with a more extensive, pretest-posttest designed evaluation.

Introduction

Beginning in the spring 2016 semester, Takako Kobayashi designed and developed the online
training module called Community Mentor Workplace Communication for Event Planning (CM
Event Planning) in order to help CMs understand the whole process of organizing community
events and acquire procedural skills to complete the specific paperwork for purchasing items.
This self-paced online module consists of a scenario-based learning content and assessment
portion. The plans for the official deployment as a required training upon hire are under
consideration.

Since this is the prototype of the online module, stakeholders from University Housing are
eager to assess the modules effectiveness in order to make decisions regarding its future
revision and implementation. As such, Takako Kobayashi has been tasked with evaluating this
online module in order to answer the following main evaluation questions:

1. How effective is the online module?
2. In what ways can the online module be improved?

In order to carry out this evaluation, a variety of methods and data sources were used including
online surveys with CMs, interviews with CDs, and interviews with Administrative Assistant.

The purpose of this document is to serve as a final report for this evaluation, thereby giving the
stakeholders of the ASU University Housing department the information necessary to make
decisions regarding the online modules future. As such, this report chronicles in detail the
evaluation procedures mentioned above, and includes a summary of the key results and
findings. In addition, a discussion is provided at the end which describes possible implications of
the evaluation project and offers recommendations for the future.

Description of CM Workplace Communication Online Module

The CM Event Planning is a newly designed online training module for University Housing at
Arizona State University (ASU) which aims for student staffs also known as Community Mentors
to improve workplace communication for organizing community events. The position of a
Community Mentor (CM) refers to a student staff for ASU University Housing. One of the
responsibilities a CM has is to organize community events for student residents. The success of
organizing community events depends on concise and complete workplace communications
including the specific paperwork among fellow CMs, Community Directors (CDs), and
Administrative Assistants. Currently, it is only being trained by a lecture-style training provided
by Administrative Assistants at the beginning of every semester. Despite the current face-toface training, many CMs are not prepared well until they are actually required to complete the
paperwork by themselves. In addition, due to time constraint for events, Administrative
Assistants tend to fix mistakes in the paperwork made by CMs, which imposes extra work on
them.

The goals of the online module are threefold: 1) to enable learners to list all required
documents for the paperwork specialized for event planning; 2) to enable learners to identify
the procedure of workplace communication; and 3) to enable learners to compose a public
purpose in the paperwork.

The official deployment of the online module is under consideration in the summer of 2016,
with the prototype tryout occurring during the Spring 2016 semester. Takako Kobayashi was
the sole evaluator working on this evaluation project and the developer of the CM Event
Planning online module. She has been working as a CM at Cholla Apartments since the Fall 2015
term. Throughout the Spring 2016 term, Takako Kobayashi has been working closely with her
CD Bradley Bolin M.Ed. and Administrative Assistant Midi Brown to develop the online module
and the pilot implementation.

The online module was designed consisting of a scenario-based learning content and quizzes. In
the scenario, an experienced CM functioning as the narrator was leading a new CM (i.e.,
learners) through the process of organizing events. The quiz items were also included over the
scenario. There were five multiple choice and one fill-in question items. The scores of each user
were recorded in the web space called the Scorm Cloud system. In the system, learners who got
more than 80% of the quiz scores were identified as Completed. The training time required
for the module is estimated to be 10-15 minutes.

Evaluation Methods

Participants
During the course of this evaluation, a variety of people from University Housing participated in
the evaluation project. The participants in the project included the following from housing
communities across the Tempe campus:

Current Community Mentors

Community Directors

Administrative Assistants

Procedures
Several different methods were used to gather data during the course of this evaluation.
Included in these were online surveys of CMs, a focus group interview with CMs at Cholla
Apartments, interviews with two CDs, and interview with one Administrative Assistant for the
Sophomore Year and Beyond (SYB) community. Since this evaluation was conducted after the
prototype of the online module was distributed, data collection was organized around the
schedules of the individuals at housing communities. As a result, the data collection occurred
during the first two weeks of April 2016, as this was a time that was most convenient for
everyone. Outlined below is a brief description of the procedures used in collecting data for
each of the methods described above.

Online Surveys
CMs across the Tempe campus were administered online surveys to capture their perceptions
of the modules quality, and to provide insight into areas for improvement. The online surveys
were sent to all CDs, Administrative Assistant and University Housing officers with the
distribution of the prototype via email asking for further distribution to their CMs. The survey
consisted nine questions, can be found in Appendix A

Focus Group Interview with CMs at Cholla Apartments
A focus group interview was conducted with a group of nine CMs working for Cholla
Apartments on April 12, 2016. They were required to complete the online module prior to the
interview. The interview lasted approximately 40 minutes. The interview was designed to gauge
how their experiences taking the online module were, and whether they generally find it
effective and helpful in their learning. Appendix B contains the interview questions used with
the CMs.

Interviews with CDs
After the pilot implementation, two CDs from the SYB community were interviewed. Interviews
generally lasted between 10 and 20 minutes. During the interview, CDs were given an
opportunity to express their thoughts, concerns, and ideas pertaining to the online module. All
interviews were comprised of the same series of questions in order to maintain consistency
throughout. The interview questions are presented in Appendix C of this report.

Interview with Administrative Assistant
5

Administrative Assistant for the SYB community was interviewed in order to obtain an
administrative viewpoint of the instructional contents in the online module. The interview
lasted between 20 and 30 minutes, and was designed to give her an opportunity to explain her
current trainings goals, strengths, weaknesses, and outcomes of using the online module, and
any related long-term plans for the future. The complete list of the interview questions can be
found in Appendix E of this report.

Data Sources
In order to adequately evaluate the CM Event Planning online module, information was
obtained from several key data sources. A description of each is provided below.

Current Community Mentors
CMs for ASU housing communities across the Tempe campus served as a primary source of data
in this evaluation. Many CMs are undergraduate students in sophomore or above and some of
them have been CMs more than once. All current CMs have attended to the training session
about the process of purchasing items presented by their Administrative Assistants at least
once before. Their perspectives and experiences of taking the online module were captured via
online surveys as well as a face-to-face focus group interview.

Community Directors
CDs are direct supervisors for CMs at each housing community. CDs are also responsible for
arranging the training sessions for their CMs every semester. The perceived strengths and
weaknesses, and perceived outcomes of using the online module were solicited from the faceto-face interviews with CDs for the SYB community. To this end, the CDs offered insight into the
areas for further improvement.

Administrative Assistants
As part of the process of purchasing items, on-site specialists Administrative Assistants are
available to each of the housing communities to help with organizing of the community
programs. They currently offer face-to-face training session with CDs every semester. The
current training practices and perceptions of the instructional content in the online module
were captured via face-to-face interview with Administrative Assistant for the SYB community.
This information was valuable in better understanding the practices of online training for CMs,
and in providing insight to the implementation strategies.

Results

The following is a summary of the key findings from the evaluation. It is broken down into subsections, each corresponding to one of the data collection methods described in the Procedures
section above. Complete data for many of the instruments are available in the appendices to
this report.

Online Surveys
In total, 27 completed surveys were collected. The survey used a Likert-scale of four levels:
Strongly Disagree, Disagree, Agree, and Strongly Agree.
In general, as Tables 1 and 2 depict, the participants had positive learning experiences with the
online module. Overall approximately 80% of the learners answered either Agree or Strongly
Agree which suggested that most of the CMs satisfied with the quality of the online module.

Table 1: Online Survey Results
Strongly
Disagree

Agree

Strongly
Agree

Disagree

Q1

13

13

27

Q2

16

10

27

Q3

15

26

Q4

15

26

Q5

16

27

Q6

11

15

27

Table 2: Comparisons among Survey Questions
Strongly
Disagree

Disagree

Agree

Strongly
Agree

Q4

Q5

70%
60%
50%
40%
30%
20%
10%
0%
Q1

Q2

Q3

Q6

Based on their responses regarding ways to improve, approximately 95% of the participants
indicated the technical or design elements in the module such as:

Quiz setting (e.g., setting quiz items as non-shippable)

Sound

Clearer directions on how to control the pages

This suggests that it is necessary to polish the interface of the online module.

Focus Group Interview
Nine CMs working for Cholla Apartments were interviewed to capture their learning
experiences with the online module. What follows are the key findings from the interviews.

Overall Learning Experiences with the Online Module
Similar to the responses from the online surveys, the nine CMs experiences with the online
module were positive. Despite the different feedback on the online module among the CMs at
the focus group, they felt as though the online module seemed to be helpful as review for CMs
who continue the position, whereas there are many assumptions of background knowledge for
newly hired CMs.

Relative Strengths and Weaknesses between the Current Training and Online Module
In general, the focus group saw the strengths of the current training style as human connection
with their Administrative Assistant. In addition, they felt that it is easy to ask questions with
their Administrative Assistant during the lecture.

As far as weaknesses go, they sited the structure of overall training sessions as the primary
weakness. In general, the entire training sessions last between a week to a month. During the
sessions, CMs have to attend to the assigned trainings or complete their duties every day. With
that said, it is difficult to pay attention to the lecture due to the exhaustion from the previous
trainings.

Regarding the strengths of the online module, they indicated the self-paced learning and the
content in the module as the main strengths. In addition, they felt as though the online module
allows learners to go back the content when needed, which might help develop the required
procedural skills. In contrast, as weakness, the focus group pointed out that there is no
accountability that learners actually listen and learn from the online module.

The Ways to Improve the Online Module
In addition to the similar responses from the online surveys regarding the ways to improve, the
focus group suggested adding contents for experienced CMs. They felt as though the learning
content was very concise and provided learners with all the information required. Based on
their experiences with event planning, some even planning process might be more complicated
than the scenario in the module. With that said, offering additional information such as slides of
Frequently Asked Question and downloadable job aid files were recommended.

8

Interviews with CD
Although the two CDs seem to have slightly different feedback for areas to improve in the
online module, overall they felt using the online module is a real asset to CMs. In addition, both
CDs seem to be in agreement that training the procedural skills in item purchase is one of the
most important, on-going subjects in CMs responsibilities as student leaders.

In addition, one of CDs felt that the online module could be used as an applicable review for
returning CMs. As described above, many students tend to continue the CM positions. Instead
of attending into the face-to-face training lectured by Administrative Assistant as requirement,
the online module can be provided to returning CMs so that training sessions every semester
will be organized more appropriately.

Complete interview with one of the SYB CD is available in Appendix D.

Interview with Administrative Assistant
Administrative Assistant for the SYB community was interviewed to gain her perspective on the
instructional content in the online module, her current training practices, and any related longterm plans for the future. Despite the positive experience with the online module, several
elements in the instructional content were detected. In addition, she felt that the online
module could be used as a suitable supplementary resource accompanied by the face-to-face
training based on the goals and benefit of the current training style she expressed.

Based on her response, the intent of the current training is to introduce each document needed
for item purchasing and explain its function so that CMs will be able to select appropriate
documents when needed. In addition, unlike CDs who are living and own their office in their
assigned communities, Administrative Assistants do not necessarily have their offices in the
assigned communities. With that said, the face-to-face training session is the important
opportunity for CMs to identify who their Administrative Assistant is, recognize their role in the
housing communities, and make connections.

Currently, upon hire, CMs are required to complete several online trainings provided by Human
Resources. University Housing has never provided its own online training before. Therefore,
there will be approving process for its official implementation.

Completion Data of the Online Module Record
Originally, completion data of quizzes integrated in the online module would be used for item
analysis to detect strengths and weakness of the test contents and feedback functions. As
described previously, however, all participants received the face-to-face training of the process
for purchasing items at least once before taking this prototype. The surveys shows that 64% of
the participants have been the position of CM more than a year, which might imply that most
of the participants were already familiar with the learning content before this pilot
implementation. Due to this implication, it seems that data analysis would offer virtually no
meaningful insights to the evaluation. Therefore the details of these data will not be presented
at this time.
9

Discussion

The purpose of this evaluation is very specific. It is to determine 1) to what degree the CM
Event Planning online module contributed to the CMs understanding of the workplace
communication for event planning, and 2) what can be learned from this evaluation that can
improve the future implementation of the online module.

The overarching finding from this evaluation project is that CMs, CDs and Administrative
Assistant feel the CM Event Planning online module is a valuable tool in motivating and
assisting CMs to prepare to perform well on the process of event planning. All groups expressed
very positive feelings about it.

However, most of the evaluation methods asked the participants perceptions on the online
module. With that said, it is difficult to measure the effectiveness of the online module based
on the current responses. Furthermore, since this evaluation was conducted under a
compressed timeframe, data from newly hired CMs were not available to be obtained and
considered in the analysis. These data would likely have yielded valuable information about the
effectiveness of the online module that could help with future implementations.

Therefore, for these reasons, the following recommendations are suggested.

1. Re-design the evaluation structure. As identified above, these evaluation methods
were not appropriate to measure the effectiveness. In order to have a clear before
and after effects of the online module, the ideal evaluation project is based on the
pretest-posttest design principle. With that said, evaluation methods and procedures
must be modified.

2. If funding allows, implement the revised module next year. Based on the findings,
there still exist areas for continued development. From an evaluation standpoint,
implementing the revised module will be appropriate for any future evaluations that
may take place.

3. Re-run this evaluation prior to the training sessions next year to get a clearer answer to
the evaluation questions raised in this study. But make sure to do more upfront
planning to ensure that all data can be collected especially the survey questions. This
will also give CMs, CDs, and Administrative Assistant more flexibility in answering
surveys and participating in interviews. But most importantly, it will have given the
online module more time to be revised, hence giving Takako Kobayashi the needed
time to gain more experience with the skills in improving the interface to make better
use of it.



10

Project Cost

Takako Kobayashi was working independently and has chosen not to charge ASU University
Housing for this formative evaluation. The CM Event Planning online module has been recently
developed by Ms. Kobayashi, so she wanted to modify this online module based on this
formative evaluation prior to the actual implementation. She also wanted to acquire additional
evaluation projects and would use this online training evaluation as evidence of the quality of
her work. The below is, however, a project cost actually spent during the course of evaluation.

Operations

Refreshments for focus groups
$30
Printing and miscellaneous expenses
$100
Total Project Cost
$130

11

Appendix A: Online Survey

The purpose of this survey is to provide your valuable feedback regards learning outcomes
using the online course CM Workplace Communication for Event Planning. Please answer each
statement honestly checking in the column that represents your response to the statement:
1. This online training was well organized.
Strongly Disagree
Disagree
Agree
Strongly Agree




Comment (optional)

2. This online training was well organized.
Strongly Disagree
Disagree
Agree
Strongly Agree




Comment (optional)

3. The primary content (information & examples) were appropriate for me personally.
Strongly Disagree
Disagree
Agree
Strongly Agree




Comment (optional)

4. The practice actively involved and engaged me in my learning.
Strongly Disagree
Disagree
Agree
Strongly Agree




Comment (optional)

5. My interest and motivation were maintained throughout the course.
Strongly Disagree
Disagree
Agree
Strongly Agree




Comment (optional)

6. I would recommend this course to new CMs.
Strongly Disagree
Disagree
Agree
Strongly Agree




Comment (optional)

7. Please indicate what you liked best about this course.

8. Please share any additional insights that might help improve the online training.

9. Please indicate how many semesters you have been a CM.
1 semester
2 semesters
3 semesters
4 semesters
5 or more





10. How long did it take to complete this online course?
Less than 15 mins
Between 15 to 25 mins
More than 30 mins
Other (please specify)

12

Appendix B: Focus Group Interview Questions

Introduction:
Thank you so much for participating my online course. I have been working on creating this
online course and evaluating it as part of my studies throughout this semester. I really
appreciate you taking the time to participate.
For the next 30 minutes, I will be asking you several questions about your opinions and
experiences with the online course. Please provide honest and candid feedback since I would
like to 1) improve the online course where needed and 2) to understand the ways in which the
course is or is not effective. Of course, there are no wrong answers.
As you answer each question, I will try to note the essence of your response. I want to capture
what you say, so please let me know if Ive represented your thoughts accurately. I will also be
recording our conversation so that I can be sure to capture everything you say. Does anyone
feel uncomfortable to do so?
All of the info you provide today will be treated confidentially. I will not be using any names in
my evaluation report. I will use numbers to represent you like student staff 1. This will just
show how many different people provided the various responses. Also, please feel free to get
and help yourself to the refreshments. Do you have any questions or concerns about what
were going to do here today? Okay, lets get started!
1.

Now I would like you to think back on your experiences taking the online course... Can you
describe that experience for me?

2.

Recall the training about the paperwork we had at the beginning of every semester, please
compare the training and the online course.

3.
4.

Tell me about positive experiences you've had with the online course?
Tell me about disappointments you've had with the online course?

Tell me strengths and weakness of each learning format

What recommendations do you have for me regarding ways to improve the course?
What specific skills, abilities, experience, etc. did you gain from taking the online course?

13

Appendix C: Interview Questions for CD

Introduction:
Thank you so much for participating my online course.
For the next 30 minutes, I will be asking you several questions about your opinions and
experiences with the online course. Please provide honest and candid feedback since I would
like to 1) improve the online course where needed and 2) to understand the ways in which the
course is or is not effective. Of course, there are no wrong answers.
As you answer each question, I will try to note the essence of your response. I want to capture
what you say, so please let me know if Ive represented your thoughts accurately. I will also be
recording our conversation so that I can be sure to capture everything you say. Do you feel
uncomfortable to do so?
All of the info you provide today will be treated confidentially. I will not be using any names in
my evaluation report. I will use numbers to represent you like CD #1. This will just show how
many different people provided the various responses.
Do you have any questions or concerns about what were going to do here today?
1.

Now I would like you to think back on your experiences taking the online course... Can you
describe that experience for me?

2.
3.

Tell me about positive experiences you've had with the online course
Tell me about disappointments you've had with the online course

What recommendations do you have for me regarding ways to improve the course?
What do you foresee in the process of the actual implementation?

14

Appendix D: CD Interview Transcription

Interviewer: So, now I would like you to think back on your experiences taking the online course.
Can you describe that experience for me?

CD: Sure. I believe that initially my first response was the voice over. I think that it was
welcoming voice. I think thats so important. Those small details can be very affective. I would
say the voice over whoever the voice was well spoken..ah spoke very slowly and it was very
affective to get information across. So I think that was initially, I think that photos are very well
placed, I think. It kinda connected to the purpose of the exam, test and I feel like that flows very
easily. I think that was very good. Actual quiz flows very easily. It was not too long not too short.

Interviewer: Is there any disappointment?

CD: Ah..let me seeNo I cant say thats necessarily its disappointment. I would say the only
thing actually is to make each section mandatory to fill them in. Again, like practice makes
perfect. So, definitely everyyou did the ERF(going over the module to checking the content)
then we have BMF and Food Waiver so you have all forms, which is great. I would say may be
add examples so like you did one including food. So like do one of those, like maybe do two
quizzes like one where its kind ice and gum. Those small things that can get confusing, Oh do I
need BMF for gum? cuz people automatically think that with Canes thats food. But a lot of
people dont realize that ice, gum, those small things are still categorized as food. Thosethat
kind of like helping them understand purposely.

Interviewer: So what recommendations do you have for me regarding ways to improve the
course?

CD: I would sayI think probably having them fill out more of the form. Just would be my
recommendation. Or like you giving them more examples. Is there way that they can create
more?

Interviewer: So maybe the last question, what do you foresee in the process of the actual
implementation? What do you foresee any challenges or any benefit

I would say I am assuming approval from HR.

15

Appendix E: Interview Questions for Administrative Assistant

Introduction:
Thank you so much for participating my online course.
For the next 30 minutes, I will be asking you several questions about your opinions and
experiences with the online course. Please provide honest and candid feedback since I would
like to 1) improve the online course where needed and 2) to understand the ways in which the
course is or is not effective. Of course, there are no wrong answers.
As you answer each question, I will try to note the essence of your response. I want to capture
what you say, so please let me know if Ive represented your thoughts accurately. I will also be
recording our conversation so that I can be sure to capture everything you say. Do you feel
uncomfortable to do so?
All of the info you provide today will be treated confidentially. I will not be using any names in
my evaluation report. I will use numbers to represent you like CD #1. This will just show how
many different people provided the various responses.
Do you have any questions or concerns about what were going to do here today?
1.

Now I would like you to think back on your experiences taking the online course... Can you
describe that experience for me?

2.
3.

Tell me about positive experiences you've had with the online course
Tell me about disappointments you've had with the online course

What recommendations do you have for me regarding ways to improve the course?
What do you foresee in the process of the actual implementation?

16