Anda di halaman 1dari 8

Takako

Kobayashi
CM Event Planning

Tryout Report

CM Workplace Communication for Event Planning Tryout Report


Designer: Takako Kobayashi
Date: April 4th 12th, 2016

Program Overview
CM Workplace Communication for Event Planning (CM Event Planning) is a self-paced online module
designed for Community Mentors (CMs) in University Housing communities at Arizona State University
(ASU). The position of a Community Mentor (CM) refers to a student staff for ASU University Housing.
One of the responsibilities a CM has is to organize community events for student residents. The success
of organizing community events depends on concise and complete workplace communications including
the specific paperwork among CMs, Community Directors (CDs), and Administrative Assistants. Although
CMs receive the lecture of the paperwork specialized for organizing events as part of training sessions at
the beginning of every semester, many CMs are not prepared well until they are actually required to
complete the paperwork by themselves. In addition, due to time constraint for events, Administrative
Assistants tend to fix mistakes in the paperwork made by CMs, which imposes extra work on them.
CM Event Planning consists of a scenario-based learning content and assessment portion (i.e., quiz
items). This scenario-based e-learning project will help CMs understand the entire process of organizing
community events and acquire procedural skills for the required paperwork. Learning the entire process
will promote smooth preparation for events, which contributes to successful community building.
Furthermore, e-learning opportunity will help organize the training sessions at every semester more
properly.

Instructional Objectives
1.
2.
3.

Given available documents for housing communities, learners will be able to list all required
documents for the paperwork specialized for event planning.
Given the scenario describing entire situation of organizing an event, learners will be able to
identify the procedure of workplace communication.
Given the required documents, learners will be able to compose a public purpose in the
paperwork.

Instructional Materials
CM Event Planning is a self-paced online module consisting of a scenario-based learning content and
assessment portion (e.g., quizzes). The online module was created using Adobe Captivate 9 and
delivered using Scorm Cloud, a Learning Management System (LMS) for this tryout. The training time
required for the module is estimated to be 10 15 minutes.

Evaluation Method
Participants
The informative evaluation was done as a group evaluation with current CMs, CDs and Administrative
Assistants working across the Tempe campus. The major required qualifications to maintain the CM
position are: 1) minimum semester and overall GPA of 2.75; 2) good conduct standing with the
University; and 3) sophomore status or above by time of hire. Many CMs are undergraduate students
and some of them have been CMs more than once. The number of CM at each community may vary.
Upon hire, CMs are required to attend to the series of training sessions in the face-to-face format
provided by their CDs prior to every semester. The topic in CM Event Planning is included as part of the
trainings by inviting their Administrative Assistants to give CMs a presentation of the paperwork. With

Takako Kobayashi
CM Event Planning

Tryout Report
that said, all current CMs have attended to the training session presented by Administrative Assistants
at least once before.
Process
The prototype was distributed to all CDs, Administrative Assistants, and ASU University Housing officers
via email asking for further distribution to their CMs. With the distribution, a follow-up survey was also
attached. Nine CMs working at Cholla Apartments were required to take this online module prior to
their team meeting on April 12, 2016. A semi-structured focus group interview was conducted during
the team meeting.
Data Sources and Collection
The program developer has been working at Cholla Apartments as a CM since the Fall 2015 term.
Throughout the Spring 2016 term, the program developer has been communicating closely with her CD
Bradley Bolin M.Ed. and Administrative Assistant Midi Brown to develop the online module. The online
module consisted of six quizzes that can be reported to the Scorm Cloud system (i.e., five multiple
choice and one fill-in questions). The follow-up survey and focus group interview with the nine Cholla
CMs were designed to collect data from the small group to help evaluate the online module and identify
appropriate revisions in the program.

Results
Student Achievement
The reportage summary from the Scorm Cloud system is shown on Table 1. There were 16 individuals
who logged in the system. Among all participants, 15 people actually launched and participated in the
online module (94%).
Table 1: Reportage Summary

Course Launch
Course Completion
Average Score

Percentage (Actual #)
94% (15)
19% (3)
46%

Takako Kobayashi
CM Event Planning

Tryout Report
The pie graph represents more details of course completion. Due to the setting in the LMS, the
Incomplete condition contains three types of users: 1) people who skipped all quizzes; 2) people who
took at least one quiz but withdrew from the online module; and 3) people who got less than 80% of
quiz scores that determined pass or fail. Table 2 illustrates the numbers of participants based on three
categories.
Course Completion Pie Chart

6% (1)

19% (3)
Complete
Incomplete
75% (12)

Unknown


Table 2: Division of Incomplete Condition
Incomplete Details
Percentage (Actual #)
Group 1 (No Quiz)
17% (2)
Group 2 (Withdraw)
8% (1)
Group 3 (Less than 80%)
75% (9)
Apparently, nine users went over the entire online module and took quizzes. Since this module is a
training opportunity, which may not necessitate the grading scale as the prototype did. The grade
setting, therefore, should be discussed with ASU University Housing further in the future.
Next, the actual number of Complete learners who answered correctly and percentage score in quiz
items are shown in Table 3. Table 3 illustrates that the students were able to answer all multiple-choice
items correctly. The second question (Q2), the fill-in quiz, was not answered correctly by all three
Complete learners, which is associated with the current Captivate 9 software capacity. Generally, In
order to obtain the full score of a fill-in quiz item, the user input must exactly match the same words or
phrases validated by the program developer as an answer key. Instead of categorizing Q2 as a quiz item,
it might be better to set this item as survey and exclude from the quiz items.
Table 3: Actual Number and Percentage Score (Complete)
Quiz Items
Actual Number
Q1
3
Q2
0
Q3
3
Q4
3
Q5
3
Q6
3

Percentage Score
100%
0%
100%
100%
100%
100%

Takako Kobayashi
CM Event Planning

Tryout Report
Additionally, the actual number of Incomplete Group 3 learners who answered correctly and
percentage score in the quiz items are shown in Table 4. In addition to the lowest accuracy of Q2 like
Table 2, Table 4 illustrates that most of the learners were not able to answer the fourth quiz item (Q4)
correctly even though the accuracy rates of other items were 50% or higher. As part of the setting, the
learners were allowed to attempt infinite numbers to take those quiz items. Thus, it might be helpful to
indicate that infinite attempts of taking the quizzes are available at the beginning of the online module.

Table 4: Actual Number and Percentage Score (Incomplete Group 3)
Quiz Items
Actual Number
Percentage Score
Q1
8
89%
Q2
0
0%
Q3
5
56%
Q4
1
11%
Q5
6
67%
Q6
6
67%

Learner Attitudes (Survey Responses)
Student responses on the follow up survey are summarized in Tables 5 6. Overall, the students had
extremely positive responses to the online module. For example, 100% of the participants either
strongly agree or agree with all closed-ended questions (see Table 5).
Table 5: Follow-up Survey Part 1
Survey Question

This online course was well organized.


The primary content was emphasized clearly.
The primary content (information & examples)
were appropriate for me personally.
The practice actively involved and engaged me in
my learning.
My interest and motivation were maintained
throughout the course.
I would recommend this course to new CMs.

Strongly
Disagree

Disagree

Agree

20% (2)
40% (4)
44% (4)

Strongly
Agree
80% (8)
60% (6)
56% (5)

44% (4)

56% (5)

70% (7)

30% (3)

60% (6)

40% (4)

Takako Kobayashi
CM Event Planning

Tryout Report
When asked what they liked best about the program, the largest number of learners (5) responded, the
provided information was concise. When asked recommendations regarding the ways to improve the
module, most of feedback suggested either technical or structural improvement of the module (see
Table 6).
Table 6: Follow-up Survey Part 2
Please indicate what you liked best about this course.
Very direct and informative
I like that in included everything I needed and nothing I didn't
It gave all the information required, but did not include any extra things that might confuse people.
To the point, very short and concise
I think having the questions throughout was really helpful, especially the parts where they practice
filling out the paperwork
It was concise and short but not too short as to not be unclear.
interactive components
Very nice presentation.
It was thorough
Please share any additional insights that might help improve the online training.
Very well done and well put-together! Nice.
Mine kept selecting the first box to type in when I wanted a second, third, etc. box on the page and
that was frustrating but also maybe my fault?
Clearer directions on how to move past the page. There were times when I was unsure what to do
to keep the course moving.
Don't make the sections skip-able, otherwise people can get through the information without
really listening to it all
Having all the quiz questions at the end might help get a better sense of how much the student
remembers rather than how much they're repeating.
I think you need to explain what to click after you answer a question. I found myself sitting there
waiting for the video to resume or clicking buttons randomly to make it go again.
I think that if the course did not say lowercase letters were incorrect, those taking the module
would have an easier time completing the course.
I just found one tiny typo, so you might want to take it out if you have to submit this as an
assignment or to others. Toward the end on the "Last Preparation" page, there is an extra "a" in
"prepare."
1. There is an assumption of background knowledge. The first question is only a minute in and
theres no previous training. Also, the question involves abbreviations that have never been
presented. Maybe this could be used for returning staff, but definitely not for new staff.
2. The error sound is very abrasive.
3. The ERF I fill out is an excel spreadsheet that generally does the math for you. I understand the
purpose, of having it this way for the tutorial, but its essentially added steps that add time.
4. Reiterating that you need to reclick the play button or having another continue button come
up after you fill out the appropriate information could be helpful.
5. You can reviw the sample public purpose

Takako Kobayashi
CM Event Planning

Tryout Report
Overall, participants were most likely to have positive experiences with the online module. As described
previously, however, all participants received the face-to-face training of the paperwork, the exact same
topic of the online module, at least once before taking this prototype. The bar graph shows that 75% of
the learners have been the position of CM more than a year, which might imply that most of the
participants are already familiar with this topic. This fact seems to affect learner satisfaction and the
statistical analysis of the quiz items identified above.

Learner Demographic Information Bar Graph

Please indicate how many semesters you


have been a CM.
1 semester

12.5% (1)

2 semesters

12.5% (1)

3 semesters

37.5% (3)

4 semesters
5 or more semesters

37.5% (3)

Lastly, the length of time to complete the online module was investigated (see Table 7). 67% of learners
completed the module less than 15 minutes, which is the ideal time commitment based on the audience
analysis conducted previously.
Table 7: Demographic Information from Follow-up Survey
Survey Question
Less than 15
minutes

Between
15 to 25
minutes
33% (3)

More than 30
minutes

Other

How long did it take to complete this


67% (6)
0
0
online course?

Learner Attitudes (Focus Group Interview)
The program developer was able to conduct a focus group interview with nine Cholla CMs and their CD
to receive more insightful feedback. The list of the interview questions is located in Appendix A.
Generally, the interviewees were satisfied with the quality of the CM Event Planning online module. The
plausible comments they shared were:
1.
2.
3.

There is no accountability that learners actually listen to the module.


This online module is helpful for returning CMs to review.
For new CMs, this module can be provided as a supplementary resource in addition to the face-toface training

Takako Kobayashi
CM Event Planning

Tryout Report
4. The module should be equipped with more additional resources (e.g., cheating sheets and FAQ
section at the end)

Revisions
For the Future
The possible areas of revisions are listed below.

User experience design (UXD)


o
Quiz setting
o
Technical troubleshooting

User interface (UI)


o
Extra contents for further learning

Learning content
Generally, as the approach for the revisions above, more detailed feedback from subject matter experts
(e.g., CDs and Administrative Assistants) will be necessary to polish and finalize the online module. The
actual implementation should also be discussed and arranged while finalizing the module.

Summary
Based on the analysis of the follow-up survey and focus group interview, the online module seems to be
helpful for CMs. Despite the success of this prototype as indicated above, there still exist areas for
continued development. The program developer will continue polishing the module toward the actual
implementation in the future.

Takako Kobayashi
CM Event Planning


Appendix A

Tryout Report

Focus Group Interview Questions


1.

Now I would like you to think back on your experiences taking the online course... Can you describe
that experience for me?
a. Tell me about positive experiences you've had with the online course?

b. Tell me about disappointments you've had with the online course?


2.

Recall the training about the paperwork we had at the beginning of every semester, please
compare the training and the online course.
a. Tell me strengths and weakness of each learning format


3.

What recommendations do you have for me regarding ways to improve the course?


4.

What specific skills, abilities, experience, etc. did you gain from taking the online course?