Anda di halaman 1dari 47

Participatory Learning and

Assessment in Competency-Based
Contexts
Daniel T. Hickey
Associate Professor of Learning Sciences
Center for Research on Learning and Technology
Indiana University

Research Context
2009 in Sakai
@IU
2012 in
CourseBuilder
BOOC

2013 in Canvas
@ IU & IUHS
4

Use public contexts give


meaning to knowledge tools
Reward productive
disciplinary engagement
Grade artifacts through local
reflections
Let individuals assess their
understanding privately
Measure aggregated
achievement discreetly

James
Greeno

Ed Psych 540: Learning &


Cognition in Education
Challenging to Teach
Diverse experience & ambitions
among students
Ambitious but busy instructor!
Started adjuncts in 2011

Typical Accountability
Use textbook item bank
Include a pretest

Online in Sakai in 2009


Canvas in 2015

Gina Howard

Andi Rehak

Joshua Quick

The P540 Team

Lauren Smith

Xinyi Shen
7

Firat Soylu

Educational Assessment BOOC


P507 Assessment in Schools embedded for credit

Offered to 500 students for free in 2013 with a grant


from Google using CourseBuilder
460 registered, 160 started, 60 finished, 9 in P507

Offered a smaller streamlined version in 2014


Added videos and quizzed and automated key features
60 started, 22 completed, 12 in P507

Use public contexts to give


meaning to knowledge tools
Problematize knowledge from
learners perspective
Publically open up the discipline

Learners define (and redefine) a


personally-meaningful context
Can be disciplinary problem, project,
investigation, etc.
Embodies their experience, interests,
& aspirations
11

Rogers
Hall

15

16

20

Reward productive
disciplinary engagement
DE involves declarative knowledge and
cultural practices
PDE connects knowledge and practices,
makes connections, brings in resources,
etc.
Support PDE so it can be rewarded

Randi
Engle

Comment directly & publically on wikifolios


Work in the open and encourage lurking
No grades or mandatory posts
25

28

29

30

Grade artifacts through


local reflections
Wikifolios generate lots of text and
comments
Grading artifacts and comments
undermines participation

Grade reflections on PDE.


Critical engagement (suitability of context
for learning concepts)
Collaborative engagement (what you
learned from others)
Consequential engagement
(consequences of new knowledge)

31

Melissa
Gresalfi

33

Let individuals assess


understanding privately
Offer ungraded open-ended
items and performance tasks
Cover all of the big ideas in the
assignment
Use to re-engage rather than
remediate.
Items themselves have little formative
value beyond the correct answer

Pamela
Moss

34

35

Measure achievement
discreetly
Standards-oriented tests
Measure gains using pretests, compare
instruction, document improvement
Primary student function is motivating
prior engagement

Refine tests (but dont teach to them)


Use the item analysis routines in your LMS

Protect test security


Withhold item-level feedback

Randomly selected timed items


Proctor or use unsearchable items

Me!

Results:
Overall Public Engagement in P540
Average Words per Wikifolio & Words per Groupwiki
5000
4000
3000
Wikifolios
2000

Groupwikis

1000
0
1

5 6&7 8

10 11 12 13 14 15

Wikifolios averaged 1569 words (144-3291)


Groupwikis averaged 3477 words (2705-4630)

Average Number Comments per Wikifolio


15
10

Unthreaded

Threaded

0
1

5 6&7 8

1. Wikifolios averaged
8.9 comments & 95
words/comment
2. Groupwikis forums
averaged 61 posts
3. Groupwiki averaged
40 comments & 92
words/comment
4. 2062 comments
included 182 badges

10

11

12

13

14

15

Forum Posts by Group and


Comments by Groupwiki
100
80
60
40
20
0

Forum
Posts

Groupwiki
Comments

Agreement & Disagreement


in Badges vs. Comments
100

Percentage Frequeincy

90
80
70
60
50
40

Comments

30

Badges

20
10
0
Agreement

Disagreement

Perspective Reinforcement
Change
Content (potentially co-ccuring)

P540 Exam Scores (percent correct)


Mean Exam Scores (Percent Correct)
100
90
80
70
60
50
40
30
20
10
0
MC Pretest MC Midterm Essay Midterm MC Final
Essay Final
(25 items, 50 (40 minutes, (50 minutes, 5 (60 minutes, (120 Minutes,
minutes, 0
20 items, 10 items 5 points) 30 items 9
6 items, 6
points)
points)
points)
points)

Course Evaluation Results


Campus-Wide Items
Overall, I would rate the quality of this
COURSE as outstanding.

Overall, I would rate this INSTRUCTOR as


outstanding.
My instructor treats students with
RESPECT
My instructor is WELL-PREPARED for class
meetings.
0
(11/15, 5-point Likert, SD to SA)

14 Other Sections

This Course

Course Evaluation Results


School-Wide Online Learning Items

The instructor used the available TECHNOLOGY well.


I would take ANOTHER ONLINE COURSE from the
School of Education
The level of INTERACTION in this distance course
was sufficient.

The ONLINE FORMAT was well suited for this course.


The online technology used in this course was
RELIABLE
This distance education format helped me feel IN
CONTROL of my own learning
0
(11/15, 5-point Likert, SD to SA)

14 Other Sections

This Course

Issues with Semi-Formal (Cohorted)


Course Format
Assumes regular instructor effort
Is not what online learners expect
Not flexible
Cant get behind

Led many students to drop out of the BOOC


Led many IUHS students to stay with
distance ed courses

Moving to Self-Paced Courses


Not simple to get interaction
Need to find like-minded peers
Need to find peers working on same unit
Need a large number of learners

Currently creating a self-paced BOOC


New participant list shows current learners
New wikifolio feature allowing archiving
Learners can look at archived work

Currently creating self-paced IUHS courses


Modules with exams and Canvabadge

Moving to Open Learning Resources


Need to automate many features
Automated registration and homepage
Real-time participatory feedback
Exam scoring and badges issuer

Relying extensively on open ed resources


Use ranking feature with OERs

Pending NSF proposals for IUHS and IU

Questions?
Thank you very much!
More info at www.RemediatingAssessment.blogspot.com
Hickey, D. T. (in press). Situative approaches to aligning summative, formative, and
transformative assessment functions. Assessment in Education: Principles,
Policies, and Practices.
Hickey, D. T., & Rehak, A. (2013). Wikifolios and participatory assessment for
engagement, understanding, and achievement in online courses. Journal of
Educational Media and Hypermedia, 22 (4), 229-263.
Hickey, D. T., & Zuiker, S. J. (2012). Multi-level assessment for discourse,
understanding, and achievement in innovative learning contexts. The Journal of
the Learning Sciences, 22, (4) 1-65
Hickey, D. T., Kelly, T. A, & Shen, X. (2014). Small to big before massive: Scaling up
participatory learning and assessment. Proceedings of the Fourth International
Conference on Learning Analytics and Knowledge, Indianapolis, IN (pp. 93-97)