HR Lsa Trends

Advancements
in Large-scale
Assessment
Heiko Rlke & Krisz:na Toth
DIPF - Deutsches Ins:tut fr Interna:onale
Pdagogische Forschung
Frankfurt am Main
TAO Days | 10.09.2012
Seite 1
Overview
Trends in LSA
Closer Look at Trends
Example: Making Use of Data
Challenges in LSA
TAO Days| 10.09.2012
Seite 2
Trends
Main trend: CBA
Sub-trends/eects of CBA:
CAT
towards forma:ve assessment
Complex items
Simula:on, interconnec:on
Interweaving
dierent tests, ques:onnaires, etc.
Big data
TAO Days | 10.09.2012
Seite 3
Trend: CBA
Computer-based Assessment has to serve a purpose
Costs?
Time!
Validity!
TAO Days | 10.09.2012
Seite 4
CAT
Also not prac:ced for its own sake
Not as important in pure summa:ve assessment
But:
Strict :me limits (e.g. PIAAC)
Growing demand for forma:ve aspects in summa:ve
assessment
TAO Days | 10.09.2012
Seite 5
Complex Items
Closer at reality
Simula:on
Most important:
Ability to assess new domains
TAO Days | 10.09.2012
Seite 6
Examples of Complex Items

Simula+on of an Email / Web Scenario
The test person receives an email and should book

cinema :ckets online.

Dynamic Model (MicroDYN)
The test person should explore and master a dynamic

system with input (exogenous) variables inuencing
output (endogenous) variables.
Automaton with Finite State Machine (MicroFIN)
The test person should interact with a mobile phone and set :me to
summer:me.
Interweaving of Tests and

Ques:onnaires
Framing
test only if precondi:ons are fullled
Double-check
ndings of ques:onnaire
TAO Days | 10.09.2012
Seite 8
Big Data
Log data, not only results
Find out what is going on
Make use of data
E.g. par:al scoring
-> Examples from

PIAAC PS-TRE
TAO Days | 10.09.2012
Seite 9
Simula:on-based assessment
Modern assessments:
real life situa:ons: e.g. web environments
complexity of instrument various ways
Monitoring students ac:ons - log les
Analysis of test-taking paths is a rela:vely new eld in educa:on
(reason: printed vs. computer-based data collec:on)
Process data requires to consider and evaluate methods
Berlin, 10.9.2012 | H. Rlke & K. Tth | TAO Days 2012 | Advancements in Large-Scale Assessments
10
Aim
to inves:gate how log data can be integrated into the process
of educa+onal assessment and evalua:on to support

researchers and prac::oners in making use of the data
assembled in computer-based test delivery
11
Aim
to inves:gate how log data can be integrated into the process
of educa+onal assessment and evalua:on to support

researchers and prac++oners in making use of the data
assembled in computer-based test delivery
12
Hypertext Item
13
14
Example log
829709 15
{"sender":"15","type":"loading","data":""}

833463 15
{"sender":"15","type":"loaded","data":""}

848153 15
{"sender":"15","type":"user_interac+on","":"<?xml version=\"1.0\">
\u000d\u000a<cbaloggingmodel:EmbeddedLinkLogEntry xmlns:cbaloggingmodel=
\"htp://www.soucon.de/cba/cbaloggingmodel\" id=\- b576759f1:-7fed\" sourcePageId=
\"Item15_linklist\" targetPageId=\"Item15_website1\" textFieldId=\"cbaTextField_71"/>
\u000d\u000a"}
849094 15
{"sender":"15","type":"variable_change","data":{"name":
"snapshot_url,"value":"htp://localhost:8101/cba-run:me/itemjsessionid=
LB1?custom_servicehandler=downloadService&le=
C:\\...snapshot5537437125227002523.xml"}}
855852 15
{"sender":"15","type":"user_interac+on","data":"<?xmlversion=
\"1.0\encoding=\"UTF8\"?>\u000d\u000a
<cbaloggingmodel:BuRonLogEntryxmlns:cbaloggingmodel=\"htp://www.sou
con.de/cba/cbaloggingmodel\id=\"cbaBackBuRon_14_13454937565947\"/>
\u000d\u000a}":"15","type":"unloaded","data":""}

15
Promising methods
Sta:s:cs and visualisa:on
Clustering
Classica:on
16
Process Measures
1. Number of page visits
2. Number of dierent page visits
3. Visit of relevant page
4. Time spent on the relevant page
5. Ra:o of :me spent on the relevant page
6. Ra:o of :me spent on the opening screen
7. Comple:on :me
17
Sta:s:cs
Process measures
Mean
Number of page visits

Number of dierent page visits
Time spent on the relevant page
Ra:o of :me spent on the relevant page
Ra:o of :me spent on the opening screen
Comple:on :me
5.18
2.38
10.87
.14
.64
67.74
Standard
Devia+on
3.13
1.38
9.77
.11
.18
27.49
18
Distribu:on of dierent page

visits
19
Distribu:on of dierent page

visits
20
Visualisa:on
Berlin, 10.9.2012 | H. Rlke & K. Tth | TAO Days 2012 | Advancements in

Large-Scale Assessments 21
Visualisa:on
Berlin, 10.9.2012 | H. Rlke & K. Tth | TAO Days 2012 | Advancements in

Large-Scale Assessments 22
Cluster Analysis
Features
Cluster 1
Cluster 2
Cluster 3
Cluster 4
23%
16%
34%
27%
Nr. of page visits

Nr. of dierent page visits
Comple:on :me
Relevant page visited (Y/N)
Ra:o of :me on relevant page
Ra:o of :me on star:ng page
Distribu:on of sequences (%)
Ra:o of correct responses
23
Cluster Analysis
Features
Cluster 1
Cluster 2
Cluster 3
Cluster 4
23%
16%
34%
27%
25.0%
1.7%
92.6%
84.4%
Nr. of page visits

Comple:on :me
24
Cluster Analysis
Features
Cluster 1
Cluster 2
Nr. of page visits
.72
5.41
.36
2.54
28.38
62.08
No
No
.93
.60
23%
16%
25.0%
1.7%
Comple:on :me
Cluster 3
Cluster 4
25
Cluster Analysis
Features
Cluster 3
Cluster 4
Nr. of page visits
4.73
9.57
2.29
4.53
Comple:on :me
59.60
91.60
Yes
Yes
.20
.17
.62
.45
34%
27%
92.6%
84.4%
Cluster 1
Cluster 2
26
Cluster Analysis
Features
Cluster 2
Cluster 3
Nr. of page visits
5.41
4.73
2.54
2.29
Comple:on :me
62.08
59.60
No
Yes
.20
.60
.62
16%
34%
1.7%
92.6%
Cluster 1
Cluster 4
27
Cluster Analysis
Features
Cluster 1
Cluster 2
Cluster 3
Cluster 4
Nr. of page visits
.72
5.41
4.73
9.57
.36
2.54
2.29
4.53
28.38
62.08
59.60
91.60
No
No
Yes
Yes
.20
.17
.93
.60
.62
.45
23%
16%
34%
27%
25.0%
1.7%
92.6%
84.4%
Comple:on :me
28
Summary and Conclusions
29
Summary and Conclusions

Features
Cluster 1
Cluster 2
Cluster 3
Cluster 4
Nr. of page visits
.72
5.41
4.73
9.57
.36
2.54
2.29
4.53
28.38
62.08
59.60
91.60
No
No
Yes
Yes
.20
.17
.93
.60
.62
.45
23%
16%
34%
27%
25.0%
1.7%
92.6%
84.4%
Comple:on :me
30
Future work
Pilot study sample size
Valida:on
Other types of items require new process measures
31
32
Future work
Pilot study sample size
Valida:on
Other types of items require new process measures
We have a lot of work to do:
Souware developer
Test developer
Psychometricans
33
(Selected) Challenges
Authoring and Management
Delivery
Re-use and exchange
TAO Days | 10.09.2012
Seite 34
Delivery
Re-use and exchange
TAO Days | 10.09.2012
Seite 35
Delivery
Re-use and exchange
TAO Days | 10.09.2012
Seite 36
Delivery
Re-use and exchange
TAO Days | 10.09.2012
Seite 37

HR Lsa Trends

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

HR Lsa Trends

Diunggah oleh

Hak Cipta:

Format Tersedia

Advancements

TAO Days| 10.09.2012

TAO Days | 10.09.2012

TAO Days | 10.09.2012

TAO Days | 10.09.2012

Examples of Complex Items

The test person receives an email and should book

Dynamic Model (MicroDYN)

The test person should explore and master a dynamic

Automaton with Finite State Machine (MicroFIN)

Interweaving of Tests and

TAO Days | 10.09.2012

-> Examples from

TAO Days | 10.09.2012

of educa+onal assessment and evalua:on to support

of educa+onal assessment and evalua:on to support

Number of page visits

Distribu:on of dierent page

Distribu:on of dierent page

Berlin, 10.9.2012 | H. Rlke & K. Tth | TAO Days 2012 | Advancements in

Berlin, 10.9.2012 | H. Rlke & K. Tth | TAO Days 2012 | Advancements in

Nr. of page visits

Nr. of page visits

Nr. of page visits

Nr. of dierent page visits

Ra:o of :me on relevant page

Ra:o of :me on star:ng page

Distribu:on of sequences (%)

Ra:o of correct responses

Nr. of page visits

Nr. of dierent page visits

Relevant page visited (Y/N)

Ra:o of :me on relevant page

Ra:o of :me on star:ng page

Distribu:on of sequences (%)

Ra:o of correct responses

Nr. of page visits

Nr. of dierent page visits

Ra:o of :me on relevant page

Ra:o of :me on star:ng page

Distribu:on of sequences (%)

Ra:o of correct responses

Relevant page visited (Y/N)

Nr. of page visits

Nr. of dierent page visits

Ra:o of :me on relevant page

Ra:o of :me on star:ng page

Distribu:on of sequences (%)

Ra:o of correct responses

Summary and Conclusions

Summary and Conclusions

Nr. of page visits

Nr. of dierent page visits

Ra:o of :me on relevant page

Ra:o of :me on star:ng page

Distribu:on of sequences (%)

Ra:o of correct responses

TAO Days | 10.09.2012

TAO Days | 10.09.2012

TAO Days | 10.09.2012

TAO Days | 10.09.2012

Anda mungkin juga menyukai