|
 


 

Based on the McMaster
Undergraduate MD Programme

SIR UK User Group Conference

Aberdeen, UK, 21 June 2002

David Keane, Research Associate (keaned@mcmaster.ca)

Programme for Educational Research and Development
Faculty of Health Sciences, McMaster University
Hamilton, Ontario, Canada www.fhs.mcmaster.ca/perd
è

1. Introduction to progress testing
² definition
² purpose
² method
² goals
² special features
² basic patterns in performance data

2. Using SIR for progress testing

|
    

[ longitudinal testing of knowledge
acquisition
|
    

[ longitudinal testing of knowledge
acquisition

[ an objective method for assessing the

acquisition and retention of knowledge
over time relative to curriculum-wide
goals
|
    

an objective method for assessing the acquisition and retention of
knowledge over time relative to curriculum-wide goals

[ objective
² use multiple choice questions
[ knowledge
² test what learners know
[ over time
² test repeatedly at regular intervals
[ curriculum-wide
² address end-of-programme learning objectives
|
    

[ to determine whether the learner is
progressing

² learning enough?

² retaining enough?

² doing so quickly enough?

|
   

[ progress is relative
² compare learner to his/her peer group
[ current class or past n classes
[ standardized scores (z-scores)

² review performance on multiple tests

[ current and past

[ assessed with one measure

² percentage correct, whole test
[ adjust for guessing?
|
   

[ help the learner (formative evaluation)
² constructive feedback
[ about their knowledge base
[ about their ability to self-assess
[ has to be specific/detailed

² timely feedback
[ reassure those who are progressing
[ alert those who are not
² do so early enough to facilitate effective
remediation
|
   

[ help the Programme (summative evaluation)
² provide defensible evidence to support
critical decisions pertaining to individuals
[ mandated remediation
[ pass / fail / conditional advancement

[ the emphasis..
² on formative aspects
[ minimize negative impact on learning
behaviours
² tutorial functioning
² self/peer-assessment
|
    
[ the item bank
² a sample of the knowledge that a good
student will likely encounter by the time
that student. .
[ graduates?
[ is six months / a year beyond graduation?

- content encompasses nearly the 'entire'

domain of the field in question
[ cf. course/curriculum 'core' knowledge
|
    
[ instructions to examinees
² don't study for this test
[ 180 items, randomly selected from 2,600+

² don't guess
[ test your ability to self-assess
[ penalty for guessing (optional)

² attempt only those items for which you

have some knowledge and are reasonably
confident you know the best/correct
answer

   
 
[ Class means on whole test for..
² % attempted
² % correct
[ not adjusted for assumed guesses

[ look at patterns..
² across time
[ week in programme (x of 138)
² across classes at week x
 
 


   
 
[ % attempted, % correct
² patterns are relatively stable across tests
and classes
² means at week x are relatively consistent
across tests and classes
² examinee performance is relatively
consistent across tests and classes
[ overall test reliability 0.6 - 0.7 (8 tests)
[ test-retest correlation 0.6 - 0.8 (2 tests)
 Π
| 

untroduction to Progress Testing

Any questions?
è

1. Introduce progress testing
2. Using SIR for progress testing
² what's in an item?
² data management tasks
² managing dm tasks
[ software
[ databases and pql
² SIR \ valued features
² future enhancements
   !"
[ the examinee sees..
nn. 2






 
 


 

  
 





 











 
   
 
Stem





   

 



 !"#
$

%

 
&
2 2 

B ' 

  

C 2( 

 Options
D )


E *
   !#
[ the data manager sees..
² stem and options (text)
and
² unique item identifier
² correct response code
² content codes (6 fields, 1, 2 or 3 sets)
² item performance data
[ stats on usage, power to discriminate
[ by test, class; across tests, classes
² and more..
[ date last used, don't use flag
˜  $
1. store, retrieve, manipulate and print
large volumes of textual information
[ pre-test: test booklets
² 180 items, 21²22 pages/booklet
[ post-test: performance reports
² for examinees: 2 reports x 1²2 pages/rep
² for administrator: re. items, tests, classes and
examinees who are not progressing
[ accommodate special needs re.
² special characters ² Greek letters, math symbols
² page layout, fonts, typeface style
[ merge data into report templates
˜  $ 
%
2. read examinees' responses
² 100-item optical mark response sheets
² tab-delimited ascii records
[ Mac: 2 sheets X approx. 280|420 examinees

3. score examinees' responses

² requires item, test, class, examinee info
² compute and retain performance stats
[ key measures: % attempted, % correct
² mean & sd re. whole test (and major subdomains?)
[ for: each examinee, each class|peer group
˜  $ 
%
4. compute and retain item performance
stats
² requires item, test, class, examinee info

5. compute/retrieve data needed in

standard reports
² re. examinees, classes, tests, items

6. assemble and print reports

˜  $
7. enable support staff to do all of the
above with relative ease
² minimal reliance on the application
programmer after everything is up and
running
  




!
[ SIR is not a word processor

[ SIR is a record management and stats-

oriented reporting tool
² allows user to build powerful custom
applications
² vendor provides exceptional support
beyond the installed Help files
[ prompt, relevant and practical replies


  


[ the MD Programme's solution

² for text-intensive tasks..

Corel WordPerfect

- for numeric data and stats-intensive

tasks..
SIR


 

| 
[ a set of merge data files (database)
[ case-based by item id
[ item stems, options and other fixed info

[ export data via csv or fixed-format records

[ import data via csv-format records
[ into merge data files

[ multiple merge forms (report templates)

[ extensive use of merge and macro commands
to produce pre/post-test reports
[ custom-build merge|macro applications


   &'&#% #(((
[ 2 databases, case-based
² ITEMS re. items
² TEEX re. tests, examinees, classes

[ major reliance on (vb) PQL

² custom applications
[ csv-format records
² add/update records/fields (eg, from WP)
² write records/values (eg, for WP)
² PQL procedures
[ csv save, tabulate, save table, spss save
 #(((  "
[ DBMS
² case-based option for db type
[ system-maintained
[ easy access to any case's records
[ case id is on all dumped records

² global variables
[ pass user settings to applications

² utilities
[ Data \ File Dump, File Input

² tabfiles and tables

[ create, index, populate, delete tables
 #(((  #
[ PQL
² nested access to cases
² read csv-format records
² vb dialog boxes

[ PQL Procedures
² write csv-format records
² xtab tables, flexibility re. headers (columns)
² write SPSS system files
º    
[ upgrade to SIR 2002 (from SIR 2000)
² update custom applications (to vb pql)
² add secondary indexes
[ examinees by name, current class

[ web access
² for examinees: performance reports
² method?
[ ColdFusion (CF ²² SQL ²² ODBC driver ²² SIR db)
[ CGI scripts
 Π
| 

Using Su for progress testing

Any questions?