A concordance of Marshalls e-Learning Maturity Model with Bacsichs Pick & Mix Model

Draft 1 Stephen Marshalls e-Learning Maturity Model (eMM) is being used as the methodological basis of the benchmarking activity in the University of Manchester. However, since the University of Manchester is a member of Paul Bacsichs mini-club of four universities, all the others of which are using the Pick & Mix methodology, it was felt useful to provide a mapping between eMM and Pick & Mix in order to facilitate comparison and common dialogue. In its current form (it is due to change in the next release some details are known), eMM has 43 criteria, grouped into five categories: Learning Development Coordination & Support Evaluation Organisation

Pick & Mix in its core version has 18 criteria. They are not grouped into categories but it is likely that they could be, using for example the phases of e-learning from the Costs of Networked Learning studies or alternatively, by using the categories of Marshall. The next version of Pick & Mix is under construction it will add to the core criteria and also permit use of local criteria for an institution. The following pages provide a concordance into the emerging new release of Pick & Mix of each of the five categories of eMM. (See the Appendix for an indication of the current state.) Methodology It has to be pointed out that there are several methodological differences between eMM and Pick & Mix. These are likely to make it hard to deploy an unmodified eMM in UK HE. 1. A minor one is that each eMM criterion is expressed in terms of a narrative of good practice whereas each Pick & Mix criterion is a more stark statement with suggestions of how to measure it at various levels of compliance. 2. A slightly more major one is that the thrust (so far) of eMM has been at programme or project level so that some statements have the tone of being about courses not the whole institution but some rewording of these can be done if it causes confusion. 3. A much more major difference is that the scope of eMM goes into areas which traditionally in the UK are seen as the province of general good practice in teaching and learning (and thus the province of QAA) rather than the province of e-learning technology or pedagogy (and thus of JISC or HEA). Thus the scope of Pick & Mix is considerably narrower. 4. Coupled with this is the feeling that each of the national benchmarking criteria sets inevitably focusses on a rhetoric of features and issues that are seen as of more than average importance. This comes out clearly below with eMM and Pick & Mix but
Paul Bacsich 18 February 2006

eMM-Pick & Mix concordance draft 1

similar comparisons with US methodology would show up the far greater orientation in the US to issues of off-campus provision. And as noted in the ALT-C paper when discussing the US Quality on the Line guidelines, Comments on these guidelines from one group included the remarks that they were of an era and only a subset of what is required; the other group noted that there was no real concept of governance or legal framework in the guidelines... In several US documents there is still a rhetoric of a community of scholars, self-organising and devoid of managers and support staff. Note This material is work in progress and not destined for publication in anything like its current form. For this reason a number of niceties such as key references etc have been left out All opinions are tentative and not yet in a fit state to be checked with parties outside those involved in the creation of the opinions. The next phase is to send it to the University of Manchester team for comment. The tables In the following tables the code n/a means that the eMM criterion is deemed not applicable as a criterion in the Pick & Mix model; this is often because it is seen as out of scope for the UK approach to e-learning, not that it is not important for teaching and learning.

Paul Bacsich

21 February 2006

eMM-Pick & Mix concordance draft 1

1.

Learning: Processes that directly impact on pedagogical aspects of e-learning

There are 10 processes in this category, each described below.

eMM code eMM description P&M # P&M description or comment Notes

eMM.L1

Courses are designed to require students to engage themselves in analysis, synthesis, and evaluation as part of their course and programme requirements Student interaction with teaching staff and other students is an essential characteristic and is facilitated through a variety of ways Teaching staff clearly communicate how communication channels should be used during a course or programme Teaching staff manage student expectations over the type and timeliness of responses to student communications Feedback to student assignments and questions is constructive and provided in a timely manner

n/a

This is seen in the UK as part of good practice in teaching and learning This is seen in the UK as part of good practice in teaching and learning This is seen in the UK as part of good practice in teaching and learning This is seen in the UK as part of good practice in teaching and learning

eMM.L2

n/a

eMM.L3

n/a

eMM.L4

eMM.L5

P&M93

Communication of expectations, and conformance to them, of the assignment handling and feedback system

This is seen in the UK as part of good practice in teaching and learning but there are specific aspects for elearning, thus treated as a P&M supplementary criterion

eMM.L6

Students are instructed in the proper methods of effective research, including assessment of the validity of resources

n/a

Part of the general information literacy issue needed for all students whether on e-learning courses or not This is seen in the UK as part of good practice in teaching and learning This is seen in the UK as part of good practice in teaching and learning See the comments on L5; but note that much experience in distance learning shows that rigid deadlines increase dropout Accessibility Note that in the UK it is usual to treat this separately for legal reasons; thus L10 has a wider scope

eMM.L7 (now goes with L1) eMM.L8

Learning outcomes for each course are summarised in a clearly written, straightforward statement Assessment of students communicates high expectations Student work is subject to clearly communicated timetables and deadlines

n/a

n/a

eMM.L9

P&M93?

eMM.L10

Courses are designed to support a diversity of learning styles and to ensure accessibility

P&M05

Paul Bacsich

21 February 2006

eMM-Pick & Mix concordance draft 1

2.

Development: Processes surrounding the creation and maintenance of e-learning resources

There are 6 processes in this category, each described below.

eMM code eMM description P&M # P&M description or comment Learning material Underpinning IT/comms reliability Notes

eMM.D1

Guidelines regarding minimum standards are used for course development, design and delivery The reliability of the technology delivery system is as failsafe as possible Learning outcomes, not the availability of existing technology, determine the technology being used to deliver course content Technical assistance in course development is available to teaching staff

P&M09

P&M09 is only part of the scope of eMM.D1 This is a supplementary P&M criterion Many in the UK would regard this as unrealistic, given university financing and student demographics Perhaps both could be criticised as focussing purely on academic staff This is part of P&M16

eMM.D2

P&M52

eMM.D3

n/a

eMM.D4

P&M16

Technical assistance to academic staff Technical assistance to academic staff Instructional design and pedagogy

eMM.D5

Teaching staff are encouraged to use technical assistance when (re)developing courses Teaching staff members are assisted in the transition from classroom teaching to online instruction

P&M16

eMM.D6

P&M07?

Perhaps one should add a criterion on pedagogic support, or (more controversially) extend technical support to techno-pedagogic support

Paul Bacsich

21 February 2006

eMM-Pick & Mix concordance draft 1

3.

Coordination & Support: Processes around the day-to-day management and support of e-learning delivery

There are 11 processes in this category, each described below.

eMM code eMM description P&M # P&M description or comment Organisation (to support elearning) Notes

eMM.C1

A centralised system provides support for building and maintaining the e-learning infrastructure (Marshall recommends moving to D this seems hard to justify)

P&M15

In the UK it is by and large accepted that the technical infrastructure should be centralised, but not for the pedagogic infrastructure This is seen in the UK as part of good practice in teaching and learning This seems too closely linked to L5 and L9 to be treated separately This is seen in the UK as part of good practice in teaching and learning

eMM.C2

Students have access to sufficient library resources that may include a virtual library accessible through the World Wide Web Teaching staff and students agree upon expectations regarding times for student assignment completion and staff response Students are provided with hands-on training and information to aid them in securing material from a range of sources consistent with the discipline or subject Students have convenient access to technical assistance throughout the duration of the course/programme

n/a

eMM.C3

P&M93

eMM.C4

n/a

eMM.C5

P&M92

Competence and timeliness of help from Help Desk

For some reason this did not make it into the P&M core criteria

eMM.C6

Students are provided with detailed instructions regarding the electronic media used in a course prior to commencing it Students are able to practice with any technologies prior to commencing a course

n/a

Normally done in the last 10 years; so not a differentiator This is an important point but usually in the UK the practice time is very limited due to the rush at the beginning of semesters Links to C5; indeed Marshall recommends conflating them This is seen in the UK as part of good practice in running a university Training Level, skill and delicacy of management of plagiarism issues This may deconstruct into a variety of criteria

eMM.C7

n/a

eMM.C8

Questions directed to student service personnel are answered accurately and quickly A structured system is in place to address student complaints Instructor training and assistance continues throughout the online course Teaching staff are provided with support resources to deal with issues arising from student use of electronically-accessed data

P&M92

eMM.C9

n/a

eMM.C10 eMM.C11

P&M10 P&M65

Paul Bacsich

21 February 2006

eMM-Pick & Mix concordance draft 1

4.

Evaluation: Processes surrounding the evaluation and quality control of elearning throughout its entire lifecycle

There are 7 processes in this category, each described below.

eMM code eMM description P&M # P&M description or comment Evaluation Notes

eMM.E1

The programmes educational effectiveness is formatively and summatively assessed with multiple, standards based, and independent evaluations The programmes teaching/learning process is formatively and summatively assessed with multiple, standards based, and independent evaluations Summative data such as enrolment numbers, completion rates, and costing is used as a measure of effectiveness within course/programmes Success of technology/innovation used as a measure of effectiveness within course/programmes Intended learning outcomes are reviewed regularly to ensure clarity, utility, and appropriateness

P&M14

Marshall has recommended conflating these four criteria

eMM.E2

eMM.E3

eMM.E4

eMM.E5

n/a

This is seen in the UK as part of good practice in teaching and learning This is seen in the UK as part of good practice in teaching and learning This is related to P&M10 but is not believed in the UK to be either feasible or correlated with success in e-learning

eMM.E6.

Instructional materials are reviewed periodically to ensure they meet programme standards

n/a

eMM.E7

Teaching staff capability in making the transition from classroom to online teaching is formally assessed during training

n/a

Paul Bacsich

21 February 2006

eMM-Pick & Mix concordance draft 1

5.

Organisation: Processes surrounding the creation and maintenance of e-learning resources

There are 9 processes in this category, each described below.

eMM code eMM description P&M # P&M description or comment Decisionmaking Notes

eMM.O1

A documented set of formal criteria are used to determine access to funding and other resources which support course and programme (re)development A documented technology plan is in place and operational to ensure quality of delivery standards

P&M07

It is not the set of formal criteria so much as the processes surrounding it, including whether such a set is taken seriously In the UK the normal practice is to regard a technology plan as only part of an e-learning strategy and subservient to a pedagogic strategy in eMM the pedagogic strategy seems implicit Important but not normally in the UK seen as central to elearning success

eMM.O2

P&M06

e-learning strategy

eMM.O3

A documented technology plan is in place and operational to ensure the integrity and validity of information delivered, collected and stored Before starting a programme, students are advised of any particular requirements of that programme to ensure they possess the personal and technical skills needed for that programme Students are provided with supplemental course information that outlines course objectives, concepts and ideas Students are provided with supplemental course information that outlines admission requirements, tuition and fees and other relevant administration information Students are provided with supplemental course information that outlines requirements for additional resources such as books or other materials Students are provided with supplemental course information that outlines student support services. Before starting a programme, students are advised of any particular technological requirements of that programme to ensure they have access to the minimal technology required by the course design

n/a

eMM.O4

P&M91

Communication to students of expectations of students This is seen in the UK as part of good practice in teaching and learning (Marshall recommends merging these two.) Both are seen in the UK as part of good practice in teaching and learning in fact, as no-brainers

eMM.O5

n/a

eMM.O6

n/a

eMM.O7

eMM.O8

n/a

This is seen in the UK as part of good practice in teaching and learning and university administration Good idea; see also P&M95

eMM.O9

P&M91?

Paul Bacsich

21 February 2006

eMM-Pick & Mix concordance draft 1

6.

Conclusions

From this first cut it does not look as though the match between eMM and Pick & Mix is particularly close. However, as Marshall notes in his section on Future Development: He has cut down his set of 43 processes to 34 by skilful compositing (we have taken this into account where it is was easy to do so without distorting his original sequencing) A review of the literature suggests that as many as one hundred additional processes could potentially be incorporated.

It should also be noted that in terms of Pick & Mix that: the current core Pick & Mix of 18 criteria was in fact shrunk from a set of around 25 many other criteria are under consideration (see the Appendix), perhaps up to 20 more

Thus we may not yet be matching those methodologies in a mature enough state. In terms of both methodologies, it should be noted that they both claim to have common grounding in archetypal best practice compendia such as Quality on the Line, The Seven Principles, etc. Perhaps along the lines of their developments they have interpreted these in different ways. Some earlier work which was stopped when the current phase of funded benchmarking activity started may have to be reviewed and refined this provided both exit strategies for the various archetypal criteria and (in reverse) a pedigree for Pick & Mix. The early literature search review (the Middlesex paper http://www.cs.mdx.ac.uk/staff/profiles/p_bacsich/Benchmark-theory.pdf) contains some of this but for the usual reasons later published papers were more concise. A natural way forward is to rationally reconstruct a UK-eMM basing it initially on Pick & Mix but taking care to check out other UK benchmarking, best practice and developmental traditions (such as NLN and ELTI) noting, however, that Pick & Mix already incorporates many of these ideas. Such a UK-eMM is an admirable research topic and would also be practically very useful as a kind of ground penetrating radar compared with the alleged shallower scan of Pick & Mix. but it would lose the benefit of international comparability however, unlike with costings, that international comparability may remain a dream. And at this stage eMM is deployed in full only in New Zealand Australia is still taking it slowly in benchmarking.\

Paul Bacsich

21 February 2006

eMM-Pick & Mix concordance draft 1

Appendix: The emerging Pick & Mix extended criteria

(This is adapted from a working paper done at the instigation of the University of Leicester. Further extensions are being done as other institutions engage.) Core criteria The following version of the standard Pick & Mix table is simplified, leaving out levels 2, 4, and 6 (excellence), in order to fit more neatly and readably onto an A4 portrait page.
P&M 01 Factor Adoption phase overall (Rogers) 1 Innovators only 3 Early majority taking it up 5 All taken it up except some laggards One VLE Instrument Interviews, surveys, documentation in IT reports, etc. Observation, purchase orders Interviews, crosschecking with JISC and CETIS, etc.

02

VLE stage

No VLE

VLEs reducing in number to around two Widespread use of at least one specific tool, e.g. assignment handling, CAA Explicit usability testing of all key systems Almost all elearning material and services conform to minimum standards of accessibility e-Learning Strategy produced from time to time, e.g. under pressure from HEFCE or for particular grants E-learning decisions (e.g. for VLEs) get taken but take a long time and are contested even after the decision is taken Terms well understood within the learning and teaching centre and among some academic staff

03

Tools use

No use of tools beyond email, Web and the VLE minimum set No usability testing, no grasp of the concept e-learning material and services is not accessible

HEI-wide use of several tools

04

IT underpinning usability

All services usable, with internal evidence to back this up e-learning material and services are accessible, and key components validated by external agencies Regularly updated e-Learning Strategy, integrated with Learning and Teaching Strategy and all related strategies Effective decisionmaking for elearning across the whole institution, including variations when justified Pedagogic guidelines for the whole HEI, and acted on

(Further advice is needed from UKERNA, JISC and UCISA) (Split off separately for legal reasons)

05

Accessibility

06

e-Learning Strategy

No e-Learning Strategy. No recent Learning and Teaching Strategy

Review of HEFCE, TQEF and other documents; interview with PVC responsible

07

Decision-making

No decision making regarding e-learning each project is different

Observation and perusal of papers

08

Instructional Design/ Pedagogy

Terms not understood in the HEI

Interviews

Paul Bacsich

21 February 2006

eMM-Pick & Mix concordance draft 1

P&M 09 Factor Learning material 1 Little conformance of learning material to house style for editing or layout 3 Most learning material conforms to explicit editorial and layout guidelines 5 HEI-wide standards for learning material, which are adhered to and embedded at any early stage, e.g. by style sheets All staff trained in VLE use, appropriate to job type and retrained when needed Work planning system which recognises the main differences that e-learning courses have from traditional Activity-Based Costing being used in part Instrument Perusal of material, interviews

10

Training

No systematic training for elearning

HEI-wide training programme set up but little monitoring of attendance or encouragement to go A work planning system which makes some attempt to cope, however crudely, with e-learning courses Good informal understanding of costs in faculties and central departments but not coordinated Systematic planning process for e-learning but unintegrated with related planning such as for IT or for space Evaluation of key courses is done from time to time, by professionals

%ages plus narrative (this may not involve training courses; but is likely to)

11

Academic workload

No allowance given for the different workload pattern of elearning courses

Detailed and possibly anonymous interviews and questionnaires some union sensitivities likely in some HEIs Interviews and questionnaires (the basis here is from CNL and INSIGHT JISC projects, also Becta TCO) Interviews and questionnaires (more work to be done on levels process models may help) Interviews with key evaluators; perusal of conference and journal papers

12

Costs

No understanding of costs of elearning except possibly in (maybe) the business school

13

Planning

No planning process for elearning offerings

Integrated planning process for elearning within overall course planning and some links to IT and space planning Regular evaluation of all courses using a variety of measurement techniques and involving outside agencies where appropriate Central unit has Director-level university manager in charge and links to support teams in faculties All staff engaged in e-learning process have nearby fastresponse tech support

14

Evaluation

No evaluation of courses take place that is done by evaluation professionals

15

Organisation

No appointments of e-learning staff

Central unit or subunit set up to support e-learning developments

Interview with VC and relevant PVC(s)

16

Technical support to academic staff

No specific technical support for the typical (unfunded) academic engaged in e-learning

Key staff engaged in the main elearning projects are well supported by technical staff

Interview with both top-level staff and selective interviews with grass-roots staff

Paul Bacsich

10

21 February 2006

eMM-Pick & Mix concordance draft 1

P&M 17 Factor Quality and Excellence 1 Conformance to QAA in a minimalist way 3 Conformance to QAA precepts including those that impinge on elearning 5 Adoption of some appropriate quality methodology (EFQM, etc) integrated with course quality mechanisms derived from QAA precepts Staff engaged only in or supporting the teaching process can reach high levels of salary and responsibility Instrument Interviews, questionnaires, quality reviews, etc.

18

Staff recognition for e-learning

No recognition for staff, explicit pressure against (e.g. due to RAE)

Some recognition such as Teaching Fellows within an essentially RAEdriven paradigm

Documentary evidence

Extension criteria Extension criteria have at present been assigned to a tentative numbering system, which will change once the choice of criteria has stabilised. Seven criteria that were in the beta version of Pick & Mix that some want to bring back
P&M 51 52 52 54 55 56 57 Factor Overall NLN-style ILT adoption level Overall eMM adoption level Underpinning IT/comms reliability Underpinning IT/comms performance Foresight into technology and pedagogy Collaboration Competence in managing IPR for e-learning Notes Only of interest for HE-FE consortia Only of interest for international consortia Too IT-oriented? But it is a critical success factor so likely to stay. Too IT-oriented? But it is a critical success factor so likely to stay (and maybe in new core) Specialised but likely to stay (though not core) But is more collaboration better? So how to score? Specialised but likely to stay (though not core)

Additional institution-oriented extension criteria under consideration at more than one HEI This list has benefited from the authors notes of the presentations at the Launch Meeting but a second pass through the presentations is required to refine this.
58 59 60 61 62 63 64 Market research Competitor research Competence and frequency of benchmarking Public relations external and communications internally with stakeholders Level of integration of e-learning with admin systems Level of exploitation of pre-existing student IT skills Level of development of Computer Aided Assessment The whole MLE versus VLE debate The NetGen issue. Fashionable and likely to stay; Specialised; likely to stay but not in new core. Not a strength of some UK universities. Likely to stay (and in the new core) Often ignored. Likely to stay (but not core). Too internal a criterion?

Paul Bacsich

11

21 February 2006

eMM-Pick & Mix concordance draft 1 65 66 66 67` 68 69 Level, skill and delicacy of management of plagiarism Contribution of e-learning to WP agenda Level of integration/interaction of physical space management with management of IT Effective management of risks associated with e-learning Amount and level of research leveraged on elearning Cost-effectiveness of e-learning A component of the overall HEI Risk Register Important to some HEIs but not many; likely to stay; but not in new core. A tough one. Likely to stay coming back into fashion in Australia and never out of fashion in US Specialised; likely to stay but not in new core Hard to tease out? Important to some HEIs.

Additional student-oriented criteria

91 92 93 Communication of expectations of students to students Competence and timeliness of IT help from Help Desk Communication of expectations and conformance to them of the assignment handling and feedback system Overall student satisfaction with e-learning Understanding and management of the student-borne costs of e-learning Convergence of functionality and usability of IT-based systems of technical, academic and pastoral help with face to face systems Needs much discussion. A component of overall student satisfaction

94 95 99

Paul Bacsich

12

21 February 2006