Anda di halaman 1dari 105

Recruitment to

specialist training:
a resource guide for selectors
Contents
1. Objectives, Timetable & Workshop Slides

2. Resource Pack: Section A


Key Concepts & Best Practice

3. Resource Pack: Section B


National Standards & Upskilling

4. Example Person Specification

5. Reference Report

6. Application Form Draft

7. Shortlisting Indicators & Example Responses

8. Example Interview Report Form

9. Upskilling in Selection Workshop: Selector Skills,


Trainer Skills, Trainer Slides, FAQs

10. Evaluation & Audit Documentation


Workshops
National
National Workshops
Selection into Specialty Training: Upskilling selectors

National Specialty Selection Workshop:


Methods & best practice for up-skilling selectors

9.30am Registration & Coffee


10am-4pm Workshop

Workshop facilitators
Dr Sarah Thomas, Professor Fiona Patterson & colleagues.

Aims & objectives


To review the overarching principles and processes that have been agreed for national
specialty selection
To review key concepts underpinning best practice selection and recruitment
To introduce the national documentation for specialty selection (person specification,
reference report, application form)
To review the key skills required of selectors involved in selection (shortlisting & interviews)
To provide an opportunity to experience these skills using practical exercises
To provide supporting training materials for use by trainers in deaneries/ unit of
application (UoA) in upskilling selectors for 2007.
To establish next steps in cascading training of selector skills in deaneries/ UoA’s.

Target audience & outcomes


This workshop is aimed at those responsible for training selectors within their deanery/UoA in
best practice selection principles and methods. The outcomes of the workshop are to 1) raise
awareness and upskill participants in key concepts, principles and methods in best practice
selection, such that they are able to deliver training effectively within their UoA (using a
cascade approach); 2) to provide UoA representatives with relevant documentation and
materials required for specialty selection; 3) to establish next steps in cascading training within
UoA, including the role of the selection methodology team in the individual deanery visits.

Format
This is a participative workshop with a variety of practical exercises related to selector skills and
will include plenary discussions, presentations and small group work.

Suggested pre-reading
Suggested pre-reading includes the documents attached. Those who have attended the
planning and development meeting on 19th September will already have received these:
Document 1: Overarching framework for selection into specialty training
Document 2: Overview of the development of the selection methodology presented to
PMETB for national selection
National Selection for Specialty Training
Methods & Best Practice
for Upskilling Selectors
Timetable
10-10.20 Introduction
• Objectives & framework for National Specialty Selection

10.20-10.35 Selection Process Overview

10.35-11.15 Key Concepts in Best Practice Selection


• Research evidence
• What’s different?

11.15 Tea/Coffee Break

11.30-11.50 National Documentation for Specialty Selection


• Person Specifications
• Application form
• Reference report & reference process

11.50-1pm Selector Skills: Application Form & Shortlisting


• Applying indicators & rating scale to application form questions

1-2pm Lunch

2-3pm Selector Skills: Interviews


• Sources of error & bias, questioning styles
• Applying indicators & rating scale to interview questions

3-3.15pm Tea/Coffee Break

3.15-3.45pm Implementation & Practical Considerations


• FAQs
• Next steps

3.45-4pm Review & Questions

4 pm Finish
Key Topics
ƒ Selection Process Overview
− The plan for delivery
National Selection into − The Specialist Training programmes
− The timeframe
Specialty Training: − How information is being cascaded

Launch of Training Materials ƒ Key concepts in best practice selection


ƒ The National Documentation for Specialty Selection
− Person Specification, Application Form, Structured
Reference
17th October 2006 ƒ Selector Skills
− Short-listing, The interview
ƒ Implementation & practical considerations

Timetable:
4 Sessions Framework for National Selection
10.00-10.20 Introduction
10.20-10.35 Selection Process Overview Key Points http://www.mmc.nhs.uk/
10.35-11.15 Key Concepts in Best Practice Selection
11.15 Tea/Coffee Break ƒ UK Strategy Group agreed framework for
11.30-11.50 National Documentation for Specialty Selection national selection July 2006
11.50-1pm Selector Skills: Application Form & Shortlisting ƒ MTAS will be used by all 4 UK countries
1-2pm Lunch ƒ Nationally agreed person specs for each
2-3pm Selector Skills: Interviews specialty & for each entry level (ST1, ST2, ST3)
3-3.15pm Tea/Coffee Break ƒ Nationally agreed application form, reference
3.15-3.45pm Implementation & Practical Considerations
form, minimum standards for interview
3.45-4pm Review & Questions
ƒ PMETB approval September 2006
4 pm Finish

Stakeholder & expert consultation Ownership: Are COPMeD/


COPMeD/ Deans
collectively signed up to the process?
Chairs PMETB
PMETB Current
Current
Chairs of
of Trainees/
Trainees/
STCs
STCs Applicants
Program
Program
Applicants
Medical
Medical
ƒ COPMeD Steering Group for Recruitment &
Directors/
Directors/
College
CollegeTutors
Tutors
Directors/
Directors/
HR/Trusts
HR/Trusts
Selection into Specialty Training responsible for
the work (reporting to UK Strategy Group)
COPMED/
COPMED/
COGPED
COGPED
Stakeholders
MMC
MMC ƒ Close involvement of stakeholders in development,
Deaneries
Deaneries Stakeholders
&&Experts
Experts
implementation & evaluation
Patient
Patient AoMRC,
Groups
Groups
AoMRC,
JACSTAG
JACSTAG
ƒ Deans have nominated YOU to come here today
to deliver implementation in your locality
BMA
BMA GMC
GMC ƒ Deans will oversee evaluation of selection panels
NHS
NHS Department
Department
Employers
Employers of
of Health
Health

1
What Specialist Training Core Run Through Training
Programmes will be available? Recruit in 2007 to:
Medicine in General
Surgery in General OFMS Ophthalmology
Psychiatry
+ ST4
Paediatrics
Basic Neuroscience Training
Medicine in General
Acute Care Common Stem ST3
Surgery in General
Histopathology Chemical Pathology Medical Microbiology
Ophthalmology ST2
Obstetrics & Gynaecology
Acute Care Common Stem
Paediatrics
Basic Neuroscience Training ST1
Anaesthesia
Obstetrics & Gynaecology
Psychiatry
Anaesthesia
All recruited to after ƒ Existing SHO
Radiology
F2 and during
programmes offer core
transition from the
current SHO pool
Public Health
Core training 
General Practice

Non-Core Run Through Training


from ST1
Specialist Training
Psychiatry Non-Core
ƒ Currently do not have Non-Core Paediatrics
core training at SHO Radiology
Medicine in General
grade Radiology
OFMS
OFMS Surgery in General
ƒ In 2007 will recruit to Chemical Pathology
ST1 from F2 and Chemical Pathology Ophthalmology
Medical Microbiology
SHO grades Medical Microbiology Acute Care Common Stem
Histopathology
ƒ General Practice will Histopathology Basic Neuroscience Training
Public Health
recruit at ST1-3 in Public Health Obstetrics & Gynaecology
2007 General Practice
General Practice Anaesthesia

Core

How do trainees apply for


What levels will be selected into
Specialist Training?
Specialist Training in 2007?
ƒ Advance information in Medical Press
ƒ Single alert in Medical Press & electronic
Specialty-specific information
− Levels recruited to depend on comparison ƒ Single electronic application per recruitment
between competencies detailed in the new & round
old curricula ƒ Long-
Long-listed nationally against person
Recruitment will occur at ST1- ST4 in some specification
Specialties ƒ Apply for up to 2 preferred Specialty groups in 2
preferred Units of Application*
− Distinct application forms
− GP 1 UoA
− Different recruitment rounds − May no longer express ‘no geographical preference’
*Unit of application- a Deanery or
cluster of Deaneries working together

2
Recruitment & Selection Offers to Applicants
National Advert
1
Round 1 E- Applications co-ordinated & sent to chosen UOAs* Applicants will receive a
Run-Through Shortlisting Shortlisting Shortlisting Shortlisting single e-letter listing their
Grade Posts:
2 Specialties, Assessment Assessment Assessment Assessment offers and will have a limited
2 UOA & Ranking Ranking Ranking Ranking period in which to reply
FTSTAs Selection Selection Selection Selection
o Acceptance is a binding
Central Co-ordination of Offers of Posts: commitment to the UOA
0 1 2 3 4 and the employer
Unsuccessful o Applicants breaking the
Re-enter Applications co-ordinated & agreement are disqualified
Round 2 sent to chosen UOAs* from any further
Shortlisting Assessment Training participation in the
Vacant Ranking Programme recruitment and selection
Run-Through Appointed to
Grade Posts
Selection
Training
10 Vacancies 45 process except in very
& FTSTAs Central Co-ordination of Offer of Post programme Appointable exceptional and unforeseen
Trainees circumstances
Unsuccessful Successful Ranked 1- 45

Ranking of Applicants 06
Aug
Sept
Identify No of Programmes/ Posts
Agree person specification for ST1, 2 & 3
1 Jan Finalise complete selection/ interview schedule
6 Jan Alert in Medical Press- Application start
22 Jan- 2 Feb Submit applications
2 Feb Application form to selection panels
There will be a By 23 Feb Shortlist applicants
continuous process of Round 1 24 Feb One email confirming interviews or not
issuing more offers to 2007
27 Feb Trainees confirm attendance or not
appointable candidates 28 Feb- 13 Apr Complete selection activities:
Identify appointable applicants
in Round 1 of Selection 19-26 Apr Round 1 Offers made & accepted
27 Apr Trainees with no offers asked to reapply
Training
Programme
Indicative 28 Apr Advertise unfilled posts
10 Vacancies 45
Round 2 By 27 May Shortlist applicants
4- 22 June Complete Selection
45 Trainees
Appointable
Timeframe 23 June One email of offers/ rejections
August 2006-7 26 June Acceptance/rejection of offer
Ranked 1- 45

Implications of Timeframe Feasibility:


Selectors will need to have the following availability: Can the Selection process be rolled out uniformly?

Provisional Dates
ƒ Shortlisting Mechanisms for 2007
− 2- 23 Feb ƒ Targeting of agreed selection criteria within specialties
− 14- 27 May ƒ Nationally agreed documentation, appeals system
ƒ Selection ƒ UoA leads for selection identified using train-the-trainer
− 28 Feb- 13 April cascade.
− 4- 22 June ƒ Today is Phase 2 – Launch of training materials.
ƒ Auditing & evaluation procedures, annual QA report
ƒ Selection Panel should be available to complete
short-listing within the timeframe as well as to
attend the selection process for candidates

3
Feasibility: Selection Methodology Cascade
Are there barriers to implementation in
particular areas?
Phase 1: Concepts & Planning (19 Sept)
Potential barriers Phase 2:
2: Launch of Training Materials (3,
ƒ Units of Application with differing needs (→ 10, 17 Oct)
training tailored accordingly)
Phase 3: Individual Deanery visits - support
ƒ Timeframe & resource
for roll out as appropriate (Oct &
ƒ Resistance to change? Nov)

Scope of Our Work Background


In “Proposals for Reform of the SHO grade”
grade”
Our role: To advise on Selection methods for entering Sir Liam Donaldson argues that:
Specialty training
“Reform must take account of…
of…
Phase 1 Person Specs for National Selection 2007
Phase 2 Pre-
weak selection & appointment procedures:
Pre-application information for candidates
Phase 3 Methods design: References, Application form & these are not standardised & are frequently
Interview not informed by core competencies’
competencies’.
Phase 4 Upskilling assessors/admin
Phase 5 Evaluation
UK Strategy Group agreement to national selection
ƒ Consultation (especially trainees)
PMETB Panel review 25th August 2006

Why Change? What’s new?

ƒ Historically we have done an effective job. ƒ This is the first step. For 2007, there will be
ƒ Little formal evaluation uniform targeting of agreed selection
ƒ Our aim is to improve efficiency & effectiveness criteria (via the Person Specifications).
Specifications).
for all stakeholders (trainees, admin, interviewers, HR) ƒ Minimum requirement of a 30 minute
− cost, fairness, reliability & validity (standards) structured interview
ƒ ST1 is new: We must deliver a process that is ‘fit ƒ Identifying best practice (no re-
for purpose’
re-invention)
ƒ Resource pack & opportunities to further
upskill selectors

4
Research on Selection Systems Research on selection methods
¾ Structured interviews can be reliable & valid selection tools.
ƒ Research & implementation projects to ¾ Reliability of assessment & of assessors using
standardised rating scales & documentation. Rigorous
develop selection tools that are standardised,
standardised, training is critical.
fair, defensible, reliable, valid, cost-
cost-effective &
feasible. Research shows Selection Centres (designed
appropriately) are best predictor of future job performance
1. Develop & validate selection criteria for various
Reliability & validity gains - combining different selection
specialties tools, standardised scoring systems to measure key
2. Design & validate selection tools that assess core competencies eg.eg. 3 x 10 minute stations (= 30 minutes, with 2
assessors per panel)
criteria
Research consistently demonstrates
demonstrates that application form
¾ Growing literature in medicine data & references have
have limited validity
Patterson et al. (BMJ
(BMJ,, 2005; BMJ 2001; BJGP,
BJGP, 2000; Arch. Dis. Child. Validity can be significantly improved by providing
2006; TOG in press, 2006, RCS) standardised competency-
competency-based rating scales

Defining reliability & validity Evaluating validity in selection


1. Faith validity 4. Criterion validity
“The person who Predictive validity
sold me the “The tool/test predicts who
Reliability: “The instrument measures consistently selection tool/ test will be competent doctors”
under varying conditions” was very plausible” 5. Incremental validity
2. Face validity “How much additional value
does using another test/tool
Validity: “The instrument measures what it “The test looks provide?”
claims to measure” plausible”
6. Construct validity
3. Content validity
“The test looks “The tool measures
plausible to experts” something meaningful”
meaningful”

Research on selection in medicine Candidate reactions


Practically, for some specialties we can’t interview all applicants.
We need to shortlist. Options? Factors affecting reactions use
– Psychometric testing, MCQs? organisational justice theories:
ƒ Academic success predicts very little of the variance in
subsequent clinical performance (e.g. Ferguson et al, BMJ; 2002) − Distributive justice: Perceived fairness
ƒ For ST2, ST3 exams/research more relevant to indicate regarding equity (outcome consistent with
competence. The is not possible for differentiating at ST1 expectations) & equality (opportunities).
ƒ We recommend a combination of questions to add − Procedural justice: Information, feedback,
predictive power – this has been piloted job-relatedness, recruiter effectiveness.
ƒ MTAS can search for plagiarism
ƒ For 2007, this methodology is fit for purpose.
ƒ For the future, evaluation is necessary, on a national basis. Applicants prefer multiple
ƒ Future innovations could be a Machine marked test for opportunities to demonstrate skills
shortlisting

5
Person Specifications
Purpose
− Usability (especially candidate perspective)
− Standardisation within specialties
− Operationalisation of assessment (e.g. AF,
National Documentation- interview questions, scoring framework)

The Person Specification Two sections:


−Eligibility Criteria:
Tab 4 Minimum entry requirements, all essential
−Selection Criteria:
Used to rank candidates
Essential/ desirable criteria
Specialty-specific criteria (some common domains)
Commitment to the specialty

Reference Reports

ƒ Standardised, structured report form


National Documentation- nationally agreed
The Structured Reference ƒ Minimum of 2 referees
Tab 5 ƒ Guidelines in how reference report used
in the selection process

Application Forms
Questions directly linked to person specification

ƒ Eligibilty
− Personal details, GMC Registration, Professional
National Documentation- Qualifications, Eligibility for entry into Specialist
The Application Form training, Previous posts, Referees
ƒ Selection Criteria
Tab 6 − Clinical, Academic & Research skills, Probity,
Evidence of commitment to the Specialty, Personal
skills relevant to the Specialty

Currently undergoing substantial development

6
Application Process

ƒ Specialty specific forms


ƒ Long listing via MTAS (electronic portal)
Shortlisting Indicators
ƒ Shortlisting in UoA by trained shortlisters
(minimum of 2) Tab 7
Reference Framework for Target Selection Criteria
ƒ Lay involvement, determined by the UoA
ƒ Scoring frames provided

Framework for Target Selection Criteria Short-listing Process


Clinical, Academic & Research Skills
ƒ All interview panel members (or a sub-set) carry out
shortlisting (Orange Guide)
− Panel members do not have to mark ALL of a single
application form.
− Deaneries may choose to split questions between panel
members.
ƒ Benefits in terms of speed & calibration
ƒ Different methods/options:
− All in one room – marking/ decisions
− Sent out to short-listers & returned
− electronic

ƒ GMC reporting for falsification added to Application


Form

Rating Scale Evaluation & rating process

„ Standardised system of documentation &


Applicant responses scored on 4-point scale:
scoring framework allows for more valid &
4 = good to excellent
fair assessments
3 = satisfactory
„ Reduces potential for error & bias
2 = areas of concern
1 = poor [0] = No evidence „ Increases transparency

7
Practical considerations in
Shortlisting

Selector Skills Exercise 1 ƒ Calibration and reliability


ƒ Each question marked by 2 different short
listers
Evaluating & scoring the application form
ƒ Returning scores

Interviews in Selection

ƒ 2 main types of interview in selection


− Behavioural (past behaviour = best predictor of future behaviour)
− Situational (scenarios)
Selector Skills - The Interview − Both useful, dependent on interviewer skills

Skills: Peeling, probing, familiarisation, observe


record, classify, evaluate

Management Scenario:
Anaesthetic SpR refuses to anaesthetise a
patient for appendicectomy

Example Situational Interview EVALUATION 1 2 3 4

Question Positive Indicators


•consider the reasons (eg
Negative Indicators
•insist on operating
overnight, co-morbidity) •threaten a critical incident
•review the decision to operate •consults senior anaesthetist
and assess priority without discussing with SpR
•discuss it with the anaesthetist •tell the patient anaesthetist
•involve the surgical senior cover “refusing” to anaesthetise
•deal with the patient •just accepts it
appropriately

8
Clinical Scenario: Familiarise
12 yr old boy – who has come off his BMX –
he has a broken arm (supracondylar fracture)
and is complaining of breathlessness. ƒ Interviewers need to know:
− the questions they are asking
− how they relate to the person specification
EVALUATION 1 2 3 4
− what probes are appropriate
Positive Indicators Negative Indicators
• Structured approach to the question • Failing to take a history of the nature ƒ Be familiar with the relevant indicators for
• Awareness of possible vascular of the accident
injury to arm • Failure to take adequate history the question
• Awareness of possible including possibility of asthma
pneumothorax
• Of risk of associate intra-abdominal
• Failure to examine patient including
trachea
ƒ Be familiar with the rating scale to be used
injury – ruptured spleen • Sending patient for CXR without
• Awareness of risk of head injury excluding tension pneumothorax
• Failing to assess circulation in arm

What’s the difference between a


skilled interviewer & novice?
styles…..open questions, one at a time
ƒ Questioning styles…..open
ƒ Managing silences….
ƒ Making the question feel authentic! Follow-
Follow-up
essential to uncover what “you” (the candidate) Example Interview Report Form
actually did
Tab 8
ƒ Peel the layers 1.facts 2. meaning 3. attitude
ƒ Probe for evidence who, what, why, when…etc 1
2

ƒ Verify what the trainee actually did, or would do 3

in scenarios

Observation: Sources of error & bias


ƒ Haloes & Horns effects
ƒ Prejudice (race, gender, etc.)
Selector Skills Exercise 2 ƒ
ƒ
Stereotyping & ‘Red Rags’
Concentration
− ‘Primacy & recency’ effects
ƒ Observation bias
Questioning Styles − ‘Similar-to-me effect’/ Personal Liking
ƒ Temporal extension
− Decision made within 3 seconds
ƒ First impressions last
− “Can I suspend judgement?”

9
‘Real World’ Selector Stereotypes Recording
ƒ Legible!
ƒ white stilettos ƒ designer stubble
ƒ Accurate
ƒ limp handshakes ƒ close-
close-set eyes
ƒ Complete
ƒ white socks ƒ double-
double-barrel − If in doubt, include it
ƒ male earrings surnames ƒ Non-judgmental
ƒ tattoos ƒ Volvo drivers − Record appropriately, do not judge or evaluate
ƒ peroxide hair while recording
ƒ Millwall fans
ƒ heavy make- ƒ Specific – need to be able to provide
ƒ male long hair make-up
explanation for judgment later on
ƒ sweaty palms ƒ belly showing!

Evaluation & Rating Process

4 = good to excellent
Selector skills Exercise 3 3 = satisfactory
2 = areas of concern
1 = poor [0] = No evidence
Effective recording & evaluating
Standardised system of markers allows for
more valid & fair assessments &
increases transparency

Interviews/Selection Centres Interviews/Selection Centres


Logistics
ƒ Chairperson/ facilitator appointed
Methods ƒ Venue, admin support, materials planned in
ƒ Minimum of 30 minute structured interview advance
ƒ Questions/exercises must be; Content
− Based on the relevant Person Spec. ƒ Minimum of 30 minutes contact
− Consistent across interviewers/ interviewees − Panel interview; Split-panel (two 15 minutes); Three
− Previously agreed scoring framework that links to the 10 minute exercises
Person Spec ƒ Multiple selection tools encouraged
ƒ Example questions & calibration exercises ƒ Selection centres (focus also on demonstrating,
provided not reporting)
Selectors
ƒ All trained, lay involvement essential

10
Feedback
ƒ Selection decision uploaded to MTAS; offers ONLY
by MTAS at the end of the selection process, or the
national system will not work
National Documentation- ƒ Feedback offered only after the whole of the
Candidate evaluation selection process.
ƒ This is essential so there is fairness to all
Tab 10 candidates
ƒ We must ensure fairness to all candidates
ƒ UoA s will be asked to audit to ensure national
adherence

Practical considerations

ƒ Timetable
Implementation &
ƒ Logistics
Practical Considerations ƒ Candidate experience

Review
ƒ National specialty selection in 2007 includes national
person specifications, application forms & references
ƒ Key concepts in best practice selection inform new
practice
ƒ Short-
Short-listing & interview questions MUST be directly
Final questions?
linked to person specs
ƒ Selector skills training vital to contribute to
standardised, fair, defensible, reliable, valid, cost-
cost- Thank you……
effective & feasible process.
ƒ Selection system not static – continuous
improvement

11
Resource Pack;
Section A
Resource Pack: Section A
Section A. Key Concepts and Best Practice

1. Background

2. Why Change?

3. What’s New?

4. Person Specifications

5. The Selection Process

6. Research on Selection Methods

7. Application Process

8. Interviews/Selection Centres

9. Reference Reports

10. Feedback

Attachment 1: Orange Guide Excerpt.

Attachment 2: PMETB Generic Standards for Training Excerpt.

1
Across all four UK countries, specialties currently apply selection criteria inconsistently. For
2007, there will be uniform targeting of agreed selection criteria, nationally agreed
documentation and a nationally agreed appeals system. This is a first step towards achieving
national selection for all specialties. This document provides guidance for staff involved in
delivering the selection process for specialty training. Minimum standards are presented and
a guide to best practice at each stage of the selection process is outlined. By following this
guidance, the aim is to deliver a selection processes for specialty training is efficient, valid
and fair. The Resource Pack is designed to allow Units of Application [UoAs]/Deaneries to
evaluate their current selection processes against best practice in selection.

1. Background
Postgraduate medical education and training has undergone a period of rapid change, partly
to ensure standards for safety, efficacy, and accountability of health care. In 2002, the Chief
Medical Officer for England, Sir Liam Donaldson, highlighted, in his report called “Proposals
for Reform of the SHO grade,” the need for best practice selection:
“Reform must take account of… weak selection and appointment
procedures: these are not standardised and are frequently not informed by
core competencies”.
Since Sir Donaldson’s report, the four UK Health Departments published a policy statement
on “Modernising Medical Careers” (MMC), which set out principles for a major reform in
postgraduate medical education. The MMC programme places emphasis on delivering robust
methods for assessing and developing doctors throughout their career life-cycle.
Implementation of MMC has led to trainees making career decisions earlier, so the selection
systems must be robust and transparent.
The main regulatory body behind these planned changes is the Postgraduate Medical
Education and Training Board (PMETB). The PMETB, established in 2003, is independent of
government. It bears responsibility for the standards and quality assurance of all postgraduate
education, training and assessment in medicine in the UK. In August 2006, PMETB reviewed
the principles for national selection and the key concepts are presented here.

2. Why Change?
Obviously, selection has been effective in the past. However, there is scope for improvement.
Specifically, the objective for 2007 is to improve the efficiency and the effectiveness of the
selection system and methodologies used therein. Although considerable effort will be spent
on delivering these changes, achieving national selection will have significant advantages for
all stakeholders (trainees, admin, interviewers, HR). For example, for many years there have
been ‘multiple applications’, where trainees complete many applications to different training
programmes for the same specialty throughout the UK. This potentially duplicates effort for
applicants, administration teams, HR and various staff conducting shortlisting and interviews.
By having a national portal for applications, valuable resource will be saved.

3. What’s New?
The intention is to build on good practice and current expertise - not to re-invent! For 2007,
there will be a train-the-trainer cascade of the new national documentation for speciality
selection, taking place in the various Units of Application (usually Deaneries). Specifically,
there will be nationally agreed Person Specifications, speciality specific shortlisting
procedures and a minimum standard for the interview process. In addition, auditing and
evaluation procedures will be introduced. The auditing and evaluation process is overseen by
the Postgraduate Deans and used to identify areas for modification and improvement.

4. Person Specifications
A Person Specification describes the qualifications, skills, experience, knowledge and other
attributes a candidate needs in order to perform effectively on the training programme. Person
Specifications are the cornerstone of the selection process and should be derived from a
thorough analysis of the post in question. The Person Specification outlines the selection

2
criteria that will form the basis of selection decisions at shortlisting and interview/selection
centre. Selection criteria must be specific, justifiable, measurable, fair and pitched at a level
appropriate for the training programme. The content of interviews/selection centre exercises
must derive from the person specification and be designed to elicit evidence from candidates
in relation to the criteria. All selection tools used (whether at shortlisting or interview) must be
directly relevant to the criteria listed on the Person Specification.

5. The Selection Process


The four UK countries have agreed to a common process and further detail is available via
the MMC website at http://www.mmc.nhs.uk/. The recruitment process for specialty training is
summarised in Figure 1. Key features to note are that applications and longlisting process
will be delivered via a centralised electronic portal (called the Medical Training Application
Service; MTAS). There is a minimum standard of a 30 minute structured interview. However,
1
other options are encouraged such as split panel interviews, and selection centres . Note that
additional, specific training is essential before selectors can use a selection centre approach.

Advert & Pre-application Information


Pre-application information for each specialty accessible via MTAS, including;
Brief description of the specialty, attraction & challenges, future of the specialty
A trainee’s view
Person specification: Eligibility and Selection criteria
Information on selection methods, the selection process
Links to relevant websites, references, etc

Application Form Submitted


Long listing against eligibility criteria via MTAS

Short Listing
Person specification criteria must be used so that applications can be measured against
the published selection criteria

References collected
Structured, competency-based rating scale, used a pre-allocation check

Interview/Selection centre
Minimum requirement: 30 minute, structured interview
Options: 2 split-level panel interviews (15 minutes each, 3 selectors per panel, lay
involvement) OR 3 X 10 minutes interviews, 2 selectors per panel.
For a specialty, the same competencies measured across all UoA
Selectors use national documentation with validated scoring framework

References used

Allocation

Figure 1. The Selection Process

1
Selection ‘centres’ are a process, not a place, involving multiple selection activities such as simulations, group
challenges and structured interviews.
3
6. Research on Selection Methods
There has been a considerable amount of research conducted (over many years) on
2
developing robust selection criteria and methodologies in medicine (e.g. Patterson, 2005 ).
The ongoing research and development aims to develop tools and techniques for selection
that are standardised, fair, defensible, reliable, valid, cost-effective and feasible. This body of
research has been used to develop selection criteria across all specialties for 2007.

7. Application Process
LongListing & Eligibility Criteria
• The first step is to ensure candidates meet all the eligibility criteria for the training
post at the appropriate level. For example, for entry to ST1, applicants must have F2
competencies. Candidates who do not meet all eligibility criteria will be rejected at
this stage (and do not proceed to shortlisting).
• This process will be conducted by the electronic system via MTAS. Certain criteria
might be flagged so they can be assessed at interview stage (e.g. evidence of F2
competencies for some candidates might be assessed at interview stage).
• Candidates who meet all eligibility criteria will go through to shortlisting and assessed
against the relevant selection criteria.

Shortlisters
• All staff involved in the shortlisting process must be trained in the selection principles
and the shortlisting process.
• There should be a minimum of 2 shortlisters per application form. To improve
reliability, shortlisters could chose to divide the application form questions (where a
shortlister focuses on one or more questions across an applicant pool). Ideally,
responses to individual questions should be separated and marked by different
shortlisters.
• Ensure that all applications are treated confidentially and that applications are only
circulated to those involved in the shortlisting process.

Shortlisting & Shortlisting Criteria


• Applications will be sent to the relevant UoA for shortlisting.
• When shortlisting, each candidate must be assessed objectively and consistently
against the selection criteria identified in the relevant person specification.
Candidates should only be assessed against these criteria.
• Scoring frameworks are provided for each criterion and must be applied consistently
across shortlisters. Training and calibration is essential in this respect.
• Selectors must avoid making assumptions about candidates’ skills, experience or
qualifications without demonstrable evidence.
• Candidates are scored on a scale of 1-4 for each criterion (1=poor, 2=areas of
concern, 3=satisfactory, 4=good to excellent, and 0 if no evidence is presented).
• Shortlisting criteria must be consistently applied. This is the responsibility of the UoA.
• Marks for all selection criteria should be combined to create a final total that will be
used to rank candidates. (Eligibility criteria are not included in the ranking process).
• A cut-off score for inviting candidates to interview/selection centre should be agreed
by the panel and applied consistently in the UoA.

8. Interviews/Selection Centres
Methods
• The aim of the interview/selection centre is to determine candidates’ suitability for the
post by obtaining detailed evidence to judge them against the relevant criteria.
• Interviews/selection centres also give candidates an opportunity to find out more
about the specialty/programme/deanery and thus enable self-selection.

2
Patterson et al (2005) A new selection system to recruit GP registrars: Preliminary findings from a validation study.
British Medical Journal.

4
• Research clearly demonstrates that interviews can represent an important and valid
means of selecting candidates provided it is structured and interviewers are trained
appropriately.
• For best practice, interviewer questions must be:
based on the relevant Person Specification,
consistent across interviewers and interviewees
based on the interviewer panel using a consistent set of indicators and
scoring criteria to evaluate interviewee responses.
• The two most common forms of structured interview used in selection are:
Behavioural Interviewing and Situational Interviewing.
Behavioural interviewing involves asking interviewees to describe previous
behaviour in past situations that are job relevant.
Situational interviews present interviewees with hypothetical job-related
situations/scenarios and ask them to indicate how he or she would respond.
• A combination of both types of questions could be used alongside questions about
commitment to the specialty. In all cases interviewers must use appropriate scoring
frames, detailing a good, average and poor response (as appropriate) in advance of
the interview taking place. Using indicators to rate interviewee responses, ensures
greater reliability and consistency across interviews and interviewers. Example
scoring keys and rating scale using indicators are provided in the resource pack.
• All interviewers must be trained in interview skills and equal opportunities.

Logistics & Panel Composition


• Advance planning is essential to ensure the necessary staff, administrative support,
venue, equipment and materials are available for the interviews/selection centres.
• The Orange Guide provides specific guidance on the appointment panel composition.
A chairperson must be appointed to ensure the interview/selection centre is
conducted professionally and fairly and that questions/exercises are within best
practice guidelines (see Attachment 1 below).
• Lay involvement is essential as a requirement of the PMETB Generic Standards for
Training (see Attachment 2 below).
• The interview panel (or facilitators) are responsible for ensuring the interview/exercise
runs to time.
• Sufficient time should be allowed between interviews/exercises for selectors to
complete scoring sheets/report forms. Similarly, sufficient time for rest breaks for
selectors and candidates should also be built into the schedule (as appropriate).
• Each interview/exercise should take place in a different room where possible. Where
this is not possible, soundproof screens must be provided to ensure fairness.
• A selector or facilitator should be responsible for welcoming candidates and
explaining the process for the interview/exercise. Candidates must be made aware of
the (maximum) length of each interview/exercise at the start and of the timing of any
subcomponents if appropriate (e.g. 5 minutes’ briefing time then 15 minutes’
simulation exercise).
• All candidates must be asked in their invitation letter whether they require any special
arrangements in order to attend the interview/selection centre. All
reasonable/necessary adjustments should be made (in compliance with the Orange
Guide).

Interview Questions/Selection Centre Content


• Content of the interviews/selection centre must clearly relate to the criteria identified
in the Person Specification.
• Each candidate must receive a minimum of 30 minutes assessment time (interviews
or selection centre exercises). The purpose of this is to ensure candidates have
sufficient opportunity to provide evidence of the required competencies.
• Second stage assessments should have face validity (i.e. be relevant/reasonable to
the candidate).
• Interviews must be competency-based and structured. The same question areas
should be asked of all candidates applying of the same post. A range of acceptable
probing questions should be developed that can be used to elicit further information
5
from candidates as necessary. Structured interviews that use the same questions for
every candidate based on the selection criteria will help ensure candidates are
assessed fairly and only against the selection criteria, in addition to help defending
discrimination claims from unsuccessful candidates.
• Using multiple assessments is strongly encouraged such as using a split-panel
interviewer with three 10 minute interviews conducted in three separate rooms. The
purpose of this is to increase reliability and validity by ensuring candidates receives
multiple assessments by multiple selectors.

Selectors
• All selectors involved in interview/selection centres must be trained in selection
principles and the interview/selection centre process. Selectors must be trained in
recording appropriate information at interview/selection centre. Lay involvement is
essential where appropriate (see Attachments 1 and 2 below).
• Each selector on a panel should assign marks to candidates independently which can
be aggregated to give a final mark for the interview/exercise where appropriate.
• Standardised, competency-based scoring frameworks must be developed and used
for each interview/exercise. This should include a proforma for collecting observation
notes made during the interview/exercise and evaluations/justifications for marks
completed after the interview/exercise. Selectors must ensure that sufficient detail is
recorded to justify a selection decision (and to reference feedback at a later date, as
appropriate). Example interview report forms are provided in the trainers resource
pack.

9. Reference Reports
• References should be requested at application stage and will be used to inform the
selection process. References will be used as a pre-allocation check once selection
decisions have been made. Guidelines for UoA on use of reference reports will be
provided.
• The nationally agreed standard, structured reference form must be used.
• Candidates will be asked to provide two references, one of which must be from their
current employer.

10. Feedback
• A standardised procedure for providing feedback to successful and unsuccessful
candidates should be agreed in the UoA, in line with the national process and
implemented consistently.
• Informal feedback, delivered immediately after the interview, by the interview panel, is
not permitted.
• Feedback can only be given after the selection decision has been formally
communicated to the candidate, in line with the national system.
• Staff providing feedback to candidates must be appropriately qualified.
• Feedback must directly relate to the selection criteria and be based on evidence
gathered during the selection process. Feedback should be descriptive, accurate and
as objective as possible and must be substantiated by documentation completed
during the selection process, as appropriate.
• Under data protection legislation, candidates have the right to access any data held
on their application, including shortlisting/interview/selection centre documentation
and references.

6
Attachment 1: ORANGE GUIDE Page 10.

Who sits on the appointment committee?

9. In England, Wales and Northern Ireland, the membership of an appointment


committee should be:

i. a lay chairman who has the confidence of the participating hospitals and locations. The
chair ensures that their interests are fully considered in the appointment process. Chairmen
will be appointed from a list compiled by the postgraduate dean and endorsed by trusts,
colleagues and where appropriate, universities. A lay chairman may be an executive or non
executive member of a trust board or other senior non-medical member of management. A
senior doctor, for example, a trust medical director may act in a lay capacity as chairman. In
this case the vital distinction between the lay role of the chairman and the roles of the clinical
and academic members of the committee should be born in mind. A lay chairman cannot
undertake the roles of any other members of the committee while acting as chairman;

ii. the regional college adviser or a nominated deputy ;

iii. the relevant postgraduate dean or a nominated deputy ;

iv. representatives of the consultant staff in the training location(s) involved in the (rotational)
training programme. The composition will depend on local circumstances but will be a
minimum of two and normally a maximum of four consultants. Where more than four trusts
are involved in the training programme representatives should sit on appointment committees
on a rotational basis ;

v. a nominee from the appropriate university in the deanery ;

vi. the programme director or chairman of the deanery specialty training committee;

vii. a representative of senior management in an employing trust in the training rotation.

The appointment committee for a dual certification programme must reflect the interests of
each of the specialties concerned. Postgraduate deans, in convening appointment
committees, should also take in to account the diverse nature of the modern medical work
force which includes significant numbers of women doctors and doctors from ethnic
minorities. Committees should, therefore, reflect this diversity and include, where possible,
members who are women and/or from ethnic minorities.

10. In Scotland the membership of the Committee will comprise at least five members
including:

i. a chairman selected from a panel drawn up by the regional postgraduate dean in


consultation with the trusts in the region ;

ii. a member from the appropriate section of the National Panel of Specialists ;

iii. a member of the regional Postgraduate Medical Education Committee (usually the regional
postgraduate dean or a deputy) ;

iv. a senior medical representative of the services principally involved in the training
programme for the post in question (e.g. clinical director or consultant); and

v. a consultant appointed by the relevant university.

11. If extra or alternate membership is proposed to take account of a particular


discipline, placement or rotation, the postgraduate dean should be consulted and will
be responsible for arranging this where it is necessary. A proper balance of
membership should be care fully preserved.

7
Attachment 2: PMETB GENERIC STANDARDS FOR TRAINING April 2006.

Domain 4. Recruitment, selection and appointment.

The purpose of this domain is to ensure that the processes for entry into postgraduate training
programmes are fair and transparent. Processes must be consistent with PMETB Principles
for Entry to Specialist Training.

Responsibility: Deaneries.

Evidence: Deanery data, trainee surveys.

Standard: Processes for recruitment, selection and appointment must be open,


fair, and effective.

Recruitment and selection

Mandatory

4.1 Candidates will be eligible for consideration for entry into a specialist training
programme if they:
a. are a fully registered medical practitioner or hold limited registration with the
General Medical Council or are eligible for any such registration;
b. are fit to practise.

4.2 The selection process (which may be conducted by interview or by other process)
must:
• ensure that information about places on training programmes, eligibility and
selection criteria and the application process is made widely available in sufficient
time to doctors who may be eligible to apply;

• use criteria and processes which treat eligible candidates fairly;

• select candidates on the basis of open competition;

• have an appeals system against non-selection on the grounds that the criteria
were not applied correctly, or were unfairly discriminatory;

• seek from candidates only such information (apart from information sought for
equalities monitoring purposes) as is relevant to the published criteria and which
potential candidates have been told will be required.

4.3 Selection panels must consist of persons who have been trained in selection
principles and processes.

Developmental

4.4 In addition to 4.1, to be eligible for consideration for entry into a specialist training
programme, candidates must be able to demonstrate the competences required to
complete Foundation Training. (This covers candidates who have completed
Foundation Training, candidates who apply before completion and those who have
not undertaken Foundation training, but can demonstrate the competences in another
way.)

4.5 Selection panels must include a lay person.

8
Resource Pack;
Section B
Resource Pack: Section B
Section B. National Standards & Resource Pack for Upskilling
Methodology National Standard Documentation Trainer Resource
Person There will be nationally agreed person specifications for each specialty and for each entry level of Tab [
Specification each specialty (e.g. ST1, ST2, ST3). Agreement will be via the Academy of Medical Royal Example Person
Colleges Specialty Training Committee and COPMeD. Specification
Example Person
Person Specification
Person Specifications were requested from the 16 Specialties for ST1, ST2 and ST3 entry (as
Specifications for Anaesthetics
appropriate). These will follow a generic format.
delivered by at ST1
COPMeD Steering
Group (Autumn
The Person Specification identifies eligibility and selection criteria relevant to each Specialty.
2006)
Selection criteria include competencies common across all specialties and competencies identified
from research as important to particular specialties.

The Person Specifications describe how and when the selection criteria are evaluated. These will
be agreed via AoMRC and COPMeD.

Reference There will be a nationally agreed structured reference form designed by the selection methodology Tab \
Report team and agreed with the AoMRC, COPMeD and other relevant stakeholders. National Reference Form

Form delivered by Applicants will be required to submit the names of two referees, both of whom must have National
COPMeD Steering supervised the applicant in the past 2 years. References will be requested for short listed Reference Form
(Draft)
Group (Autumn candidates only.
2006)
Reference Reports will be structured and require referees to provide information relevant to the
Person Specification.

Application There will be a nationally agreed application form, with nationally agreed specialty-specific Tab ] Tab `Trainer Slides &
Form sections, designed by the selection methodology team and agreed with the AoMRC, COPMeD and
National Example Timetable for a ½
other relevant stakeholders.
Application Form day workshop
Form delivered by (Draft)
COPMeD Steering All applications will be submitted electronically via MTAS.
Tab ^Example Applicant
Group (Autumn Tab ^ Responses to Application
2006) Applicants will be invited to apply to two specialties and to two units of application (UoA) for each Example
(unless one of the specialties selected is general practice, in which case they will be restricted to a Questions (for use in
Upskilling & Indicators for calibration)
single geography as is currently the case for GP recruitment). Shorlisting
Shortlisting
delivered by
Applicants will be asked to submit a single application for vacancies at the most appropriate level of
UoA/Deanery Tab ^
training within a specialty (ST1 or ST2 or ST3, etc.)
Rating Scale

Tab X=where to find documentation and materials in the resource pack


Application form questions will reflect those competencies identified as important to each specialty
(and linked to the Person Specification).

Shortlisting will be scored locally by selection panels including trained selectors in the specialty
against national, specialty specific criteria.

Shortlisting results will be communicated to applicants via e-mail only in accordance with the
agreed national timetable.

Interview/ As an absolute minimum, selection will consist of a 30 minute competency based structured Tab _ Tab `Trainer Slides &
Selection interview.
Example Interview Example Timetable for a ½
Centre The panel composition must conform to the guidance provided in the Orange Guide (and must Format with day workshop
have lay involvement). Indicators
Upskilling & Workshop Exercises:
Process delivered Selection will be scored locally by a selection panel including trained selectors in the specialty Tab _ Exercise: Questioning
by UoA/Deanery against national, specialty specific criteria described on the relevant Person Specification. Example Blank Styles (Onion Exercise)
Interview Form Exercise: How and What
Selection results will be communicated to applicants via e-mail only in accordance with the agreed
to Record
national timetable.
Tab _
Specialties/UoA may opt to go beyond the minimum requirement and use additional selection Example Interview Frequently Asked
methods to enhance validity (e.g. 3x10mins interviews, selection centres using multiple selection Questions Questions (FAQs)
tools such as simulations, scenarios, etc).

A national appeals system will ensure consistency of appeal processes for Units of Application
(UoA). This will be agreed via COPMeD, AoMRC and other stakeholders as appropriate.

Evaluation & Evaluation is critical to identify areas for improvement and ensuring adherence to PMETB Tab a
Audit principles and standards.
Evaluation &
Evaluation proforma must be completed and signed by the chairperson of a selection panel. Audit Form (Draft for
Form(s) delivered Approval)
Specifically, the panel must audit the process and methodology to monitor compliance with national
by COPMeD standards.
Steering Group
Tab a
(Autumn 2006) Candidate
After interview, all candidates will be asked to complete an anonymised evaluation proforma
Evaluation Form
Upskilling & regarding recruiter effectiveness. (Draft for Approval)
Process delivered
by UoA/Deanery Evaluation information will be collated centrally by the UoA. Results will be reported to relevant
stakeholder groups for modification of the existing process as appropriate.

Tab X=where to find documentation and materials in the resource pack


Example Person Specification

Specification
Example
Person
PERSON SPECIFICATION
APPLICATION TO ENTER SPECIALTY TRAINING at ST1: ANAESTHESIA

ENTRY CRITERIA
ESSENTIAL DESIRABLE WHEN EVALUATED1
QUALIFICATIONS • MBBS or equivalent medical Application form
qualification
ELIGIBILITY • Eligible for full or limited Application form
registration with the GMC at
time of appointment
• Evidence of achievement of Application form
Foundation competencies by Interview / Selection
August 2007 in line with centre
2

GMC standards/Good
Medical Practice including:
• Good clinical care
• Maintaining good medical
practice
• Good relationships and
communication with patients
• Good working relationships with
colleagues
• Good teaching and training
• Professional behaviour and
probity
• Delivery of good acute clinical
care
• Eligibility to work in the UK Application form

FITNESS TO • Is up to date and fit to Application form


PRACTISE practise safely References
LANGUAGE • All applicants to have Application form
SKILLS demonstrable skills in written Interview / Selection
and spoken English that are centre
adequate to enable effective
communication about
medical topics with patients
and colleagues which could
be demonstrated by one of
the following:
• a) that applicants have undertaken
undergraduate medical training in
English; or
• b) have the following scores in the
academic lnternational English
Language Testing System (IELTS)
– Overall 7, Speaking 7, Listening
6, Reading 6, Writing 6.
• However, if applicants
believe that they have
adequate communication
skills but do not fit into one
of the examples they need
to provide evidence

1
‘when evaluated’ is indicative, but may be carried out at any time throughout the selection process
2
A selection centre is a process not a place. It involves a number of selection activities that may be delivered within the Unit of Application.
HEALTH • Meets professional health Application form
requirements (in line with Pre-employment health
GMC standards/Good Medical screening
Practice)

CAREER • No unexplained career gaps Application form


PROGRESSION • Less than 12 months’
3
experience (at SHO level)
in this specialty (not
including Foundation
modules) by August 2007
APPLICATION • ALL sections of application Application form
COMPLETION form FULLY completed
according to written
guidelines

SELECTION CRITERIA
CLINICAL SKILLS • Clinical Knowledge & • Personal Attributes: Application form
Expertise: Capacity to apply Shows aptitude for practical Interview / Selection
sound clinical knowledge & skills, e.g. manual dexterity centre
judgement. Able to prioritise
clinical need. Works to References
maximise safety & minimise risk

ACADEMIC / • Research Skills: • Evidence of relevant Application form


RESEARCH SKILLS Demonstrates understanding of academic & research Interview / Selection
the principles of audit & achievements , e.g. centre
research degrees, prizes, awards,
distinctions, publications,
presentations, other
achievements
• Evidence of active
participation in audit
• Teaching: Evidence of
interest and experience in
teaching

3
Any time periods specified in this person specification refer to full time equivalent
PERSONAL SKILLS • Vigilance & Situational Application form
Awareness: Capacity to be Interview / Selection
alert to dangers or problems, centre
particularly in relation to clinical
governance. Demonstrates References
awareness of developing
situations
• Coping with Pressure:
Capacity to operate under
pressure. Demonstrates
initiative & resilience to cope
with setbacks & adapt to rapidly
changing circumstances.
Awareness of own limitations &
when to ask for help
• Managing Others & Team
Involvement: Capacity to work
cooperatively with others and
demonstrate leadership when
appropriate. Capacity to work
effectively in multi-professional
teams
• Problem Solving &
Decision Making: Capacity to
use logical/lateral thinking to
solve problems & make
decisions
• Empathy & Sensitivity:
Capacity to take in others’
perspectives and see patients
as people
• Communication Skills:
Demonstrates clarity in
written/spoken communication
and capacity to adapt language
as appropriate to the situation
• Organisation & Planning:
Capacity to organise oneself &
prioritise own work.
Demonstrates punctuality,
preparation & self-discipline.
Understands importance of
information technology

PROBITY • Professional Integrity & Application form


Respect for Others: Interview / Selection
Capacity to take responsibility centre
for own actions and
demonstrate a non-judgemental References
approach towards others.
Displays honesty, integrity,
awareness of confidentiality &
ethical issues

COMMITMENT TO • Learning & Personal • Extracurricular activities / Application form


SPECIALTY Development: Demonstrates achievements relevant to Interview / Selection
interest and realistic insight into anaesthesia, intensive centre
anaesthesia, intensive care & care and/or acute care
acute care. Demonstrates self- References
awareness & ability to accept
feedback.
Reference Report

Reference
Report
Page 1 of 4:

REF 1
AUG 2007

Structured Reference
The candidate to whom this reference refers has applied for Specialty training post(s) and has given your
name as a referee. We would be grateful if you could provide us with information requested below. Please
note that we can only accept references on this structured reference form.

This professional reference should verify factual information and comment on the strengths and weaknesses
of the candidate as an indicator of his/her suitability for appointment. This is not a personal testimonial but
your objective assessment, from personal observation, of competencies based on the relevant person
specification(s). If the candidate is successful, we may also use the information you provide to help inform
the formative assessment process and identify potential learning needs. In such cases the comments you
make may be discussed with the candidate named below and/or his/her trainer. Your reference may also be
made available to other departments within the NHS.

This reference form has been developed with the General Medical Council publication “Good Medical
Practice” in mind. Your attention is drawn to the following paragraph:

“When providing references for colleagues, your comments must be honest and justifiable; you must
include all relevant information which has a bearing on the colleague’s competence, performance,
reliability and conduct”

(GMC Good Medical Practice, Second Edition, July 1998 – The duties of a doctor registered with the General Medical Council, Item 11
– References.)

Candidate Name: (MTAS to complete)

Candidate GMC No: (MTAS to complete) Candidate Ref No: (MTAS to complete)

Post(s)/Specialties (1st specialty applied for - MTAS to complete)


Applied For: (2nd specialty applied for - MTAS to complete)

Relationship to Educational Supervisor Clinical Supervisor/Employer Other (please specify)


candidate:
Please state the dates the candidate was supervised by/worked with you:

Date started: Date finished:


Position held / Grade:
Location:

Are you aware of any formal disciplinary procedure(s) the candidate was subject to during their time with you?

YES NO If Yes, please give details:

Form REF 1
Issued by (organisation)
(website)
Page 2 of 4:

REF 1
AUG 2007

Structured Reference
Please give your opinion regarding the candidate’s present knowledge, skills and personal attributes by
ticking the appropriate boxes on the next three pages. Statements are provided to give examples of
behaviours that would constitute different levels of performance, although this is not intended to be an
exhaustive list. Please use the space provided to give examples of the candidate’s behaviour that support
the rating you have given them in each area; this is essential if you have given a rating of D or C.

[Medical graduates only] Clinical Knowledge & Expertise: Appropriate knowledge base and capacity to apply sound clinical
judgement to problems. Capacity to prioritise clinical need.
A B C D
Good to excellent Satisfactory Weak Cause for concern
Comments/evidence:

[Non-medical graduates for public health only] Technical Knowledge & Expertise: Appropriate knowledge base relevant to
public health and capacity to apply sound judgement to problems.
A B C D
Good to excellent Satisfactory Weak Cause for concern
Comments/evidence:

Communication Skills: Capacity to communicate clearly and effectively with others, adjusting behaviour and language
(written/spoken) as appropriate to needs of differing situations.
A B C D
Always speaks clearly, gives Usually communicates clearly, Can be lacking in clarity and Uses technical language that
others time to speak and tends to use appropriate coherence when others do not understand,
checks understanding language communicating with others ignores what they have to say
Comments/evidence:

Empathy & Sensitivity: Capacity and motivation to take in patients’/colleagues’ perspectives and treat others with
understanding.
A B C D
Always shows empathy and Usually demonstrates Shows some interest in the Is not sensitive to the feelings
sensitivity, gives reassurance empathy towards others individual and occasionally of others and treats them in an
to others reassures others impersonal manner
Comments/evidence:

Professional Integrity: Capacity and motivation to take responsibility for own actions. Respect for position, patients and
protocol.
A B C D
Takes full responsibility for Often shows respect to others, Sometimes seeks to blame Does not take responsibility for
their own actions, is generally aware of ethical others for their actions their actions, does not
demonstrates respect for all issues demonstrate respect for others
Comments/evidence:

Form REF 1
Issued by (organisation)
(website)
Page 3 of 4:

REF 1
AUG 2007

Structured Reference
Problem Solving & Decision Making: Capacity to think beyond the obvious, with analytical but flexible mind. Capacity to
bring a range of approaches to problem solving and decision making.
A B C D
Thinks beyond surface Usually thinks beyond surface Often relies on surface Lets assumptions guide
information, uses a range of information, able to make information, can be hesitant diagnosis/decision making,
problem solving strategies decisions when making decisions does not think around issues
Comments/evidence:

Organisation & Planning: Capacity to organise time/information in a structured and planned manner. Capacity to prioritise
conflicting demands and deliver on time.
A B C D
Excellent at managing own Usually able to prioritise tasks Is often late for meetings and Is always late for
time and prioritising workload and meet deadlines deadlines and disorganised meetings/deadlines and
with paperwork etc. unable to prioritise tasks
Comments/evidence:

Learning & Development: Capacity and motivation to learn from experience, commits time and resources to appropriate
personal and professional development activities.
A B C D
Actively seeks out constructive Often learns from experience, Needs assistance in Reacts badly to constructive
criticism/feedback and generally reacts well to identifying own development criticism or feedback, not
development opportunities constructive criticism/feedback needs/targets interested in own development
Comments/evidence:

Managing Others & Team Involvement: Capacity to work effectively in partnership with others and demonstrate
leadership when appropriate.
A B C D
Is excellent at supporting and Recognises contribution of Tends to take a ‘back seat’ Sticks rigidly to their own
motivating others, able to lead others, usually able to rather than participating, agenda, critical of others’
when appropriate compromise reluctant to lead ideas
Comments/evidence:

Working under Pressure: Capacity to work effectively under pressure, remaining calm and objective. Demonstrates initiative
and resilience to cope with setbacks and rapidly changing circumstances.
A B C D
Always remains calm under Often recognises when to Finds it difficult to remain calm Loses temper easily, refuses
pressure, rapidly adapts to share workload, able to cope under pressure or to switch off to share workload, unable to
changing situations with changing circumstances after work adapt to change
Comments/evidence:

Form REF 1
Issued by (organisation)
(website)
Page 4 of 4:

REF 1
AUG 2007

Structured Reference
Was the candidate’s attendance/timekeeping satisfactory?

YES NO If No, please give details

Would you be happy to work with this candidate again? YES NO

Recommendation of candidate for training in (1st specialty - MTAS to insert)


Strongly without reservation A Would have some reservations C
Could recommend as competent B Could not recommend for training D
If you have any other comments regarding this candidate’s application for this specialty, please give details here:

Recommendation of candidate for training in (2nd specialty - MTAS to insert)


Strongly without reservation A Would have some reservations C
Could recommend as competent B Could not recommend for training D
If you have any other comments regarding this candidate’s application for this specialty, please give details here:

This reference is based upon (tick all that apply):


General impression A Collective opinion of tutors/consultants/supervisors C
Close observation B Employer’s views D

SIGNATURE NAME (print in block capitals)

POSITION HELD CONTACT TELEPHONE NO.


Name of deanery/
DATE (dd/mm/yyyy)
hospital/practice/organisation

It is essential that this form is stamped with an official deanery, hospital, training practice or organisation stamp. If no stamp is available,
please attach a compliment slip signed by the referee. Forms received without a stamp or a signed compliment slip will be returned.
Official stamp (deanery/hospital/training practice/organisation)
Thank you for completing this reference.
This form should be returned to the address
given on the accompanying e-mail.
Please ensure that a signed paper copy is also given to the candidate.

Form REF 1
Issued by (organisation)
(website)
RECOMMENDED PROCESS

FOR

THE USE OF REFERENCES DURING RECRUITMENT AND SELECTION


INTO SPECIALIST TRAINING DURING 2007

Introduction

1. The principle of applicants being required to provide details of two


referees, both of whom must have supervised the applicant in the past 2
years, has been agreed by the four UK health departments.

2. It has also been agreed that structured references will be requested for
short-listed candidates only.

3. Equally, it has been agreed that the Medical Training Application Service
(MTAS) will be used by all four UK countries for recruitment into specialist
training programmes and FTSTAs.

4. The four UK health departments have stated that there will be nationally
agreed structured reference forms for each specialty or specialty group.

5. This paper sets out the recommended process for how references will fit
into the overall process and timeline for recruitment to specialist training
during the national recruitment episode planned for 2007.

The issues

6. Anecdotal evidence suggests references are not received for up to 40% of


applicants by the time of interview. Deaneries and trusts have to chase up
the missing references, which is a time consuming exercise. When we
move to national recruitment within a very tight timescale, it will not be
possible to chase references in this way. .

7. The process and supporting MTAS system must provide a mechanism for
ensuring references are available at the time of the interview for every
applicant.

8. There is considerable concern across deaneries and trusts regarding


falsified or altered references.

9. Every applicant has a legal right to see their references.

10. It is felt that some referees will not give an accurate reference if they know
it will be seen by the applicant (i.e. they may be reluctant to raise issues of
poor performance). However, applicants are currently allowed to see their
references under the freedom of information act.
11. Where and how references are used within the selection process is
currently variable across deaneries and trusts.

Recommended process

12. The national, structured reference form will be the only reference
accepted.

13. Applicants must provide valid e-mail addresses for two referees. They will
have the option to nominate different referees for each specialty they apply
to.

14. Once an application is submitted, the MTAS system will generate an


automatic e-mail message to nominated referees to inform them that they
have been nominated (and by whom) and to request that they respond to
the message to prove that the communication channel is open.

15. Referees will also be able to submit the completed reference form at this
stage if they wish to do so. They will also be able to view a copy of the
applicant’s submitted application form.

16. If the referee does not respond we can use the MTAS system to send
reminders and/or to send notes to the applicant telling them that there is a
problem with the referee’s e-mail address. There is the three week short-
listing period to do this. Precisely how this will be developed will be
determined during the MTAS requirements workshops being held in
September/October 2006.

17. Once short-listing is complete, the MTAS system will automatically send a
message to the nominated referees of short-listed applicants who have not
already submitted a completed reference requesting they complete the
structured reference form and telling them by when it must be submitted.
They will also be able to view a copy of the applicant’s submitted
application form.

18. Referees will need to complete and submit the form in a single session.
There will be no facility to save the form and return to it at a later date.

19. There will be a facility to print copies of the completed form. Copies
should be printed and then a signed copy provided to the applicant.

20. Short-listed applicants will be instructed to bring a copy of all signed


/certified references with them to the selection interview.

21. References will only be seen by the selection panel chair prior to the
interview. The chair may choose to explore an area with the candidate as
appropriate during the interview.

22. Following the selection interview and scoring, the reference may be seen
by the selection panel members as supplementary support for decisions
made, under the guidance of the chair.
23. Where there is cause for concern between the outcome of the selection
interview and the content of the reference(s), or major differences between
the references provided by the two referees, recruitment staff will be able
to cross-check the content of the signed/certified reference with the
electronic copy held on the MTAS system to ensure that the printed
references have not been altered in any way.
Application Form Draft

Application
Form Draft
ANY CANDIDATE FALSIFYING EVIDENCE ON THIS APPLICATION FORM OR IN
SUPPORTING EVIDENCE WILL BE AUTOMATICALLY REFERRED TO THE GMC

SAMPLE APPLICATION FORM FOR SPECIALTY TRAINING


PROGRAMME ENTRY AT ST1

PLEASE NOTE:

• THIS DOCUMENT IS FOR ILLUSTRATION ONLY TO DEMONSTRATE THE SELECTION


METHODOLOGY.

• THE ONLINE APPLICATION FORM WILL HAVE THE SAME SECTIONS BUT WILL BE
PRESENTED DIFFERENTLY AS APPROPRIATE FOR THE ONLINE APPLICATION SYSTEM.

• THE ONLINE APPLICATION FORM WILL INCLUDE DECLARATIONS, FITNESS TO PRACTISE,


CRB DECLARATION, EQUAL OPPORTUNITIES MONITORING FORM AND OTHER RELEVANT
EMPLOYMENT CHECKS.

YOU WILL BE EXPECTED TO PROVIDE DOCUMENTARY EVIDENCE WHERE


APPROPRIATE TO SUPPORT THE RESPONSES YOU HAVE GIVEN HERE IF YOU
ATTEND THE INTERVIEW STAGE OF THE SELECTION PROCESS
ANY CANDIDATE FALSIFYING EVIDENCE ON THIS APPLICATION FORM OR IN
SUPPORTING EVIDENCE WILL BE AUTOMATICALLY REFERRED TO THE GMC
STANDARDISED APPLICATION FORM FOR
SPECIALTY TRAINING PROGRAMME: Entry at ST1
Declaration: I understand that any candidate falsifying evidence on this application form
or in supporting evidence will be automatically referred to the General Medical Council (GMC)

Advertisement Reference Number


Please indicate which Unit(s) of Application and Specialty/Specialties you wish to apply to by completing the
boxes below

Which Specialty/Specialties are you Unit of Application/ Deanery


applying to work in.
A 1
2
B 1
2

Section 1 - Personal Details


Surname Forenames

Title If previously known by another name, please specify

Home Address Address for correspondence

Home telephone number Work telephone number


Home fax number Work fax number
Home email address Work email address
Mobile phone number Preferred email address

Section 2 – Entry Qualifications


Medical School/Dental School/University (NB Only include here the Dates
qualification(s) that meet(s) the eligibility requirements listed in the relevant person
specification(s). Additional qualifications are covered in Section 7)

YOU WILL BE EXPECTED TO PROVIDE DOCUMENTARY EVIDENCE WHERE


APPROPRIATE TO SUPPORT THE RESPONSES YOU HAVE GIVEN HERE IF YOU ATTEND
THE INTERVIEW STAGE OF THE SELECTION PROCESS
ANY CANDIDATE FALSIFYING EVIDENCE ON THIS APPLICATION FORM OR IN
SUPPORTING EVIDENCE WILL BE AUTOMATICALLY REFERRED TO THE GMC
Evidence of IELTS (International English Language Testing System) or Date awarded
equivalent (for candidates who have not graduated from an English medium
university) – state level achieved

SHORTLISTED CANDIDATES WILL BE REQUIRED TO BRING ORIGINAL CERTIFICATES


AND DOCUMENTATION AS EVIDENCE TO THE INTERVIEWS
Section 3 - General Medical/Dental Council Registration
Type of registration GMC/GDC Number Renewal date Name in which you are
registered

Section 4 – Eligibility for Entry into Specialty Training


Have you been issued with a Certificate of Completion of Foundation training? YES NO
You will need to bring the original certificate if you are called for interview
Will you be issued with a Certificate of Completion of Foundation training by August 2007? YES NO
If not applying directly from Foundation, have you gained Foundation competencies? YES NO
If not applying directly from Foundation, please
describe how you have gained Foundation
competencies

Are you a United Kingdom (UK), European Community (EC) or European Economic Area
YES NO
(EEA) national? (Please circle as appropriate)

If not, do you have evidence of entitlement to enter and work permanently in the United
YES NO
Kingdom ie settled status? (Please circle as appropriate)
If you have circled no to both of the above questions please tick those boxes that relate to your current
immigration status:
Tick Status Expiry date
Highly Skilled Migrant Programme

Permit Free Training

Refugee

Work Permit

Any other – please specify ……………………………………………………………

Residence Permit valid until:


Please attach a copy of your residence permit to the application form
Date available to start if offered the post

YOU WILL BE EXPECTED TO PROVIDE DOCUMENTARY EVIDENCE WHERE


APPROPRIATE TO SUPPORT THE RESPONSES YOU HAVE GIVEN HERE IF YOU ATTEND
THE INTERVIEW STAGE OF THE SELECTION PROCESS
ANY CANDIDATE FALSIFYING EVIDENCE ON THIS APPLICATION FORM OR IN
SUPPORTING EVIDENCE WILL BE AUTOMATICALLY REFERRED TO THE GMC

Section 5 – Current and Previous Employment


Current employer’s name
Address
Telephone number
Current position please Grade Date Length of Length of Do you currently
state if locum position appointed contract notice hold an NTN or
required VTN? (please give
number)
YES/NO

Please list previous employment below:


Specialty, Hospital, Consultant Grade From To Months in post

Will you have completed less than 12 months’ experience in the specialty you are
YES NO
applying for by August 2007? [note: not for all specialties]
Do you have any unexplained career gaps (of more than 4 weeks’ duration?) since
qualification? If YES, please specify YES NO

YOU WILL BE EXPECTED TO PROVIDE DOCUMENTARY EVIDENCE WHERE


APPROPRIATE TO SUPPORT THE RESPONSES YOU HAVE GIVEN HERE IF YOU ATTEND
THE INTERVIEW STAGE OF THE SELECTION PROCESS
ANY CANDIDATE FALSIFYING EVIDENCE ON THIS APPLICATION FORM OR IN
SUPPORTING EVIDENCE WILL BE AUTOMATICALLY REFERRED TO THE GMC

Section 6 – Commitment to the Specialty


6.1 Please provide a brief statement describing the reasons why you have chosen to train for a career in this
specialty. Please include any information about yourself that you feel is important or relevant in this
regard. (max. 75 words)

6.2 What steps have you taken to develop your understanding of this specialty? (max. 150 words)

6.3 Please provide evidence of extracurricular activities and achievements relevant to this specialty (these
could be work or non-work related). Indicate date and place relating to the evidence (max. 150 words)

YOU WILL BE EXPECTED TO PROVIDE DOCUMENTARY EVIDENCE WHERE


APPROPRIATE TO SUPPORT THE RESPONSES YOU HAVE GIVEN HERE IF YOU ATTEND
THE INTERVIEW STAGE OF THE SELECTION PROCESS
ANY CANDIDATE FALSIFYING EVIDENCE ON THIS APPLICATION FORM OR IN
SUPPORTING EVIDENCE WILL BE AUTOMATICALLY REFERRED TO THE GMC

Section 7 – Clinical, Academic & Research Skills


7.1 Give an example of a time when you personally made a significant difference to clinical outcome for a
patient by applying your judgement.. What did you do and how do you think the outcome was improved by
your judgement? (max. 150 words).

7.2 What do you think is the relevance of medical research to a trainee doctor? If you have been involved in
research, use examples from your own research to illustrate this. (max. 150 words).

7.3 Please provide evidence of your undergraduate and/or postgraduate academic and research achievements
under the following headings (if appropriate).
a) Additional qualifications, e.g. Awarding Body Date of Qualification
degrees/diplomas – state class of
degree awarded

OMFS only: MFDS

b) Relevant courses/training Place of training/name of training Date of training


completed, e.g. ALS etc. [note: provider
not for all specialties]

YOU WILL BE EXPECTED TO PROVIDE DOCUMENTARY EVIDENCE WHERE


APPROPRIATE TO SUPPORT THE RESPONSES YOU HAVE GIVEN HERE IF YOU ATTEND
THE INTERVIEW STAGE OF THE SELECTION PROCESS
ANY CANDIDATE FALSIFYING EVIDENCE ON THIS APPLICATION FORM OR IN
SUPPORTING EVIDENCE WILL BE AUTOMATICALLY REFERRED TO THE GMC

c) Prizes and other academic Awarding Body Date of Award


distinctions

d) Presentations/posters at conferences/ Publications (shortlisted candidates will be asked to bring copies of


all abstracts and publications to the interview)

e) Describe your experience of clinical audit. Indicate clearly your level of involvement and what you learned
about the process.

f) Experience of delivering teaching. What experience of delivering teaching do you have (including outside
medicine)?

YOU WILL BE EXPECTED TO PROVIDE DOCUMENTARY EVIDENCE WHERE


APPROPRIATE TO SUPPORT THE RESPONSES YOU HAVE GIVEN HERE IF YOU ATTEND
THE INTERVIEW STAGE OF THE SELECTION PROCESS
ANY CANDIDATE FALSIFYING EVIDENCE ON THIS APPLICATION FORM OR IN
SUPPORTING EVIDENCE WILL BE AUTOMATICALLY REFERRED TO THE GMC

Section 8 – Personal Skills - EXAMPLE


A range of personal skills have been identified as important for this specialty. For each of the skills indicated
below, please give an example (preferably recent) from your own experience to illustrate how you dealt with
particular situations. You may draw your experiences from work or other activities (unless otherwise specified).
8.1 [Vigilance & Situational Awareness] Describe a time when you identified a clinical situation that was
changing rapidly. How did you identify this and what did you do? (max. 150 words)

8.2 [Coping with Pressure] Describe a time when you had a particularly difficult day at work due to workload
pressures. What strategies did you use to cope with this situation both during and after this time? What
other strategies do you regularly use to cope with stress? (max. 150 words)

8.3 [Managing Others & Team Involvement] Describe a time when you had to negotiate something to
achieve something important with a colleague(s) or multiprofessional team. What approach did you take?
What was the outcome? (max. 150 words)

YOU WILL BE EXPECTED TO PROVIDE DOCUMENTARY EVIDENCE WHERE


APPROPRIATE TO SUPPORT THE RESPONSES YOU HAVE GIVEN HERE IF YOU ATTEND
THE INTERVIEW STAGE OF THE SELECTION PROCESS
ANY CANDIDATE FALSIFYING EVIDENCE ON THIS APPLICATION FORM OR IN
SUPPORTING EVIDENCE WILL BE AUTOMATICALLY REFERRED TO THE GMC

Section 9 – Probity
9.1 [Professional Integrity] Please outline a time when you had to be open and honest with a patient, even
though you found it hard to do so. Why was this? What was the outcome? (max. 150 words)

Section 10 - Referees
1 2

Telephone Number Telephone Number


Fax Number Fax Number
Email address Email address

Name, job title and address of two Consultant referees, covering the last two years of employment. One of
the referees must be your present or most recent Consultant/Educational supervisor. The Unit of Application
must obtain information from two referees who are able to comment, from experience, on your ability and
suitability for the post.

YOU WILL BE EXPECTED TO PROVIDE DOCUMENTARY EVIDENCE WHERE


APPROPRIATE TO SUPPORT THE RESPONSES YOU HAVE GIVEN HERE IF YOU ATTEND
THE INTERVIEW STAGE OF THE SELECTION PROCESS
Shortlisting Indicators &
Example Responses

Shortlisting
Indicators
Example Responses Only

Please indicate the level of entry (eg ST1, ST2 etc): ST1

Commitment to the Specialty


• What steps have you taken to develop your understanding of this specialty? (max. 150 words)

During my medical course I spent 3 weeks in the Cambridge clinical genetics department as part of an “SSC
(student selected component)” including a range of adult and paediatric clinics covering the spectrum from
adult cancer to pre-natal diagnosis. I also attended numerous genetics clinics apart from the SSC, which
together with the SSC confirmed my attraction to the specialty. I have already been in contact with some of the
Exeter department, and hope to spend some time in clinics during my F2 year. My obstetrics & gynaecology
audit will look at the interface between obstetrics and genetics, and I hope to be involved in the links with
genetics in my paediatrics F2 job as well.

• Please provide evidence of extracurricular activities and achievements relevant to this specialty. Indicate
date and place relating to the evidence (max. 150 words)

During my medical course I was the BMA representative (2003-4) and the CMF (Christian Medical Fellowship)
(2003-4) representative for Cambridge University. At BMA student conferences I proposed motions on various
ethical issues, including informed consent for vaccination (2002, London), palliative care and euthanasia (2003,
London), the law on driving offences and homicide (2004, Cambridge), and physician assisted suicide (2005,
Keele). In 2003 I proposed a motion on discrimation in medicine at the BMA Annual Members’ Meeting
(Torquay). All this indicates a strong interest in medical ethics which will be applicable to the ethical questions
involved in genetics (e.g. the question of whether someone should have a genetic test, or whether ignorance is
better).
Genetics has a strong research focus. During my spare time from 2002-4 on my medical course in Cambridge
I worked in clinical research, which should help in applying research results (both clinical and laboratory) to
patient care.

THANK YOU FOR COMPLETING THIS FORM


Example Responses Only

Please indicate the level of entry (eg ST1, ST2 etc): ST1

Clinical, Academic & Research Skills


• Give an example of a time when you personally made a significant difference to clinical outcome for a
patient by applying your judgement. What did you do and how do you think the outcome was improved by
your judgement? (max. 150 words).

I was called to see an elderly surgical patient who had become anuric. Her urine output had been poor for
three days and had been reviewed by several doctors, including senior members of the team. Each time she
had been given a fluid challenge, but was never reassessed following this. On reviewing her charts she now
had a severely positive fluid balance. Clinically assessing the patient she was fluid overloaded with pitting
oedema to the level of her scapula, with a markedly raised jugular venous pressure. On this assessment she
did not need further fluid challenges, but input from the renal team. I discussed the case with the renal registrar
who suggested transfer to the high dependency unit and haemodialysis. Her renal failure could have been
missed and further inappropriate fluid challenges given if I had not assessed the patient carefully and trusted
my clinical judgement.

• What do you think is the relevance of medical research to a doctor? If you have been involved in research,
use examples from your own research to illustrate this. (max. 150 words).

Without medical research, medicine would never progress. If no one is asking “How can we do this better?”
and thinking of ways to test their hypotheses, the quality of medical care would never improve (and would
probably stagnate). Evidence for therapies and medicines needs trials to prove or disprove the usefulness of
an intervention. Prior to training in medicine I was involved in pre-clinical research, including researching new
analgesics, then during my medicine course on clinical research on pain models in healthy volunteers. This
showed me the scope there is for doctors to be involved in both pre-clinical and clinical research, and my
research has given me an enquiring mind, so that I am not content to assume that we already have the best
way of doing things. It has also shown me the advantages in doing research in a team, and how productivity
improves when a team works together.

THANK YOU FOR COMPLETING THIS FORM


Example Responses Only

Please indicate the level of entry (eg ST1, ST2 etc): ST1

• Please provide evidence of your undergraduate and/or postgraduate academic and research achievements
under the following headings (if appropriate).

a) Additional qualifications, e.g. Awarding Body Date of Qualification


degree(s)/diploma(s) – state class
of degree awarded
BmedSci 2:1 University of Nottingham December 2003

BMBS University of Nottingham July 2005

b) Prizes and other academic Awarding Body Date of Award


distinctions
None to date

c) Presentations/posters at conferences & relevant publications (shortlisted candidates will be asked to bring
copies of all abstracts and publications to the interview)

None to date

d) Describe your experience of clinical audit. Indicate clearly your level of involvement and what you learned
about the process.
I instigated, developed and performed an audit on the effects of advice given when prescribing emergency
contraception on future effective contraception use. I established current guidelines, and set standards with
reference to these. Through retrospective review of notes I compared recent practice to these standards. This
enabled me to develop advice on future practice. This was then implemented. I did not have the opportunity to
complete the audit cycle through re-audit after the advice was implemented. This was useful experience in the
audit cycle, and on extracting data from patient notes.
I am currently auditing complications for potentially curative resections for bowel malignancy. I am collating the
data retrospectively from patient notes. This data will compare results in our region with national standards.

e) What experience of delivering teaching do you have?


I had a final year medical student shadow me for two weeks at the end of my first foundation year, allowing him
practical experience before starting as a house officer himself. He gave me good feedback on the two weeks,
allowing him plenty of practical experience while assisting him when needed. I currently hold regular teaching
sessions with the fifth year medical student attached to our firm, combining sessions on history taking and
examination skills, with background theory on subjects of her choice. I note interesting cases I have
encountered and share them with the medical student and house officers. I am supportive to the house
officers, giving them relevant teaching when they encounter new clinical problems. I have given presentations
at first year foundation teaching sessions, at educational radiology meetings, and at surgical training meetings.

THANK YOU FOR COMPLETING THIS FORM


Example Responses Only

Please indicate the level of entry (eg ST1, ST2 etc): ST1

Personal Skills: Specialty-Specific Questions


A range of personal skills have been identified as important for this specialty. For each of the skills indicated
below, please give an example (preferably recent) from your own experience to illustrate how you dealt with
particular situations. You may draw your experiences from work or other activities

• [Vigilance & Situational Awareness] Describe a time when you identified a clinical situation that was
changing rapidly. How did you identify this and what did you do? (max. 150 words)
A patient had been referred to the surgical unit by the general practitioner with possible biliary colic. On arrival
he appeared unwell, on assessment he was clinically shocked and I began initial resuscitation. He improved
markedly becoming more orientated, with improving observations. During further assessment however he
deteriorated further. I found he had unequal blood pressures in both arms, and a palpable expansile abdominal
mass. I phoned for assistance from my senior, arranged an urgent CT angio and arranged for an anaesthetist
to accompany me to the scanner. A thoracic aortic dissection with leaking abdominal aortic aneurysm was
seen on the scan and arrangements were made for the patient to go straight to theatre for an open repair.

• [Coping with Pressure] Describe a time when you had a particularly difficult day at work due to workload
pressures. What strategies did you use to cope with this situation both during and after this time? What
other strategies do you regularly use to cope with stress? (max. 150 words)

I was the surgical senior house officer on call, on a day when my registrar was away leaving me with main
responsibility for the ward patients. Due to a rota error there was no house officer, and I was asked to cover
their responsibilities as well. I also had a presentation to give. I conducted a brief ward round, ensuring my
patients were stable and essential house officer tasks were finished. I arranged with a fellow senior house
officer to hold my bleep during the presentation. When I collected this several referrals had arrived. I
prioritised these by clinical urgency and began clerking. During a gap in referrals I reviewed my ward patients
and checked all jobs had been cleared. By prioritising, regularly reviewing the situation, and utilizing help from
colleagues I managed to juggle several different roles effectively.

• [Managing Others & Team Involvement] Describe a time when you had to negotiate something with a
colleague(s) or multiprofessional team to achieve something important. What approach did you take? What
was the outcome? (max. 150 words)
A patient was investigated under vascular for a possible psuedoaneurysm, which was excluded. His admission
chest X-ray showed a large, suspicious lung lesion. He had previously had resection of a bladder tumour, and
was under follow-up by urology. The vascular team were unwilling to investigate this further. On discussion
with the respiratory team they suggested urology review. The urology team refused and said it was now a
respiratory problem. After these difficulties with referral I organised an urgent CT chest and abdomen myself,
and discussed the problem with the lung cancer nurse. She made him an urgent appointment in lung cancer
clinic and listed him for the multi-disciplinary team meeting. I was then happy for him to be discharged,
knowing that he would get the appropriate follow-up and investigations he needed.

THANK YOU FOR COMPLETING THIS FORM


Example Responses Only

THANK YOU FOR COMPLETING THIS FORM


ILLUSTRATION
Reference Framework for Target Selection Criteria

CLINICAL, ACADEMIC & RESEARCH SKILLS


CLINICAL KNOWLEDGE & EXPERTISE: Capacity to apply sound clinical knowledge and judgement.
Positive Indicators Negative Indicators
• identifies key issues involved, gets to the root cause • overlooks important issues
• elicits necessary detail from patient/colleague • fails to explore important information / signals
• shows sound/systematic judgement in making decisions • is too quick/unsystematic in making decisions
• is aware of appropriate options • suggests too narrow a range of options
• is able to anticipate possible issues • needs the “full picture” before understanding problem
RESEARCH SKILLS: Understanding of the importance and basic principles of research.
Positive Indicators Negative Indicators
• provides clear and reasoned explanation / argument for • explanation / argument is unclear or lacks relevance
importance of research • shows little understanding of purpose and impact of research
• demonstrates accurate understanding of purpose and impact of • demonstrates little awareness of research methodology
research • provides no evidence of personal experience of research
• shows awareness of basic principles of research methodology • unable to link own experience to his/her argument
• provides evidence of personal experience of research
• provides evidence of personal experience to support argument

PERSONAL SKILLS
EMPATHY & SENSITIVITY: Capacity and motivation to take in others’ perspectives and treat others with understanding.
Positive Indicators Negative Indicators
• responds to needs/concerns with interest /understanding • shows little apparent interest in others’ situations
• acts in open, non-judgemental manner • is quick to judge, makes assumptions
• makes effort to understand others’ perspectives • little evidence of understanding others’ needs/concerns
• reassures others with appropriate words/actions • unable to reassure others effectively
• attempts to generate safe/trusting atmosphere • fails to generate safe/trusting atmosphere
VIGILANCE & SITUATIONAL AWARENESS: Capacity to monitor developing situations and anticipate dangers/problems.
Positive Indicators Negative Indicators
• is alert to symptoms and signs suggesting conditions which • pays little attention to changes in clinical picture
might progress or de-stabilise rapidly • fails to pick up on signs that indicate that original plans need to
• picks up subtle changes in clinical condition be revised
• is able to anticipate possible dangers/problems • is unable to anticipate dangers / problems
• demonstrates awareness of performance of team members and • lacks awareness of level of competence of team members
social dynamics within clinical situations and/or social dynamics within the clinical team
COPING with PRESSURE: Capacity to operate under pressure. Demonstrates initiative and resilience to cope with changing
circumstances.
Positive Indicators Negative Indicators
• remains calm and in control when under pressure • becomes tense, agitated or over-sensitive under pressure
• rarely looses sight of wider needs of situation • shifts focus to immediate worries/needs, looses perspective
• recognises own limitations and is able to compromise • becomes defensive or uncompromising under pressure
• is able to seek help when necessary • tries unsuccessfully to deal with situation alone
• uses strategies to deal with pressure/stress • uses inappropriate strategies to deal with pressure/stress
ILLUSTRATION
MANAGING OTHERS & TEAM INVOLVEMENT: Capacity to work effectively in partnership with others and demonstrate
leadership when appropriate.
Positive Indicators Negative Indicators
• participative, non-confrontational approach • confrontational style, is critical of others and their views
• negotiates with others, is willing to compromise • sticks rigidly to own agenda and does not negotiate /
• respects others’ views, is able to build on others’ ideas compromise
• able to delegate and demonstrate leadership when appropriate • treats some colleagues more favourably than others
• gives constructive feedback & support where appropriate • fails to delegate or demonstrate leadership when appropriate
• provides little support for others (verbal and/or enacted)
COMMUNICATION SKILLS: Capacity for clarity in written/spoken communication and to adapt language as appropriate to
the situation.
Positive Indicators Negative Indicators
• adjusts style of questioning / response as appropriate • is unable to adapt language / behaviour as needed
• is able to express ideas clearly to others (written/spoken) • is patronising / domineering in communicating with others
• uses inventive language (e.g. humour, analogy) to explain • use of language is too functional / narrow / technical
• uses active listening and open, patient-centred questions • limited evidence of active listening, overuse of closed questions
• makes effective use of non-verbal behaviour (voice, posture etc) • fails to engage others at non-verbal level
ORGANISATION & PLANNING: Capacity to organise time/information in a structured/planned manner. Able to think ahead
and prioritise conflicting demands.
Positive Indicators Negative Indicators
• thinks ahead, prepares and plans effectively • misses deadline based on reasonable expectations
• builds contingencies to deal with shifting demands • fails to think ahead, pre-plan, & build effective contingencies
• prioritises conflicting demands, delivers on time • disorganised, fails to manage time effectively
• understands limitations and works within them • unable to juggle competing demands

PROBLEM SOLVING & DECISION MAKING: Capacity to think beyond the obvious, analytical but flexible mind. Capacity to
apply knowledge to new situations and generate functional solutions.
Positive Indicators Negative Indicators
• attempts to think “around” issue, open to new ideas • makes immediate assumptions about problems
• generates functional solutions • unable to suggest “workable” outcome
• prioritises information/time effectively, identifies key points • disorganised/unsystematic approach
• able to sift peripheral information to detect root cause • focuses on peripheral issues, unable to identify root cause

PROBITY
PROFESSIONAL INTEGRITY: Capacity to take responsibility for own actions. Demonstrates respect for all.
Positive Indicators Negative Indicators
• demonstrates clear respect for patient(s)/colleague(s) • lacks sufficient respect for others
• takes full responsibility for own actions, able to admit mistakes • avoids taking responsibility for poor decisions/ideas
• backs own judgement, expresses suggestions / decisions with • is tentative when explaining decisions/actions
appropriate confidence • fails to prioritise patient(s)/others’ needs when appropriate
• puts patient(s)/others’ needs before their own when appropriate • lacks a sense of operating within clear professional boundaries
• consistently acts within professional boundaries
ILLUSTRATION

COMMITMENT TO THE SPECIALTY


Reasons for choice of specialty and relevant information.
Positive Indicators Negative Indicators
• provides clear, comprehensive rationale for choice of specialty • unfocused, unclear reasons for choice of specialty
• reasoning demonstrates understanding of the specialty and of • reasoning demonstrates a limited / inaccurate understanding of
own abilities / needs the specialty and/or own abilities
• demonstrates insight into own career motivations / • shows little insight into own career motivations /
goals/development needs goals/development needs
• additional information provided is relevant and supports • additional information provided is irrelevant or unclear
rationale / choice of specialty
LEARNING & PERSONAL DEVELOPMENT: Demonstrates realistic insight into and understanding of the specialty.
Commitment to personal and professional development.
Positive Indicators Negative Indicators
• realistic insight into positive & negative aspects of the specialty • superficial and/or overly positive understanding of the specialty
• provides evidence of efforts made to increase knowledge • provides little evidence of actions taken to learn about specialty
• is able to reflect and learn from experience/opportunities • shows little evidence of learning from experience
• recognises & makes use of available learning opportunities • demonstrates limited awareness of potential learning resources
• identifies gaps in knowledge & organises appropriate • takes a passive approach towards learning opportunities,
development; takes an active, structured approach demonstrated little commitment to own development
RELEVANT EXTRACURRICULAR ACTIVITIES / ACHIEVEMENTS

Positive Indicators Negative Indicators


• provides a range of examples or 1-2 significant examples • fails to provide examples of activities/achievements
• evidence is verifiable with dates and places indicated • gives little/no indication of dates / places, evidence is
• provides clear explanation of how activities are relevant to the unverifiable
specialty, e.g. in content or in relation to transferable skills • examples given are unclear and their relevance to the specialty
• examples given demonstrates active commitment to own is not explained
development (professional and/or personal) • range of examples provided is very narrow or superficial
• demonstrates little active commitment to own development
Page 1 of 2:
ENTRY AT ST1
Scoring Indicators for Shortlisting
Statements are provided below to give examples of behaviours that would indicate different levels of
performance for each shortlisting question, although this is not intended to be an exhaustive list.

COMMITMENT TO THE SPECIALTY

Personal Statement: Reasons for choice of specialty and relevant information.


NE 1 2 3 4
Unclear reasons for choice Comprehensive rationale
Rationale for choice of
of specialty, lacked Clear reasons for choice of for choice of specialty,
specialty somewhat
understanding of specialty, evidence of clear understanding of
No evidence unclear/unfocused, limited
specialty/own motivation, some insight into specialty specialty/own motivations
insight into specialty
no or irrelevant additional and/or own motivations & relevant additional
and/or own motivation
information information
Learning & Personal Development: Demonstrates realistic insight into and understanding of the specialty. Commitment to
personal and professional development.
NE 1 2 3 4
Realistic insight into
Inaccurate understanding
Superficial understanding Realistic understanding of (positive & negative)
of specialty, little evidence
of specialty, steps taken to some aspects of specialty, aspects of specialty,
of actions taken to
No evidence increase knowledge are evidence of active efforts evidence of active,
increase knowledge or of
limited and/or passive in made to increase structured & reflective
commitment to own
approach knowledge approach to learning &
development
development

Relevant Extracurricular Activities/Achievements


NE 1 2 3 4
A range or 1-2 significant
Example(s) limited in Verifiable example(s),
Unclear & unverifiable & verifiable examples,
scope and/or somewhat some evidence of
examples, no attempt to relevance to specialty &
No evidence vague/unverifiable, relevance to specialty
explain relevance to own development clearly
superficial attempt to link and/or personal
specialty explained (e.g. content or
to specialty development
transferable skills)

CLINICAL, ACADEMIC & RESEARCH SKILLS

Clinical Knowledge & Expertise: Capacity to apply sound clinical knowledge and judgement.
NE 1 2 3 4
Evidence of sound,
Unsystematic approach to Narrow or rigid approach, Identified key issues
systematic judgement,
decision-making, failed to needed the full picture involved, elicited
thorough investigation of
No evidence explore or anticipate before understanding necessary detail from
key issues / options,
important information / problem, little awareness others and was aware of
understood impact of own
issues of own impact appropriate options
actions

Research Skills: Understanding of the importance and basic principles of research.


NE 1 2 3 4
Clear & accurate
Little understanding of Limited or superficial Understanding of the understanding of the
purposes or impact of understanding of the importance of research, importance & purposes of
No evidence research, explanation / importance of research, awareness of basic research, used personal
argument was unclear or little awareness of principles of research experience to support
lacked relevance research methods methodology argument where
appropriate
Page 2 of 2:
ENTRY AT ST1
Scoring Indicators for Shortlisting
Relevant Academic & Research Achievements: Including involvement in audit and experience of delivering teaching.
0 points 1 point 2 points
Additional qualifications No evidence Any additional degree(s) or relevant PhD
diploma(s)
Membership exam (OMFS only) No evidence MFDS

Courses (some specialties only) No evidence ALS, ALERTS or equivalent


Paediatric life support course
(paediatrics)
Prizes / distinctions No evidence Any undergraduate or postgraduate
prizes / distinctions
Presentations / posters / Presentation / poster Regional / national presentation / poster International presentation
publications within hospital / workplace Non-peer-reviewed publication Peer-reviewed publication

Experience of audit No direct experience Experience of audit e.g. collected data, Initiated or managed audit
completed forms, attended meetings project, evidence of
learning from experience
Experience of delivering teaching Informal teaching Formal teaching

PERSONAL SKILLS
Vigilance & Situational Awareness: Capacity to monitor developing situations and anticipate dangers/problems.
NE 1 2 3 4
Alert to subtle signs and
Able to identify symptoms
Picked up only on clear / minimal cues suggesting
Paid little attention to and signs suggesting
obvious signs of rapidly conditions that might
changes in clinical picture, conditions that might
No evidence changing situations, destabilise rapidly, able to
was unable to anticipate progress or destabilise
struggled to maintain anticipate problems, aware
dangers / problems rapidly, maintains vigilance
vigilance for long periods of team members’
despite distractions
performance
Coping with Pressure: Capacity to operate under pressure. Demonstrates initiative and resilience to cope with changing
circumstances.
NE 1 2 3 4
Became tense or Remained calm under
Found it difficult to remain Recognised own
defensive under pressure, pressure, retained
calm under pressure or limitations and was able to
tried unsuccessfully to perspective and sought
No evidence seek help, limited compromise, identified
deal with situation alone, help when necessary,
strategies to deal with strategies to deal with
inappropriate strategies to appropriate strategies to
pressure & stress stress
deal with stress deal with stress
Managing Others & Team Involvement: Capacity to work effectively in partnership with others and demonstrate
leadership when appropriate.
NE 1 2 3 4
Confrontational approach, Little active involvement Participative & non-
Worked in partnership with
stuck rigidly to own with others in achieving confrontational approach,
others and respected their
No evidence agenda & was goals, treated some able to compromise &
views, provided support
unable/unwilling to colleagues more build on others’ ideas,
for others
compromise favourably than others delegated effectively

PROBITY
Professional Integrity: Capacity to take responsibility for own actions. Demonstrates respect for all.
NE 1 2 3 4
Avoided taking Recognised
Was tentative when Took full responsibility for
responsibility for poor responsibilities & backed
explaining own actions, demonstrated
decisions/ideas, lacked own judgement with
No evidence decisions/actions, tried to clear respect for all,
respect for others, little appropriate confidence,
find excuses for errors or consistently acted within
understanding of demonstrated respect for
misunderstandings professional boundaries
professional boundaries others
Example Interview Report Form

Interview
Example
STRUCTURED INTERVIEW EVALUATION FORM
SPECIALTY ________________ LOCATION __________________

Candidate Name: Date: / /


(MM DD YY)
Selector Name(s):
(Provide all selector names)

INSTRUCTIONS:
The interview contains ________ question areas. The interview should last no more than _____ minutes.
We recommend that one interviewer records the candidate responses.

The question areas to be addressed are directly linked to the Person Specification. Each question area should be covered
in turn. Interviewer 1 asks the questions while the candidate responses are recorded by interviewer 2.
Candidate responses should be written into the boxes provided.
Ensure that this report form has the candidate’s name recorded and the date and time of the interview.

The indicators on this Interview Scoring Form should be used in rating the candidate responses on a 4 point scale (see
below). Procedures and definitions for scoring the interview appear in the evaluation manual.

Introduce the interview by saying: “Thank you for coming, please take a seat. I am going to ask you to reflect on some
specific experiences at work. You will have some questions regarding these events, so don’t be surprised if I stop you at
various points. My colleague will be writing throughout”

The following scale should be used to determine a score for each question area.

0 No evidence • No evidence reported

• Little evidence of specified positive behavioural indicators


1 Poor
• Mostly negative indicators displayed, many of which decisively
• Limited number of specified positive behavioural indicators displayed
2 Areas for Concern
• Many negative indicators displayed, one or more decisively
• Satisfactory display of specified positive behavioural indicators
3 Satisfactory
• Some negative indicators displayed, but not clearly and decisively
• Strong display of specified positive behavioural indicators (and possibly others)
4 Good to Excellent
• Few negative indicators displayed, and these considered minor in status

NOTES (TO BE COMPLETED AFTER THE INTERVIEW)

Blank Interview Proforma


STRUCTURED INTERVIEW EVALUATION FORM 1 - POOR 3 - SATSIFACTORY
2 - AREA FOR CONCERN 4 - GOOD TO EXCELLENT
QUESTION SET 1 POSITIVE INDICATORS NEGATIVE INDICATORS
(COMMITMENT TO THE SPECIALTY)
• •
Q1 • •
• •
Probes: • •
• •

Notes to Justify Rating RATING: 1 2 3 4

QUESTION SET 2 POSITIVE INDICATORS NEGATIVE INDICATORS


(JUDGEMENT UNDER PRESSURE)
Q2
• •
• •
Probes: • •
• •
• •

Notes to Justify Rating RATING: 1 2 3 4

QUESTION SET 3 POSITIVE INDICATORS NEGATIVE INDICATORS


(SITUATION AWARENESS)
• •
• •
• •
Probes: • •
• •

Notes to Justify Rating RATING: 1 2 3 4

OVERALL EVALUATION RATING: 1 2 3 4

Blank Interview Proforma


STRUCTURED INTERVIEW EVALUATION FORM
SPECIALTY ________________ LOCATION __________________

Candidate Name: Date: / /


(MM DD YY)
Selector Name(s):
(Provide all selector names)

INSTRUCTIONS:
The interview contains ________ question areas. The interview should last no more than _____ minutes.
We recommend that one interviewer records the candidate responses.

The question areas to be addressed are directly linked to the Person Specification. Each question area should be covered
in turn. Interviewer 1 asks the questions while the candidate responses are recorded by interviewer 2.
Candidate responses should be written into the boxes provided.
Ensure that this report form has the candidate’s name recorded and the date and time of the interview.

The indicators on this Interview Scoring Form should be used in rating the candidate responses on a 4 point scale (see
below). Procedures and definitions for scoring the interview appear in the evaluation manual).

Introduce the interview by saying: “Thank you for coming, please take a seat. I am going to ask you to reflect on some
specific experiences at work. You will have some questions regarding these events, so don’t be surprised if I stop you at
various points. My colleague will be writing throughout”

The following scale should be used to determine a score for each question area.

0 No evidence • No evidence reported

• Little evidence of specified positive behavioural indicators


1 Poor
• Mostly negative indicators displayed, many of which decisively
• Limited number of specified positive behavioural indicators displayed
2 Areas for Concern
• Many negative indicators displayed, one or more decisively
• Satisfactory display of specified positive behavioural indicators
3 Satisfactory
• Some negative indicators displayed, but not clearly and decisively
• Strong display of specified positive behavioural indicators (and possibly others)
4 Good to Excellent
• Few negative indicators displayed, and these considered minor in status

NOTES (TO BE COMPLETED AFTER THE INTERVIEW)

Example Interview questions and recording form


STRUCTURED INTERVIEW EVALUATION FORM 1 - POOR 3 - SATSIFACTORY
2 - AREA FOR CONCERN 4 - GOOD TO EXCELLENT
QUESTION SET 1 POSITIVE INDICATORS NEGATIVE INDICATORS
(EMPATHY & SENSITIVITY) • responded to needs/concerns with • showed little apparent interest in patient
Give an example of how you won the trust of interest/understanding situation
a worried and sceptical patient. • acted in open, non-judgmental manner • was quick to judge, made assumptions
• attempted to generate safe/trusting • little evidence of understanding patient
Probes: atmosphere needs/concerns
Describe what you did and why? • made efforts to understand patient • unable to reassure patient effectively
What effect did it have on both you and the concerns • appeared over-sensitive as if personally
patient? • reassured patient with appropriate involved
words/actions • failed to generate safe/trusting atmosphere
Observation Notes to Justify Rating RATING: 1 2 3 4

QUESTION SET 2 POSITIVE INDICATORS NEGATIVE INDICATORS


(COMMUNICATION SKILLS) • adjusted style of questioning/response • unable to adapt language/behaviour as
Please describe a time when you have as appropriate needed
needed to explain a complicated procedure • able to express ideas clearly to others • imprecise response, rambled in
or term to a patient. • clarity in both verbal and written communicating, lacked focus
communication. • limited evidence of active listening
Probes: • flexible style to suit recipient(s) • confused/ambiguous verbal communication
How did you do this? • tailored communication, established • communication style was not tailored to suit
What was the outcome? relationship of respect with others recipient(s), is functional, narrow

Observation Notes to Justify Rating RATING: 1 2 3 4

QUESTION SET 3 POSITIVE INDICATORS NEGATIVE INDICATORS


(PROBLEM-SOLVING) • attempted to think “around” issue • made immediate assumptions about
Describe an example of when you have used • open to new ideas/possibilities problem
creative thinking to solve a problem at work. • generated functional, workable • dealt with issue narrowly or dogmatically
(related to a patient or to the organisation) solution • was unable to suggest “workable” outcome
• prioritised information/time effectively • was disorganised/unsystematic
Probes: • identified key points • focused on non-important/peripheral issues
What did you do?
• sifted peripheral information to detect • unable to identify key issues/root cause
What was the result? root cause
Observation Notes to Justify Rating RATING: 1 2 3 4

Example Interview questions and recording form


QUESTION SET 4 POSITIVE INDICATORS NEGATIVE INDICATORS
(ORGANISATION & PLANNING) • thinks ahead, prepares, plans • little evidence of thinking ahead, pre-
People organise their time in different ways. effectively planning, considering knock-on effects
• co-ordinates activities appropriately • unable to cope effectively with unexpected
What approaches & strategies do you use to
• builds contingencies to deal with occurrences
plan/protect your time for training?
shifting demands • failed to re-prioritise & build effective
Probes: • understands limitations, constraints & contingencies
How do you prioritise conflicting demands? works within them • fails to juggle competing demands
How do you justify your actions? • prioritises conflicting demands • disorganised, fails to manage time
What is most important? • juggles competing demands, effectively
delivered on time
Observation Notes to Justify Rating RATING: 1 2 3 4

QUESTION SET 5 POSITIVE INDICATORS NEGATIVE INDICATORS


(PROFESSIONAL INTEGRITY) • demonstrates respect for • lacked sufficient respect for others
Describe a time when you had to tell a patient(s)/colleague(s) • treated issues as problems rather than
patient that a mistake had been made, either • positive when dealing with problems challenges
by you or someone else. • able to admit mistake/learn from it • avoided taking responsibility for poor
• committed to equality of care for all decisions/ideas
Probes: • put patient(s) needs before own when • failed to prioritise patient(s) needs when
What did you do? appropriate appropriate
How did you justify your actions? • backs own judgment appropriately • tentative in justifying decisions/actions
How did it affect your work afterwards?
Observation Notes to Justify Rating RATING: 1 2 3 4

QUESTION SET 6 POSITIVE INDICATORS NEGATIVE INDICATORS


(COPING WITH PRESSURE) • remained calm/under control • shifted focus largely to immediate
Please describe a time when pressure at • aware of wider needs of situation worries/needs
work has caused you to feel upset or angry. • recognises own limitations & able to • defensive or uncompromising
compromise • tried unsuccessfully to deal with situation
Probes: • knows when to seek help alone
What was the cause of your feeling pressured? • uses strategies to deal with • used inappropriate strategies to deal with
What did you do? pressure/stress the issue(s)
What was the outcome? • responds quickly & decisively to • hesitant & unsure when faced with
unexpected circumstances unforeseen circumstances
Observation Notes to Justify Rating RATING: 1 2 3 4

Example Interview questions and recording form


QUESTION SET 7
(COMMITMENT TO THE SPECIALTY) POSITIVE INDICATORS NEGATIVE INDICATORS
What has influenced your decision to follow • •
a career in this specialty?
Probes:
Why do you think you are particularly suited to
this specialty?
What other medical careers have you
considered and why?
Observation Notes to Justify Rating RATING: 1 2 3 4

OVERALL EVALUATION RATING: 1 2 3 4

Example Interview questions and recording form


Upskilling in Selection Workshop:
Selector Skills, Trainer Skills,
Trainer Slides, FAQs

Upskilling in

Workshop
Selection
Selector Skills – Best Practice

1. Introduction to Observing, Recording, Classifying and Evaluating (ORCE)

2. Shortlisting skills

3. Advanced Skills for Interviewers


1. Observing, Recording, Classifying & Evaluating (ORCE)
Selectors must observe, record, classify and evaluate responses during the various
elements of the selection process in order to implement the selection methods accurately and
fairly. Research clearly shows that selectors require thorough and rigorous training for the
following reasons:

• To ensure that selectors can make decisions with confidence. The only way that this
can be enhanced is if each selector interprets the rating/scoring schedule in the
same way and therefore uses it in the same manner as other selectors. This is a pre-
requisite to achieve reliability of assessment.

• To ensure that selectors understand the various selection criteria on which the
selection methods (and ultimately the selection decisions) are based. This entails
becoming familiar with the scoring indicators that accompany each competency so
that components may be recognised during shortlisting/interview. A reference
framework for target selection criteria is provided to help guide selectors. These are
shown as positive and negative indicators against each selection criteria (see over
the page).

• To understand the aims of shortlisting/interview thoroughly, familiarise selectors with


all the necessary materials, and develop an understanding of what candidates are
required to do in each element of the selection process.

• To upskill selectors in the assessment procedure and in particular -

During the interview:

Observing the candidates’ responses.

During the interview:

Recording responses.

After shortlisting/interview:

Classifying recorded information.

Evaluating evidence.
2. Shortlisting Skills
In the case of shortlisting, selectors should:

1. Familiarise yourself with the targeted question area prior to scoring. The scoring
indicators will help selectors in doing this. Scoring indicators are anchored to each
point on a four point scale (1-4).

2. Read through the written response fully to gain an understanding of the response.

3. Indicate on the 1-4 descriptors your score for the response.

4. Note, there is a box for no evidence (NE)

General Rating Guidelines


You may find the following guidelines useful in ensuring that ratings are consistent and fair:

1. A scale of 1-4 is given to assess candidates’ performance for most parts of the
selection process. There is a tendency among raters to prefer the middle of the scale
(2-3). This is ‘Central Tendency’. You should not be afraid of using the extremes of
the scale as this will help to differentiate between candidates.

2. Beware of consistently over-rating or under-rating a candidate’s performance. This


means that you have tended to score at the top or lower end of the scale. This is
important to ensure fairness.
3. Advanced Skills for Interviewers

In this section we explore advanced skills for interviewers including levels of questioning, using
probe questions and issues regarding note taking. We are assuming that everyone who comes
forward for training has had experience in interviewing in the past and has been trained in Equal
Opportunities.

3.1 Levels of Questioning in Interviews


Questioning in a structured interview can be a like “peeling an onion”, but without the tears! This
section outlines the concepts of questioning at different levels as a strategy for collecting useful and
valid information during interviews. It is a proven strategy when you need to get information beyond
the ‘facts’ and surface level (or even coached…) responses.

Values/
3 Attitudes

2 Attractio
n
Meaning/
1 Facts/
Data

When you want to understand and judge a


candidate’s values and attitudes, this
involves 3 key levels of questioning (as
highlighted in the diagram). That’s why it’s
referred to as ‘peeling the layers of an onion
By probing further in an interview, the
assessment allows you to elicit useful and
valid information on which to make your
judgement. Here we give examples of
questions which may be asked at each leve

Level 1. Factual Information/Data


Questions relevant at this level are the least threatening to the
candidate and they are necessary to piece together the nature of
their background, such as their employment history, qualifications
and other relevant information. In this way, they provide the
interviewer with a more complete and ‘whole person’ understanding
of the candidate.
An example Level 1 questions are as follows;

Where did you complete you Foundation training?

What attachments have you completed?

How often did you meet with your educational supervisor?

Who helped you complete you work base assessments?

Level 2. Meaning, Attraction, Interest

A third level of question relates to the meaning and importance something holds for the
candidate. Questions at this level usually begin with:

What do you feel about……….?

What does ………… mean to you?

What do you get out of ……….?

At this level of questioning we are moving more towards the candidates ‘feeling’ about the
specialty of choice (for example) and what meaning it has to them. This is important as an
interviewer could make assumptions which could lead to invalid decisions being made. The
intention is to peel off the layers to start uncovering meaning, attraction and interest.

Examples:
Here are some examples of the style of questions that might be asked at level 2.

What do you feel about working in a specialty that requires you to do ………..?

What does it mean to you to pursue a career in ………….?

What does it mean to you when ……………………?

When a patient does ……. how do you feel?


As an interviewer you may already have a certain amount of information on the candidate
from their application form. However, you may need to check out some facts and you may
need to establish facts about equivalence of certain qualifications, such as F2 competencies.
The interviewer will need to ask several factual questions with follow-up probes to ascertain
the facts.

Level 3. Values/Attitudes
The next level of question is aimed at probing those items the candidate sees as important.
The answers provided by the candidate will tell the interviewer what he or she places value on
and why. They also provide information on the candidate’s attitudes. Therefore the answers
from these questions are important in coming to a more complete understanding of the
individual. This is particularly important because the interviewer will find this information very
useful in reaching a judgement about a candidate’s attitudes and values. Questions here are
usually of the type:

Why do you feel that ……………..?

Why is ……………… important to you?

For example, if the question area relates coping with pressure and resilience, answers to this
style of questioning might reveal (for example) that the candidate clearly places more value
on patient safety than on workload. It might also reveal that whilst the candidate is willing to
work long hours under pressure, they also recognise the need for a positive work-life balance.
In this way, the interviewer could probe further to gain evidence and find out what the
candidate actually does to maintain a positive work-life balance.

However, it may be at this stage that you still need further evidence in order to make your
judgement. It’s then necessary to probe further. Continuing to probe answers leads to more
information about the candidate’s attitudes and values.

3.2 Using Probe Questions

Many people think that structured interviews are about delivering exactly the same question to
all candidates and not following up on the candidate response. It is obviously important to
deliver the same opening question to all candidates applying for the same post in any given
area. Best practice structured interviews do have previously agreed question areas that directly
link to the relevant Person Specification. However, the main responsibility (and skill) of the
interviewer is to collect appropriate evidence relating to the question domain. Once the
evidence is collected, an appropriate professional judgement can be made.

The open question in an area should be consistently delivered between candidates. However,
since interviews are an encounter between at least two human beings they are, by their very
nature, idiosyncratic (no matter how much structure we impose). Best practice clearly shows
that what makes the difference in quality of interviews is the extent and quality of interviewer
probing. Probe questions are the follow-up to the opening questions. They help uncover the
evidence needed to make a fair and valid assessment. A skilled interviewer, through the use of
effective (not aggressive) probe questions a potentially ‘coached’ response can be exposed.
On the positive side, a skilled interviewer will provide an opportunity for the candidate to fully
describe their capabilities.
The following are a series of sample probing questions that will assist in exploring the level of
depth and understanding about the applicant’s experience and the related competencies;
and the actions taken specifically by the applicant, what lead to the actions, the results of the
actions taken and what was learned from the experience. These questions elicit specific
details about the who, when, where, what, why and how of the applicant's experience.

Who:

• Who else was involved?


• Who else helped you with this situation?

When:

• How long ago did this happen?


• When did all this take place?

Where:

• Where did this take place?

What:

• What was the [situation/ issue / problem]?


• What were the [results / outcomes]?
• What actions did you decide to take, and when?
• What was your role?
• What part did you play in making the decision to _______?
• What did you say, or do that [did / didn’t] work?
• What did they do?
• What was the presentation about?
• What was the message you were trying to get across?
• What were you trying to convince or persuade them of?
• What options were available?
• What steps did you take to ensure you met deadlines?
• What techniques did you use to keep yourself and your work organized?
• What did you do when your work started to get behind?
• What steps did you take to get them all done?
• What was the deadline?
• What were the consequences of missing the deadline?
• What actions did you take to ensure you met the deadline?
• What needed organizing?
• What improvements did you hope to see?
• What did you learn from this experience?
• What would you do differently if [you were asked to do it/ happened] again?

Why:

• Why was this situation a particular challenge?


• Why was it important to convince or persuade them?
• Why was this person difficult?
• Why did you take that approach?
• Why was it important to meet the deadline?

How:

• How did you develop this idea?


• How did you convince your supervisor to adopt it?
• How did it help the organization?
• How did you prepare for it?
• How did you handle [it / that person]?
• How did you size up the situation?
• How was the challenge met or the problem finally solved?
• How did you determine priorities in scheduling your time?
• How well did this meet the needs of the business?

Other:

• In the past [6 months / 1 year], how often have you been called upon to do this?
• In the past year, how often have experienced this type of difficulty?
• Did you receive any feedback on your actions? If yes, from whom? What was said?

3.3 Observation & Recording in Interviews


Extensive research shows over many years in the selection literature shows that there are
many sources of error and bias that can act to reduce the reliability, validity and fairness of
interviews as follows. Many of these sources of error are with regard to observation biases,
as described below.

3.3.1 Observation: Potential Sources of Error


The primary function of a selector during the interview is to observe and record what
candidates say and do in response to the question they have been asked. This is not an easy
task and certain pitfalls may be encountered which could influence the objectivity of the
selector. These pitfalls have been identified by rigorous research on selection and
assessment over several years and are outlined below.

• Discrimination: Most of us hold some form of stereotype about a person or group of


people. These can be powerful barriers to accurate observation. Recognising your
personal stereotypes can be useful in helping you overcome the prejudice they might
normally create.

• ‘Haloes and Horns’: First impressions we make of people influence how we see
them thereafter. Research consistently demonstrates that the Halo Effect occurs
when a person makes a positive impression and we then allow it to colour or
influence everything they say or do. The opposite can also apply, so that a negative
impression created in the first place could result in a poor outcome for the person. It
is therefore extremely important that when we are observing behaviour we do so with
a clear mind and we are aware of these potential sources of bias. We must aim to be
fair and unbiased when observing and recording behaviour.

• Prejudice and ‘Red Rags’: We all have our ‘petty’ dislikes and points of view and
usually we don’t appreciate having these views questioned or ridiculed. It is therefore
imperative when observing that these prejudices do not influence the way we observe
and record the candidate’s responses. To ensure this objectivity, make a point of
recording the behaviour as it actually happened and not how you interpret it.
Similarly, the candidate may use a word, or may have a habit, which to you is like a
‘red rag to a bull’. The temptation is to become irritated and to stop listening. This
must be avoided.

• Attribution Effects: Sometimes when we observe ‘inappropriate’ behaviour in a


person we attribute it to an inherent part of the person’s personality (whilst we see the
same behaviour in ourselves as being due to circumstances or the situation). Be
aware of this and remember that a person’s behaviour is a product of personality and
the environment and during the selection process the environment can be stretching
and pressured.
• Primacy and Recency Effects: Another human tendency in social encounters is to
remember the behaviour observed or the comment made either at the beginning or
the end of the encounter. The bit in the middle is often neglected. Ensure you have
based your assessment on the whole of the interview by recording your observations
as you go along.

• Concentration: It is extremely difficult to concentrate on something for more than 10


minutes without concentration starting to wane. Be aware of this issue as you might
unintentionally miss important information. The more you can concentrate the more
accurate your observations will be.

3.3.2 Recording
Each selector must make written notes of what the candidate’s response was during the
selection process. For panel interviews, we recommend at one interviewer delivers the
question area, whilst another records the information. It is difficult to do both reliably. An
observation/recording sheet should be available for panel members this purpose. You should
make factual recordings and should not include any inferences about what the response or
behaviour means. For example, you might write: “asked for question to be repeated twice”, or
“sighed when asked for further details of research project” (rather than “did not understand
question”, or “quickly became exasperated”). Factual information is transparent, easier to
classify, helps scoring and is also more useful when discussing the candidate’s response and
justifying the score. Avoiding inference at this stage has also been shown to improve the
quality of the assessment. When recording behaviour you should try and follow the following
guidelines:

• Accurate Recording: An accurate record should reflect all the relevant information
for the candidates you are assessing. Do not leave the writing up until after the
interview as information will become distorted. Always record during the interview
itself.

• Complete Recording: Your records should be as complete an account as is


possible in the time allowed. This will provide an overall picture of what the candidate
said and when and allow you to judge the comments of candidates in context. This
will not be possible if you have failed to make thorough notes or missed chunks of
dialogue. If you develop shorthand, make sure that it is easily understood and
remembered and that the same shorthand applies to all candidates.

• Language and Movement: Record actions as well as words where appropriate, but
be descriptive rather than judgemental e.g. “maintained eye contact with all panel
members”, rather than “good eye contact”

• Non-judgemental Records: All the records that you take of an interview should be
non-biased accounts of what actually happened. They should not reflect what
personal feelings you had of what was said by the candidate, e.g. if the candidate
points out that a decision would not make clinical sense, record “Candidate put
forward point that decision would not make clinical sense”. Do not make judgements
on the statement, e.g. “Candidate made a poor decision”. Again, your judgement on
whether this was poor or excellent would come later in the process, after the interview
has finished and you have gathered your evidence.

• Specific Records: Recorded statements should be specific and describe what took
place or was said, e.g. “Candidate stated that her particular strengths were honesty,
an empathic approach and logical thinking skills” is more specific and accurate than
“Candidate listed her key strengths”.
Selector Exercise: Examples of Interviewer Notes
The following are examples of observation notes recorded by selectors as evidence
of a candidate response in a structured interview. Some are effective and record
what the candidate actually said. Other notes are ineffective examples as they are
subjective judgements or are vague, general and do not describe the actual
response.

Place a tick in the appropriate column indicating if the statement is an effective or


ineffective example.

Interviewer Notes Effective Ineffective


Example Example

Was very creative in his solutions to


1. the problem posed _______ _______

Contacted a senior colleague (the


2. consultant) in the first instance to gain _______ _______
his views on the matter

Tested the understanding of the


3. relatives before decision was made
_______ _______

Was very insensitive to the needs of


4. her colleagues
_______ _______

Mature, considered approach to


5. questions
_______ _______

Suggested to colleagues that he make


6. a note of the situation regarding the _______ _______
difficult patient

Presented a superb rationale for his


7. decision
_______ _______

Inappropriately dressed, not


8. professional
_______ _______

Couldn’t understand a word he was


9. saying
_______ _______

Referred him to a cardiologist for


10. another opinion
_______ _______

A wide ranging discussion of the


11. issues
_______ _______

Demonstrated a good knowledge of


12. treatment options
_______ _______

Recorded the conversation with the


13. patient noting his refusal to take tablets _______ _______
to discuss with colleagues
Her standard of clinical competence
14. leaves much to be desired
_______ _______

Gave the patient an information leaflet,


15. made a follow up appointment
_______ _______

Spent 20 minutes talking with the


16. patient, listened to his concerns
_______ _______

17. Lack of insight to the issues presented _______ _______

Unable to cope with the pressures and


18. demands of the situation
_______ _______

Described the benefits and


19. consequences of not taking the _______ _______
treatment to the patient

20. Disrespectful towards her seniors _______ _______


Trainer Notes

Exercise 1 Purpose Learning Points Process/ Format Hints and Tips

Time Allowed Demonstrates Observation notes recorded in Stage 1. Participants work on their own Allow 5-10 minutes for
(15 minutes) how candidate interviews should be descriptive, initially to complete the exercise. delegates to complete
based on 25 responses factual and indicate the candidate’s on their own.
delegates in a should be response. Stage 2. Facilitators read the first item
workshop recorded. aloud (“Was very creative in his solutions Correct answers
Notes should not be judgemental, to the problem”). Ask delegates to raise delivered and
5 minutes for evaluative and contain information their right arm as to whether they believe it discussed in 10-15
participants to that isn’t relevant and inappropriate. is an effective or ineffective example of minutes.
complete the observation notes in an interview.
exercise. Recording information
inappropriately reduces validity and Then ask those who give the incorrect
10 minutes to in some circumstances could have answer to justify their response. Listen to
run through legal implications. the response and do not give the correct
answers in answer. Then, invite the participants who
plenary gave the correct answer to explain why.
After this discussion, give the whole group
the correct answer.
Objectives
Best Practice Selection ƒ Provide background & rationale to
Methods: national specialty selection
Shortlisting & Advanced ƒ Introduce key concepts to best practice
Interviewing training selection
ƒ Upskill selectors in short listing &
interview methods
National Selection into Specialty Training
ƒ Provide opportunities to practice
selector skills

Timetable
1.00 Context, key concepts & evidence
1.30 Selection in practice I: Short listing
− Selector skills Exercise 1-
1- Evaluating CB questions
− Practical considerations
2.30 Selection in practice II: Interviews
− Introduction to core skills
− Selector skills Exercise 2
Context for National Selection
3.00 Tea
3.15 Selection in practice II: Interviews cont’d
− Selector skills Exercise 3
− Practical considerations
3.45 Timetabling, logistics & evaluation
4.30 Review & Close

Context Framework for National Selection


COPMeD Steering Group for Selection & Recruitment into Specialty Training
In “Proposals for Reform of the SHO grade”
grade”
Sir Liam Donaldson argues that: Key Points
“Reform must take account of…
of… ƒ UK Strategy Group agreed framework for
weak selection & appointment procedures: national selection July 2006
these are not standardised & are frequently ƒ MTAS will be used by all 4 UK countries
not informed by core competencies’
competencies’. ƒ Nationally agreed person specs for each
specialty & for each entry level (ST1, ST2, ST3)
ƒ Nationally agreed application form, reference
UK Strategy Group agreement to national selection form, minimum standards for interview
PMETB Panel review 25th August 2006 ƒ Selector training for short-listing & interview
essential

1
Why Change? What’s new?

ƒ Historically we have done an effective job. ƒ This is the first step. For 2007, there will be
ƒ Little formal evaluation uniform targeting of agreed selection
ƒ Our aim is to improve efficiency & effectiveness criteria (via the Person Specifications).
Specifications).
for all stakeholders (trainees, admin, interviewers, HR) ƒ Minimum requirement of a 30 minute
− cost, fairness, reliability & validity (standards) structured interview
ƒ ST1 is new: We must deliver a process that is ‘fit ƒ Identifying best practice (no re-
for purpose’
re-invention)
ƒ Resource pack & opportunities to further
upskill selectors

How do trainees apply for


What levels will be selected into
Specialist Training?
Specialist Training in 2007?
ƒ Advance information in Medical Press
ƒ Single alert in Medical Press & electronic
Specialty-specific information
− Levels recruited to depend on comparison ƒ Single electronic application per recruitment
between competencies detailed in the new & round
old curricula ƒ Long-
Long-listed nationally against person
Recruitment will occur at ST1- ST4 in some specification
Specialties ƒ Apply for up to 2 preferred Specialty groups in 2
preferred Units of Application*
− Distinct application forms
− GP 1 UoA
− Different recruitment rounds − May no longer express ‘no geographical preference’
*Unit of application- a Deanery or
cluster of Deaneries working together

Recruitment & Selection Offers to Applicants


National Advert
1
Round 1 E- Applications co-ordinated & sent to chosen UOAs* Applicants will receive a
Run-Through Shortlisting Shortlisting Shortlisting Shortlisting single e-letter listing their
Grade Posts:
2 Specialties, Assessment Assessment Assessment Assessment offers and will have a limited
2 UOA & Ranking Ranking Ranking Ranking period in which to reply
FTSTAs Selection Selection Selection Selection
o Acceptance is a binding
Central Co-ordination of Offers of Posts: commitment to the UOA
0 1 2 3 4 and the employer
Unsuccessful o Applicants breaking the
Re-enter Applications co-ordinated & agreement are disqualified
Round 2 sent to chosen UOAs* from any further
Shortlisting Assessment Training participation in the
Vacant Ranking Programme recruitment and selection
Run-Through Appointed to
Grade Posts
Selection
Training
10 Vacancies 45 process except in very
& FTSTAs Central Co-ordination of Offer of Post programme Appointable exceptional and unforeseen
Trainees circumstances
Unsuccessful Successful Ranked 1- 45

2
Ranking of Applicants Allocation of Posts
1 1
Round A of Offers
Electronic generation of offers
There will be a to trainees ranked 1-10
continuous process of
issuing more offers to
appointable candidates
in Round 1 of Selection
Training Training
Programme Programme
10 Vacancies 45 10 Vacancies 45
45 Trainees 45 Trainees
Appointable Appointable
Ranked 1- 45 Ranked 1- 45

Allocation of Posts Allocation of Posts


1 1
Round A of Offers Round B of Offers
Electronic generation of offers Electronic generation of offers
to trainees ranked 1-10 to trainees ranked 11-15

5 Posts accepted by the


trainees → 5 Vacancies filled

Training Training
Programme Programme
10 Vacancies 45 10 Vacancies 45
45 Trainees 45 Trainees
Appointable Appointable
Ranked 1- 45 Ranked 1- 45

Allocation of Posts Allocation of Posts


1 1
Round B of Offers Round C of Offers
Electronic generation of offers Electronic generation of offers
to trainees ranked 11-15 to trainees ranked 16-17

3 Posts accepted by the


trainees → 8 Vacancies filled

Training Training
Programme Programme
10 Vacancies 45 10 Vacancies 45
45 Trainees 45 Trainees
Appointable Appointable
Ranked 1- 45 Ranked 1- 45

3
Allocation of Posts Allocation of Posts
1 1
Round C of Offers Round C of Offers
Electronic generation of offers Electronic generation of offers
to trainees ranked 16-17 to trainees ranked 16-17

1 Post accepted by the 1 Post accepted by the


trainees → 9 Vacancies filled trainees → 9 Vacancies filled

1 Post entered into Round 2

Training Training
Programme Programme
10 Vacancies 45 10 Vacancies 45
45 Trainees 45 Trainees
Appointable Appointable
Ranked 1- 45 Ranked 1- 45

Aug Identify No of Programmes/ Posts


06 Sept Agree person specification for ST1, 2 & 3
Jan Finalise complete selection/ interview schedule
6 Jan Alert in Medical Press- Application start
22 Jan- 2 Feb Submit applications
2 Feb Application form to selection panels
By 23 Feb Shortlist applicants
Round 1 24 Feb One email confirming interviews or not
Selection Process Overview- 2007
27 Feb Trainees confirm attendance or not

The Timeframe 28 Feb- 13 Apr Complete selection activities:


Identify appointable applicants
19-26 Apr Round 1 Offers made & accepted
27 Apr Trainees with no offers asked to reapply
Indicative 28 Apr Advertise unfilled posts
Round 2 By 27 May Shortlist applicants
4- 22 June Complete Selection
Timeframe 23 June One email of offers/ rejections
August 2006-7 26 June Acceptance/rejection of offer

Implications of Timeframe
Selectors will need to have the following availability:
Provisional Dates
ƒ Shortlisting
− 2- 23 Feb
− 14- 27 May Key Concepts & Evidence:
ƒ Selection Best Practice Selection
− 28 Feb- 13 April
− 4- 22 June

ƒ Selection Panel should be available to complete


short-listing within the timeframe as well as to
attend the selection process for candidates

4
Key concepts Research on Selection Systems
ƒ Selection tools must be standardised, fair,
ƒ Research & implementation projects to
defensible, reliable, valid, cost-
cost-effective &
develop selection tools that are standardised,
standardised,
feasible.
fair, defensible, reliable, valid, cost-
cost-effective &
ƒ To achieve this, the skills & attributes tested at
feasible.
selection must be based on the appropriate
Person Specification 1. Develop & validate selection criteria for various
specialties
ƒ The Person Spec details the selection criteria & 2. Design & validate selection tools that assess core
methods.
methods. criteria
ƒ Candidate reactions are increasingly important
¾ Growing literature in medicine
ƒ Selector skills central to reliability & validity Patterson et al. (BMJ
(BMJ,, 2005; BMJ 2001; BJGP,
BJGP, 2000; Arch. Dis. Child.
2006; TOG in press, 2006, RCS)

Research on selection methods


¾ Structured interviews can be reliable & valid selection tools.
¾ Reliability of assessment & of assessors using
standardised rating scales & documentation. Rigorous
training is critical.

Research shows Selection Centres (designed


appropriately) are best predictor of future job performance Principles of Reliability & Validity
Reliability & validity gains - combining different selection in Selection
tools, standardised scoring systems to measure key
competencies eg.eg. 3 x 10 minute stations (= 30 minutes, with 2
assessors per panel)

Research consistently demonstrates


demonstrates that application form
data & references have
have limited validity
Validity can be significantly improved by providing
standardised competency-
competency-based rating scales

Defining reliability & validity Evaluating validity in selection


1. Faith validity 4. Criterion validity
“The person who Predictive validity
sold me the “The tool/test predicts who
Reliability: “The instrument measures consistently selection tool/ test will be competent doctors”
under varying conditions” was very plausible” 5. Incremental validity
2. Face validity “How much additional value
does using another test/tool
Validity: “The instrument measures what it “The test looks provide?”
claims to measure” plausible”
6. Construct validity
3. Content validity
“The test looks “The tool measures
plausible to experts” something meaningful”
meaningful”

5
Research on selection in medicine Research on selection in medicine
Practically, for some specialties we can’t interview all applicants.
We need to shortlist. Options? Relevant research references for those interested:
– Psychometric testing, MCQs? ƒ Patterson, F., Ferguson, E., Norfolk, T., Lane, P. (2005).
ƒ Academic success predicts very little of the variance in A new selection system to recruit general practice
subsequent clinical performance (e.g. Ferguson et al, BMJ; 2002) registrars: preliminary findings from a validation study.
BMJ, 330, 711-
711-714
ƒ For ST2, ST3 exams/research more relevant to indicate
competence. The is not possible for differentiating at ST1 ƒ Patterson, F., Lane, P., Ferguson, E., Norfolk, T. (2001).
A competency based selection system for GP trainees.
ƒ We recommend a combination of questions to add BMJ, Career Focus, 323:2
predictive power – this has been piloted ƒ Patterson, F., Ferguson, E., Lane, P.W., Farrell, K.,
ƒ MTAS can search for plagiarism Martlew, J., Wells, A.A. (2000). A Competency model
ƒ For 2007, this methodology is fit for purpose. for general practice: implications for selection, training
and development. Br J Gen Pract, 50, 188-188-193
ƒ For the future, evaluation is necessary, on a national basis. ƒ Randall, R., Davies, H., Patterson, F. & Farrell, K. (in
ƒ Future innovations could be a Machine marked test for press) Selecting doctors for postgraduate training in
shortlisting paediatrics using a competency based assessment
centre Arch. Dis. Child. 91: 444 - 448.

Shortlisting

• Documentation & scoring mechanisms for


shortlisting will be on a national basis
Selection in Practice 1: • Nationally agreed person specifications,
application forms & scoring frames created for
Shortlisting each specialty at ST1, 2 and 3.
• This section is important for selectors to
understand the paperwork & their role in the
short listing process

Person Specifications
Purpose
− Usability (especially candidate perspective)
− Standardisation within specialties
− Operationalisation of assessment (e.g. AF,
interview questions, scoring framework)
National Documentation-
Two sections:
The Person Specification −Eligibility Criteria:
Minimum entry requirements, all essential
−Selection Criteria:
Used to rank candidates
Essential/ desirable criteria
Specialty-specific criteria (some common domains)
Commitment to the specialty

6
Person Specification eg ST1 Anaesthesia Person Specification ST2 and ST3 Anaesthesia
ƒ Differ from ST1 in the entry criteria
Essential Eligibility: Career Progression:

ST1 • GMC Registration • No unexplained career gaps


Essential Eligibility: • FP Competencies • Less than 12 months experience
• GMC Registration Personal Skills: (at SHO level) in this Specialty
• FP Competencies Vigilance & Situational Awareness, (not including FP modules) by
Coping with Pressure, August 2007
Managing others & team
involvement, ST2 • GMC Registration • No unexplained career gaps
Problem-solving & decision making, • FP & ST1 Competencies • At least 12 months and less
Empathy & Sensitivity, than 3 years experience (at SHO
Communication Skills, level) in this Specialty (not incl. FP
Career Progression: Organisation & Planning modules) by Aug 2007
• No unexplained career gaps
• Less than 12 months experience ST3 • GMC Registration • No unexplained career gaps
(at SHO level) in this Specialty • FP & ST1&2 Competencies • At least 2 years experience (at
(not including FP modules) by • Primary FRCA examination SHO level) in this Specialty (not
August 2007 passed including FP modules) by August
2007

Application Forms
Questions directly linked to person specification

ƒ Eligibilty
− Personal details, GMC Registration, Professional
National Documentation- Qualifications, Eligibility for entry into Specialist
training, Previous posts, Referees
The Application Form ƒ Selection Criteria
− Clinical, Academic & Research skills, Probity,
Evidence of commitment to the Specialty, Personal
skills relevant to the Specialty

Application Process

ƒ Specialty specific forms


ƒ Long listing via MTAS (electronic portal)
ƒ Shortlisting in UoA by trained shortlisters
(minimum of 2)
Shortlisting Process
ƒ Lay involvement, determined by the UoA
(as per Orange Guide)
ƒ Scoring frames provided

7
Short-listing Process
ƒ Panel members carry out shortlisting (as per Orange
Guide)
− Panel members do not have to mark ALL of a single
application form.
− Deaneries may choose to split questions between panel
members.
ƒ Benefits in terms of speed & calibration
Shortlisting Scoring & Indicators
ƒ Different methods/options:
− All in one room – marking/ decisions
− Sent out to short-listers & returned
− electronic

ƒ GMC reporting for falsification on Application Form

What paperwork do I need Example Questions: Application Form


Generic
for shortlisting? Clinical, academic & research skills
Materials ƒ What do you think is the relevance of medical research to a
doctor? If you have been involved in research, use
1. Framework reference for targeted criteria examples from your own research to illustrate this. (max.
ƒ This helps to ensure that all selectors have a 150 words).
shared understanding of the criteria being
assessed ƒ Describe your experience of clinical audit. Indicate clearly
your level of involvement and what you learned about the
2. Shortlisting scoring indicators process.
ƒ This is where scores are recorded for each
question against specific descriptors/ ‘word
pictures’ Specialty-Specific
3. Candidates response Clinical, academic & research skills
ƒ This may be paper based or on-line, depending ƒ [Vigilance & Situational Awareness] Describe a time when
on your UoA you identified a clinical situation that was changing rapidly.
How did you identify this and what did you do? (max. 150
words)

Example Questions: Application Form Framework for Target Selection Criteria


Clinical, Academic & Research Skills
Please provide a brief statement describing the reasons
why you have chosen to train for a career in this specialty.
Please include any information about yourself that you feel
is important or relevant in this regard. (max. 75 words)

What steps have you taken to develop your understanding of


this specialty? (max. 150 words)

8
Indicators for Target Selection Criteria at Shortlisting-
Framework for Target Selection Criteria
Commitment to the Specialty
Personal Skills

Framework for Target Selection Criteria Shortlisting Scoring Indicators


Probity
COMMITMENT TO THE SPECIALTY

Personal Statement: Reasons for choice of specialty and relevant information.

NE 1 2 3 4

Unclear reasons for


Clear reasons for Comprehensive
choice of Rationale for choice of
choice of rationale for choice
specialty, lacked specialty
specialty, of specialty, clear
understanding of somewhat
evidence of understanding of
No evidence specialty/own unclear/unfocused,
some insight specialty/own
motivation, no or limited insight into
into specialty motivations &
irrelevant specialty and/or
and/or own relevant additional
additional own motivation
motivations information
information

Candidate Responses I’m shortlisting, what do I do?


ƒ Read the question you are scoring
ƒ Familiarise yourself with the framework
indicators for the targeted criteria to
ensure understanding
ƒ Read the candidates response
ƒ Using the scoring indicators, circle one
score based on the evidence presented

Training essential to ensure understanding


of criteria & callibration of assessors

9
Evaluation & rating process

„ Standardised system of documentation &


scoring framework allows for more valid & Selector Skills Exercise 1
fair assessments
„ Reduces potential for error & bias

„ Increases transparency
Evaluating & scoring the application form

An example……..

Practical considerations in
Shortlisting
ƒ Work needed on calibration & reliability
ƒ Each question marked by 2 different short
listers
ƒ Scores aggregated to get total & range
Selection in Practice 2:
ƒ Scores returned to MTAS including UoA cut off Interviews
ƒ System allocates interview according to UoA
interview capacity & candidate scores
− eg if 100 interview slots, the top 100 candidates on
shortlisting offered interview. As these slots are
accepted or declined, interview is offered to next on
list until you reach cut off, or none left.

Interviews in Selection

ƒ 2 main types of interview in selection


− Behavioural (past behaviour = best predictor of future behaviour)
− Situational (scenarios)
Selector Skills - The Interview − Both useful, dependent on interviewer skills

Skills: Peeling, probing, familiarisation, observe


record, classify, evaluate

10
Management Scenario:
Anaesthetic SpR refuses to anaesthetise a
patient for appendicectomy

Example Situational Interview EVALUATION 1 2 3 4

Question Positive Indicators


•consider the reasons (eg
Negative Indicators
•insist on operating
overnight, co-morbidity) •threaten a critical incident
•review the decision to operate •consults senior anaesthetist
and assess priority without discussing with SpR
•discuss it with the anaesthetist •tell the patient anaesthetist
•involve the surgical senior cover “refusing” to anaesthetise
•deal with the patient •just accepts it
appropriately

Clinical Scenario:
12 yr old boy – who has come off his BMX –
he has a broken arm (supracondylar fracture)
and is complaining of breathlessness.

EVALUATION
Positive Indicators
1 2 3 4
Fairness in Interviews
Negative Indicators
• Structured approach to the question • Failing to take a history of the nature
• Awareness of possible vascular of the accident
injury to arm • Failure to take adequate history
• Awareness of possible including possibility of asthma
pneumothorax • Failure to examine patient including
• Of risk of associate intra-abdominal trachea
injury – ruptured spleen • Sending patient for CXR without
• Awareness of risk of head injury excluding tension pneumothorax
• Failing to assess circulation in arm

Forbidden Questions Forbidden Questions


Sex Discrimination Racial Discrimination
ƒ We like our employees to look smart. Do you
ƒ Do you intend to have a family in the near have to wear a turban and a beard?
future?
ƒ Why do you call yourself British with an
ƒ I see you are engaged to be married. Does accent like that?
that mean you will be moving away from
the area? ƒ Would you, like so many other Indians I have
employed in the past, keep wanting long
ƒ Do you think you would be able to gain the holidays to go back to India?
respect of males who report to you?
ƒ Being Irish, are you going to lose your
temper if annoyed with fellow workers?

11
Familiarise What’s the difference between a
skilled interviewer & novice?
ƒ Interviewers need to know: styles…..open questions, one at a time
ƒ Questioning styles…..open
− the questions they are asking ƒ Managing silences….
ƒ Making the question feel authentic! Follow-
Follow-up
− how they relate to the person specification
essential to uncover what “you” (the candidate)
− what probes are appropriate actually did
ƒ Be familiar with the relevant indicators for ƒ Peel the layers 1.facts 2. meaning 3. attitude
the question ƒ Probe for evidence who, what, why, when…etc 1
2

ƒ Be familiar with the rating scale to be used ƒ Verify what the trainee actually did, or would do 3

in scenarios

Exemplar Interview Report Form

Example Interview Report Form

Observation: Sources of error & bias


ƒ Haloes & Horns effects
ƒ Prejudice (race, gender, etc.)
Selector Skills Exercise 2 ƒ
ƒ
Stereotyping & ‘Red Rags’
Concentration
− ‘Primacy & recency’ effects
ƒ Observation bias
Questioning Styles − ‘Similar-to-me effect’/ Personal Liking
ƒ Temporal extension
− Decision made within 3 seconds
ƒ First impressions last
− “Can I suspend judgement?”

12
‘Real World’ Selector Stereotypes
A TRAINING POST
ƒ white stilettos ƒ designer stubble
ƒ limp handshakes ƒ close-
close-set eyes
ƒ white socks ƒ double-
double-barrel
ƒ male earrings surnames
ƒ tattoos ƒ Volvo drivers
ƒ Millwall fans ƒ peroxide hair
ƒ male long hair ƒ heavy make-
make-up
ƒ sweaty palms ƒ belly showing!

Recording
ƒ Legible!
ƒ Accurate
ƒ Complete Selector skills Exercise 3
− If in doubt, include it
ƒ Non-judgmental
− Record appropriately, do not judge or evaluate
while recording Effective recording & evaluating
ƒ Specific – need to be able to provide
explanation for judgment later on

Evaluation & Rating Process Interviews/Selection Centres

Methods
4 = good to excellent
ƒ Minimum of 30 minute structured interview
3 = satisfactory
ƒ Questions/exercises must be;
2 = areas of concern
− Based on the relevant Person Spec.
1 = poor [0] = No evidence − Consistent across interviewers/ interviewees
− Previously agreed scoring framework that links to the
Person Spec

Standardised system of markers allows for ƒ Example questions & calibration exercises
provided
more valid & fair assessments &
increases transparency

13
Interviews/Selection Centres Candidate reactions
Logistics
ƒ Chairperson/ facilitator appointed Factors affecting reactions use
ƒ Venue, admin support, materials planned in organisational justice theories:
advance
− Distributive justice: Perceived fairness
Content regarding equity (outcome consistent with
ƒ Minimum of 30 minutes contact expectations) & equality (opportunities).
− Panel interview; Split-panel (two 15 minutes); Three
10 minute exercises − Procedural justice: Information, feedback,
ƒ Multiple selection tools encouraged job-relatedness, recruiter effectiveness.
ƒ Selection centres (focus also on demonstrating,
not reporting)
Selectors Applicants prefer multiple
ƒ All trained, lay involvement essential opportunities to demonstrate skills

Reference Reports
ƒ Evidence suggests that choice of reference provider &
structure of the form have the most impact on the quality
of information provided
National Documentation- ƒ Designed appropriately & in relation to the person
specification, they can be improved
The Structured Reference ƒ A standardised, structured report form has been
designed and nationally agreed
Tab 5 ƒ Minimum of 2 referees, covering the last two years of
employment. Candidate guidance on selecting these.
ƒ Guidelines on how & where the reference report is used
in the selection process is provided (at interview stage
only and via the Chair of the panel)

Structured Reference (Draft)

National Documentation-
Candidate evaluation
Tab 10

14
Feedback
ƒ Selection decision uploaded to MTAS; offers ONLY
by MTAS at the end of the selection process, or the
national system will not work
ƒ Feedback offered only after the whole of the
selection process.
Implementation &
ƒ This is essential so there is fairness to all Practical Considerations
candidates
ƒ We must ensure fairness to all candidates
ƒ UoA s will be asked to audit to ensure national
adherence

Review
Practical considerations ƒ National specialty selection in 2007 includes national
person specifications, application forms & references
ƒ Key concepts in best practice selection inform new
practice
ƒ Timetable ƒ Short-
Short-listing & interview questions MUST be directly
linked to person specs
ƒ Logistics ƒ Selector skills training vital to contribute to
ƒ Candidate experience standardised, fair, defensible, reliable, valid, cost-
cost-
effective & feasible process.
ƒ Selection system not static – continuous
improvement

Final questions?

Thank you……

15
Evaluation & Audit Documentation

ocumentation
Evaluation &
Audit
Best Practice Selection

Evaluation

1. Principles of best practice evaluation in selection

2. Auditing process

1
Best Practice Selection

1 Principles of best practice evaluation in selection


Evaluation is critical in the development of the future selection process to identify areas for
improvement and ensure adherence to PMETB principles and standards for entry to Specialist
Training. Figure 1 provides an outline of the main elements involved in designing and implementing
a personnel selection procedure and how evaluation forms an integral part of this.

Fig. 1 The personnel selection, design and validation process

Job Analysis
To derive a;
• Person specification
• Job description
• Competency model

Identify Selection Criteria

Choose Selection Methods


Judgements based on reliability, validity, legality,
fairness, cost and candidate reactions

Attract Implement Selection Methods and Make Selection


Candidates Decisions
Self-selection
to the job role

Evaluation
• Evaluate candidate reactions
• Empirical validation studies
• Utility assessment

Before considering how well selection methods work it is necessary to be clear about what it means
for a selection method to work. A list of various features of personnel selection methods that are
important is presented in Table 1. A comprehensive evaluation of any selection method would require
a thorough examination of the method in relation to each of the features given below.

Table 1 Major evaluative standards for personnel selection procedures

1. Discrimination

The measurement procedures involved should provide for clear discrimination


between candidates.

2. Validity and Reliability

The technical qualities of the measurement procedures must be adequate.


Several different (e.g. content, predictive or concurrent) validation processes are
available for assessing validity.

2
Best Practice Selection

For example, if you demonstrate that your assessment tool (for example, a
structured interview) is measuring competencies that are critical to successfully
performing a given job, you have content validity evidence.
If you demonstrate that your assessment tool (for example, a written test) predicts
actual performance on the job, you have evidence of the predictive validity of the
assessment tool.

3. Legality & Fairness

The measures must not discriminate unfairly against members of any specific
subgroup of the population (e.g. ethnic minorities).

4. Administrative Convenience/Practicality

The procedures should be acceptable within the organisation and capable of being
implemented effectively within the organisation’s administrative structure. Those
administering the procedures may need appropriate training.
5.
Cost and Development Time

Given the selection decisions (e.g. number of jobs, number of candidates, type of
jobs) involved, the costs involved and the time taken to develop adequate
procedures need to be balanced with the potential benefits. This is essentially a
question of utility.

6. Candidate Reactions

Candidate reactions have important consequences for an organisation. In an


extreme case, an applicant who has a negative reaction to a selection procedure
could make a legal challenge on the basis of unfair discrimination.

Research has tended to explain the different factors that affect applicant reactions
using theories of organisational justice:

• Distributive justice focuses on perceived fairness regarding equity (where the


selection outcome is consistent with the applicant’s expectation) and equality
(the extent to which applicants have the same opportunities in the selection
process).
• Procedural justice refers to the formal characteristics of the selection process
such as information and feedback offered, job-relatedness of the procedures
and methods, and recruiter effectiveness.
Four main factors seem to account for positive or negative applicants reactions
where selection methods are;
(1) based on a thorough job analysis and appear more job relevant
(2) less personally intrusive,
(3) do not contravene procedural of distributive justice expectations, and
(4) allow applicants to meet in person with the recruiters.
Candidates also seem to prefer multiple opportunities to demonstrate their skills and
that the selection system is administered consistently for all candidates.

3
Best Practice Selection

10.2. Auditing process

The information collected at selection is invaluable at helping design training and developments plans
and activities. In accordance with best practice guidelines in terms of evaluation of the selection
process, the following has been agreed with the PMETB (August 2006):
.
(1) Selectors, applicants and administrators will be asked to complete feedback proforma and
evaluation forms regarding the selection process (including candidate reactions regarding
perceptions of fairness and procedural justice). Results will be reported to relevant
stakeholder groups for modification of the existing process if appropriate

(2) Deaneries/UoAs will be asked to complete a self-assessment of selection methods as part of


the auditing of this process, which could inform the statutory quality assurance process

Examples of feedback proformas for this purpose are presented in this section.

4
Candidate’s Evaluation of the Selection Process
This form asks about your experience of going through the specialty training selection process. The answers
you give will not form part of your selection, they are only to help us improve the selection system in the
future. Please answer each question by ticking the appropriate box according to the scale provided, we
would also be grateful to receive your comments in the spaces provided.

Age: _____ Gender: Male Female


Ethnic origin: Area of Qualification:
White Mixed UK Africa
Asian Chinese Europe Other
Black Other Asia

1= strongly disagree, 2= disagree, 3=neither agree nor disagree, 4=agree, 5=strongly agree

Application Form Questions 1 2 3 4 5


1. The content of the application form questions was clearly related to the job.......
2. A person who scored well on the application form questions will be a good doctor
in this specialty
3. The content of the application form questions did not appear to be unfair...........
4. The content of the application form questions seemed appropriate….................
5. How long did it take you to complete the application form questions?

Hours: ___________ Minutes: ______________

Comments on application form questions

1= strongly disagree, 2= disagree, 3=neither agree nor disagree, 4=agree, 5=strongly agree
The Interview 1 2 3 4 5
1. The content of the interview was clearly related to the job………………
2. A person who scored well on the interview would be a good doctor in this
specialty……….
3. The interview gives applicants the opportunity to show what they can really
do………………………………………………………………………………

The whole Selection Process


1. I was treated politely during the selection process………………………………
2. The staff involved in the interviews treated candidates with respect during
the interview process…………………………………………….
3. I was satisfied with my treatment at the interview…………………….
4. The content of the interview did not appear to be unfair………………..
5. The content of the interview seemed appropriate……………………….
6. Overall I was given a good opportunity to show my skills and abilities today...
7. The interview was conducted with a high degree of professionalism….
8. The interview centre helped me to learn something about myself……………..

We are interested in your experience today in comparison to other selection procedures you have been
through during your medical career. Please circle the appropriate answer for each question.

1. In comparison to other selection processes I have been through, the level of professionalism shown at
interview was…

A lot less Less Slightly less The same Slightly more More A lot more

2. In comparison to other selection processes I have been through, the relevance of the content of the
interview questions to work as a doctor was…

A lot less Less Slightly less The same Slightly more More A lot more

3. In comparison to other selection processes I have been through, the level of opportunity I had to show my
skills and abilities today was…

A lot less Less Slightly less The same Slightly more More A lot more

Please comment on your experience of going through the selection process. We would be particularly
interested in how your experience today compares to other selection procedures you have been
through, and whether it has helped you to learn anything about your own skills and development
needs.

Thank you for completing this evaluation form


Selectors Evaluation of the Selection Process
This selection round sees a large change to the selection process for specialty training. We are very
keen to receive feedback from you about how this has worked. This page asks for your comment on
the short listing and interviewing process.

Interviews: we would like to know your views about:


a) The content of the interview questions – did they give candidates a proper chance to demonstrate
competencies? Were there any issues around fairness?
b) Was there anything about the questions that made them easy or difficult for you to assess?
c) Was there anything that made the practical set up and administration of the interviews particularly
easy or difficult?

Application form questions – Short Listing:

Interviews:

General Comments on the selection process:

Thank you for completing this evaluation form


Audit of Interview/ Selection Centre Process: Chair to complete
Deaneries/UoAs will be asked to complete a self-assessment of selection methods as part of the auditing of
the selection process. This will inform the statutory quality assurance process and will be monitored by
PMETB as part of the associated statutory quality assurance process.

Unit of Application Appointment Level


Specialty Name of Chair

Selection Process: Please indicate with a TICK either Yes or No against the statements
YES NO
1. Membership of the interview panels/ selectors at selection centres complies
with guidelines set out in the Orange Guide to appointments to specialty
training (or future equivalent documentation)
2. A chair has been appointed locally for the selection process, whose
responsibility it is to quality assure delivery of selection methods to ensure
compliance with PMETB standards
3. All persons involved in interview panels/ selection centres have been trained
in the principles of best practice selection.

4. Selection panels include a lay person to participate in the selection process.


The lay person/people involved must also be trained in best practice
selection and assessment
5. Interview questions/selection centre exercises are directly and clearly related
to criteria contained in the person specification

6. Interview questions/selection centre exercises were scored according to the


national standardised scoring framework

7. Relevant feedback will be provided to applicants if requested

8. Selectors and candidates have completed evaluation forms regarding the


selection process

Comments on Selection Process

Signature of Chair
Signature of HR
Date