How effective are selection methods in medical education?: A systematic review

Fiona Patterson; Alec Knight; Jon Dowell; Sandra Nicholson; Fran Cousans; Jennifer Cleland

doi:10.1111/medu.12817

How effective are selection methods in medical education? A systematic review

Fiona Patterson^* (Corresponding Author)
, Alec Knight
, Jon Dowell
, Sandra Nicholson
, Fran Cousans
, Jennifer Cleland

^*Corresponding author for this work

City, University of London
Work Psychology Group, Derby
University of Dundee
Queen Mary University of London

Research output: Contribution to journal › Article › peer-review

377 Citations (Scopus)

29 Downloads (Pure)

Abstract

CONTEXT: Selection methods used by medical schools should reliably identify whether candidates are likely to be successful in medical training and ultimately become competent clinicians. However, there is little consensus regarding methods that reliably evaluate non-academic attributes, and longitudinal studies examining predictors of success after qualification are insufficient. This systematic review synthesises the extant research evidence on the relative strengths of various selection methods. We offer a research agenda and identify key considerations to inform policy and practice in the next 50 years.

METHODS: A formalised literature search was conducted for studies published between 1997 and 2015. A total of 194 articles met the inclusion criteria and were appraised in relation to: (i) selection method used; (ii) research question(s) addressed, and (iii) type of study design.

RESULTS: Eight selection methods were identified: (i) aptitude tests; (ii) academic records; (iii) personal statements; (iv) references; (v) situational judgement tests (SJTs); (vi) personality and emotional intelligence assessments; (vii) interviews and multiple mini-interviews (MMIs), and (viii) selection centres (SCs). The evidence relating to each method was reviewed against four evaluation criteria: effectiveness (reliability and validity); procedural issues; acceptability, and cost-effectiveness.

CONCLUSIONS: Evidence shows clearly that academic records, MMIs, aptitude tests, SJTs and SCs are more effective selection methods and are generally fairer than traditional interviews, references and personal statements. However, achievement in different selection methods may differentially predict performance at the various stages of medical education and clinical practice. Research into selection has been over-reliant on cross-sectional study designs and has tended to focus on reliability estimates rather than validity as an indicator of quality. A comprehensive framework of outcome criteria should be developed to allow researchers to interpret empirical evidence and compare selection methods fairly. This review highlights gaps in evidence for the combination of selection tools that is most effective and the weighting to be given to each tool.

Original language	English
Pages (from-to)	36-60
Number of pages	25
Journal	Medical Education
Volume	50
Issue number	1
Early online date	23 Dec 2015
DOIs	https://doi.org/10.1111/medu.12817
Publication status	Published - Jan 2016

Bibliographical note

Acknowledgements

the authors thank the UK General Medical Council (GMC) for commissioning an initial rapid review of the literature on selection and widening access to medicine in 2013. Further thanks are due to the UK Medical Schools Council (MSC) for commissioning a significantly updated review of selection methods in 2014, funding for which was provided by Health Education England (HEE) and the Office for Fair Access (OFFA). Working together on both of these projects encouraged us to further develop our ideas and produce an updated systematic review for publication in 2015. We also thank those who contributed to the original project funded by the GMC, notably John McLachlan, Member of the Centre for Medical Education Research, Durham University, and Emma Dunlop, Medical Admissions, University of Aberdeen.

Funding

funding was provided by HEE and the OFFA.

Access to Document

10.1111/medu.12817Licence: Unspecified

Accepted Manuscript
This is the peer reviewed version of the following article: Medical Education 2016: 50: 36–60, which has been published in final form at doi: 10.1111/medu.12817. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Self-Archiving.
Accepted author manuscript, 854 KBLicence: Unspecified

Cite this

@article{75b30062b4264a06b6d19a3ee037fbe1,

title = "How effective are selection methods in medical education?: A systematic review",

abstract = "CONTEXT: Selection methods used by medical schools should reliably identify whether candidates are likely to be successful in medical training and ultimately become competent clinicians. However, there is little consensus regarding methods that reliably evaluate non-academic attributes, and longitudinal studies examining predictors of success after qualification are insufficient. This systematic review synthesises the extant research evidence on the relative strengths of various selection methods. We offer a research agenda and identify key considerations to inform policy and practice in the next 50 years.METHODS: A formalised literature search was conducted for studies published between 1997 and 2015. A total of 194 articles met the inclusion criteria and were appraised in relation to: (i) selection method used; (ii) research question(s) addressed, and (iii) type of study design.RESULTS: Eight selection methods were identified: (i) aptitude tests; (ii) academic records; (iii) personal statements; (iv) references; (v) situational judgement tests (SJTs); (vi) personality and emotional intelligence assessments; (vii) interviews and multiple mini-interviews (MMIs), and (viii) selection centres (SCs). The evidence relating to each method was reviewed against four evaluation criteria: effectiveness (reliability and validity); procedural issues; acceptability, and cost-effectiveness.CONCLUSIONS: Evidence shows clearly that academic records, MMIs, aptitude tests, SJTs and SCs are more effective selection methods and are generally fairer than traditional interviews, references and personal statements. However, achievement in different selection methods may differentially predict performance at the various stages of medical education and clinical practice. Research into selection has been over-reliant on cross-sectional study designs and has tended to focus on reliability estimates rather than validity as an indicator of quality. A comprehensive framework of outcome criteria should be developed to allow researchers to interpret empirical evidence and compare selection methods fairly. This review highlights gaps in evidence for the combination of selection tools that is most effective and the weighting to be given to each tool.",

author = "Fiona Patterson and Alec Knight and Jon Dowell and Sandra Nicholson and Fran Cousans and Jennifer Cleland",

note = "Acknowledgements the authors thank the UK General Medical Council (GMC) for commissioning an initial rapid review of the literature on selection and widening access to medicine in 2013. Further thanks are due to the UK Medical Schools Council (MSC) for commissioning a significantly updated review of selection methods in 2014, funding for which was provided by Health Education England (HEE) and the Office for Fair Access (OFFA). Working together on both of these projects encouraged us to further develop our ideas and produce an updated systematic review for publication in 2015. We also thank those who contributed to the original project funded by the GMC, notably John McLachlan, Member of the Centre for Medical Education Research, Durham University, and Emma Dunlop, Medical Admissions, University of Aberdeen. Funding funding was provided by HEE and the OFFA.",

year = "2016",

month = jan,

doi = "10.1111/medu.12817",

language = "English",

volume = "50",

pages = "36--60",

journal = "Medical Education",

issn = "0308-0110",

publisher = "John Wiley and Sons Inc.",

number = "1",

}

TY - JOUR

T1 - How effective are selection methods in medical education?

T2 - A systematic review

AU - Patterson, Fiona

AU - Knight, Alec

AU - Dowell, Jon

AU - Nicholson, Sandra

AU - Cousans, Fran

AU - Cleland, Jennifer

N1 - Acknowledgements the authors thank the UK General Medical Council (GMC) for commissioning an initial rapid review of the literature on selection and widening access to medicine in 2013. Further thanks are due to the UK Medical Schools Council (MSC) for commissioning a significantly updated review of selection methods in 2014, funding for which was provided by Health Education England (HEE) and the Office for Fair Access (OFFA). Working together on both of these projects encouraged us to further develop our ideas and produce an updated systematic review for publication in 2015. We also thank those who contributed to the original project funded by the GMC, notably John McLachlan, Member of the Centre for Medical Education Research, Durham University, and Emma Dunlop, Medical Admissions, University of Aberdeen. Funding funding was provided by HEE and the OFFA.

PY - 2016/1

Y1 - 2016/1

N2 - CONTEXT: Selection methods used by medical schools should reliably identify whether candidates are likely to be successful in medical training and ultimately become competent clinicians. However, there is little consensus regarding methods that reliably evaluate non-academic attributes, and longitudinal studies examining predictors of success after qualification are insufficient. This systematic review synthesises the extant research evidence on the relative strengths of various selection methods. We offer a research agenda and identify key considerations to inform policy and practice in the next 50 years.METHODS: A formalised literature search was conducted for studies published between 1997 and 2015. A total of 194 articles met the inclusion criteria and were appraised in relation to: (i) selection method used; (ii) research question(s) addressed, and (iii) type of study design.RESULTS: Eight selection methods were identified: (i) aptitude tests; (ii) academic records; (iii) personal statements; (iv) references; (v) situational judgement tests (SJTs); (vi) personality and emotional intelligence assessments; (vii) interviews and multiple mini-interviews (MMIs), and (viii) selection centres (SCs). The evidence relating to each method was reviewed against four evaluation criteria: effectiveness (reliability and validity); procedural issues; acceptability, and cost-effectiveness.CONCLUSIONS: Evidence shows clearly that academic records, MMIs, aptitude tests, SJTs and SCs are more effective selection methods and are generally fairer than traditional interviews, references and personal statements. However, achievement in different selection methods may differentially predict performance at the various stages of medical education and clinical practice. Research into selection has been over-reliant on cross-sectional study designs and has tended to focus on reliability estimates rather than validity as an indicator of quality. A comprehensive framework of outcome criteria should be developed to allow researchers to interpret empirical evidence and compare selection methods fairly. This review highlights gaps in evidence for the combination of selection tools that is most effective and the weighting to be given to each tool.

AB - CONTEXT: Selection methods used by medical schools should reliably identify whether candidates are likely to be successful in medical training and ultimately become competent clinicians. However, there is little consensus regarding methods that reliably evaluate non-academic attributes, and longitudinal studies examining predictors of success after qualification are insufficient. This systematic review synthesises the extant research evidence on the relative strengths of various selection methods. We offer a research agenda and identify key considerations to inform policy and practice in the next 50 years.METHODS: A formalised literature search was conducted for studies published between 1997 and 2015. A total of 194 articles met the inclusion criteria and were appraised in relation to: (i) selection method used; (ii) research question(s) addressed, and (iii) type of study design.RESULTS: Eight selection methods were identified: (i) aptitude tests; (ii) academic records; (iii) personal statements; (iv) references; (v) situational judgement tests (SJTs); (vi) personality and emotional intelligence assessments; (vii) interviews and multiple mini-interviews (MMIs), and (viii) selection centres (SCs). The evidence relating to each method was reviewed against four evaluation criteria: effectiveness (reliability and validity); procedural issues; acceptability, and cost-effectiveness.CONCLUSIONS: Evidence shows clearly that academic records, MMIs, aptitude tests, SJTs and SCs are more effective selection methods and are generally fairer than traditional interviews, references and personal statements. However, achievement in different selection methods may differentially predict performance at the various stages of medical education and clinical practice. Research into selection has been over-reliant on cross-sectional study designs and has tended to focus on reliability estimates rather than validity as an indicator of quality. A comprehensive framework of outcome criteria should be developed to allow researchers to interpret empirical evidence and compare selection methods fairly. This review highlights gaps in evidence for the combination of selection tools that is most effective and the weighting to be given to each tool.

U2 - 10.1111/medu.12817

DO - 10.1111/medu.12817

M3 - Article

C2 - 26695465

SN - 0308-0110

VL - 50

SP - 36

EP - 60

JO - Medical Education

JF - Medical Education

IS - 1

ER -

How effective are selection methods in medical education? A systematic review

Abstract

Bibliographical note

Access to Document

Fingerprint

Cite this