Does following an “excellent” candidate in the Objective Structured Clinical Examination (OSCE) affect your checklist score?

The Objective Structured Clinical Examination (OSCE) is regarded as the gold standard of competence assessment in healthcare related curricula, however, a host of internal and external sources of variations contribute to the checklist marking scheme including station effects, patient/actor effects, examiner effects and, expectedly, student effects. In this study, we assessed if the average checklist scores differed depending on who a candidate follows accounted for different sources of variations. We evaluated retrospective assessment data from the final year MBChB OSCEs 2017-2019 at the University of Aberdeen and categorised each candidate based on who they followed for each station. Candidates were categorised into three levels based on the preceding candidate’s examiner awarded global score: followed an ‘excellent’ candidate (global score awarded within the station as ‘excellent’), followed an ‘unsatisfactory’ candidate and followed ‘other’ (who did not follow either an ‘excellent’ or ‘unsatisfactory candidate’). We modelled the data using a linear mixed model incorporating appropriate fixed and random effects. A total of 349 students appeared in the OSCEs resulting 3838 examination encounters. The predicted mean (95% confidence interval) score for students who followed an ‘excellent’ candidate was 21.6 (20.3, 22.9), followed ‘other’ was 21.9 (20.6, 23.2), and for those who followed an ‘unsatisfactory’ student was 22.1 (20.7, 23.5). When accounted for individual, examiner and station levels variabilities, the data did not show any evidence (p=0.117) that the variations in checklist scores of candidates could be attributed to who they followed in a high-stake OSCE. Further studies including multiple centres and larger sample sizes with predefined limits of equivalence may be warranted to assess the effect size more rigorously and assuring the perceived fairness of the OSCE examination and its administration to candidates and educators alike.