Abstract
Introduction
Absolute methods of standard setting (SS) are more suitable for OSCEs. However, previous studies have theorised that borderline regression method (BRM) is not reliable for small sample sizes.
Materials and methods
OSCE results for the 2017–2019 cohorts were analysed to compare BRM versus Modified Angoff. We reported on whether the stations in multiple cohorts were sufficiently equivalent to aggregate the results together to calculate which method of SS was more reliable. We finally used the bootstrapping method to compare the accuracy of BRM for small versus simulated larger cohorts.
Results
BRM was a valid method for SS OSCEs in this dataset when station quality was sufficiently high. However, a large gap between the Angoff and BRM in some of the OSCEs was observed, which could be explained by poor use of the grading scale. Model fit statistics were generally adequate even with low sample sizes. Using the bootstrap of datasets, the error rate was much higher for low-quality stations but was not an issue in high-quality ones.
Discussion
This study adds to the evidence that well-designed OSCEs can use BRM for small cohorts. However, there is a need for the institutions to properly assess their stations and their assessors, before embarking into using this method, to prevent from having to remove stations.
Conclusions
This analysis suggests that BRM is an acceptable replacement for Angoff SS in small cohorts, where there is a range of candidates undertaking the assessments and there are well-designed OSCES with well-trained examiners.
Absolute methods of standard setting (SS) are more suitable for OSCEs. However, previous studies have theorised that borderline regression method (BRM) is not reliable for small sample sizes.
Materials and methods
OSCE results for the 2017–2019 cohorts were analysed to compare BRM versus Modified Angoff. We reported on whether the stations in multiple cohorts were sufficiently equivalent to aggregate the results together to calculate which method of SS was more reliable. We finally used the bootstrapping method to compare the accuracy of BRM for small versus simulated larger cohorts.
Results
BRM was a valid method for SS OSCEs in this dataset when station quality was sufficiently high. However, a large gap between the Angoff and BRM in some of the OSCEs was observed, which could be explained by poor use of the grading scale. Model fit statistics were generally adequate even with low sample sizes. Using the bootstrap of datasets, the error rate was much higher for low-quality stations but was not an issue in high-quality ones.
Discussion
This study adds to the evidence that well-designed OSCEs can use BRM for small cohorts. However, there is a need for the institutions to properly assess their stations and their assessors, before embarking into using this method, to prevent from having to remove stations.
Conclusions
This analysis suggests that BRM is an acceptable replacement for Angoff SS in small cohorts, where there is a range of candidates undertaking the assessments and there are well-designed OSCES with well-trained examiners.
| Original language | English |
|---|---|
| Pages (from-to) | 686-691 |
| Number of pages | 6 |
| Journal | European Journal of Dental Education |
| Volume | 26 |
| Issue number | 4 |
| Early online date | 7 Jan 2022 |
| DOIs | |
| Publication status | Published - 1 Nov 2022 |
Bibliographical note
ACKNOWLEDGEMENTwe would like to thank the Institute of Dentistry at the University of Aberdeen for internally financing this study.
Data Availability Statement
The data that support the findings of this study are available from the corresponding author, RML, upon reasonable request.Keywords
- Standard setting
- OSCE
- borderline-regression
- Modified Angoff