Service and clinical impacts of reader bias in breast cancer screening: a retrospective study

Clarisse F de Vries; Roger T Staff; Jaroslaw A Dymiter; Moragh Boyle; Lesley A Anderson; Gerald Lip; iCAIRD Radiology Collaboration

doi:10.1093/bjr/tqad024

Service and clinical impacts of reader bias in breast cancer screening: a retrospective study

Clarisse F de Vries^* (Corresponding Author), Roger T Staff, Jaroslaw A Dymiter, Moragh Boyle, Lesley A Anderson, Gerald Lip, iCAIRD Radiology Collaboration

^*Corresponding author for this work

Aberdeen Royal Infirmary

Research output: Contribution to journal › Article › peer-review

Abstract

OBJECTIVES: To determine factors influencing reader agreement in breast screening and investigate the relationship between agreement level and patient outcomes.

METHODS: Reader pair agreement for 83 265 sets of mammograms from the Scottish Breast Screening service (2015-2020) was evaluated using Cohen's kappa statistic. Each mammography examination was read by two readers, per routine screening practice, with the second initially blinded but able to choose to view the first reader's opinion. If the two readers disagreed, a third reader arbitrated. Variation in reader agreement was examined by: whether the reader acted as the first or second reader, reader experience, and recall, cancer detection and arbitration recall rate.

RESULTS: Readers' opinions varied by whether they acted as the first or second reader. Furthermore, reader 2 was more likely to agree with reader 1 if reader 1 was more experienced than they were, and less likely to agree if they themselves were more experienced than reader 1 (P < .001). Agreement was not significantly associated with cancer detection rate, overall recall rate or arbitration recall rates (P > .05). Lower agreement between readers led to a higher arbiter workload (P < .001).

CONCLUSIONS: In mammography screening, the second reader's opinion is influenced by the first reader's opinion, with the degree of influence dependent on the readers' relative experience levels.

ADVANCES IN KNOWLEDGE: While less-experienced readers relied on their more experienced reading partner, no adverse impact on service outcomes was observed. Allowing access to the first reader's opinion may benefit newly qualified readers, but reduces independent evaluation, which may lower cancer detection rates.

Original language	English
Pages (from-to)	120-125
Number of pages	6
Journal	British Journal of Radiology
Volume	97
Issue number	1153
Early online date	12 Dec 2023
DOIs	https://doi.org/10.1093/bjr/tqad024
Publication status	Published - 23 Jan 2024

Bibliographical note

Acknowledgements
iCAIRD Radiology Collaboration: Corri Black, Alison D. Murray and Katie Wilde, University of Aberdeen. James D. Blackwood, NHS Greater Glasgow and Clyde. Claire Butterly and John Zurowski, University of Glasgow. Jon Eilbeck and Colin McSkimming, NHS Grampian. Canon Medical Research Europe Ltd. – SHAIP platform.
We would like to acknowledge the support of the Grampian Data Safe Haven (DaSH) facility within the Aberdeen Centre for Health Data Science and the associated financial support of the University of Aberdeen, and NHS Research Scotland (through NHS Grampian investment in DaSH).
Funding
This work is supported by the Industrial Centre for Artificial Intelligence Research in Digital Diagnostics (iCAIRD). iCAIRD was funded by Innovate UK on behalf of UK Research and Innovation (UKRI) [project number: 104690]. The funders were not involved in the study’s design; the collection, analysis or interpretation of the data; or the decision to submit the manuscript for publication.

Keywords

Humans
Female
Early Detection of Cancer
Breast Neoplasms
Retrospective Studies
Mammography
Breast

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1093/bjr/tqad024Licence: CC BY

De_Vries_etal_BIR_Services_And_Clinical_VoR
Copyright © 2023, © The Author(s) 2023. Published by Oxford University Press on behalf of the British Institute of Radiology. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.
Final published version, 487 KBLicence: CC BY

Aberdeen Biomedical Imaging Centre
Gordon Waiter (Manager) & Teresa Morris (Facilities Co-ordinator)
Aberdeen Biomedical Imaging Centre
Research Facilities: Facility
Grampian Data Safe Haven (DaSH)
Katie Wilde (Manager)
Institute of Applied Health Sciences
Research Facilities: Facility

Cite this

@article{d2641c2044f04c37aac2eab509d28f93,

title = "Service and clinical impacts of reader bias in breast cancer screening: a retrospective study",

abstract = "OBJECTIVES: To determine factors influencing reader agreement in breast screening and investigate the relationship between agreement level and patient outcomes.METHODS: Reader pair agreement for 83 265 sets of mammograms from the Scottish Breast Screening service (2015-2020) was evaluated using Cohen's kappa statistic. Each mammography examination was read by two readers, per routine screening practice, with the second initially blinded but able to choose to view the first reader's opinion. If the two readers disagreed, a third reader arbitrated. Variation in reader agreement was examined by: whether the reader acted as the first or second reader, reader experience, and recall, cancer detection and arbitration recall rate.RESULTS: Readers' opinions varied by whether they acted as the first or second reader. Furthermore, reader 2 was more likely to agree with reader 1 if reader 1 was more experienced than they were, and less likely to agree if they themselves were more experienced than reader 1 (P < .001). Agreement was not significantly associated with cancer detection rate, overall recall rate or arbitration recall rates (P > .05). Lower agreement between readers led to a higher arbiter workload (P < .001).CONCLUSIONS: In mammography screening, the second reader's opinion is influenced by the first reader's opinion, with the degree of influence dependent on the readers' relative experience levels.ADVANCES IN KNOWLEDGE: While less-experienced readers relied on their more experienced reading partner, no adverse impact on service outcomes was observed. Allowing access to the first reader's opinion may benefit newly qualified readers, but reduces independent evaluation, which may lower cancer detection rates.",

keywords = "Humans, Female, Early Detection of Cancer, Breast Neoplasms, Retrospective Studies, Mammography, Breast",

author = "{de Vries}, {Clarisse F} and Staff, {Roger T} and Dymiter, {Jaroslaw A} and Moragh Boyle and Anderson, {Lesley A} and Gerald Lip and {iCAIRD Radiology Collaboration}",

note = "Acknowledgements iCAIRD Radiology Collaboration: Corri Black, Alison D. Murray and Katie Wilde, University of Aberdeen. James D. Blackwood, NHS Greater Glasgow and Clyde. Claire Butterly and John Zurowski, University of Glasgow. Jon Eilbeck and Colin McSkimming, NHS Grampian. Canon Medical Research Europe Ltd. – SHAIP platform. We would like to acknowledge the support of the Grampian Data Safe Haven (DaSH) facility within the Aberdeen Centre for Health Data Science and the associated financial support of the University of Aberdeen, and NHS Research Scotland (through NHS Grampian investment in DaSH). Funding This work is supported by the Industrial Centre for Artificial Intelligence Research in Digital Diagnostics (iCAIRD). iCAIRD was funded by Innovate UK on behalf of UK Research and Innovation (UKRI) [project number: 104690]. The funders were not involved in the study{\textquoteright}s design; the collection, analysis or interpretation of the data; or the decision to submit the manuscript for publication.",

year = "2024",

month = jan,

day = "23",

doi = "10.1093/bjr/tqad024",

language = "English",

volume = "97",

pages = "120--125",

journal = "British Journal of Radiology",

issn = "0007-1285",

publisher = "Oxford University Press (OUP)",

number = "1153",

}

TY - JOUR

T1 - Service and clinical impacts of reader bias in breast cancer screening

T2 - a retrospective study

AU - de Vries, Clarisse F

AU - Staff, Roger T

AU - Dymiter, Jaroslaw A

AU - Boyle, Moragh

AU - Anderson, Lesley A

AU - Lip, Gerald

AU - iCAIRD Radiology Collaboration

N1 - Acknowledgements iCAIRD Radiology Collaboration: Corri Black, Alison D. Murray and Katie Wilde, University of Aberdeen. James D. Blackwood, NHS Greater Glasgow and Clyde. Claire Butterly and John Zurowski, University of Glasgow. Jon Eilbeck and Colin McSkimming, NHS Grampian. Canon Medical Research Europe Ltd. – SHAIP platform. We would like to acknowledge the support of the Grampian Data Safe Haven (DaSH) facility within the Aberdeen Centre for Health Data Science and the associated financial support of the University of Aberdeen, and NHS Research Scotland (through NHS Grampian investment in DaSH). Funding This work is supported by the Industrial Centre for Artificial Intelligence Research in Digital Diagnostics (iCAIRD). iCAIRD was funded by Innovate UK on behalf of UK Research and Innovation (UKRI) [project number: 104690]. The funders were not involved in the study’s design; the collection, analysis or interpretation of the data; or the decision to submit the manuscript for publication.

PY - 2024/1/23

Y1 - 2024/1/23

N2 - OBJECTIVES: To determine factors influencing reader agreement in breast screening and investigate the relationship between agreement level and patient outcomes.METHODS: Reader pair agreement for 83 265 sets of mammograms from the Scottish Breast Screening service (2015-2020) was evaluated using Cohen's kappa statistic. Each mammography examination was read by two readers, per routine screening practice, with the second initially blinded but able to choose to view the first reader's opinion. If the two readers disagreed, a third reader arbitrated. Variation in reader agreement was examined by: whether the reader acted as the first or second reader, reader experience, and recall, cancer detection and arbitration recall rate.RESULTS: Readers' opinions varied by whether they acted as the first or second reader. Furthermore, reader 2 was more likely to agree with reader 1 if reader 1 was more experienced than they were, and less likely to agree if they themselves were more experienced than reader 1 (P < .001). Agreement was not significantly associated with cancer detection rate, overall recall rate or arbitration recall rates (P > .05). Lower agreement between readers led to a higher arbiter workload (P < .001).CONCLUSIONS: In mammography screening, the second reader's opinion is influenced by the first reader's opinion, with the degree of influence dependent on the readers' relative experience levels.ADVANCES IN KNOWLEDGE: While less-experienced readers relied on their more experienced reading partner, no adverse impact on service outcomes was observed. Allowing access to the first reader's opinion may benefit newly qualified readers, but reduces independent evaluation, which may lower cancer detection rates.

AB - OBJECTIVES: To determine factors influencing reader agreement in breast screening and investigate the relationship between agreement level and patient outcomes.METHODS: Reader pair agreement for 83 265 sets of mammograms from the Scottish Breast Screening service (2015-2020) was evaluated using Cohen's kappa statistic. Each mammography examination was read by two readers, per routine screening practice, with the second initially blinded but able to choose to view the first reader's opinion. If the two readers disagreed, a third reader arbitrated. Variation in reader agreement was examined by: whether the reader acted as the first or second reader, reader experience, and recall, cancer detection and arbitration recall rate.RESULTS: Readers' opinions varied by whether they acted as the first or second reader. Furthermore, reader 2 was more likely to agree with reader 1 if reader 1 was more experienced than they were, and less likely to agree if they themselves were more experienced than reader 1 (P < .001). Agreement was not significantly associated with cancer detection rate, overall recall rate or arbitration recall rates (P > .05). Lower agreement between readers led to a higher arbiter workload (P < .001).CONCLUSIONS: In mammography screening, the second reader's opinion is influenced by the first reader's opinion, with the degree of influence dependent on the readers' relative experience levels.ADVANCES IN KNOWLEDGE: While less-experienced readers relied on their more experienced reading partner, no adverse impact on service outcomes was observed. Allowing access to the first reader's opinion may benefit newly qualified readers, but reduces independent evaluation, which may lower cancer detection rates.

KW - Humans

KW - Female

KW - Early Detection of Cancer

KW - Breast Neoplasms

KW - Retrospective Studies

KW - Mammography

KW - Breast

U2 - 10.1093/bjr/tqad024

DO - 10.1093/bjr/tqad024

M3 - Article

C2 - 38263824

SN - 0007-1285

VL - 97

SP - 120

EP - 125

JO - British Journal of Radiology

JF - British Journal of Radiology

IS - 1153

ER -

Service and clinical impacts of reader bias in breast cancer screening: a retrospective study

Abstract

Bibliographical note

Keywords

UN SDGs

Access to Document

Fingerprint

Equipment

Aberdeen Biomedical Imaging Centre

Grampian Data Safe Haven (DaSH)

Cite this