Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions

Jack Wilkinson; Calvin Heal; George A Antoniou; Ella Flemyng; Zarko Alfirevic; Alison Avenell; Virginia Barbour; Nicholas J L Brown; John Carlisle; Mike Clarke; Patrick Dicker; Jo Dumville; Andrew Grey; Lyle C Gurrin; Jill A Hayden; James Heathers; Kylie E Hunter; Toby Lasserson; Sarah Lensen; Tianjing Li; Wentao Li; Elizabeth Loder; Andreas Lundh; Gideon Meyerowitz-Katz; Ben W Mol; Neil E O' Connell; Lisa Parker; Barbara K Redman; Anna Lene Seidler; Kyle A Sheldrick; Emma Sydenham; David J Torgerson; Madelon van Wely; Rui Wang; Lisa Bero; Jamie J Kirkham; Emily Lam

doi:10.1101/2023.09.21.23295626

Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions

Jack Wilkinson^* (Corresponding Author), Calvin Heal, George A Antoniou, Ella Flemyng, Zarko Alfirevic, Alison Avenell, Virginia Barbour, Nicholas J L Brown, John Carlisle, Mike Clarke, Patrick Dicker, Jo Dumville, Andrew Grey, Lyle C Gurrin, Jill A Hayden, James Heathers, Kylie E Hunter, Toby Lasserson, Sarah Lensen, Tianjing LiWentao Li, Elizabeth Loder, Andreas Lundh, Gideon Meyerowitz-Katz, Ben W Mol, Neil E O' Connell, Lisa Parker, Barbara K Redman, Anna Lene Seidler, Kyle A Sheldrick, Emma Sydenham, David J Torgerson, Madelon van Wely, Rui Wang, Lisa Bero, Jamie J Kirkham, Emily Lam

^*Corresponding author for this work

Research output: Working paper › Preprint

Abstract

INTRODUCTION: Randomised controlled trials (RCTs) inform healthcare decisions. It is now apparent that some published RCTs contain false data and some appear to have been entirely fabricated. Systematic reviews are performed to identify and synthesise all RCTs that have been conducted on a given topic. While it is usual to assess methodological features of the RCTs in the process of undertaking a systematic review, it is not usual to consider whether the RCTs contain false data. Studies containing false data therefore go unnoticed and contribute to systematic review conclusions. The INSPECT-SR project will develop a tool to assess the trustworthiness of RCTs in systematic reviews of healthcare related interventions.

METHODS AND ANALYSIS: The INSPECT-SR tool will be developed using expert consensus in combination with empirical evidence, over five stages: 1) a survey of experts to assemble a comprehensive list of checks for detecting problematic RCTs, 2) an evaluation of the feasibility and impact of applying the checks to systematic reviews, 3) a Delphi survey to determine which of the checks are supported by expert consensus, culminating in 4) a consensus meeting to select checks to be included in a draft tool and to determine its format, 5) prospective testing of the draft tool in the production of new health systematic reviews, to allow refinement based on user feedback. We anticipate that the INSPECT-SR tool will help researchers to identify problematic studies, and will help patients by protecting them from the influence of false data on their healthcare.

Original language	English
Publisher	MedRxiv
Number of pages	16
DOIs	https://doi.org/10.1101/2023.09.21.23295626
Publication status	Published - 10 Oct 2023

Bibliographical note

Funding Statement
This research is funded by the NIHR Research for Patient Benefit programme (NIHR203568). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

Data Availability
This is a research protocol. Datasets arising from the project will be made freely available.

Version History

This article is a preprint and has not been peer-reviewed. It reports new medical research that has yet to be evaluated and so should not be used to guide clinical practice.

Access to Document

10.1101/2023.09.21.23295626Licence: CC BY

Cite this

Wilkinson, J., Heal, C., Antoniou, G. A., Flemyng, E., Alfirevic, Z., Avenell, A., Barbour, V., Brown, N. J. L., Carlisle, J., Clarke, M., Dicker, P., Dumville, J., Grey, A., Gurrin, L. C., Hayden, J. A., Heathers, J., Hunter, K. E., Lasserson, T., Lensen, S., ... Lam, E. (2023). Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions. MedRxiv. https://doi.org/10.1101/2023.09.21.23295626

Wilkinson, J, Heal, C, Antoniou, GA, Flemyng, E, Alfirevic, Z, Avenell, A, Barbour, V, Brown, NJL, Carlisle, J, Clarke, M, Dicker, P, Dumville, J, Grey, A, Gurrin, LC, Hayden, JA, Heathers, J, Hunter, KE, Lasserson, T, Lensen, S, Li, T, Li, W, Loder, E, Lundh, A, Meyerowitz-Katz, G, Mol, BW, O' Connell, NE, Parker, L, Redman, BK, Seidler, AL, Sheldrick, KA, Sydenham, E, Torgerson, DJ, van Wely, M, Wang, R, Bero, L, Kirkham, JJ & Lam, E 2023 'Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions' MedRxiv. https://doi.org/10.1101/2023.09.21.23295626

@techreport{f116c030ba3145abab64ddc7796d7192,

title = "Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions",

abstract = "INTRODUCTION: Randomised controlled trials (RCTs) inform healthcare decisions. It is now apparent that some published RCTs contain false data and some appear to have been entirely fabricated. Systematic reviews are performed to identify and synthesise all RCTs that have been conducted on a given topic. While it is usual to assess methodological features of the RCTs in the process of undertaking a systematic review, it is not usual to consider whether the RCTs contain false data. Studies containing false data therefore go unnoticed and contribute to systematic review conclusions. The INSPECT-SR project will develop a tool to assess the trustworthiness of RCTs in systematic reviews of healthcare related interventions.METHODS AND ANALYSIS: The INSPECT-SR tool will be developed using expert consensus in combination with empirical evidence, over five stages: 1) a survey of experts to assemble a comprehensive list of checks for detecting problematic RCTs, 2) an evaluation of the feasibility and impact of applying the checks to systematic reviews, 3) a Delphi survey to determine which of the checks are supported by expert consensus, culminating in 4) a consensus meeting to select checks to be included in a draft tool and to determine its format, 5) prospective testing of the draft tool in the production of new health systematic reviews, to allow refinement based on user feedback. We anticipate that the INSPECT-SR tool will help researchers to identify problematic studies, and will help patients by protecting them from the influence of false data on their healthcare.",

author = "Jack Wilkinson and Calvin Heal and Antoniou, {George A} and Ella Flemyng and Zarko Alfirevic and Alison Avenell and Virginia Barbour and Brown, {Nicholas J L} and John Carlisle and Mike Clarke and Patrick Dicker and Jo Dumville and Andrew Grey and Gurrin, {Lyle C} and Hayden, {Jill A} and James Heathers and Hunter, {Kylie E} and Toby Lasserson and Sarah Lensen and Tianjing Li and Wentao Li and Elizabeth Loder and Andreas Lundh and Gideon Meyerowitz-Katz and Mol, {Ben W} and {O' Connell}, {Neil E} and Lisa Parker and Redman, {Barbara K} and Seidler, {Anna Lene} and Sheldrick, {Kyle A} and Emma Sydenham and Torgerson, {David J} and {van Wely}, Madelon and Rui Wang and Lisa Bero and Kirkham, {Jamie J} and Emily Lam",

note = "Funding Statement This research is funded by the NIHR Research for Patient Benefit programme (NIHR203568). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. Data Availability This is a research protocol. Datasets arising from the project will be made freely available.",

year = "2023",

month = oct,

day = "10",

doi = "10.1101/2023.09.21.23295626",

language = "English",

publisher = "MedRxiv",

type = "WorkingPaper",

institution = "MedRxiv",

}

TY - UNPB

T1 - Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions

AU - Wilkinson, Jack

AU - Heal, Calvin

AU - Antoniou, George A

AU - Flemyng, Ella

AU - Alfirevic, Zarko

AU - Avenell, Alison

AU - Barbour, Virginia

AU - Brown, Nicholas J L

AU - Carlisle, John

AU - Clarke, Mike

AU - Dicker, Patrick

AU - Dumville, Jo

AU - Grey, Andrew

AU - Gurrin, Lyle C

AU - Hayden, Jill A

AU - Heathers, James

AU - Hunter, Kylie E

AU - Lasserson, Toby

AU - Lensen, Sarah

AU - Li, Tianjing

AU - Li, Wentao

AU - Loder, Elizabeth

AU - Lundh, Andreas

AU - Meyerowitz-Katz, Gideon

AU - Mol, Ben W

AU - O' Connell, Neil E

AU - Parker, Lisa

AU - Redman, Barbara K

AU - Seidler, Anna Lene

AU - Sheldrick, Kyle A

AU - Sydenham, Emma

AU - Torgerson, David J

AU - van Wely, Madelon

AU - Wang, Rui

AU - Bero, Lisa

AU - Kirkham, Jamie J

AU - Lam, Emily

N1 - Funding Statement This research is funded by the NIHR Research for Patient Benefit programme (NIHR203568). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. Data Availability This is a research protocol. Datasets arising from the project will be made freely available.

PY - 2023/10/10

Y1 - 2023/10/10

N2 - INTRODUCTION: Randomised controlled trials (RCTs) inform healthcare decisions. It is now apparent that some published RCTs contain false data and some appear to have been entirely fabricated. Systematic reviews are performed to identify and synthesise all RCTs that have been conducted on a given topic. While it is usual to assess methodological features of the RCTs in the process of undertaking a systematic review, it is not usual to consider whether the RCTs contain false data. Studies containing false data therefore go unnoticed and contribute to systematic review conclusions. The INSPECT-SR project will develop a tool to assess the trustworthiness of RCTs in systematic reviews of healthcare related interventions.METHODS AND ANALYSIS: The INSPECT-SR tool will be developed using expert consensus in combination with empirical evidence, over five stages: 1) a survey of experts to assemble a comprehensive list of checks for detecting problematic RCTs, 2) an evaluation of the feasibility and impact of applying the checks to systematic reviews, 3) a Delphi survey to determine which of the checks are supported by expert consensus, culminating in 4) a consensus meeting to select checks to be included in a draft tool and to determine its format, 5) prospective testing of the draft tool in the production of new health systematic reviews, to allow refinement based on user feedback. We anticipate that the INSPECT-SR tool will help researchers to identify problematic studies, and will help patients by protecting them from the influence of false data on their healthcare.

AB - INTRODUCTION: Randomised controlled trials (RCTs) inform healthcare decisions. It is now apparent that some published RCTs contain false data and some appear to have been entirely fabricated. Systematic reviews are performed to identify and synthesise all RCTs that have been conducted on a given topic. While it is usual to assess methodological features of the RCTs in the process of undertaking a systematic review, it is not usual to consider whether the RCTs contain false data. Studies containing false data therefore go unnoticed and contribute to systematic review conclusions. The INSPECT-SR project will develop a tool to assess the trustworthiness of RCTs in systematic reviews of healthcare related interventions.METHODS AND ANALYSIS: The INSPECT-SR tool will be developed using expert consensus in combination with empirical evidence, over five stages: 1) a survey of experts to assemble a comprehensive list of checks for detecting problematic RCTs, 2) an evaluation of the feasibility and impact of applying the checks to systematic reviews, 3) a Delphi survey to determine which of the checks are supported by expert consensus, culminating in 4) a consensus meeting to select checks to be included in a draft tool and to determine its format, 5) prospective testing of the draft tool in the production of new health systematic reviews, to allow refinement based on user feedback. We anticipate that the INSPECT-SR tool will help researchers to identify problematic studies, and will help patients by protecting them from the influence of false data on their healthcare.

U2 - 10.1101/2023.09.21.23295626

DO - 10.1101/2023.09.21.23295626

M3 - Preprint

C2 - 37873409

BT - Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions

PB - MedRxiv

ER -

Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions

Abstract

Bibliographical note

Version History

Access to Document

Fingerprint

Cite this