A good use of time?: Providing evidence for how effort is invested in primary and secondary outcome data collection in trials

Heidi Gardner; Adel Elfeky; David Pickles; Annabel Dawson; Katie Gillies; Violet Warwick; Shaun Treweek

doi:10.1186/s13063-022-06973-8

A good use of time? Providing evidence for how effort is invested in primary and secondary outcome data collection in trials

Heidi Gardner^* (Corresponding Author), Adel Elfeky , David Pickles, Annabel Dawson, Katie Gillies, Violet Warwick, Shaun Treweek

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

Abstract

BACKGROUND: Data collection is a substantial part of trial workload for participants and staff alike. How these hours of work are spent is important because stakeholders are more interested in some outcomes than others. The ORINOCO study compared the time spent collecting primary outcome data to the time spent collecting secondary outcome data in a cohort of trials.

METHODS: We searched PubMed for phase III trials indexed between 2015 and 2019. From these, we randomly selected 120 trials evaluating a therapeutic intervention plus an additional random selection of 20 trials evaluating a public health intervention. We also added eligible trials from a cohort of 189 trials in rheumatology that had used the same core outcome set. We then obtained the time taken to collect primary and secondary outcomes in each trial. We used a hierarchy of methods that included data in trial reports, contacting the trial team and approaching individuals with experience of using the identified outcome measures. We calculated the primary to secondary data collection time ratio and notional data collection cost for each included trial.

RESULTS: We included 161 trials (120 phase III; 21 core outcome set; 20 public health), which together collected 230 primary and 688 secondary outcomes. Full primary and secondary timing data were obtained for 134 trials (100 phase III; 17 core outcome set; 17 public health). The median time spent on primaries was 56.1 h (range: 0.0-10,746.7, IQR: 226.89) and the median time spent on secondaries was 190.7 hours (range: 0.0-1,356,832.9, IQR: 617.6). The median primary to secondary data collection time ratio was 1.0:3.0 (i.e. for every minute spent on primary outcomes, 3.0 were spent on secondaries). The ratio varied by trial type: phase III trials were 1.0:3.1, core outcome set 1.0:3.4 and public health trials 1.0:2.2. The median notional overall data collection cost was £8015.73 (range: £52.90-£31,899,140.70, IQR: £20,096.64).

CONCLUSIONS: Depending on trial type, between two and three times as much time is spent collecting secondary outcome data than collecting primary outcome data. Trial teams should explicitly consider how long it will take to collect the data for an outcome and decide whether that time is worth it given importance of the outcome to the trial.

Original language	English
Article number	1047
Number of pages	12
Journal	Trials
Volume	23
DOIs	https://doi.org/10.1186/s13063-022-06973-8
Publication status	Published - 27 Dec 2022

Bibliographical note

Acknowledgements
The authors would like to thank Dr Beatriz Goulao from the University of Aberdeen's Health Services Research Unit for her support and statistical expertise when responding to reviewer comments. The Health Services Research Unit, University of Aberdeen, receives core funding from the Chief Scientist Office of the Scottish Government Health Directorates.

Funding
ORINOCO was funded by the Chief Scientist Office, grant number HIPS/18/04.

Data Availability Statement

The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Supplementary data available at https://figshare.com/articles/journal_contribution/Additional_file_1_of_A_good_use_of_time_Providing_evidence_for_how_effort_is_invested_in_primary_and_secondary_outcome_data_collection_in_trials/22607285

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1186/s13063-022-06973-8Licence: CC BY

Gardner_etal_Tri_AGoodUse_VOR
This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Final published version, 3.48 MBLicence: CC BY

Additional file 3 of A good use of time? Providing evidence for how effort is invested in primary and secondary outcome data collection in trials
Gardner, H. (Creator), Elfeky, A. (Creator), Pickles, D. (Creator), Dawson, A. (Creator), Gillies, K. (Creator), Warwick, V. (Creator) & Treweek, S. (Creator), Figshare, 2023
DOI: 10.6084/m9.figshare.22607291, https://springernature.figshare.com/articles/dataset/Additional_file_3_of_A_good_use_of_time_Providing_evidence_for_how_effort_is_invested_in_primary_and_secondary_outcome_data_collection_in_trials/22607291 and one more link, https://springernature.figshare.com/articles/dataset/Additional_file_3_of_A_good_use_of_time_Providing_evidence_for_how_effort_is_invested_in_primary_and_secondary_outcome_data_collection_in_trials/22607291/1 (show fewer)
Dataset
Additional file 2 of A good use of time? Providing evidence for how effort is invested in primary and secondary outcome data collection in trials
Gardner, H. (Creator), Elfeky, A. (Creator), Pickles, D. (Creator), Dawson, A. (Creator), Gillies, K. (Creator), Warwick, V. (Creator) & Treweek, S. (Creator), Figshare, 2023
DOI: 10.6084/m9.figshare.22607288.v1, https://springernature.figshare.com/articles/dataset/Additional_file_2_of_A_good_use_of_time_Providing_evidence_for_how_effort_is_invested_in_primary_and_secondary_outcome_data_collection_in_trials/22607288/1 and one more link, https://springernature.figshare.com/articles/dataset/Additional_file_2_of_A_good_use_of_time_Providing_evidence_for_how_effort_is_invested_in_primary_and_secondary_outcome_data_collection_in_trials/22607288 (show fewer)
Dataset
Additional file 4 of A good use of time? Providing evidence for how effort is invested in primary and secondary outcome data collection in trials
Gardner, H. (Creator), Elfeky, A. (Creator), Pickles, D. (Creator), Dawson, A. (Creator), Gillies, K. (Creator), Warwick, V. (Creator) & Treweek, S. (Creator), Figshare, 2023
DOI: 10.6084/m9.figshare.22607294.v1, https://springernature.figshare.com/articles/dataset/Additional_file_4_of_A_good_use_of_time_Providing_evidence_for_how_effort_is_invested_in_primary_and_secondary_outcome_data_collection_in_trials/22607294/1 and one more link, https://springernature.figshare.com/articles/dataset/Additional_file_4_of_A_good_use_of_time_Providing_evidence_for_how_effort_is_invested_in_primary_and_secondary_outcome_data_collection_in_trials/22607294 (show fewer)
Dataset

Cite this

@article{9e02b09db10c4d6a9526a996d3eac756,

title = "A good use of time?: Providing evidence for how effort is invested in primary and secondary outcome data collection in trials",

abstract = "BACKGROUND: Data collection is a substantial part of trial workload for participants and staff alike. How these hours of work are spent is important because stakeholders are more interested in some outcomes than others. The ORINOCO study compared the time spent collecting primary outcome data to the time spent collecting secondary outcome data in a cohort of trials.METHODS: We searched PubMed for phase III trials indexed between 2015 and 2019. From these, we randomly selected 120 trials evaluating a therapeutic intervention plus an additional random selection of 20 trials evaluating a public health intervention. We also added eligible trials from a cohort of 189 trials in rheumatology that had used the same core outcome set. We then obtained the time taken to collect primary and secondary outcomes in each trial. We used a hierarchy of methods that included data in trial reports, contacting the trial team and approaching individuals with experience of using the identified outcome measures. We calculated the primary to secondary data collection time ratio and notional data collection cost for each included trial.RESULTS: We included 161 trials (120 phase III; 21 core outcome set; 20 public health), which together collected 230 primary and 688 secondary outcomes. Full primary and secondary timing data were obtained for 134 trials (100 phase III; 17 core outcome set; 17 public health). The median time spent on primaries was 56.1 h (range: 0.0-10,746.7, IQR: 226.89) and the median time spent on secondaries was 190.7 hours (range: 0.0-1,356,832.9, IQR: 617.6). The median primary to secondary data collection time ratio was 1.0:3.0 (i.e. for every minute spent on primary outcomes, 3.0 were spent on secondaries). The ratio varied by trial type: phase III trials were 1.0:3.1, core outcome set 1.0:3.4 and public health trials 1.0:2.2. The median notional overall data collection cost was £8015.73 (range: £52.90-£31,899,140.70, IQR: £20,096.64).CONCLUSIONS: Depending on trial type, between two and three times as much time is spent collecting secondary outcome data than collecting primary outcome data. Trial teams should explicitly consider how long it will take to collect the data for an outcome and decide whether that time is worth it given importance of the outcome to the trial.",

author = "Heidi Gardner and Adel Elfeky and David Pickles and Annabel Dawson and Katie Gillies and Violet Warwick and Shaun Treweek",

note = "Acknowledgements The authors would like to thank Dr Beatriz Goulao from the University of Aberdeen's Health Services Research Unit for her support and statistical expertise when responding to reviewer comments. The Health Services Research Unit, University of Aberdeen, receives core funding from the Chief Scientist Office of the Scottish Government Health Directorates. Funding ORINOCO was funded by the Chief Scientist Office, grant number HIPS/18/04. ",

year = "2022",

month = dec,

day = "27",

doi = "10.1186/s13063-022-06973-8",

language = "English",

volume = "23",

journal = "Trials",

issn = "1745-6215",

publisher = "BioMed Central",

}

TY - JOUR

T1 - A good use of time?

T2 - Providing evidence for how effort is invested in primary and secondary outcome data collection in trials

AU - Gardner, Heidi

AU - Elfeky , Adel

AU - Pickles, David

AU - Dawson, Annabel

AU - Gillies, Katie

AU - Warwick, Violet

AU - Treweek, Shaun

N1 - Acknowledgements The authors would like to thank Dr Beatriz Goulao from the University of Aberdeen's Health Services Research Unit for her support and statistical expertise when responding to reviewer comments. The Health Services Research Unit, University of Aberdeen, receives core funding from the Chief Scientist Office of the Scottish Government Health Directorates. Funding ORINOCO was funded by the Chief Scientist Office, grant number HIPS/18/04.

PY - 2022/12/27

Y1 - 2022/12/27

N2 - BACKGROUND: Data collection is a substantial part of trial workload for participants and staff alike. How these hours of work are spent is important because stakeholders are more interested in some outcomes than others. The ORINOCO study compared the time spent collecting primary outcome data to the time spent collecting secondary outcome data in a cohort of trials.METHODS: We searched PubMed for phase III trials indexed between 2015 and 2019. From these, we randomly selected 120 trials evaluating a therapeutic intervention plus an additional random selection of 20 trials evaluating a public health intervention. We also added eligible trials from a cohort of 189 trials in rheumatology that had used the same core outcome set. We then obtained the time taken to collect primary and secondary outcomes in each trial. We used a hierarchy of methods that included data in trial reports, contacting the trial team and approaching individuals with experience of using the identified outcome measures. We calculated the primary to secondary data collection time ratio and notional data collection cost for each included trial.RESULTS: We included 161 trials (120 phase III; 21 core outcome set; 20 public health), which together collected 230 primary and 688 secondary outcomes. Full primary and secondary timing data were obtained for 134 trials (100 phase III; 17 core outcome set; 17 public health). The median time spent on primaries was 56.1 h (range: 0.0-10,746.7, IQR: 226.89) and the median time spent on secondaries was 190.7 hours (range: 0.0-1,356,832.9, IQR: 617.6). The median primary to secondary data collection time ratio was 1.0:3.0 (i.e. for every minute spent on primary outcomes, 3.0 were spent on secondaries). The ratio varied by trial type: phase III trials were 1.0:3.1, core outcome set 1.0:3.4 and public health trials 1.0:2.2. The median notional overall data collection cost was £8015.73 (range: £52.90-£31,899,140.70, IQR: £20,096.64).CONCLUSIONS: Depending on trial type, between two and three times as much time is spent collecting secondary outcome data than collecting primary outcome data. Trial teams should explicitly consider how long it will take to collect the data for an outcome and decide whether that time is worth it given importance of the outcome to the trial.

AB - BACKGROUND: Data collection is a substantial part of trial workload for participants and staff alike. How these hours of work are spent is important because stakeholders are more interested in some outcomes than others. The ORINOCO study compared the time spent collecting primary outcome data to the time spent collecting secondary outcome data in a cohort of trials.METHODS: We searched PubMed for phase III trials indexed between 2015 and 2019. From these, we randomly selected 120 trials evaluating a therapeutic intervention plus an additional random selection of 20 trials evaluating a public health intervention. We also added eligible trials from a cohort of 189 trials in rheumatology that had used the same core outcome set. We then obtained the time taken to collect primary and secondary outcomes in each trial. We used a hierarchy of methods that included data in trial reports, contacting the trial team and approaching individuals with experience of using the identified outcome measures. We calculated the primary to secondary data collection time ratio and notional data collection cost for each included trial.RESULTS: We included 161 trials (120 phase III; 21 core outcome set; 20 public health), which together collected 230 primary and 688 secondary outcomes. Full primary and secondary timing data were obtained for 134 trials (100 phase III; 17 core outcome set; 17 public health). The median time spent on primaries was 56.1 h (range: 0.0-10,746.7, IQR: 226.89) and the median time spent on secondaries was 190.7 hours (range: 0.0-1,356,832.9, IQR: 617.6). The median primary to secondary data collection time ratio was 1.0:3.0 (i.e. for every minute spent on primary outcomes, 3.0 were spent on secondaries). The ratio varied by trial type: phase III trials were 1.0:3.1, core outcome set 1.0:3.4 and public health trials 1.0:2.2. The median notional overall data collection cost was £8015.73 (range: £52.90-£31,899,140.70, IQR: £20,096.64).CONCLUSIONS: Depending on trial type, between two and three times as much time is spent collecting secondary outcome data than collecting primary outcome data. Trial teams should explicitly consider how long it will take to collect the data for an outcome and decide whether that time is worth it given importance of the outcome to the trial.

U2 - 10.1186/s13063-022-06973-8

DO - 10.1186/s13063-022-06973-8

M3 - Article

C2 - 36575542

SN - 1745-6215

VL - 23

JO - Trials

JF - Trials

M1 - 1047

ER -

A good use of time? Providing evidence for how effort is invested in primary and secondary outcome data collection in trials

Abstract

Bibliographical note

Data Availability Statement

UN SDGs

Access to Document

Fingerprint

Datasets

Additional file 3 of A good use of time? Providing evidence for how effort is invested in primary and secondary outcome data collection in trials

Additional file 2 of A good use of time? Providing evidence for how effort is invested in primary and secondary outcome data collection in trials

Additional file 4 of A good use of time? Providing evidence for how effort is invested in primary and secondary outcome data collection in trials

Cite this