External validation of the QCovid risk prediction algorithm for risk of COVID-19 hospitalisation and mortality in adults: national validation cohort study in Scotland

Colin R. Simpson; Chris Robertson; Steven Kerr; Ting Shi; Eleftheria Vasileiou; Emily Moore; Colin McCowan; Utkarsh Agrawal; Annemarie Docherty; Rachel Mulholland; Josie Murray; Lewis Duthie Ritchie; Jim McMenamin; Julia Hippisley-Cox; Aziz Sheikh

doi:10.1136/thoraxjnl-2021-217580

External validation of the QCovid risk prediction algorithm for risk of COVID-19 hospitalisation and mortality in adults: national validation cohort study in Scotland

Colin R. Simpson, Chris Robertson, Steven Kerr, Ting Shi, Eleftheria Vasileiou, Emily Moore, Colin McCowan, Utkarsh Agrawal, Annemarie Docherty, Rachel Mulholland, Josie Murray, Lewis Duthie Ritchie, Jim McMenamin, Julia Hippisley-Cox, Aziz Sheikh

Research output: Contribution to journal › Article › peer-review

12 Citations (Scopus)

2 Downloads (Pure)

Abstract

BACKGROUND: The QCovid algorithm is a risk prediction tool that can be used to stratify individuals by risk of COVID-19 hospitalisation and mortality. Version 1 of the algorithm was trained using data covering 10.5 million patients in England in the period 24 January 2020 to 30 April 2020. We carried out an external validation of version 1 of the QCovid algorithm in Scotland.

METHODS: We established a national COVID-19 data platform using individual level data for the population of Scotland (5.4 million residents). Primary care data were linked to reverse-transcription PCR (RT-PCR) virology testing, hospitalisation and mortality data. We assessed the performance of the QCovid algorithm in predicting COVID-19 hospitalisations and deaths in our dataset for two time periods matching the original study: 1 March 2020 to 30 April 2020, and 1 May 2020 to 30 June 2020.

RESULTS: Our dataset comprised 5 384 819 individuals, representing 99% of the estimated population (5 463 300) resident in Scotland in 2020. The algorithm showed good calibration in the first period, but systematic overestimation of risk in the second period, prior to temporal recalibration. Harrell's C for deaths in females and males in the first period was 0.95 (95% CI 0.94 to 0.95) and 0.93 (95% CI 0.92 to 0.93), respectively. Harrell's C for hospitalisations in females and males in the first period was 0.81 (95% CI 0.80 to 0.82) and 0.82 (95% CI 0.81 to 0.82), respectively.

CONCLUSIONS: Version 1 of the QCovid algorithm showed high levels of discrimination in predicting the risk of COVID-19 hospitalisations and deaths in adults resident in Scotland for the original two time periods studied, but is likely to need ongoing recalibration prospectively.

Original language	English
Pages (from-to)	497-504
Number of pages	8
Journal	Thorax
Volume	77
Issue number	5
Early online date	15 Nov 2021
DOIs	https://doi.org/10.1136/thoraxjnl-2021-217580
Publication status	Published - May 2022

Bibliographical note

Acknowledgments
The authors would like to thank staff at Public Health Scotland, Albasoft Ltd, the general practices that contributed data to the study and the EAVE II Collaborators. AS, JM and CR serve on The Scottish Government’s COVID-19 Chief Medical Officer’s Advisory Group and the New and Emerging Respiratory Virus Threats Advisory Group (NERVTAG) Risk Stratification Subgroup.

Ethics approval
Ethical permission for this study was granted from South East Scotland Research Ethics Committee 02 (12/SS/0201). The Public Benefit and Privacy Panel Committee of Public Health Scotland, approved the linkage and analysis of the deidentified datasets for this project (1920-0279).

Data Availability Statement

All code, metadata and documentation for this project is publicly available at https://github.com/EAVE-II/Qcovid-validation. A data dictionary is available at https://github.com/EAVE-II/EAVE-II-data-dictionary. Most of the data that were used in this study are highly sensitive and will not be made available publicly

Keywords

clinical epidemiology
COVID-19

Access to Document

10.1136/thoraxjnl-2021-217580Licence: CC BY-NC

Simpson_etal_BMJT_ExternalValidationoftheQcovid_VOR
This article is available under the Creative Commons CC-BY-NC 4.0 International license and permits non-commercial use, distribution and reproduction in any medium, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. https://creativecommons.org/licenses/by-nc/4.0/
Final published version, 2.67 MBLicence: CC BY-NC

Cite this

Simpson, C. R., Robertson, C., Kerr, S., Shi, T., Vasileiou, E., Moore, E., McCowan, C., Agrawal, U., Docherty, A., Mulholland, R., Murray, J., Ritchie, L. D., McMenamin, J., Hippisley-Cox, J., & Sheikh, A. (2022). External validation of the QCovid risk prediction algorithm for risk of COVID-19 hospitalisation and mortality in adults: national validation cohort study in Scotland. Thorax, 77(5), 497-504. https://doi.org/10.1136/thoraxjnl-2021-217580

Simpson, CR, Robertson, C, Kerr, S, Shi, T, Vasileiou, E, Moore, E, McCowan, C, Agrawal, U, Docherty, A, Mulholland, R, Murray, J, Ritchie, LD, McMenamin, J, Hippisley-Cox, J & Sheikh, A 2022, 'External validation of the QCovid risk prediction algorithm for risk of COVID-19 hospitalisation and mortality in adults: national validation cohort study in Scotland', Thorax, vol. 77, no. 5, pp. 497-504. https://doi.org/10.1136/thoraxjnl-2021-217580

@article{f841d5f1520d4eca82c95d01061ce433,

title = "External validation of the QCovid risk prediction algorithm for risk of COVID-19 hospitalisation and mortality in adults: national validation cohort study in Scotland",

abstract = "BACKGROUND: The QCovid algorithm is a risk prediction tool that can be used to stratify individuals by risk of COVID-19 hospitalisation and mortality. Version 1 of the algorithm was trained using data covering 10.5 million patients in England in the period 24 January 2020 to 30 April 2020. We carried out an external validation of version 1 of the QCovid algorithm in Scotland. METHODS: We established a national COVID-19 data platform using individual level data for the population of Scotland (5.4 million residents). Primary care data were linked to reverse-transcription PCR (RT-PCR) virology testing, hospitalisation and mortality data. We assessed the performance of the QCovid algorithm in predicting COVID-19 hospitalisations and deaths in our dataset for two time periods matching the original study: 1 March 2020 to 30 April 2020, and 1 May 2020 to 30 June 2020. RESULTS: Our dataset comprised 5 384 819 individuals, representing 99% of the estimated population (5 463 300) resident in Scotland in 2020. The algorithm showed good calibration in the first period, but systematic overestimation of risk in the second period, prior to temporal recalibration. Harrell's C for deaths in females and males in the first period was 0.95 (95% CI 0.94 to 0.95) and 0.93 (95% CI 0.92 to 0.93), respectively. Harrell's C for hospitalisations in females and males in the first period was 0.81 (95% CI 0.80 to 0.82) and 0.82 (95% CI 0.81 to 0.82), respectively. CONCLUSIONS: Version 1 of the QCovid algorithm showed high levels of discrimination in predicting the risk of COVID-19 hospitalisations and deaths in adults resident in Scotland for the original two time periods studied, but is likely to need ongoing recalibration prospectively.",

keywords = "clinical epidemiology, COVID-19",

author = "Simpson, {Colin R.} and Chris Robertson and Steven Kerr and Ting Shi and Eleftheria Vasileiou and Emily Moore and Colin McCowan and Utkarsh Agrawal and Annemarie Docherty and Rachel Mulholland and Josie Murray and Ritchie, {Lewis Duthie} and Jim McMenamin and Julia Hippisley-Cox and Aziz Sheikh",

note = "Acknowledgments The authors would like to thank staff at Public Health Scotland, Albasoft Ltd, the general practices that contributed data to the study and the EAVE II Collaborators. AS, JM and CR serve on The Scottish Government{\textquoteright}s COVID-19 Chief Medical Officer{\textquoteright}s Advisory Group and the New and Emerging Respiratory Virus Threats Advisory Group (NERVTAG) Risk Stratification Subgroup. Ethics approval Ethical permission for this study was granted from South East Scotland Research Ethics Committee 02 (12/SS/0201). The Public Benefit and Privacy Panel Committee of Public Health Scotland, approved the linkage and analysis of the deidentified datasets for this project (1920-0279).",

year = "2022",

month = may,

doi = "10.1136/thoraxjnl-2021-217580",

language = "English",

volume = "77",

pages = "497--504",

journal = "Thorax",

issn = "0040-6376",

publisher = "BMJ Publishing Group Ltd",

number = "5",

}

TY - JOUR

T1 - External validation of the QCovid risk prediction algorithm for risk of COVID-19 hospitalisation and mortality in adults

T2 - national validation cohort study in Scotland

AU - Simpson, Colin R.

AU - Robertson, Chris

AU - Kerr, Steven

AU - Shi, Ting

AU - Vasileiou, Eleftheria

AU - Moore, Emily

AU - McCowan, Colin

AU - Agrawal, Utkarsh

AU - Docherty, Annemarie

AU - Mulholland, Rachel

AU - Murray, Josie

AU - Ritchie, Lewis Duthie

AU - McMenamin, Jim

AU - Hippisley-Cox, Julia

AU - Sheikh, Aziz

N1 - Acknowledgments The authors would like to thank staff at Public Health Scotland, Albasoft Ltd, the general practices that contributed data to the study and the EAVE II Collaborators. AS, JM and CR serve on The Scottish Government’s COVID-19 Chief Medical Officer’s Advisory Group and the New and Emerging Respiratory Virus Threats Advisory Group (NERVTAG) Risk Stratification Subgroup. Ethics approval Ethical permission for this study was granted from South East Scotland Research Ethics Committee 02 (12/SS/0201). The Public Benefit and Privacy Panel Committee of Public Health Scotland, approved the linkage and analysis of the deidentified datasets for this project (1920-0279).

PY - 2022/5

Y1 - 2022/5

N2 - BACKGROUND: The QCovid algorithm is a risk prediction tool that can be used to stratify individuals by risk of COVID-19 hospitalisation and mortality. Version 1 of the algorithm was trained using data covering 10.5 million patients in England in the period 24 January 2020 to 30 April 2020. We carried out an external validation of version 1 of the QCovid algorithm in Scotland. METHODS: We established a national COVID-19 data platform using individual level data for the population of Scotland (5.4 million residents). Primary care data were linked to reverse-transcription PCR (RT-PCR) virology testing, hospitalisation and mortality data. We assessed the performance of the QCovid algorithm in predicting COVID-19 hospitalisations and deaths in our dataset for two time periods matching the original study: 1 March 2020 to 30 April 2020, and 1 May 2020 to 30 June 2020. RESULTS: Our dataset comprised 5 384 819 individuals, representing 99% of the estimated population (5 463 300) resident in Scotland in 2020. The algorithm showed good calibration in the first period, but systematic overestimation of risk in the second period, prior to temporal recalibration. Harrell's C for deaths in females and males in the first period was 0.95 (95% CI 0.94 to 0.95) and 0.93 (95% CI 0.92 to 0.93), respectively. Harrell's C for hospitalisations in females and males in the first period was 0.81 (95% CI 0.80 to 0.82) and 0.82 (95% CI 0.81 to 0.82), respectively. CONCLUSIONS: Version 1 of the QCovid algorithm showed high levels of discrimination in predicting the risk of COVID-19 hospitalisations and deaths in adults resident in Scotland for the original two time periods studied, but is likely to need ongoing recalibration prospectively.

AB - BACKGROUND: The QCovid algorithm is a risk prediction tool that can be used to stratify individuals by risk of COVID-19 hospitalisation and mortality. Version 1 of the algorithm was trained using data covering 10.5 million patients in England in the period 24 January 2020 to 30 April 2020. We carried out an external validation of version 1 of the QCovid algorithm in Scotland. METHODS: We established a national COVID-19 data platform using individual level data for the population of Scotland (5.4 million residents). Primary care data were linked to reverse-transcription PCR (RT-PCR) virology testing, hospitalisation and mortality data. We assessed the performance of the QCovid algorithm in predicting COVID-19 hospitalisations and deaths in our dataset for two time periods matching the original study: 1 March 2020 to 30 April 2020, and 1 May 2020 to 30 June 2020. RESULTS: Our dataset comprised 5 384 819 individuals, representing 99% of the estimated population (5 463 300) resident in Scotland in 2020. The algorithm showed good calibration in the first period, but systematic overestimation of risk in the second period, prior to temporal recalibration. Harrell's C for deaths in females and males in the first period was 0.95 (95% CI 0.94 to 0.95) and 0.93 (95% CI 0.92 to 0.93), respectively. Harrell's C for hospitalisations in females and males in the first period was 0.81 (95% CI 0.80 to 0.82) and 0.82 (95% CI 0.81 to 0.82), respectively. CONCLUSIONS: Version 1 of the QCovid algorithm showed high levels of discrimination in predicting the risk of COVID-19 hospitalisations and deaths in adults resident in Scotland for the original two time periods studied, but is likely to need ongoing recalibration prospectively.

KW - clinical epidemiology

KW - COVID-19

UR - http://www.scopus.com/inward/record.url?scp=85122953580&partnerID=8YFLogxK

U2 - 10.1136/thoraxjnl-2021-217580

DO - 10.1136/thoraxjnl-2021-217580

M3 - Article

C2 - 34782484

AN - SCOPUS:85122953580

SN - 0040-6376

VL - 77

SP - 497

EP - 504

JO - Thorax

JF - Thorax

IS - 5

ER -

External validation of the QCovid risk prediction algorithm for risk of COVID-19 hospitalisation and mortality in adults: national validation cohort study in Scotland

Abstract

Bibliographical note

Data Availability Statement

Keywords

Access to Document

Other files and links

Fingerprint

Cite this