Towards objectively evaluating the quality of generated medical summaries

Francesco Moramarco; Aleksandar  Savkov; Damir  Juric; Ehud Reiter

Towards objectively evaluating the quality of generated medical summaries

Francesco Moramarco, Aleksandar Savkov, Damir Juric, Ehud Reiter

Babylon Health

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

1 Citation (Scopus)

3 Downloads (Pure)

Abstract

We propose a method for evaluating the quality of generated text by asking evaluators to count facts, and computing precision, recall, f-score, and accuracy from the raw counts. We believe this approach leads to a more objective and easier to reproduce evaluation. We apply this to the task of medical report summarisation, where measuring objective quality and accuracy is of paramount importance.

Original language	English
Title of host publication	Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)
Subtitle of host publication	EACL 2021
Editors	Anya Belz, Shubham Agarwal, Yvette Graham, Ehud Reiter, Anastasia Shimorina
Publisher	ACL Anthology
Pages	56-61
Number of pages	6
ISBN (Electronic)	978-1-954085-10-7
Publication status	Published - 19 Apr 2021
Event	Workshop on Human Evaluation of NLP Systems - virtual Duration: 19 Apr 2021 → 19 Apr 2021 https://www.virtual2021.eacl.org/workshop_WS-5.html

Workshop

Workshop	Workshop on Human Evaluation of NLP Systems
Period	19/04/21 → 19/04/21
Internet address	https://www.virtual2021.eacl.org/workshop_WS-5.html

Access to Document

Moramarco_etal_ACL_Towards_Objectively_Evaluating_VoR
ACL materials are Copyright © 1963–2022 ACL; other materials are copyrighted by their respective copyright holders. Materials prior to 2016 here are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 International License. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License. https://creativecommons.org/licenses/by-nc-sa/3.0/
Final published version, 925 KBLicence: CC BY

https://arxiv.org/abs/2104.04412Licence: Unspecified
https://aclanthology.org/2021.humeval-1.6Licence: CC BY

Cite this

Towards objectively evaluating the quality of generated medical summaries. / Moramarco, Francesco; Savkov, Aleksandar ; Juric, Damir et al.
Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval): EACL 2021. ed. / Anya Belz; Shubham Agarwal; Yvette Graham; Ehud Reiter; Anastasia Shimorina. ACL Anthology, 2021. p. 56-61.

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

Moramarco, F, Savkov, A, Juric, D & Reiter, E 2021, Towards objectively evaluating the quality of generated medical summaries. in A Belz, S Agarwal, Y Graham, E Reiter & A Shimorina (eds), Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval): EACL 2021. ACL Anthology, pp. 56-61, Workshop on Human Evaluation of NLP Systems, 19/04/21. <https://arxiv.org/abs/2104.04412>

@inproceedings{75fa93ebfd2a4722bae72b5750e43cdb,

title = "Towards objectively evaluating the quality of generated medical summaries",

abstract = "We propose a method for evaluating the quality of generated text by asking evaluators to count facts, and computing precision, recall, f-score, and accuracy from the raw counts. We believe this approach leads to a more objective and easier to reproduce evaluation. We apply this to the task of medical report summarisation, where measuring objective quality and accuracy is of paramount importance.",

author = "Francesco Moramarco and Aleksandar Savkov and Damir Juric and Ehud Reiter",

year = "2021",

month = apr,

day = "19",

language = "English",

pages = "56--61",

editor = "Anya Belz and Shubham Agarwal and Yvette Graham and Ehud Reiter and Anastasia Shimorina",

booktitle = "Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)",

publisher = "ACL Anthology",

note = "Workshop on Human Evaluation of NLP Systems ; Conference date: 19-04-2021 Through 19-04-2021",

url = "https://www.virtual2021.eacl.org/workshop_WS-5.html",

}

TY - GEN

T1 - Towards objectively evaluating the quality of generated medical summaries

AU - Moramarco, Francesco

AU - Savkov, Aleksandar

AU - Juric, Damir

AU - Reiter, Ehud

PY - 2021/4/19

Y1 - 2021/4/19

N2 - We propose a method for evaluating the quality of generated text by asking evaluators to count facts, and computing precision, recall, f-score, and accuracy from the raw counts. We believe this approach leads to a more objective and easier to reproduce evaluation. We apply this to the task of medical report summarisation, where measuring objective quality and accuracy is of paramount importance.

AB - We propose a method for evaluating the quality of generated text by asking evaluators to count facts, and computing precision, recall, f-score, and accuracy from the raw counts. We believe this approach leads to a more objective and easier to reproduce evaluation. We apply this to the task of medical report summarisation, where measuring objective quality and accuracy is of paramount importance.

M3 - Published conference contribution

SP - 56

EP - 61

BT - Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)

A2 - Belz, Anya

A2 - Agarwal, Shubham

A2 - Graham, Yvette

A2 - Reiter, Ehud

A2 - Shimorina, Anastasia

PB - ACL Anthology

T2 - Workshop on Human Evaluation of NLP Systems

Y2 - 19 April 2021 through 19 April 2021

ER -