Abstract
Reflection is a crucial counselling skill where the therapist conveys to the client their interpretation of what the client said. Language models have recently been used to generate reflections automatically, but human evaluation is challenging, particularly due to the cost of hiring experts. Laypeople-based evaluation is less expensive and easier to scale, but its quality is unknown for reflections. Therefore, we explore whether laypeople can be an alternative to experts in evaluating a fundamental quality aspect: coherence and context-consistency. We do so by asking a group of laypeople and a group of experts to annotate both synthetic reflections and human reflections from actual therapists. We find that both laypeople and experts are reliable annotators and that they have moderate-to-strong inter-group correlation, which shows that laypeople can be trusted for such evaluations. We also discover that GPT-3 mostly produces coherent and consistent reflections, and we explore changes in evaluation results when the source of synthetic reflections changes to GPT-3 from the less powerful GPT-2.
Original language | English |
---|---|
Title of host publication | Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |
Editors | Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki |
Place of Publication | Toronto, Canada |
Publisher | Association for Computational Linguistics |
Pages | 6906-6930 |
Number of pages | 25 |
DOIs | |
Publication status | Published - 2023 |
Event | The 61st Annual Meeting of the Association for Computational Linguistics - Toronto, Canada Duration: 9 Jul 2023 → 14 Jul 2023 Conference number: 61 https://2023.aclweb.org/ |
Conference
Conference | The 61st Annual Meeting of the Association for Computational Linguistics |
---|---|
Country/Territory | Canada |
City | Toronto |
Period | 9/07/23 → 14/07/23 |
Internet address |