CholecTriplet2022: Show me a tool and tell me the triplet — An endoscopic vision challenge for surgical action triplet detection

Chinedu Innocent Nwoye; Tong Yu; Saurav Sharma; Aditya Murali; Deepak Alapatt; Armine Vardazaryan; Kun Yuan; Jonas Hajek; Wolfgang Reiter; Amine Yamlahi; Finn Henri Smidt; Xiaoyang Zou; Guoyan Zheng; Bruno Oliveira; Helena R. Torres; Satoshi Kondo; Satoshi Kasai; Felix Holm; Ege Özsoy; Shuangchun Gui; Han Li; Sista Raviteja; Rachana Sathish; Pranav Poudel; Binod Bhattarai; Ziheng Wang; Guo Rui; Melanie Schellenberg; João L. Vilaça; Tobias Czempiel; Zhenkun Wang; Debdoot Sheet; Shrawan Kumar Thapa; Max Berniker; Patrick Godau; Pedro Morais; Sudarshan Regmi; Thuy Nuong Tran; Jaime Fonseca; Jan Hinrich Nölke; Estevão Lima; Eduard Vazquez; Lena Maier-Hein; Nassir Navab; Pietro Mascagni; Barbara Seeliger; Cristians Gonzalez; Didier Mutter; Nicolas Padoy

doi:10.1016/j.media.2023.102888

CholecTriplet2022: Show me a tool and tell me the triplet — An endoscopic vision challenge for surgical action triplet detection

Chinedu Innocent Nwoye^*, Tong Yu, Saurav Sharma, Aditya Murali, Deepak Alapatt, Armine Vardazaryan, Kun Yuan, Jonas Hajek, Wolfgang Reiter, Amine Yamlahi, Finn Henri Smidt, Xiaoyang Zou, Guoyan Zheng, Bruno Oliveira, Helena R. Torres, Satoshi Kondo, Satoshi Kasai, Felix Holm, Ege Özsoy, Shuangchun GuiHan Li, Sista Raviteja, Rachana Sathish, Pranav Poudel, Binod Bhattarai, Ziheng Wang, Guo Rui, Melanie Schellenberg, João L. Vilaça, Tobias Czempiel, Zhenkun Wang, Debdoot Sheet, Shrawan Kumar Thapa, Max Berniker, Patrick Godau, Pedro Morais, Sudarshan Regmi, Thuy Nuong Tran, Jaime Fonseca, Jan Hinrich Nölke, Estevão Lima, Eduard Vazquez, Lena Maier-Hein, Nassir Navab, Pietro Mascagni, Barbara Seeliger, Cristians Gonzalez, Didier Mutter, Nicolas Padoy

^*Corresponding author for this work

Computing Science

Research output: Contribution to journal › Short survey › peer-review

2 Citations (Scopus)

Abstract

Formalizing surgical activities as triplets of the used instruments, actions performed, and target anatomies is becoming a gold standard approach for surgical activity modeling. The benefit is that this formalization helps to obtain a more detailed understanding of tool-tissue interaction which can be used to develop better Artificial Intelligence assistance for image-guided surgery. Earlier efforts and the CholecTriplet challenge introduced in 2021 have put together techniques aimed at recognizing these triplets from surgical footage. Estimating also the spatial locations of the triplets would offer a more precise intraoperative context-aware decision support for computer-assisted intervention. This paper presents the CholecTriplet2022 challenge, which extends surgical action triplet modeling from recognition to detection. It includes weakly-supervised bounding box localization of every visible surgical instrument (or tool), as the key actors, and the modeling of each tool-activity in the form of ‹instrument, verb, target› triplet. The paper describes a baseline method and 10 new deep learning algorithms presented at the challenge to solve the task. It also provides thorough methodological comparisons of the methods, an in-depth analysis of the obtained results across multiple metrics, visual and procedural challenges; their significance, and useful insights for future research directions and applications in surgery.

Original language	English
Article number	102888
Number of pages	21
Journal	Medical Image Analysis
Volume	89
Early online date	12 Jul 2023
DOIs	https://doi.org/10.1016/j.media.2023.102888
Publication status	Published - Oct 2023

Bibliographical note

The organizers would like to thank the IHU and IRCAD research teams for their help with the initial data annotation during the CONDOR project. We also thank Stefanie Speidel, Lena Maier-Hein, Danail Stoyanov, and the entire EndoVis 2022 organizing committee for providing the platform for this challenge.

Funding
This work was supported by French state funds managed within the Plan Investissements d’Avenir by the ANR under references: National AI Chair AI4ORSafety [ANR-20-CHIA-0029-01], Labex CAMI [ANR-11-LABX-0004], DeepSurg [ANR-16-CE33-0009], IHU Strasbourg [ANR-10-IAHU-02] and by BPI France under references: project CONDOR, project 5G-OR [DOS0180017/00].

Software validation and evaluation were performed with servers managed by CAMMA at University of Strasbourg and IHU Strasbourg, as well as HPC resources from Unistra Mésocentre, and GENCI-IDRIS [Grant 2021-AD011011638R2, 2021-AD011011638R3].

Awards for the challenge winners were sponsored by IHU Strasbourg, NVIDIA, and Medtronic Ltd.

Participating teams would like to acknowledge the following funding: CITI: Shanghai Municipal Science and Technology Commission, China [20511105205]. SDS-HD: Twinning Grant [DKFZ+RBCT]; the Surgical Oncology Program of the National Center for Tumor Diseases (NCT) Heidelberg, by the German Federal Ministry of Health under the reference number 2520DAT0P1 as part of the pAItient project, and by HELMHOLTZ IMAGING, a platform of the Helmholtz Information & Data Science Incubator, Germany. European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (NEURAL SPICING; grant agreement No. [101002198]) and Surgical Oncology Program of the National Center for Tumor Diseases (NCT) Heidelberg, Germany. 2AI-ICVS: Fundação para a Ciência e a Tecnologia (FCT), Portugal, Portugal and the European Social Fund, European Union, for funding support through the “Programa Operacional Capital Humano” (POCH) in the scope of the Ph.D. grants [SFRH/BD/136721/2018, SFRH/BD/136670/2018]. Grants [NORTE-01-0145-FEDER-000045,NORTE-01-0145-FEDER-000059], supported by Northern Portugal Regional Operational Programme (NORTE 2020), under the Portugal 2020 Partnership Agreement, through the European Regional Development Fund (FEDER). Also funded by national funds, through the FCT and FCT/MCTES in the scope of the project [UIDB/05549/2020, UIDP/05549/2020]. SHUANGCHUN: Guangdong Climbing Plan, China under Grant [pdjh2023c21602]. CAMP: partially supported by Carl Zeiss AG, Germany .

Data Availability Statement

The CholecT50 dataset and the validation data used in the challenge have been made available to the public and accessible via https://github.com/CAMMA-public/cholect50. The test set spatial labels will be released publicly. The baseline model code will be released as well. Participants can release their code on their own volition. All released code would be linked to the central GitHub repository for the challenge: https://github.com/CAMMA-public/cholectriplet2022.

Keywords

Action detection
CholecT50
Computer-assisted surgery
Fine-grained activity recognition
Surgical action triplet
Tool localization
Weak supervision

Access to Document

10.1016/j.media.2023.102888Licence: Unspecified

Cite this

Nwoye, C. I., Yu, T., Sharma, S., Murali, A., Alapatt, D., Vardazaryan, A., Yuan, K., Hajek, J., Reiter, W., Yamlahi, A., Smidt, F. H., Zou, X., Zheng, G., Oliveira, B., Torres, H. R., Kondo, S., Kasai, S., Holm, F., Özsoy, E., ... Padoy, N. (2023). CholecTriplet2022: Show me a tool and tell me the triplet — An endoscopic vision challenge for surgical action triplet detection. Medical Image Analysis, 89, Article 102888. https://doi.org/10.1016/j.media.2023.102888

Nwoye, CI, Yu, T, Sharma, S, Murali, A, Alapatt, D, Vardazaryan, A, Yuan, K, Hajek, J, Reiter, W, Yamlahi, A, Smidt, FH, Zou, X, Zheng, G, Oliveira, B, Torres, HR, Kondo, S, Kasai, S, Holm, F, Özsoy, E, Gui, S, Li, H, Raviteja, S, Sathish, R, Poudel, P, Bhattarai, B, Wang, Z, Rui, G, Schellenberg, M, Vilaça, JL, Czempiel, T, Wang, Z, Sheet, D, Thapa, SK, Berniker, M, Godau, P, Morais, P, Regmi, S, Tran, TN, Fonseca, J, Nölke, JH, Lima, E, Vazquez, E, Maier-Hein, L, Navab, N, Mascagni, P, Seeliger, B, Gonzalez, C, Mutter, D & Padoy, N 2023, 'CholecTriplet2022: Show me a tool and tell me the triplet — An endoscopic vision challenge for surgical action triplet detection', Medical Image Analysis, vol. 89, 102888. https://doi.org/10.1016/j.media.2023.102888

@article{d1645ba6405e45f2a0e4b170bbe86c0f,

title = "CholecTriplet2022: Show me a tool and tell me the triplet — An endoscopic vision challenge for surgical action triplet detection",

abstract = "Formalizing surgical activities as triplets of the used instruments, actions performed, and target anatomies is becoming a gold standard approach for surgical activity modeling. The benefit is that this formalization helps to obtain a more detailed understanding of tool-tissue interaction which can be used to develop better Artificial Intelligence assistance for image-guided surgery. Earlier efforts and the CholecTriplet challenge introduced in 2021 have put together techniques aimed at recognizing these triplets from surgical footage. Estimating also the spatial locations of the triplets would offer a more precise intraoperative context-aware decision support for computer-assisted intervention. This paper presents the CholecTriplet2022 challenge, which extends surgical action triplet modeling from recognition to detection. It includes weakly-supervised bounding box localization of every visible surgical instrument (or tool), as the key actors, and the modeling of each tool-activity in the form of ‹instrument, verb, target› triplet. The paper describes a baseline method and 10 new deep learning algorithms presented at the challenge to solve the task. It also provides thorough methodological comparisons of the methods, an in-depth analysis of the obtained results across multiple metrics, visual and procedural challenges; their significance, and useful insights for future research directions and applications in surgery.",

keywords = "Action detection, CholecT50, Computer-assisted surgery, Fine-grained activity recognition, Surgical action triplet, Tool localization, Weak supervision",

author = "Nwoye, {Chinedu Innocent} and Tong Yu and Saurav Sharma and Aditya Murali and Deepak Alapatt and Armine Vardazaryan and Kun Yuan and Jonas Hajek and Wolfgang Reiter and Amine Yamlahi and Smidt, {Finn Henri} and Xiaoyang Zou and Guoyan Zheng and Bruno Oliveira and Torres, {Helena R.} and Satoshi Kondo and Satoshi Kasai and Felix Holm and Ege {\"O}zsoy and Shuangchun Gui and Han Li and Sista Raviteja and Rachana Sathish and Pranav Poudel and Binod Bhattarai and Ziheng Wang and Guo Rui and Melanie Schellenberg and Vila{\c c}a, {Jo{\~a}o L.} and Tobias Czempiel and Zhenkun Wang and Debdoot Sheet and Thapa, {Shrawan Kumar} and Max Berniker and Patrick Godau and Pedro Morais and Sudarshan Regmi and Tran, {Thuy Nuong} and Jaime Fonseca and N{\"o}lke, {Jan Hinrich} and Estev{\~a}o Lima and Eduard Vazquez and Lena Maier-Hein and Nassir Navab and Pietro Mascagni and Barbara Seeliger and Cristians Gonzalez and Didier Mutter and Nicolas Padoy",

note = "The organizers would like to thank the IHU and IRCAD research teams for their help with the initial data annotation during the CONDOR project. We also thank Stefanie Speidel, Lena Maier-Hein, Danail Stoyanov, and the entire EndoVis 2022 organizing committee for providing the platform for this challenge. Funding This work was supported by French state funds managed within the Plan Investissements d{\textquoteright}Avenir by the ANR under references: National AI Chair AI4ORSafety [ANR-20-CHIA-0029-01], Labex CAMI [ANR-11-LABX-0004], DeepSurg [ANR-16-CE33-0009], IHU Strasbourg [ANR-10-IAHU-02] and by BPI France under references: project CONDOR, project 5G-OR [DOS0180017/00]. Software validation and evaluation were performed with servers managed by CAMMA at University of Strasbourg and IHU Strasbourg, as well as HPC resources from Unistra M{\'e}socentre, and GENCI-IDRIS [Grant 2021-AD011011638R2, 2021-AD011011638R3]. Awards for the challenge winners were sponsored by IHU Strasbourg, NVIDIA, and Medtronic Ltd. Participating teams would like to acknowledge the following funding: CITI: Shanghai Municipal Science and Technology Commission, China [20511105205]. SDS-HD: Twinning Grant [DKFZ+RBCT]; the Surgical Oncology Program of the National Center for Tumor Diseases (NCT) Heidelberg, by the German Federal Ministry of Health under the reference number 2520DAT0P1 as part of the pAItient project, and by HELMHOLTZ IMAGING, a platform of the Helmholtz Information & Data Science Incubator, Germany. European Research Council (ERC) under the European Union{\textquoteright}s Horizon 2020 research and innovation programme (NEURAL SPICING; grant agreement No. [101002198]) and Surgical Oncology Program of the National Center for Tumor Diseases (NCT) Heidelberg, Germany. 2AI-ICVS: Funda{\c c}{\~a}o para a Ci{\^e}ncia e a Tecnologia (FCT), Portugal, Portugal and the European Social Fund, European Union, for funding support through the “Programa Operacional Capital Humano” (POCH) in the scope of the Ph.D. grants [SFRH/BD/136721/2018, SFRH/BD/136670/2018]. Grants [NORTE-01-0145-FEDER-000045,NORTE-01-0145-FEDER-000059], supported by Northern Portugal Regional Operational Programme (NORTE 2020), under the Portugal 2020 Partnership Agreement, through the European Regional Development Fund (FEDER). Also funded by national funds, through the FCT and FCT/MCTES in the scope of the project [UIDB/05549/2020, UIDP/05549/2020]. SHUANGCHUN: Guangdong Climbing Plan, China under Grant [pdjh2023c21602]. CAMP: partially supported by Carl Zeiss AG, Germany .",

year = "2023",

month = oct,

doi = "10.1016/j.media.2023.102888",

language = "English",

volume = "89",

journal = "Medical Image Analysis",

issn = "1361-8415",

publisher = "Elsevier",

}

TY - JOUR

T1 - CholecTriplet2022

T2 - Show me a tool and tell me the triplet — An endoscopic vision challenge for surgical action triplet detection

AU - Nwoye, Chinedu Innocent

AU - Yu, Tong

AU - Sharma, Saurav

AU - Murali, Aditya

AU - Alapatt, Deepak

AU - Vardazaryan, Armine

AU - Yuan, Kun

AU - Hajek, Jonas

AU - Reiter, Wolfgang

AU - Yamlahi, Amine

AU - Smidt, Finn Henri

AU - Zou, Xiaoyang

AU - Zheng, Guoyan

AU - Oliveira, Bruno

AU - Torres, Helena R.

AU - Kondo, Satoshi

AU - Kasai, Satoshi

AU - Holm, Felix

AU - Özsoy, Ege

AU - Gui, Shuangchun

AU - Li, Han

AU - Raviteja, Sista

AU - Sathish, Rachana

AU - Poudel, Pranav

AU - Bhattarai, Binod

AU - Wang, Ziheng

AU - Rui, Guo

AU - Schellenberg, Melanie

AU - Vilaça, João L.

AU - Czempiel, Tobias

AU - Wang, Zhenkun

AU - Sheet, Debdoot

AU - Thapa, Shrawan Kumar

AU - Berniker, Max

AU - Godau, Patrick

AU - Morais, Pedro

AU - Regmi, Sudarshan

AU - Tran, Thuy Nuong

AU - Fonseca, Jaime

AU - Nölke, Jan Hinrich

AU - Lima, Estevão

AU - Vazquez, Eduard

AU - Maier-Hein, Lena

AU - Navab, Nassir

AU - Mascagni, Pietro

AU - Seeliger, Barbara

AU - Gonzalez, Cristians

AU - Mutter, Didier

AU - Padoy, Nicolas

N1 - The organizers would like to thank the IHU and IRCAD research teams for their help with the initial data annotation during the CONDOR project. We also thank Stefanie Speidel, Lena Maier-Hein, Danail Stoyanov, and the entire EndoVis 2022 organizing committee for providing the platform for this challenge. Funding This work was supported by French state funds managed within the Plan Investissements d’Avenir by the ANR under references: National AI Chair AI4ORSafety [ANR-20-CHIA-0029-01], Labex CAMI [ANR-11-LABX-0004], DeepSurg [ANR-16-CE33-0009], IHU Strasbourg [ANR-10-IAHU-02] and by BPI France under references: project CONDOR, project 5G-OR [DOS0180017/00]. Software validation and evaluation were performed with servers managed by CAMMA at University of Strasbourg and IHU Strasbourg, as well as HPC resources from Unistra Mésocentre, and GENCI-IDRIS [Grant 2021-AD011011638R2, 2021-AD011011638R3]. Awards for the challenge winners were sponsored by IHU Strasbourg, NVIDIA, and Medtronic Ltd. Participating teams would like to acknowledge the following funding: CITI: Shanghai Municipal Science and Technology Commission, China [20511105205]. SDS-HD: Twinning Grant [DKFZ+RBCT]; the Surgical Oncology Program of the National Center for Tumor Diseases (NCT) Heidelberg, by the German Federal Ministry of Health under the reference number 2520DAT0P1 as part of the pAItient project, and by HELMHOLTZ IMAGING, a platform of the Helmholtz Information & Data Science Incubator, Germany. European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (NEURAL SPICING; grant agreement No. [101002198]) and Surgical Oncology Program of the National Center for Tumor Diseases (NCT) Heidelberg, Germany. 2AI-ICVS: Fundação para a Ciência e a Tecnologia (FCT), Portugal, Portugal and the European Social Fund, European Union, for funding support through the “Programa Operacional Capital Humano” (POCH) in the scope of the Ph.D. grants [SFRH/BD/136721/2018, SFRH/BD/136670/2018]. Grants [NORTE-01-0145-FEDER-000045,NORTE-01-0145-FEDER-000059], supported by Northern Portugal Regional Operational Programme (NORTE 2020), under the Portugal 2020 Partnership Agreement, through the European Regional Development Fund (FEDER). Also funded by national funds, through the FCT and FCT/MCTES in the scope of the project [UIDB/05549/2020, UIDP/05549/2020]. SHUANGCHUN: Guangdong Climbing Plan, China under Grant [pdjh2023c21602]. CAMP: partially supported by Carl Zeiss AG, Germany .

PY - 2023/10

Y1 - 2023/10

N2 - Formalizing surgical activities as triplets of the used instruments, actions performed, and target anatomies is becoming a gold standard approach for surgical activity modeling. The benefit is that this formalization helps to obtain a more detailed understanding of tool-tissue interaction which can be used to develop better Artificial Intelligence assistance for image-guided surgery. Earlier efforts and the CholecTriplet challenge introduced in 2021 have put together techniques aimed at recognizing these triplets from surgical footage. Estimating also the spatial locations of the triplets would offer a more precise intraoperative context-aware decision support for computer-assisted intervention. This paper presents the CholecTriplet2022 challenge, which extends surgical action triplet modeling from recognition to detection. It includes weakly-supervised bounding box localization of every visible surgical instrument (or tool), as the key actors, and the modeling of each tool-activity in the form of ‹instrument, verb, target› triplet. The paper describes a baseline method and 10 new deep learning algorithms presented at the challenge to solve the task. It also provides thorough methodological comparisons of the methods, an in-depth analysis of the obtained results across multiple metrics, visual and procedural challenges; their significance, and useful insights for future research directions and applications in surgery.

AB - Formalizing surgical activities as triplets of the used instruments, actions performed, and target anatomies is becoming a gold standard approach for surgical activity modeling. The benefit is that this formalization helps to obtain a more detailed understanding of tool-tissue interaction which can be used to develop better Artificial Intelligence assistance for image-guided surgery. Earlier efforts and the CholecTriplet challenge introduced in 2021 have put together techniques aimed at recognizing these triplets from surgical footage. Estimating also the spatial locations of the triplets would offer a more precise intraoperative context-aware decision support for computer-assisted intervention. This paper presents the CholecTriplet2022 challenge, which extends surgical action triplet modeling from recognition to detection. It includes weakly-supervised bounding box localization of every visible surgical instrument (or tool), as the key actors, and the modeling of each tool-activity in the form of ‹instrument, verb, target› triplet. The paper describes a baseline method and 10 new deep learning algorithms presented at the challenge to solve the task. It also provides thorough methodological comparisons of the methods, an in-depth analysis of the obtained results across multiple metrics, visual and procedural challenges; their significance, and useful insights for future research directions and applications in surgery.

KW - Action detection

KW - CholecT50

KW - Computer-assisted surgery

KW - Fine-grained activity recognition

KW - Surgical action triplet

KW - Tool localization

KW - Weak supervision

UR - http://www.scopus.com/inward/record.url?scp=85164670806&partnerID=8YFLogxK

U2 - 10.1016/j.media.2023.102888

DO - 10.1016/j.media.2023.102888

M3 - Short survey

AN - SCOPUS:85164670806

SN - 1361-8415

VL - 89

JO - Medical Image Analysis

JF - Medical Image Analysis

M1 - 102888

ER -

CholecTriplet2022: Show me a tool and tell me the triplet — An endoscopic vision challenge for surgical action triplet detection

Abstract

Bibliographical note

Data Availability Statement

Keywords

Access to Document

Other files and links

Fingerprint

Cite this