Passing a USA National Bar Exam: a First Corpus for Experimentation

Adam Zachary Wyner; Biralatei James Fawei; Jeff Z Pan

Passing a USA National Bar Exam: a First Corpus for Experimentation

Adam Zachary Wyner, Biralatei James Fawei, Jeff Z Pan

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

12 Citations (Scopus)

15 Downloads (Pure)

Abstract

Bar exams provide are a key watershed by which legal professionals demonstrate their knowledge of the law and its application. Passing the bar entitles one to practice the law in a given jurisdiction. The bar provides a excellent benchmark for the performance of legal information systems since passing the bar would arguably signal that the system has acquired key aspects of legal reason on a par with a human lawyer. The paper presents a first experiment with actual bar exam material, treating it as a form of textual entailment from the question to an answer. The Gold Standard is provided by the company which provides the bar exam. The paper reports a textual entailment student on a US bar exam, running `out of the box' the Excitement Open Platform for textual entailment. The results and evaluation show that the tool can identify wrong answers (non-entailment) with a high F1 score, it performs poorly in identifying the correct answer (entailment). The results provide a baseline performance measure against which to evaluate future improvements. The reasons for the poor performance are examined, and proposals are made to augment the tool in the future.

Original language	English
Title of host publication	LREC 2016, Tenth International Conference on Language Resources and Evaluation
Editors	Nicoletta Calzolari , Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asunción Moreno, Jan Odijk, Stelios Piperidis
Publisher	LREC
Number of pages	7
ISBN (Print)	978-2-9517408-9-1
Publication status	Published - 23 May 2016
Event	Tenth International Conference on Language Resources and Evaluation - Grand Hotel Bernardin Conference Center, Portorož, Slovenia Duration: 23 May 2016 → 28 May 2016

Conference

Conference	Tenth International Conference on Language Resources and Evaluation
Country/Territory	Slovenia
City	Portorož
Period	23/05/16 → 28/05/16

Keywords

textual entailment
bar exam
legal reasoning
natural language processing

Access to Document

Passing a USA National Bar Exam
The LREC 2016 Proceedings are licensed under a Creative Commons Attribution-NonCommercial 4.0 International License https://creativecommons.org/licenses/by-nc/4.0/
Final published version, 113 KBLicence: CC BY-NC

http://www.lrec-conf.org/proceedings/lrec2016/index.html

Passing a USA National Bar Exam: a First Corpus for Experimentation: Article Data
Wyner, A. Z. (Creator), Fawei, B. J. (Creator), Pan, J. Z. (Creator) & Edwards, P. (Data Manager), University of Aberdeen, 31 Mar 2016
Dataset

Cite this

Wyner, A. Z., Fawei, B. J., & Pan, J. Z. (2016). Passing a USA National Bar Exam: a First Corpus for Experimentation. In N. Calzolari , K. Choukri, T. Declerck, S. Goggi, M. Grobelnik, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, & S. Piperidis (Eds.), LREC 2016, Tenth International Conference on Language Resources and Evaluation LREC. http://www.lrec-conf.org/proceedings/lrec2016/index.html

Passing a USA National Bar Exam: a First Corpus for Experimentation. / Wyner, Adam Zachary; Fawei, Biralatei James; Pan, Jeff Z.
LREC 2016, Tenth International Conference on Language Resources and Evaluation. ed. / Nicoletta Calzolari ; Khalid Choukri; Thierry Declerck; Sara Goggi; Marko Grobelnik; Bente Maegaard; Joseph Mariani; Helene Mazo; Asunción Moreno; Jan Odijk; Stelios Piperidis. LREC, 2016.

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

Wyner, AZ, Fawei, BJ & Pan, JZ 2016, Passing a USA National Bar Exam: a First Corpus for Experimentation. in N Calzolari , K Choukri, T Declerck, S Goggi, M Grobelnik, B Maegaard, J Mariani, H Mazo, A Moreno, J Odijk & S Piperidis (eds), LREC 2016, Tenth International Conference on Language Resources and Evaluation. LREC, Tenth International Conference on Language Resources and Evaluation, Portorož, Slovenia, 23/05/16. <http://www.lrec-conf.org/proceedings/lrec2016/index.html>

Wyner, Adam Zachary ; Fawei, Biralatei James ; Pan, Jeff Z. / Passing a USA National Bar Exam : a First Corpus for Experimentation. LREC 2016, Tenth International Conference on Language Resources and Evaluation. editor / Nicoletta Calzolari ; Khalid Choukri ; Thierry Declerck ; Sara Goggi ; Marko Grobelnik ; Bente Maegaard ; Joseph Mariani ; Helene Mazo ; Asunción Moreno ; Jan Odijk ; Stelios Piperidis. LREC, 2016.

@inproceedings{6e1e0a8352914ba8a01a047d429cc6c1,

title = "Passing a USA National Bar Exam: a First Corpus for Experimentation",

abstract = "Bar exams provide are a key watershed by which legal professionals demonstrate their knowledge of the law and its application. Passing the bar entitles one to practice the law in a given jurisdiction. The bar provides a excellent benchmark for the performance of legal information systems since passing the bar would arguably signal that the system has acquired key aspects of legal reason on a par with a human lawyer. The paper presents a first experiment with actual bar exam material, treating it as a form of textual entailment from the question to an answer. The Gold Standard is provided by the company which provides the bar exam. The paper reports a textual entailment student on a US bar exam, running `out of the box' the Excitement Open Platform for textual entailment. The results and evaluation show that the tool can identify wrong answers (non-entailment) with a high F1 score, it performs poorly in identifying the correct answer (entailment). The results provide a baseline performance measure against which to evaluate future improvements. The reasons for the poor performance are examined, and proposals are made to augment the tool in the future.",

keywords = "textual entailment, bar exam, legal reasoning, natural language processing",

author = "Wyner, {Adam Zachary} and Fawei, {Biralatei James} and Pan, {Jeff Z}",

year = "2016",

month = may,

day = "23",

language = "English",

isbn = "978-2-9517408-9-1",

editor = "{Calzolari }, {Nicoletta } and Choukri, {Khalid } and Declerck, {Thierry } and Goggi, {Sara } and Marko Grobelnik and Bente Maegaard and Joseph Mariani and Helene Mazo and Moreno, {Asunci{\'o}n } and Jan Odijk and Stelios Piperidis",

booktitle = "LREC 2016, Tenth International Conference on Language Resources and Evaluation",

publisher = "LREC",

note = "Tenth International Conference on Language Resources and Evaluation ; Conference date: 23-05-2016 Through 28-05-2016",

}

TY - GEN

T1 - Passing a USA National Bar Exam

T2 - Tenth International Conference on Language Resources and Evaluation

AU - Wyner, Adam Zachary

AU - Fawei, Biralatei James

AU - Pan, Jeff Z

PY - 2016/5/23

Y1 - 2016/5/23

N2 - Bar exams provide are a key watershed by which legal professionals demonstrate their knowledge of the law and its application. Passing the bar entitles one to practice the law in a given jurisdiction. The bar provides a excellent benchmark for the performance of legal information systems since passing the bar would arguably signal that the system has acquired key aspects of legal reason on a par with a human lawyer. The paper presents a first experiment with actual bar exam material, treating it as a form of textual entailment from the question to an answer. The Gold Standard is provided by the company which provides the bar exam. The paper reports a textual entailment student on a US bar exam, running `out of the box' the Excitement Open Platform for textual entailment. The results and evaluation show that the tool can identify wrong answers (non-entailment) with a high F1 score, it performs poorly in identifying the correct answer (entailment). The results provide a baseline performance measure against which to evaluate future improvements. The reasons for the poor performance are examined, and proposals are made to augment the tool in the future.

AB - Bar exams provide are a key watershed by which legal professionals demonstrate their knowledge of the law and its application. Passing the bar entitles one to practice the law in a given jurisdiction. The bar provides a excellent benchmark for the performance of legal information systems since passing the bar would arguably signal that the system has acquired key aspects of legal reason on a par with a human lawyer. The paper presents a first experiment with actual bar exam material, treating it as a form of textual entailment from the question to an answer. The Gold Standard is provided by the company which provides the bar exam. The paper reports a textual entailment student on a US bar exam, running `out of the box' the Excitement Open Platform for textual entailment. The results and evaluation show that the tool can identify wrong answers (non-entailment) with a high F1 score, it performs poorly in identifying the correct answer (entailment). The results provide a baseline performance measure against which to evaluate future improvements. The reasons for the poor performance are examined, and proposals are made to augment the tool in the future.

KW - textual entailment

KW - bar exam

KW - legal reasoning

KW - natural language processing

M3 - Published conference contribution

SN - 978-2-9517408-9-1

BT - LREC 2016, Tenth International Conference on Language Resources and Evaluation

A2 - Calzolari , Nicoletta

A2 - Choukri, Khalid

A2 - Declerck, Thierry

A2 - Goggi, Sara

A2 - Grobelnik, Marko

A2 - Maegaard, Bente

A2 - Mariani, Joseph

A2 - Mazo, Helene

A2 - Moreno, Asunción

A2 - Odijk, Jan

A2 - Piperidis, Stelios

PB - LREC

Y2 - 23 May 2016 through 28 May 2016

ER -

Passing a USA National Bar Exam: a First Corpus for Experimentation

Abstract

Conference

Keywords

Access to Document

Fingerprint

Datasets

Passing a USA National Bar Exam: a First Corpus for Experimentation: Article Data

Cite this