Passing a USA National Bar Exam: a First Corpus for Experimentation

Adam Zachary Wyner, Biralatei James Fawei, Jeff Z Pan

Research output: Chapter in Book/Report/Conference proceedingPublished conference contribution

12 Citations (Scopus)
17 Downloads (Pure)


Bar exams provide are a key watershed by which legal professionals demonstrate their knowledge of the law and its application. Passing the bar entitles one to practice the law in a given jurisdiction. The bar provides a excellent benchmark for the performance of legal information systems since passing the bar would arguably signal that the system has acquired key aspects of legal reason on a par with a human lawyer. The paper presents a first experiment with actual bar exam material, treating it as a form of textual entailment from the question to an answer. The Gold Standard is provided by the company which provides the bar exam. The paper reports a textual entailment student on a US bar exam, running `out of the box' the Excitement Open Platform for textual entailment. The results and evaluation show that the tool can identify wrong answers (non-entailment) with a high F1 score, it performs poorly in identifying the correct answer (entailment). The results provide a baseline performance measure against which to evaluate future improvements. The reasons for the poor performance are examined, and proposals are made to augment the tool in the future.
Original languageEnglish
Title of host publicationLREC 2016, Tenth International Conference on Language Resources and Evaluation
EditorsNicoletta Calzolari , Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asunción Moreno, Jan Odijk, Stelios Piperidis
Number of pages7
ISBN (Print)978-2-9517408-9-1
Publication statusPublished - 23 May 2016
EventTenth International Conference on Language Resources and Evaluation - Grand Hotel Bernardin Conference Center, Portorož, Slovenia
Duration: 23 May 201628 May 2016


ConferenceTenth International Conference on Language Resources and Evaluation


  • textual entailment
  • bar exam
  • legal reasoning
  • natural language processing


Dive into the research topics of 'Passing a USA National Bar Exam: a First Corpus for Experimentation'. Together they form a unique fingerprint.

Cite this