The Evidence Behind Adding Civ Pro to the Multistate Bar Exam

In February 2013, the National Conference of Bar Examiners (NCBE) made an announcement of a major change to the Multistate Bar Exam (MBE).  Starting with the February 2015 administration of the MBE, civil procedure will be added the list of subject areas being tested. Currently constitutional law, contracts, criminal law/criminal procedure, evidence, real property and torts are tested on the MBE. Since 1972, the MBE has been a high stakes licensing exam for attorneys, and a large aspect of passing the bar in most U.S. jurisdictions. (Today, Louisiana and Puerto Rico are the only exceptions as Washington state was added this year.) The new MBE will still consist of 200 multiple-choice items (of which 190 are actually scored), but there will be fewer items for each of the current subjects in order to make room for civil procedure.  

The addition of civil procedure is of little surprise, for experimental civil procedure-like questions had appeared in recent past administrations of the MBEs.  But the justification for this change was not apparent. A review of the NCBE’s 2012 “job study” makes it clear why civil procedure was added. The heart of the “job study” was the results of a survey to recent law graduates with 1 to 3 years of law practice experience.  Survey participants were asked to rate the significance of specified legal tasks, knowledge areas, and skills/abilities. The laws of civil procedure had the highest rating (3.08 in a scale from 1 to 4) among all the knowledge areas, and 86% of the participants of the survey said that civil procedure was both significant and frequently used in their work.

NCBE had contracted a private consultant to do the  job study  “to determine what new lawyers do, and what knowledge, skills, and abilities newly licensed lawyers believe that they need to carry out their work.”  The job study and its survey was part of NCBE’s effort to establish “content validity” for future and current versions of the MBE–in other words,  is the content of the MBE reflective of the actual knowledge or skills that is required of newly admitted attorneys.

Survey participants consisted of law graduates with 1 to 3 years of practice experience, and the makeup of the participants is reflective of the racial composition of law graduates nationally (78% were non-minority while 22%, minority) and the majority (about 52%) worked in private law firms.  The survey methodology for the job study is sound, and law schools should think about using the NCBE’s survey results and survey methodology to evaluate their own curriculum. Take a look at the study full study and the summary of the results. However the jobs study was only a first step toward establishing the MBE’s test validity; the study does establish relevance of the knowledge areas being tested—in other words, content validity.

But the bigger issue is that the evidence for the criterion and construct validity of the MBE hasn’t been strong. These types of validity are essential qualities of a good test    Validity raises the following issues: How well does the MBE predict whether an individual will engage in  a specified level of legal thinking? What exactly are the constructs that the MBE seeks to measure and how well does it measure these constructs? Educational psychologist Sarah Bonner (2012) points out  “the existing criterion and construct-related validity evidence in support of the argument that the MBE is a measure of domain-specific legal thinking is not strong, suggesting a need for an inquiry into processes underlying performance.”  Bonner then proceeded to conduct a study on whether the MBE measures domain specific and domain general skills. But more about the study in a future blog.