As thousands of law graduates take the bar exam, women graduates may wonder whether an important section of the bar exam—the Multistate Bar Exam (MBE)–is stacked against them. Evidence of this is tucked away in an obscure report of the NY Board of Bar Examiners (2006, Oct. 4), the agency that is responsible for administering the licensing exam. The numbers below show men outperform women on the MBEs[1] by about 27 points while women do slightly better –10 points–than men on the Essay[2] portion of the NY State Bar exam.
Score Means, Standard Deviations and Standard Errors
Domestic-Educated First Time Takers, Females and Males, July 2005**
| Gender | MBE Scaled Score x 5 | Essay Scaled Score | NYMC Scaled Score | Total NY Bar Score* |
| FemaleMean | 713.28 | 734.08 | 719.75 | 724.34 |
| (SD )(n-3,264, SEM = 1.2) | (72.53) | (69.21) | (76.85) | (63.74) |
| MaleMean | 740.04 | 724.12 | 724.62 | 730.54 |
| (SD)(n=3,299), SEM 1.30 | (73.96) | (70.80) | (77.84) | (64.47) |
| Total Mean | 726.69 | 729.07 | 722.20 | 727.44 |
| Total (SD) | (73.96) | (70.80) | (77.84) | (64.47) |
*Total score is computed by taking the weighted average of the adjusted MBE scaled score (40%), Essay score (50%) and NYMC (10%). New York Board of Bar Examiners (2006, Oct.4).
**New York Board of Bar Examiners (2006, Oct.4).
On the MBE, men scored better than women by 26.76 adjusted points. This 27-point difference between men and women seems big and pervasive. The gap is consistent across ethnic/racial groups, first-time versus repeat takers as well as between domestically and foreign schooled graduates. But a closer analysis of the numbers tells us that the gap isn’t that alarming.
Presumably the 27-point gap meets the statistical test of significance, which tells us whether two averages from two different groups are statistically different or not. Typically a .05 probability is set and that means that there is a 5% or less chance that the two means are really different due to pure dumb luck. Put differently, 5% or less represents the acceptable level of risk of being wrong when we conclude that two averages from two groups are different.
But the fact that two means are statistically different doesn’t tell us whether that difference is big enough to really have practical significance. To paraphrase Cummins (2014), it’s wrong-headed to make too much of the statistical significance because it is largely a product of the enhanced sensitivity that comes from with a sample size as large as the one here—over 6,000 people. Instead of relying on a test of statistical significance, we need an analysis of “effect size.”
You can use a formula to determine effect size, and there’s a handy website calculator for that. Rounded off, the effect size of sex is .37. This means that sex has a moderate effect on MBE performance. The other statistic –effect size r squared =.03 –tells us that 3% of the difference between men and women is due to sex.
In other words, sex has a small relationship to performance on the MBE; in fact, the contribution of sex to performance on the MBE is only 3%. That means that the remaining 97% of the variability in performance is due to factors other than sex. Thus, it would be hard to claim that the MBE is biased against women examinees.
[1] The Multistate Bar Exam is a 6-hour test that requires examinees to answer 200 multiple questions on the application of law several doctrinal areas.
[2] The essay portion tests the ability of examinees to identify the applicable New York state law and write essayd on the application of that law to fact patterns.
Sam Sue, Copyright, 2014.
eddoctorinhouse 