New Study Shows How Often Juries Get It Wrong

In a set of 271 cases from four areas, juries gave wrong verdicts in at least one out of eight cases, according to "Estimating the Accuracy of Jury Verdicts," a paper by a Northwestern University statistician that is being published in the July issue of Journal of Empirical Legal Studies.

"Contrary to popular belief, this study strongly suggests that DNA or other after-the-fact evidence is not the only way to know how often jury verdicts are correct," said Bruce Spencer, the study's author, professor of statistics and faculty fellow at the Institute for Policy Research at Northwestern. "Based on findings from a limited sample, I am optimistic that larger, carefully designed statistical studies would have much to tell us about the accuracy of jury verdicts."

Spencer cautions that the numerical findings should not be generalized to broader sets of cases, for which additional study would be needed, but the study strongly suggests that jury verdicts can be studied statistically. If such studies were conducted on a large scale, they might lead to better understanding of the prevalence of incorrect verdicts -- false convictions and false acquittals, he said.

To conduct the study, Spencer employed a replication analysis of jury verdicts, comparing decisions of actual jurors with decisions of judges who were hearing the cases they were deciding. In other words, as a jury was deliberating about a particular verdict, its judge filled out a questionnaire giving what he or she believed to be the correct verdict.

"Consider the analogy to sample surveys, where sampling error is estimated even though the true value may never be known," Spencer said. "The key is replication. To assess the accuracy of jury verdicts, we need a second opinion of what the verdict should be."

By comparing agreement rates of judges and juries over time and across jurisdictions, and even across types of cases, Spencer's statistical analysis could give insights into the comparative accuracy of verdicts in different sets of cases.

For his analysis, Spencer utilized a study with a special set of cases that was recently conducted in the United States by the National Center for State Courts (NCSC). An earlier study was conducted by Kalven and Zeisel in the 1950s.

The agreement rate was 77 percent in the NCSC study and 80 percent for the earlier study. Allowing for chance agreement, the agreement rates were not high. (With chance agreement, for example, if two people tossed coins heads or tails independently to see if they matched, one would expect agreement, heads-heads or tails-tails, 50 percent of the time.)

To obtain a numerical estimate of jury accuracy, some assumptions were made, as is the case for virtually any statistical analysis of social groups or programs. A key assumption of Spencer's study is that, on average, the judge's verdict is at least as likely to be correct as the jury's verdict.

Without assumptions, a 77 percent agreement rate could reflect 100 percent accuracy by the judge and 77 percent accuracy by the jury, or 100 percent accuracy by the jury and 77 percent accuracy by the judge, or 88 percent accuracy by both, or even 50 percent accuracy by both if they often agreed on the incorrect verdict.

With the assumption of the Spencer analysis that judges are at least as accurate as jurors after completion of all testimony, we can get an estimate of jury accuracy that is likely to be higher than the actual accuracy. Thus, the 77 percent agreement rate means that juries are accurate up to 87 percent of the time or less, or reach an incorrect verdict in at least one out of eight cases.

"Some of the errors are incorrect acquittals, where the defendant goes free, and some are incorrect convictions," Spencer said. "As a society can we be satisfied if 10 percent of convictions are incorrect" Can we be satisfied knowing that innocent people go to jail for many years for wrongful convictions""

Spencer envisions that statistical studies would complement nationwide efforts to expose wrongful convictions, including the work of the Center on Wrongful Convictions at Northwestern University School of Law. The center's work in exposing flaws in Illinois' capital punishment system played a significant role in former Gov. George Ryan's decision to commute Illinois death row inmates' pending executions to sentences of life in prison.

The NCSC study is not representative of a larger set of cases, Spencer stressed. He hopes that nationally representative studies will be carried out in the future.

Using additional assumptions and statistical models, the extent of wrongful convictions and wrongful acquittals also can be estimated, according to Spencer. The methods also could be extended to estimate accuracy of verdicts in non-jury trials.

While the studies on verdict accuracy will not tell whether the verdict for a particular case was correct or not, they will help assess what proportion of verdicts are correct.

"If you were on trial and not guilty, you certainly would want the jury to do the right thing," Spencer said. "Now, subject to these assumptions, studies could be employed to give us an idea of how often that happens."

A technical report is available at http://www.northwestern.edu/ipr/publications/papers/2006/wp0605.pdf .