Wednesday, August 25, 2010

Statistical Analysis of the Twombly Era

Guest post by Stephanie Thomas of MCG.

In his August 17 post, Phil Miles discussed an interesting data point reported by Joseph Seiner: approximately 80% of summary judgment motions in employment discrimination cases decided by district courts were granted. This finding was reported in Mr. Seiner’s paper “The Trouble with Twombly: A Proposed Pleading Standard for Employment Discrimination Cases”.

Mr. Seiner, with the help of the Federal Judicial Center, identified 3,983 employment discrimination cases during fiscal year 2006 in which a defendant filed a motion for summary judgment that was decided by the district court. Among those 3,893 cases, summary judgment was granted – either in whole or in part – in 3,219 cases:

This is certainly an interesting data point drawn from anecdotal evidence. Unfortunately, it’s just that - anecdotal evidence. The “80% statistic” is a descriptive statistic, but not a statistical analysis. It doesn’t formally address any underlying expectations about a summary judgment motion being granted. Given the information available here, it’s not possible to do a comparative statistical analysis. There’s nothing to compare this 80% result to.

But for those of you who were looking forward to a statistical analysis – all is not lost. There is an interesting statistical analysis contained in Mr. Seiner’s paper: the impact of Twombly on the dismissal rates in Title VII employment discrimination cases. This analysis looks at whether we can infer that Twombly - in effect – applies a “heightened pleading standard” to these kinds of cases.

Mr. Seiner examines the decisions of district courts on motions to dismiss brought for failure to state a claim. Two data sets were compared against one another:

A. Conley Decisions: decisions in the year right before Twombly that relied on the Supreme Court’s decision in Conley;

B. Twombly Decisions: decisions in the year immediately after Twombly.

Mr. Seiner notes that his study didn’t look at absolute dismissal rates. Rather, he was looking at whether judges relying on Twombly were more likely to dismiss a Title VII case than those judges relying on Conley. In other words, Mr. Seiner is looking at percentages.

Using the Westlaw database, 264 “Conley” opinions and 268 “Twombly” opinions were identified, for a total of 532 decisions. For each decision, the following information was recorded:

1. The name of the case and the citation;
2. The jurisdiction where the case was filed;
3. Whether the motion to dismiss was:
      a. Granted;
      b. Denied;
      c. Granted-in-part.

Upon reviewing the decisions, 73 “Conley” opinions and 63 “Twombly” opinions were excluded from the study for reasons such as addressing a motion for summary judgment rather than a motion to dismiss, they were brought under a statute other than Title VII, or involved only magistrate recommendations to a district court.

The final sample of 396 opinions – 191 “Conley” opinions and 205 “Twombly” opinions – were then compared:

Among the decisions studied, 54.5% of those relying on Conley were granted in whole, while 57.1% of those relying on Twombly were granted in whole. We see a similar pattern among decisions in which the motions were denied: 24.6% of those relying on Conley versus 22.4% of those relying on Twombly.

Mr. Seiner also notes that “the rate at which district courts have been granting motions to dismiss in Title VII decisions has begun to rise when the court relies on Twombly.” To demonstrate this, he restricts his “Twombly” decisions to those occurring during the most recent six months of the study period.

Regarding this six month study, Mr. Seiner states:
In the short time following Twombly, then, the data already demonstrate a higher percentage of decisions that grant a motion to dismiss in the Title VII context when the courts rely on the new decision.
Before you get too excited about this and conclude that Twombly is a game-changer, you should know that the differences found by Mr. Seiner in his six-month study are not statistically significant (he mentions this on page 22 of his paper).

I replicated Mr. Seiner’s analysis and compared the likelihood of “granted” (defined as granted either in part or in whole) under Conley and Twombly. I found that there was no statistically significant difference in the likelihood of “granted” in either the full study period or when using the last six months of data:

Comparison of Conley and Twombly – entire Twombly period: Z = 0.39

Comparison of Conley and Twombly – six-month Twombly period: Z = 0.97

As you know, Hazelwood School District v. US (1977) held that a disparity of at least 2 or 3 units of standard deviation is statistically significant. In the Conley / Twombly comparisons, we see only 0.39 units and 0.97 units of standard deviation, respectively.

This means that we cannot conclude that the likelihood of a motion to dismiss being granted is any different under Twombly than under Conley. Because the difference is not statistically significant, we cannot rule out the possibility that the difference we see is attributable to random chance. From a purely statistical perspective, the effect of Twombly on motions to dismiss in employment discrimination cases is the same as that of Conley.

Stephanie Thomas is the author of The Proactive Employer blog, and hosts the podcast of the same name. Ms. Thomas is also on Twitter @ProactiveStats.