Daubert In Antitrust

This table summarizes the results of hypothesis tests on all of the explanatory variables. Recall that a t-statistic of 2.0 is the rule-of-thumb cutoff for the hypothesis test to support the hypothesis that G-COMP affects P. An economist would say that Table 1A shows that the coefficient on G-COMP is not statistically significant because its t-statistic is 0.67, or that the coefficient is not differentiable from zero. In a Daubert hearing, the same information would be asserted as “the hypothesis test on [G-COMP] fails to show that it affected the price of Ampicillin when tested with a 5% rate of error.” On the other hand, the hypothesis that C does not affect the price of Ampicillin is rejected at an error rate of 5% because its t-statistic is above 2.0.

The other interesting thing to note about Table 1 is that it is divided into two parts. Table 1B exists because there is concern that costs may influence the price of Ampicillin in a more complicated way than is captured by the regression specification used to generate the results found in Table 1A. In particular, the concern is that production cost increases may affect price more when cost is already fairly high. As a result, the proper specification may include the square of cost as an explanatory variable. Table 1B shows results for a regression equation that includes this variable. This is an example of the model specification problem discussed in §13.24. Although the coefficients in the table change, the coefficient on N is very small and continues to be statistically insignificant (and fails the Daubert factor-one test at any rate of error that is used for such tests). In this case, both specifications present evidence that the number of generic houses has no statistically significant impact on the price of Ampicillin.

b. [§13.35] Daubert Analysis Of Ampicillin Example

Will the Ampicillin example developed above be admissible under Daubert? The summary of the investigation into the impact of the presence of generic houses on the price of Ampicillin shows that the testimony has been Daubert tested, and the test prescribed by factor one of Daubert evinces that, in the relevant range and at the levels of error that economists apply to their nonlitigation work, the number of generic houses bidding on Ampicillin contracts does not affect the price at which Ampicillin is sold. So far, the testimony seems admissible.

There are two important things to note about the preceding descrip¬tion. The first is that this is a typical application of regression in expert testimony, and the discussion is typical of how economists review their results. Therefore, this is not just an antitrust example; it is an example of how regression analysis is applied in the scientific community and of how regression is required to be applied forensically.

The second is that the basis of economists’ work is the hypothesis testing that some commentators believe cannot be widely done in the economic analysis that underlies expert testimony in antitrust litigation. Indeed, the most important point to be made about this table of coefficient estimates and t-statistics is that its purpose is to report on the testing of hypotheses of the sort that the Daubert court discusses and that scientifically unsophisticated commentators have dubbed as being impossible in most economics expert testimony work. See §§13.3, 13.17. Such a table of hypothesis tests is present in almost all credibly published empirical economics research, and usually the point of such a table is to show the results of the hypothesis tests that are the core of the economist’s work. Hypothesis testing, far from being impossible in most situations involving economic analysis, is indeed required in most such analysis. Suggestions to the contrary are almost always ill-informed.

Antitrust price-fixing damages are measured by the difference between the prices actually paid by the plaintiff and the market price that would have obtained in the absence of the defendants’ alleged conspiracy. Much of the application of regression analysis to antitrust is solving the data and econometric problems associated with estimating these but-for prices. See Finkelstein & Levenbach, Regression Estimates of Damages in Price-Fixing Cases, 46 Law & Contemp. Probs. 145 (1983) (illustrating this proposition in discussing regression estimation in cases including In re Corrugated Container Antitrust Litigation, 441 F.Supp. 921 (Jud.Pan.Mult.Lit. 1977), In re Chicken Antitrust Litigation, 560 F.Supp. 963 (N.D. Ga. 1980), and Ohio Valley Electric Corp. v. General Electric Co., 244 F.Supp. 914 (S.D. N.Y. 1965)). Finkelstein & Levenbach is almost a companion piece to Rubinfeld & Steiner, Quantitative Methods in Antitrust Litigation, 46 Law & Contemp. Probs. 69 (1983), and continues its sophisticated analysis.

The analogue to what could be called “but for” returns is the event study technique of §§13.25–13.32, in which regression techniques are used to estimate what the return on a security would have been “but for” the release of fraudulent information. Although there are surely very large differences in actual practice detail, the conceptual problem is the same: to estimate the market price of the security or the product in the absence of the fraud or the antitrust activity.

3. Post-Daubert Expert Testimony In Eleventh Circuit

a. [§13.36] Introduction

Because virtually all credible economics so routinely utilize the kind of scientific analysis that the Daubert Court articulated, there is little room to argue against requiring that economics-based expert testimony be tested for reliability and admissibility by the Daubert factors. This is evident in the two cases described in §13.37. Both cases were decided by the same Eleventh Circuit trial judge in the same month, and they are somewhat complementary. These opinions look carefully at econometric issues such as model selection, heteroskedasticity, and other regression model assumptions that inform the trial court’s reliability investigation required by Daubert. There is, however, a contrasting view that manifests itself in the case discussed in §13.38.

b. [§13.37] Prototype Antitrust Daubert Hearing

The court in In re Polypropylene Carpet Antitrust Litigation, 996 F.Supp. 18, 26 (N.D. Ga. 1997), held a hearing to determine whether an expert’s proposed methodology would “comport with the basic principles of econometric theory,” and agreed that the economist’s “multiple regression analysis is a scientific endeavor whose admissibility in court proceedings must be determined using the test set forth in Daubert.” The court analyzed the econometric model selection, including some of the possible different independent variables that could be included, then analyzed the hidden perils of trying a large number of variables in a regression and excluding the ones that do not seem to fit.

In Estate of Hill v. ConAgra Poultry Co., 1997 WL 538887 (N.D. Ga. 1997), the court articulated some of the particular shortcomings that can cause regression to fail a Daubert test. In this case, the defendants challenged the reliability of the plaintiff’s expert’s testimony on the ground that it failed to satisfy the regression assumptions, which the court calls “principles.” In particular, the defendants alleged that the plaintiff had failed to test whether the error term of the regression formula was independent of the included explanatory variables. This is the specification error problem of §13.24. The court investigated and determined that the defendants failed to show that the regression violated the independence assumption “to the point that the analysis fails to follow standard, acceptable econometric practices.” Estate of Hill at *5. The court reasoned that the evidence cited by the defendants did not demonstrate the failure to test for the independence assumption but rather the failure to satisfy the constant variance assumption (what economists call the heteroskedasticity problem), which the defendants neither defined nor established must be fulfilled to perform a valid regression analysis.

A subsequent section of the court’s analysis is interesting for the depth of its probing into the violation of the regression assumptions. It is somewhat lengthy, but it is this author’s belief that it reflects where sophisticated courts are in their application of Daubert to regression-based damages:

[T]he Court believes sufficient evidence exists to support a reasonable inference that Dr. Jackson properly tested the assumptions specified in Defendants’ argument . . . Dr. Jackson agrees that his model indeed contains heteroscedastic disturbances, and that he did not correct these disturbances before reaching the conclusions in his report. (Jackson Aff. ¶ 18-19.) Dr. Jackson cites two reasons for this course of action. First, Dr. Jackson states that, to compute t statistics for formulas that are based on very large sample sizes (as is the case here), a regression analysis may use an estimate of the standard errors for its coefficients in lieu of the classic least squares estimates for these coefficients. FN9 (Id. at ¶ 15.) This procedure, according to Dr. Jackson, eliminates the need to correct for heteroscedasticity. (Id.) Second, Dr. Jackson states that, because the disturbance creating the heteroscedasticity varies systematically, the presence of the heteroscedasticity alone can be interpreted as further support for a conclusion that Defendants manipulated the tare weights. (Id. ¶ 19.) In other words, the systematic nature of the variation suggests deliberate manipulation of the tare weights, according to Dr. Jackson. (Id.)

Defendants point to no evidence that contradicts Dr. Jackson’s assertions. Moreover, the Court is aware of several studies that have been admitted into evidence or published in journals in spite of the presence of heteroscedasticity in regression formulas. Cf. Denny v. Westfield State College, 669 F.Supp. 1146, 1149 (D.Mass.1987) (although expert admitted that hetero¬scedasticity was present in regression study to some degree, court concluded that “presence of heteroscedasticity itself [does not] detract from the validity” of the expert’s study); Roy F. Gilbert, Estimates of Earnings Growth Rates Based on Earnings Profiles, 4-Sum J. Legal Econ. 1, 14 n. 4 (1994) (“The results in this paper ignore the possibility of heteroscedasticity of the errors. . . . Even in the presence of the heteroscedasticity, however, least squares estimators are still consistent and unbiased.”)

For these reasons, the Court concludes Defendants’ evidence is insufficient to show that Dr. Jackson failed to test properly the assumptions underlying his regression analysis.

Estate of Hill, 1997 WL 538887 at *5–*6. Having dispatched the defendants’ allegation that the plaintiff’s regression residuals were heteroskedastic, the court turned its attention to the defendants’ allegation that the plaintiff’s regression did not include certain significant variables. The court cited Bazemore v. Friday, 478 U.S. 385, 400, 106 S.Ct. 3000, 92 L.Ed.2d 315 (1986), a pre-Daubert Supreme Court opinion, for the following proposition:

While the omission of variables from a regression analysis may render the analysis less probative than it otherwise might be, it can hardly be said, absent some other infirmity, that an analysis which accounts for the major factors “must be considered unacceptable as evidence.” . . . Normally, failure to include variables will affect the analysis’ probativeness, not its admissibility.

As the econometric discussion of §§13.17–13.24 demonstrates, when variables are improperly omitted from a regression study, the resulting estimators lose the desirable properties that make them scientifically reliable; lost with this scientific reliability is the estimate’s evidentiary reliability. Although econometrics is a powerful tool, when it is used other than in accord with the assumptive structure on which it is built, the answers that it produces are really answers to questions other than the ones that were thought to have been asked of it. A mis¬specified model must fail a Daubert examination because its tests are invalid, the error rates of those tests are unknowable, and, given these problems, the model itself is not generally accepted for peer-reviewed publication. Bazemore may accommodate model misspecification. Daubert does not.

Judge Posner provides an accessible but sophisticated discussion of specification error in Sheehan v. Daily Racing Form, Inc., 104 F.3d 940 (7th Cir. 1997), an employment discrimination case discussed in §13.42.

c. [§13.38] Another View Of Daubert And
Economics Expert Testimony

One commentator has observed that “[i]t is doubtful that much economic testimony would survive a strict and literal application of the Daubert factors. . . . [F]ew economic techniques of the ilk utilized in antitrust litigation could be ‘tested’ in the sense contemplated by Daubert, i.e., falsified.” Gavil, After Daubert: Discerning the Increasingly Fine Line Between the Admissibility and Sufficiency of Expert Testimony in Antitrust Litigation, 65 Antitrust L. J. 663, 673–674 (1997). The cases cited in §13.37 seem to refute this assertion, and it is interesting to contrast it with the views of the scientifically sophisticated commentators cited elsewhere in this chapter. See Rubinfeld & Steiner, Quantitative Methods in Antitrust Litigation, 46 Law & Contemp. Probs. 69, 70 (1983) (noting that “hypothesis testing is particularly useful for dealing with questions of whether an antitrust violation has occurred”); Rubinfeld, Econometrics in the Courtroom, 85 Colum.L.Rev. 1048, 1049 (1985) (noting that “the most prominent application of econometric methods” is “the use of significance levels for hypothesis testing” (or to use the less descriptive term employed in Daubert, falsification)); PROVING ANTITRUST DAMAGES: LEGAL AND ECONOMIC ISSUES 145 (ABA Section of Antitrust Law 1996) (beginning a full chapter discussion of the use of econometrics and statistical analysis in antitrust by stating that “[r]egression analysis is a statistical technique that
. . . can assist an antitrust plaintiff in proving both the fact and the amount of its injury,” before going on to develop the notions of hypothesis testing and falsifiability for application to antitrust damage calculations). This list of contra citations could continue almost without end.

Gavil himself notes that his “exposition of ‘falsifiability’ and ‘rate of error’ is somewhat simplistic,” Gavil, supra, at 675 n.48, and the resulting observations may have given way to the preponderant evidence on the role of testing in economics and econometrics, except that these “simplistic” arguments were made in the commentator’s discussion of the trial court’s analysis of expert testimony in City of Tuscaloosa v. Harcros Chemicals, Inc., 877 F.Supp. 1504 (N.D. Ala. 1995), aff’d in part, rev’d in part, vacated in part 158 F.3d 548. When City of Tuscaloosa went up to the Eleventh Circuit on appeal, an amicus brief that echoes these misconceptions was filed in support of the City of Tuscaloosa, and the court cited to the brief with approval in a footnote that has since been cited by another Eleventh Circuit trial court. Allapattah Services, Inc. v. Exxon Corp., 61 F.Supp.2d 1335 (S.D. Fla. 1999), aff’d 333 F.3d 1248. The connection may just be coincidence because the amici did not cite to Gavil, but, regardless, this commentary is apparently adopted in the Eleventh Circuit.

d. [§13.39] Testing And Economics Expert
Testimony: City of Tuscaloosa
v. Harcros Chemicals, Inc.

In City of Tuscaloosa v. Harcros Chemicals, Inc., 158 F.3d 548, 565 n.21 (11th Cir. 1999), the Eleventh Circuit generally reversed the district court’s exclusion of proffered antitrust expert testimony and stated that many of the problems in the district court’s opinion “might have been avoided had the district court simply held a Daubert hearing to allow the parties to clarify their positions, as well as the law, regarding the admissibility of these experts’ testimony.” With respect to admissibility of the experts’ testimony, the court of appeals affirmed that

[e]xpert testimony may be admitted into evidence if: (1) the expert is qualified to testify competently regarding the matters he intends to address; (2) the methodology by which the expert reaches his conclusions is sufficiently reliable as determined by the sort of inquiry mandated in Daubert; and (3) the testimony assists the trier of fact, through the application of scientific, technical, or specialized expertise, to understand the evidence or to determine a fact in issue.

Id. at 562. The court quoted Daubert regarding the flexibility of Fed.R.Evid. 702 and the variety of factors that may be relevant to the reliability of testimony. To the five Daubert factors, the court added a fifth: the “existence and maintenance of standards controlling [its] operation” (referring to the operation of the expert’s methods). City of Tuscaloosa, 158 F.3d at 563 n.16.

The court applied a novel version of flexibility and decided against applying the testing factor at all, citing amici, who “helpfully point out that, although ‘an important aspect of assessing scientific validity (and therefore evidentiary reliability) is the ability of other scientists to test or retest a proponent’s theory,’ not every scientific or technical methodology applied by expert witnesses is susceptible to such an analysis.” Id. at 566 n.25. The court reasoned that “[e]conomic or statistical analysis of markets alleged to be collusive, for instance, cannot readily be repeatedly tested, because each such case is widely different from other such cases and because such cases cannot be made the subject of repeated experiments.” Id. The court also stated that

[t]he proper inquiry regarding the reliability of the methodologies implemented by economic and statistical experts in this context is not whether other experts, faced with substantially similar facts, have repeatedly reached the same conclusions (because there will be few or no cases that have presented substantially similar facts). Instead, the proper inquiry is whether the techniques utilized by the experts are reliable in light of the factors (other than testability) identified in Daubert and in light of other factors bearing on the reliability of the methodologies.

Id. This is difficult to square with the analysis of sophisticated com¬mentators who propose that testing is the essence of Daubert. See 1 Faigman, Saks, Sanders & Cheng, MODERN SCIENTIFIC EVIDENCE: THE LAW AND SCIENCE OF EXPERT TESTIMONY §11:6 (Thomson/West 2007–2008 ed.) (“courts will find application of Daubert difficult if they treat testability as an optional factor. The other three factors all presuppose testability; in science, a non-testable hypothesis cannot have an error rate and is exceedingly unlikely to be published in a peer-reviewed journal and achieve general acceptance.”). The field of economics generally uses a scientific methodology, Confronting the New Challenges of Scientific Evidence, 108 Harv.L.Rev. 1481 (1995), and regression, the economist’s primary tool of analysis, is prototypical of scientific methodology, In re Polypropylene Carpet Antitrust Litigation, 996 F.Supp. 18, 26 (N.D. Ga. 1997) (citing to highly informed sources and observing that “multiple regression analysis is a scientific endeavor”).

The court of appeals reversed the district court’s exclusion of the expert’s testimony, writing that his “testimony is entirely within his competence as a statistician” and that his data compilations and “testimony regarding estimated damages, are the products of simple arithmetic and algebra and of multiple regression analysis, a methodology that is well-established as reliable.” City of Tuscaloosa, 158 F.3d at 565–566. The court did exclude some nonstatistical testimony from the statistician.

Perhaps the court’s observation about testing will be limited to “arithmetic and algebra,” which surely are admissible without a testing requirement, although, at least in the former case, there is a question of how the expert testimony will assist any but the most innumerate trier of fact. However, to apply such statements to an economist’s regression analysis would fly in the face of the overwhelming preponderance of the informed literature, which generally accepts hypothesis testing as the sine qua non of regression analysis.

Finally, as discussed above, the court added to the Daubert factors a fifth factor, the existence and maintenance of standards controlling the use of the expert’s methods. The basic standards for the use of regression (see §§13.17–13.24) virtually all relate to ensuring that the hypothesis tests based on the regression are competent. Thus, this criterion alone, if applied to the expert’s regression analysis, seems to mandate hypothesis testing because it mandates that hypothesis tests be done competently. It is somewhat unclear why the court would articulate such a criterion and then pronounce its intention to disregard it.

Testing is of such critical concern because it is very easy for an expert to weave jargon together into a compelling and convincing story that has no basis in fact. Hypothesis testing can reveal such errors. Section 13.42 considers an example of a superficially compelling expert’s story that can be debunked by proper controls on the testing of hypotheses.

e. [§13.40] Allapattah Services, Inc. v. Exxon

The court in Allapattah Services, Inc. v. Exxon, 61 F.Supp.2d 1335 (S.D. Fla. 1999), aff’d 333 F.3d 1248, held several days of Daubert hearings, citing to Daubert for the Eleventh Circuit’s five factors for admissibility of expert testimony (Daubert’s four plus the “existence of standards” factor discussed in §13.39). The court then cited to City of Tuscaloosa v. Harcros Chemicals, Inc., 158 F.3d 548 (11th Cir. 1999), for the Eleventh Circuit’s opinion on the proper way to apply those factors to an economist’s testimony. The court wrote that “the Eleventh Circuit, in a pre-Kumho [Tire Co.] case, discussed, in a footnote, the proper inquiry regarding the reliability of the methodologies implemented by economic and statistical experts in the Daubert context.” Allapattah Services, Inc., 61 F.Supp.2d at 1339 n.7. The referenced footnote, set out in full in Allapattah Services, Inc., is City of Tuscaloosa, 158 F.3d at 566 n.25, which “explains,” wrongly, why testing is not applicable to an economist’s expert testimony.

The proffered expert testimony in Allapattah Services did include regression testimony, and the court made multiple references to hypothesis testing in admitting the testimony of both experts. This seems like a standard Daubert hearing, admitting testimony that seems reasonable, with some attention paid to the expert’s hypothesis tests. It remains to be seen how courts in the Eleventh Circuit apply footnote 25 of City of Tuscaloosa, which directs attention away from Daubert’s testing factor.

G. Examples Of Statistical Proof Of
Employment Discrimination

1. [§13.41] In General

Statistical testing has been an element of proof in employment discrimination litigation since the appearance of the binomial model in Castaneda v. Partida, 430 U.S. 482, 97 S.Ct. 1272, 51 L.Ed.2d 498 (1977) (concerning race of jurors), and Hazelwood School District v. United States, 433 U.S. 299, 97 S.Ct. 2736, 53 L.Ed.2d 768 (1977) (concerning race of newly hired teachers). Since that time, statistical testing has been used extensively to compare the expected number of members of some protected group to the actual number of members of that protected group who have been hired, fired, or otherwise involved in significant employment actions. See, e.g., Hazelwood School District; Sheehan v. Daily Racing Form, Inc., 104 F.3d 940 (7th Cir. 1997). The important role of Daubert in employment class actions is discussed in §§13.49–13.55.

The persuasiveness of statistical proof in employment discrimi¬nation matters seems well established. See International Brotherhood of Teamsters v. United States, 431 U.S. 324, 97 S.Ct. 1843, 52 L.Ed.2d 396 (1977) (noting importance of role of statistical analyses in establishing prima facie case of racial discrimination in both jury selection and employment dis¬crimination cases). See also Bazemore v. Friday, 478 U.S. 385, 106 S.Ct. 3000, 92 L.Ed.2d 315 (1986). The notion of using regression analysis in employment discrimination litigation dates at least to 1975 and the publication of a student note that advocated the idea. See Note, Beyond the Prima Facie Case in Employment Discrimination Law: Statistical Proof and Rebuttal, 89 Harv.L.Rev. 387 (1975). The applicability of regression to the analysis of discrimination has been extensively discussed and documented since then. See Finkelstein, The Judicial Reception of Multiple Regression Studies in Race and Sex Discrimination Cases, 80 Colum.L.Rev. 737 (1980); Fienberg, The Increasing Sophistication of Statistical Assessments as Evidence in Discrimination Litigation, 77 J.Am.Stat.Ass’n 784 (1982); Note, Title VII, Multiple Linear Regression Models, and the Courts: An Analysis, 46 Law & Contemp. Probs. 283 (1983); Rubinfeld, Econometrics in the Courtroom, 85 Colum.L.Rev. 1048 (1985); Lempert, Statistics in the Courtroom: Building on Rubinfeld, 85 Colum.L.Rev. 1098 (1985).

The regression issues raised in §§13.17–13.24 and expanded on in §§13.25–13.40 inform the use of regression analysis in employment discrimination cases much as they do in antitrust and securities litigation. Regression models must be properly specified and must meet the basic regression assumptions in employment discrimination cases just as they must in other types of litigation. With respect to model specification (the term economists apply to include all relevant explanatory variables), it is appropriate to repeat here the conflict between Daubert — which says that regression must meet the standards that economists would apply to their nonlitigation research — and Bazemore — which seems to suggest that a regression analysis is not fatally flawed just because it leaves out a relevant variable. A Seventh Circuit opinion, Sheehan, highlights why an omitted variable is fatal to the ability of statistical analysis to inform the finder of fact, and the scholarly literature cited throughout this chapter discusses the undesirable results that occur when regression models are incorrectly specified.

2. [§13.42] Model Specification Error And Inadmissibility
Of Spurious Statistics: Sheehan v. Daily
Racing Form, Inc.

In Sheehan v. Daily Racing Form, Inc., 104 F.3d 940 (7th Cir. 1997), the plaintiff, Sheehan, was a well-regarded older employee of a racing newspaper company that used manual layout procedures to generate its papers. The defendant purchased a similar company that used computerized layout procedures and converted the existing operation to the computerized techniques.

In subsequent layoffs, Sheehan and most of the other older em¬ployees were terminated, while most of the younger workers were retained. Sheehan brought a lawsuit for age discrimination, and his expert proffered a statistical study that showed a strong correlation between age and the pattern of dismissal. The court excluded the expert’s testimony, noting that the expert had failed to “correct for any potential explanatory variables other than age.” Id. at 942. Especially important was the expert’s failure to consider computer skill as an explanatory variable in his analysis of terminations and that the omitted variable — computer skill — was correlated with age. As a result, if Daily Racing Form had terminated employees who lacked computer skills, and the older workers tended to lack computer skills, a study that omitted computer skills as an explanatory variable would find a correlation between dismissal and age, regardless of whether age was a criterion for dismissal. The opinion does not identify the type of statistical analysis employed, but this failure is an example of the class of misspecification problems discussed throughout this chapter. When a regression model omits explanatory variables that are correlated with included explanatory variables, the regression coefficients and their tests and error rate calculations lose the desirable properties that make them reliable. This is a prime example why regression that omits an important variable must be excluded by the gatekeeper. When important explanatory variables are omitted, the statistical analysis is unreliable. It not only misleads but also lacks the capacity to inform, so it cannot be shown to have probative value. In terms of Fed.R.Evid. 403, it has no probative value, but it surely has the capacity to misinform the jury; the latter danger must substantially outweigh the nonexistent former probative value. Analogous statements hold for nonregression statistical models.

3. [§13.43] Model Specification Error And Admissibility
Of Spurious Statistics: Obrey
v. Johnson

The notions discussed in §13.42 are not universally understood. In Obrey v. Johnson, 400 F.3d 691 (9th Cir. 2005), the plaintiff, Obrey, alleged that the defendant, the Secretary of the Navy, had engaged in a pattern or practice of discriminating against qualified candidates of Asian-Pacific ancestry in favor of Caucasian applicants for senior management positions at the Pearl Harbor Shipyard. The district court excluded the principal expert evidence supporting Obrey’s pattern or practice claim, judgment was entered against him, and he appealed, claiming that the district court abused its discretion in failing to admit a statistical report showing a correlation between race and promotion at the Shipyard. The Ninth Circuit reversed and remanded, saying that leaving some variables out of a statistical analysis goes to weight, not to admissibility. It is instructive to compare the court’s reasoning with the Seventh Circuit’s more sophisticated analysis by Judge Posner of a similar statistical study in Sheehan v. Daily Racing Form, Inc., 104 F.3d 940 (7th Cir. 1997).

In Obrey, the government argued that “the statistical analysis was inadmissible because it failed to account for the relative qualifications of the applicants being studied,” but the court ruled that

Obrey’s statistical evidence was not rendered irrelevant under Rule 402 simply because it failed to account for the relative qualifications of the applicant pool. . . . A statistical study may fall short of proving the plaintiff’s case, but still remain relevant to the issues in dispute. . . . Thus, objections to a study’s completeness generally go to “the weight, not the admissibility[,] of the statistical evidence.”

Id. at 694–695, quoting Mangold v. California Public Utilities Commission, 67 F.3d 1470, 1476 (9th Cir. 1995). That “Obrey’s statistical evidence was not rendered irrelevant under Rule 402 simply because it failed to account for the relative qualifications of the applicant pool,” id., will come as startling news to those analysts, scientists, and statisticians who imagine that the relative qualifications of the applicant pool have something to do with who gets job offers, which is to say, all of those who understand the basic statistical concepts that govern this type of work.

The court in Obrey cited to Kumho Tire Co. and correctly stated that “[t]he Rule 702 inquiry is a ‘flexible one’ whose ‘overarching subject is the scientific validity’” of the expert’s methods. Obrey, 400 F.3d at 696. The possibility of scientific validity is precluded when conclusions are based on statistical models that use the wrong set of explanatory variables.

In a final irony, the court in Obrey cited Metabolife International, Inc. v. Wornick, 264 F.3d 832, 843 (9th Cir. 2001), for the proposition that “[r]ather than disqualify the study because of ‘incompleteness’ . . . , the district court should examine the soundness of the methodology employed,” apparently missing the fact that, for this kind of incompleteness, the two are the same. Omission of such a variable in such a study renders the study methodologically unsound at the most elementary levels. It cannot be reliable and should not be admitted. Judge Posner catches this error, and the better rule is that of Sheehan.