Volume 5, May 2005

www.psljournal.com/archives/all/haackpaper.html

 

 

                                                        Disentangling Daubert:

                               An Epistemological Study in Theory and Practice[1]  

 

                                                                  Susan Haack*

 

* Cooper Senior Scholar in Arts and Sciences, Professor of Philosophy, and Professor of Law at the University of Miami

 

 

Sometimes the word ["science"] degenerates into a vague honorific, synonymous with the advertiser's "reliable" or "guaranteed"... [JACQUES BARZUN][2]

 

In Frye (1923) the D.C. Court upheld the exclusion of testimony of the results of a then-new blood-pressure deception test on the grounds that novel scientific testimony "crosses the line between the experi­men­tal and the demonstrable," and so is admissi­ble, only if it is "sufficient­ly established to have gained general acceptance in the particular field to which it belongs."[3] Ignored for a decade, rarely cited for a quarter-century, over time the "Frye test" became increasingly influen­tial, until by the early 1980s it had been adopted by 29 states.

 

In 1975, however, newly-enacted Federal Rules of Evidence had set a seemingly less restrictive standard: the testimony of a qualified expert, including a scientific expert, is admissible provided it is relevant (unless it is excluded, under Rule 403, on grounds of unfair prejudice, waste of time, or confusing or misleading the jury). In Barefoot, a 1983 constitutional case, the Supreme Court affirmed that the rights of a Texas defendant were not violated by the jury's being allowed to hear psychiatric testimony of his future dangerousness at the sentencing hearing -- even though an amicus brief from the American Psychiatric Associa­tion reported that 2 out of 3 such predictions are mistaken. Writing for the majority, Justice White observed that state and federal rules of evidence "anticipate that relevant, unprivileged testimony should be admitted and its weight left to the fact-finder, who would have the benefit of cross-examination and contrary evidence by the opposing party."[4] Justice Blackmun wrote an angry dissent.

 

In 1991, amid increasing public concern that the tort system was getting out of hand, Peter Huber argued in his influential GALILEO'S REVENGE that under the Federal Rules worthless "junk science," which would have been excluded by the Frye test, was flooding the courts. In 1992 proposals to tighten up the Federal Rules were before Congress. In 1993 the Supreme Court issued its ruling in Daubert[5] -- the first case in its 204-year history where the central questions concerned the admissibili­ty of scientif­ic testimony. The Frye rule arose in a criminal case, and had for most of its history been cited in criminal cases; but Daubert was a tort action in which the trial court had relied on Frye in excluding the plaintiffs' experts' testimony that the morning-sickness drug Bendectin was teratogen­ic. So the Supreme Court was to determine whether the FRE had supersed­ed Frye, and in particular how Rule 702 was to be inter­preted.

 

Yes, Justice Blackmun wrote for the majority, the FRE had superseded Frye; but the Rules themselves require judges to screen proffered expert testimony not only for relevance, but also for reliability. In doing this courts must look, not to an expert's conclusions, but to his "methodolo­gy," to determine whether proffered evidence is really "scientific ... knowledge," and hence reliable. As to what that methodology is, citing an article by law professor Michael Green citing Karl Popper, and quoting an observa­tion of Carl Hempel's for good measure, the Daubert ruling suggests four factors that courts might use in assessing reliabil­ity: "falsifi­abili­ty," i.e., whether proffered evidence "can be and has been tested"; the known or potential error rate; peer review and publica­tion; and (in a nod to Frye), acceptance in the relevant community.[6]

 

In dissent, however, pointing out that the word "reliable" nowhere occurs in the text of Rule 702, Justice Rehnquist antici­pated difficulties over whether and if so how Daubert should be applied to non-scientific expert testimony; worried aloud that federal judges were being asked to become amateur scientists; and questioned the wisdom of his colleagu­es' readiness to get involved in philosophy of science. I think he was right to suspect that something was seriously amiss; in fact, what I shall have to say here might be read as an explora­tion, amplification, and partial defense of his reserva­tions about that philosophi­cal excursus.

 

                                                                        -- *** --

 

Apparently equating the question of whether expert testimony is reliable with the question of whether it is genuinely scientif­ic, taking for granted that there is some scientific "methodolo­gy" which, faithfully followed, guaran­tees reliable results, and casting about for a philosophy of science to fit this demanding bill, the Daubert Court settled on an unstable amalgam of Popper's and Hempel's very different approach­es -- neither of which, however, is suitable to the task at hand.

 

Popper describes his philosophy of science as "Falsification­ist," by contrast with the Verificationism of the Logical Positiv­ists, because his key theme is that scientific statements can never be shown conclu­sively to be true, but can sometimes be shown conclusively to be false. Hence his criterion of demarcation: to be genuinely scientific, a statement must be "testable" -- meaning, in Popper's mouth, "refutable" or "falsifiable," i.e., suscepti­ble to evidence that could potentially show it to be false (if it is false). Curiously, Popper acknowledged from the beginning that his criterion of demarcation is a "conven­tion"; and in 1959, in his Introduction to the English edition of THE LOGIC OF SCIENTIF­IC DISCOVERY, affirmed that scientific knowledge is continu­ous with common-sense knowledge.[7] Neverthe­less, his whole philoso­phy of science turns on his criterion of demarca­tion. Falsifi­ability is to discriminate real empirical science, such as Ein­stein's theory of relativity, from pre-scientific myths, from non-empirical disci­plines like pure mathemat­ics or metaphysics, from non-scientific disci­plines like history, and from such pseudo-sciences as Freud's and Adler's psychoanalytic theories and Marx's "scien­tific social­ism."[8] Falsifiability is also central to Popper's account of the method of science as "conjec­ture and refutation": making a bold, highly falsifi­able guess, testing it as severely as possible, and, if it is found to be false, giving it up and starting over rather than protect­ing it by ad hoc or "con­ventionalist" modifica­tions. (This readiness to accept falsification and eschew ad hoc strata­gems is Popper's "methodological criterion" of the genuinely scientific.)

 

Popper also describes his philosophy of science as "Deductiv­ist," by contrast with "Inductivism," whether in the strong, Baconian form that posits an inductive logic for arriving at hypotheses or in the weaker, Logical Positivist form that posits an inductive logic of confirma­tion. According to Popper, Hume showed long ago that induction is unjustifiable. But science doesn't need induction; the method of conjecture and refuta­tion requires only deductive logic -- specifi­cal­ly, modus tollens, the rule invoked when an observation­al result predicted by a theory fails.

 

Theories which have been tested but not yet falsified are "corrobo­rated," degree of corrobo­ration at a time depending on the number and severity of the tests passed. That a theory is corrobo­rat­ed, to however high a degree, doesn't show that it is true, or even probable; indeed. the degree of testability of a hypothesis is inversely related to its degree of logical probability.[9] Corrobora­tion is not a measure of verisim­il­itude, but at best an indicator of how the verisimili­tude of a theory appears, relative to other theories, at a time;[10] and that a theory is corroborated doesn't mean that it is rational to believe it. (It does mean, Popper writes, that it is rational to prefer the theory as the basis for practical action; not, however, that there are good reasons for thinking the theory will be successful in future -- there can be no good reasons for believing this.[11] So it seems that all this "concession" amounts to is that in deciding how to act we can do no better than go with theories we don't so far know to be false.)

 

The first problem with the Daubert Court's reliance on Popper is that applying his criterion of demarcation is no trivial matter; as Justice Rehnquist pointed out, observing wryly that, since he didn't really know what is meant by saying that a theory is "falsifiable," he doubted federal judges would, either.[12] Indeed, Popper himself doesn't seem quite sure how to apply his criterion. Sometimes, for example, he says that the theory of evolution is not falsifiable, and so is not science; at one point he suggests that "survival of the fittest" is a tautology, or "near-tautology," and elsewhere that evolution is really a historical theory, or perhaps metaphys­ics. Then he changes his mind: evolution is science, after all.[13] It's ironic; for Popper's criterion of demarcation had already found its way into the U.S. legal system, a decade before Daubert, in a 1982 first-amendment case: MacLean v. Arkansas Board of Educa­tion, where Michael Ruse's testimony that creation science is not science, by Popper's criterion, but the theory of evolution is, apparently persuaded Judge Overton.[14]

 

But there is an even more serious problem with the Daubert Court's reliance on Popper, of which Justice Rehnquist doesn't seem aware: Popper's philosophy of science is signally inappropriate to the Court's concern with reliability. When Popper describes his approach as "Critical Rational­ism," it is to emphasize that the rationality of the scientific enter­prise lies in the susceptibility of scientific theories to criticism, i.e., to testing, and poten­tially to falsification, not in their verifi­ability or confirmab­ility. True, early on Carnap trans­lated Popper's word "Bewahr­ung" by "confirmation"; and for a while, thinking the issue merely verbal, Popper let it go -- even, occasion­ally, using "confirm" himself. But in a footnote to the English edition of THE LOGIC OF SCIENTIFIC DISCOVERY he comments that this had been a bad mistake on his part, conveying the false impres­sion that a theory's having been corroborated means that it is probably true.[15] Except for the weak moments when he condoned Carnap's (mis)transla­tion, Popper insisted that corroboration must not be confused with confirma­tion. The degree of corroboration of a theory represents its past performance only, and "says not­hing what­ever about future perfor­mance, or about the 'reliabili­ty' of a theory"; even the best-tested theory "is not 'reliable'"[16] -- so scornful is Popper of the concept of reliabili­ty that he refuses even to use the word without putting it in precautionary scare quotes! Reiterating that he puts the emphasis "on negative argu­ments, such as negative instances or counter-examples, refutations, and attempted refuta­tions -- in short, criticism -- while the induct­ivist lays stress on 'positive instances', from which he draws 'non-demonstra­tive inferences', and which he hopes will guarantee the 'reliabil­ity' of the conclusions of these inferenc­es," Popper specifically identi­fies Hempel as representa­tive of those induct­ivists with whom he disagrees.[17]

 

Hempel is not, perhaps, the proto­typical inductivist: he describes the method of science as "hypothetico-deductive"; he affirms that scientific claims should be subject to empirical check or testing; and he doesn't follow Reichenbach and Carnap in explaining confirmation by appeal to the calculus of probabil­ities. Neverthe­less, Popper is surely right to see Hempel's approach as very significantly at odds with his own: Hempel is not centrally concerned with demarcating science; he questions the supposed asymmetry between verification and falsification, and argues that Popper's criterion "involves a very severe restriction of the possible forms of scientific hypotheses," e.g., in ruling out purely existential statements;[18] when he speaks of "testing" he envisages both disconfirmati­on and confirma­tion of a hypothe­sis; and one of his chief projects was to articulate the "logic of confirma­tion," i.e., of the support of general hypotheses by positive instances.

 

Apparently the Supreme Court hoped, by combining Hempel's account of confirma­tion with Popper's criterion of demarcation, to craft a crisp test to identify genuine, and hence reliable, science. But, though Hempel's philosophy of science is more positive than Popper's, it isn't much more help with the question of reliability. For one thing, the confirma­tion of general­izations by positive instances which preoccupies Hempel is just too simplified to apply to the enormous­ly complex congeries of epidemiologi­cal, toxicologi­cal, etc., etc., evidence at stake in a case like Daubert. For another, Hempel himself seems eventually to have concluded (rightly, I believe) that the "grue" paradox shows that confirma­tion isn't a purely syntactic or logical notion after all,[19] and late in life began to think that maybe Kuhn had been on the right track.[20]

 

But the most fundamental problem is that what Hempel offered was an account of support­iveness of evidence, or as he said, of "relative confirma­tion," the relation between observa­tion­al evidence and hypothesis, expressible as "E confirms H [to degree n]," or "H is confirmed [to degree n] by evidence E." This, as Hempel acknowl­edged, falls short of an account of "absolute confirma­tion," the warrant of a scientific claim, which would be expressed in non-relative terms, as "H is confirmed [to degree n], period." To discrimi­nate reliable testimony from unreliable, however, would require an account of the non-relative concept -- which Hempel doesn't supply.

 

                                                                        -- *** --

 

So, the Daubert Court mixes up its Hoppers and its Pempels; but isn't this just a slip, of merely scholarly interest? No: it is symptomat­ic of the serious misunder­standing of the place of the sciences within inquiry generally revealed by the Court's equation of "scien­tif­ic" and "reliable."

 

So successful have the natural sciences been that the words "science," "scientific," and "scientifically" are often used as generic terms of epistemological praise, meaning vaguely "strong, reliable, good" -- as, in television advertisements, actors in white coats urge viewers to get their clothes cleaner with new, "scientific," Wizzo. This honorific usage is unmistakably at work in the Daubert ruling; indeed, it seems to be implicit even in the way Justice Blackmun writes of "scientific ... knowledge," strategically excising a significant phrase from the reference in FRE 702 to "scientific or other technical knowledge," and apparent­ly signalling an expectation that a criterion of the genuinely scientific will also discrimi­nate reliable testimony from unreli­able.

 

If "scientific" is used honorifically, it is a tautology that "scientific" = "reliable"; but this tautology, obviously, is of no help to a judge trying to screen proffered scientific testimony. If "scien­tific" is used descriptive­ly, however, "scientific" and "reliable" come apart: for, obviously, physicists, chemists, biologists, medical scientists, etc., are sometimes incompetent, confused, self-deceived, dishonest, or simply mistaken, while histori­ans, detectives, investigative journalists, legal and literary scholars, plumbers, auto mechanics, etc., are sometimes good investigators. In short, not all, and not only, scientists are reliable inquir­ers; and not all, and not only, scientif­ic evidence is reliable. Nor is there a "scientific method" in the sense the Court assumed: no uniquely rational mode of inference or procedure of inquiry used by all scientists and only by scientists. Rather, as Einstein once put it, scientif­ic inquiry is "nothing but a refine­ment of our everyday think­ing,"[21] superim­posing on the inferences, desidera­ta, and constraints common to all serious investigation a vast variety of constantly evolving local ways and means of stretching the imagination, amplifying reasoning power, extending evidential reach, and stiffening respect for evidence.

 

Every kind of empirical inquiry, from the simplest everyday puzzling over the causes of delayed buses or spoiled food to the most complex investigations of detectives, of historians, of legal and literary scholars, and of scientists, involves making an informed guess about the explanation of some event or phenome­non, figuring out the consequences of its being true, and checking how well those consequenc­es stand up to evidence. This is the procedure of all scientists; but it is not the procedure only of scien­tists. Something like the "hypothetico-deductive meth­od," really is the core of all inquiry, scientific inquiry included. But it is not distinc­tive of scientif­ic inquiry; and the fact that scien­tists, like inquirers of every kind, proceed in this way tells us nothing substantive about whether or when their testimony is reliable.

 

The sciences have extended the senses with special­ized instruments; stretched the imagination with metaphors, analogies, and models; amplified reasoning power with numerals, the calculus, computers; and evolved a social organiza­tion that enables coopera­tion, competition, and evidence-sharing, allowing each scientist to take up his investiga­tion where others left off. Astronomers devise ever more sophisti­cated telescopes, chemists ever more sophisticat­ed techniques of analysis, medical scientists ever more sophisti­cated methods of imaging bodily states and processes, and so on; scientists work out what controls are needed to block a potential source of experimen­tal error, what statistical techniques to rule out a merely coincidental correla­tion, and so forth. But these scientific "helps" to inquiry are local and evolving, not used by all scien­tists.[22]

 

You may object that, since I have acknowledged that scien­tific inquiry is continuous with everyday empirical inquiry, I have in effect agreed with Popper that science is an extension of common sense. Indeed, I think science is well-described, in Gustav Bergmann's wonderful­ly evocative phrase, as the Long Arm of Common Sense. But the continuity is not between the content of scientific and of common-sense knowledge, but between the basic ways and means of everyday and of scientific inquiry; and it is precisely because of this continu­ity that the Popperian preoccupa­tion with the "problem of demarcation" is a distraction.

 

Or you may object that the Daubert Court's Popperian advice that courts ask whether proffered scientific testimony "can be and has been tested" surely is potentially helpful. This is true; but it is no real objection. "Check whether proffered testimony has been tested" is very good advice when a purported expert hasn't made even the most elementary effort to check how well his claims stand up to evidence: such as the knife-mark examiner in Ramirez,[23] who testified that he could infallibly identify this knife, to the exclusion of all other knives in the world, as having made the wound -- though no study had established the assumed uniqueness of individu­al knives, and his purported ability to make such infalli­ble identifi­ca­tions was untested. This is not, however, because fals­ifiability is the criterion of the scientific, but because any serious inquirer is required to seek out all the potentially available evidence, and to go where it leads, even if he would prefer to avoid, ignore, or play down information that pulls against what he hopes is true.

 

Yes, this is a requirement on scientists; as Darwin recog­nized when he wrote in his autobiog­raphy that he always made a point of recording recalcitrant examples and contrary arguments in a special notebook, to safeguard against his tendency conve­niently to forget negative evidence.[24] But it is no less a requirement on other inquirers, too; as we all realized a few years ago when a historian who announced that he had evidence that Marilyn Monroe had black­mailed President Kennedy turned out to have ignored the fact that the supposedly incriminat­ing letters were typed with correction ribbon, and that the address included a zip code -- when neither existed at the time the letters were purportedly written![25]

 

"Non-science" is an ample and diverse category, including the many human activities other than inquiry, the various forms of pseudo-inquiry, inquiry of a non-empirical character, and empirical inquiry of other kinds than the scientific; and of course there are plenty of mixed and borderline cases. The honorific use of "science" and its cognates tempts us -- like the Daubert Court -- to criticize poorly-conducted science as not really science at all; but "not scientific" is as unhelpful as generic epistemic criticism as "scientific" is as generic epistemic praise. The pejorative tone of the phrase "pseudo-science," which presumably refers to activities which purport to be science but aren't really, derives in part from its imputation of false pretenses, and in part from the favorable connotations of "scien­tific." But rather than sneering unhelpfully that this or that work is "pseudo-scientific," it is always better to specify what, exactly, is wrong with it: that it is not honestly or seriously conducted; that it rests on vague or flimsy assumptions -- assumptions there is no way to check, or for which there is no good evidence; that it seeks to impress with decorative or distracting mathemat­ical symbolism or elaborate-looking apparatus; that it fails to take essential pr­ecautions against experimental error; or whatever.

 

                                                                        -- *** --

 

So, the Daubert Court's philosophy of science was muddled; but haven't subsequent Supreme Court rulings cleared things up? Not exactly: it would be more accurate to say that in Joiner (1997) and Kumho (1999) the Supreme Court quietly backed away from Daubert's confused philosophy of science.[26] At any rate, those references to Hepper, Pompel, falsifi­ability etc., so prominent in Daubert, are conspicuous by their absence from Joiner and Kumho. But there are points of epistemological interest.

 

In Joiner there is a bit of a kerfuffle about "method­ology": Mr. Joiner's attorneys had argued that the lower court erred in excluding their proffered expert testimony because, instead of focusing exclusive­ly on their experts' methodolo­gy -- which, they maintain, was the very same "weight of evidence" methodol­ogy used by the other party's (G.E.'s) experts -- improper­ly concerned itself with the experts' conclu­sions. Apparent­ly anxious to sidestep this argument, the Joiner Court (with the exception of Justice Stevens) flatly denies the legitima­cy of the distinc­tion between methodology and conclu­sions. Opining that this is No Real Distinc­tion, the Court sounds like nothing so much as a conclave of medieval logicians; but given their citation to Paoli,[27] it seems likely that they didn't really intend to make a profound metaphysi­cal pro­nouncement, only to acknowledge, as Judge Becker had, that if an expert's conclusions are problematic enough, this alerts us to the possibility of some methodol­ogical defect.  

 

This focus on "method­olo­gy" -- an accordion concept expanded and contracted as the argument demands[28] -- obscured a much deeper epistemo­logical question. Mr. Joiner's attorneys proffered a collage of bits of informa­tion, none sufficient by itself to warrant the conclusion that exposure to PCBs promoted Mr. Joiner's cancer, but which, they argued, taken together gave strong support to that conclusion; G.E.'s attorneys replied, in effect, that piling up weak evidence can't magically trans­form it into strong evidence. In response, Mr. Joiner's attorneys refer to the EPA guidelines for assessing the combined weight of epidemio­logical, toxicological, etc., evidence. But no-one ever addresses the key question: is there a difference between a congeries of evidence so interre­lated that the whole really is greater than the sum of its parts, and a collection of unrelated and insignifi­cant bits of informa­tion, between true consilience and the "faggot falla­cy"[29] -- and if so, what is it?

 

There is a difference. Evidence of means, motive, and opportunity may interlock to support the claim that the defendant did it much more strongly than any of these pieces of evidence alone could do. Similarly, evidence of increased incidence of a disease among people exposed to a suspected substance may interlock with evidence that animals biologically similar to humans are harmed by exposure to that substance, and evidence indicating what chemical mechanism may be responsi­ble, to support the claim that this substance causes, promotes, or contributes to the disease much more strongly than any of these pieces of evidence alone could do. However, the interlocking will be less robust if, e.g., the animals are unlike humans in some relevant way, or if the mechanism postulated to cause damage is also present in other chemicals not found to be associated with an increased risk of disease, or, etc.    "Interlocking" is exactly the right word; for evidence is struc­tured like a crossword puzzle, with each claim, anchored by experiential evidence (the analogue of the clues), enmeshed in reasons (the analogue of completed intersecting entries). How reasonable a crossword entry is depends on how well it is supported by the clue and completed intersecting entries, how reason­able those other entries are, independent of this one, and how much of the crossword has been completed; similarly, how warranted a claim is depends on how supportive the evidence is, how secure the reasons are, independent of this claim its­elf, and how much of the relevant evidence the evidence includes.[30] Because of the ramifica­tion of reasons, the desirable kind of inter­lock­ing of evidence gestured at in Joiner is subtle and complex, not easily captured by any mechanical weighting of epidemio­logical data relative to animal studies or toxicologi­cal evidence. Nor, moreover -- as Justice Rehnquist already pointed out in Daubert -- can its quality readily be judged by someone who lacks the necessary background knowledge.       In Kumho the Supreme Court made a real epistemological step forward. In this products-liability case, focused on the proffered testimony of an expert on tire failure, the Court tried to sort out the problems with non-scientific experts which, as Justice Rehnquist had anticipated, soon arose in the wake of Daubert; and ruled that judges can't evade their gatekeep­ing duty on the grounds that proffered expert testimony is not science: the key word in FRE 702, after all, is "knowl­edge," not "scien­tif­ic." No longer fussing over demarcation, recognizing the gap between "scientif­ic" and "reli­able," in Kumho the Supreme Court acknowl­edges that what matters is whether proffered testimony is reliable, not whether it is scientific. Quite so.

 

Far from backing away from federal courts' gate­keeping responsi­bilities, however, the Joiner Court had affirmed that a judge's decision to allow or exclude scientific testimony, even though it may be outcome-determinative, is subject only to review for abuse of discretion, not to any more stringent standard; and the Kumho Court, pointing out that, depending on the nature of the expertise in question, the Daubert factors may or may not be appropriate, held that it is within judges' discretion to use any, all, or none of them. A year later, revised Federal Rules made explicit what according to Daubert had been implicit in Rule 702 all along: admissible expert testimony must be based on "suffi­cient" data, the product of "reliable" testimony "reliably" applied to the facts of the case. Federal judges now have large responsi­bilities and broad discre­tion in screening not only scientific testimony but expert testimo­ny generally -- but very little guidance about how to perform this difficult task.

 

In short, since Kumho's epistem­ological step forward, the other problem Justice Rehnquist worried about -- that judges generally lack the background knowledge which may be essential to a serious appraisal of the worth of scientific (or other technical) testimony -- is not merely unresolved, but more acute than ever. Bad epistemol­ogy can only get in the way; but better epistemolo­gy, unfortunately, can't by itself ensure smooth legal sail­ing.[31]