Via Mason
Porter, Danny Yee and others, I see
a news story which my kith are glossing along the lines of *a judge has
ruled that Bayes's Theorem does not apply in Britain*. Leave to one side
my "tolerate/hate" relationship with Bayesianism;
there are certainly cases, and ones of legal application at that, where Bayes's
rule amounts to a simple arithmetic statement about population counts, so it
would be very remarkable indeed if these were inadmissible in court. While I
enjoy disparaging the innumeracy of the legal profession as much as the next
mathematically-trained person, this seems like a distortion.

Let me quote from the Guardian story Mason linked to. (I can't find the actual opinion, at least not without more work than it's worth before lecture.) The story

begins with a convicted killer, "T", who took his case to the court of appeal in 2010. Among the evidence against him was a shoeprint from a pair of Nike trainers, which seemed to match a pair found at his home. While appeals often unmask shaky evidence, this was different. This time, a mathematical formula was thrown out of court. The footwear expert made what the judge believed were poor calculations about the likelihood of the match, compounded by a bad explanation of how he reached his opinion. The conviction was quashed.But more importantly, as far as mathematicians are concerned, the judge also ruled against using similar statistical analysis in the courts in future. ...

In the shoeprint murder case, for example, [applying Bayes's rule] meant figuring out the chance that the print at the crime scene came from the same pair of Nike trainers as those found at the suspect's house, given how common those kinds of shoes are, the size of the shoe, how the sole had been worn down and any damage to it. Between 1996 and 2006, for example, Nike distributed 786,000 pairs of trainers. This might suggest a match doesn't mean very much. But if you take into account that there are 1,200 different sole patterns of Nike trainers and around 42 million pairs of sports shoes sold every year, a matching pair becomes more significant.

The data needed to run these kinds of calculations, though, isn't always available. And this is where the expert in this case came under fire. The judge complained that he couldn't say exactly how many of one particular type of Nike trainer there are in the country. National sales figures for sports shoes are just rough estimates.

And so he decided that Bayes' theorem shouldn't again be used unless the underlying statistics are "firm". The decision could affect drug traces and fibre-matching from clothes, as well as footwear evidence, although not DNA.

What I take from this is that the judge was asking for reasons to believe
the numbers going *in* to Bayes's rule be accurate. This is, of course,
altogether the right reaction. Unless the component numbers in the calculation
--- the base rates and the likelihoods --- are right, the posterior probability
has
no value as evidence, because it has no connection whatsoever to the truth.
Unless those components are validated, the differences between a witness who
says "My posterior probability is 0.99" and one who says "I'm, like, really
sure" are:

- The former carries an air of precision, apt to impress juries and judges, which the latter lacks; while
- That air of precision is entirely spurious.

To reinforce just how wrong a simple-minded application of Bayes's rule can
go, I invite you to consider the saga of
the Phantom of
Heilbronn. The combined police forces of Europe spent years searching for
a criminal known from high-quality forensic evidence (DNA) left at more 40
crime scenes across a wide swathe of Europe. In the end, it turned out that
the reason all these different crime scenes turned up the same DNA, is that the
swabs used to collect the DNA from the scenes all came from the same factory,
and had been contaminated by DNA from a worker there. (Presumably the
contamination was accidental.) The case unraveled because while the common DNA
was female, it was recovered from a male corpse. If it had been recovered from
some unfortunate woman, it's very likely that this would now be regarded as a
closed case. No doubt we would then be hearing Bayesian calculations about the
odds against the suspect being anyone other than the Heilbronn serial killer
--- who, recall, did not exist. (In fact, it's instructive to do a
back-of-the-envelope version of the calculation, ignoring the contamination of
the swabs.) If you say "Well, of course those calculations are off, the
likelihood of the suspect matching a crime-scene in the *test* when the
suspect wasn't *really* there is all wrong", I can only reply,
"Exactly", and add that sensitivity analysis is no substitute for actually
understanding where and how the data arise. This is related, of course,
to the certainty of the Bayesian fortune-teller.

It is never pleasant to have claims to professional authority checked, so I
certainly *feel* where my learned British colleagues are coming from*.
But I have to conclude that, in so far as the judge said that Bayes's rule
"shouldn't ... be used unless the underlying statistics are 'firm'," he was
being entirely reasonable. He may, of course, have gone on to establish
unreasonable standards for what counts as "firm" statistics; the news stories
don't say. Unless that can be shown, however, the most damning verdict *we
statisticians* can return is (what else?) "not proven".

*Update*, later that day: A reader has kindly supplied me with a copy of the ruling. On a first scan, phrases like "Maths! Nasty, wicked, tricksy maths! We hates them, Precious, hates them forever!" are absent, but I will
try to read it and report back.

*Update*, 27 October: More than you would ever
want to know.

*: Let me remind them that one trick which is proven to help people use Bayes's rule rightly is to eschew talk of probabilities, and employ frequency formats. Since Gigerenzer and Hoffrage were able to get doctors — a tribe notorious for their mis-understanding and mis-use of inverse probability — to use Bayes's rule correctly this way, it would be rather surprising if lawyers weren't helped too.

Posted by crshalizi at October 03, 2011 08:50 | permanent link