|This heartfelt apology to Mr Loudermilk was issued in connection with correspondence with Dr Duncan Hine of IPS, please see email dated 20 January 2010|
From: David Moss
Something doesn't fit, in the story so far. I have worked out what it is.
A number of people, N, have asked to be removed from the circulation list. 1<N<3. I am adding them back pro tem so that they, too, can see the resolution.
In the talk he gave, and in our conversation afterwards, Mr Loudermilk struck me as dutiful, experienced, intelligently sceptical and confident. Confident enough to be open. Open about the limits to the reliability of biometrics.
Most people are open, it isn't noteworthy. But too many officials in the biometrics world are furtive and constipated, they don't release reports, they don't answer unambiguous questions unambiguously and they can't explain the logic of their decisions. In that world, Mr Loudermilk is a breath of fresh air.
"We are both of us in late youth", Mr Loudermilk said to me, after his talk. True. And in late youth, you don't generally make mistakes assessing people. Not on the broad assessment. So how come -- bear with me -- he is now denying what he said? It doesn't fit.
That's right. It doesn't fit. He's not denying what he said.
There are several authorities in the biometrics world, and each one has its own vocabulary.
The National Physical Laboratory (NPL) distinguishes "verification" from "identification". There is verification. Then there is an electrified, razor-wire fence. And on the other side of the fence, there is identification. The two should never be confused. The words must be used precisely. They are technical terms and there is no room for ambiguity.
Dr Tony Mansfield of NPL, and Marek Rejman-Greene, make the point in their 2003 report to the Home Office that "face recognition on its own is a long way from achieving the accuracy required for identifying one person in 50 million" (p.11). That's identification. And facial geometry isn't up to it. Or, at least, it wasn't in 2003.
But identification is the other side of the fence. It is millions of times harder than verification. So can facial recognition at least handle verification? Messrs Mansfield and Rejman-Greene voice some concerns: "In the Facial Recognition Vendor Test FRVT2000, with a longer timespan [1 to 2 months] between enrolment and verification attempts, and with less ideal illumination, performance is degraded somewhat (a false match rate of 1 in 1000 would result in a 6 in 10 false non-match rate!). Even under relatively good conditions, face recognition fails to approach the required performance" (p.15) "degraded somewhat", in this case, is another NPL technical term, meaning useless.
In 2005 we discovered that facial geometry still couldn't manage verification. In the UK Passport Service (UKPS) biometrics enrolment trial (para.188.8.131.52), the false non-match rate for facial geometry was 31% for the able-bodied participants and 52% for the disabled. And that was five minutes after registration, no need to wait for 1 to 2 months. "It wasn't a real trial", say the Identity & Passport Service (IPS), successors to UKPS. Really? Maybe not. But it may well have been an accurate indication of the way facial geometry performs in the field. Or fails to perform in the field. US-VISIT doesn't rely on facial geometry ...
Then in 2008 and 2009, we had a few clandestine reports of the trials of smart gates being carried out by the UK Border Agency (UKBA) here, here and here. There were so many false non-matches, according to the reports, that no-one could get through the gates until the operating point/threshold was turned down so low that the equipment couldn't distinguish between Osama bin Laden and Winona Ryder. At which point, there's not much point, is there?
UKBA have never released any sort of independently audited field trial report which would successfully scotch these rumours. Just like in Australia, where in 2008 "Customs refused to disclose the rates at which the [smart gates] system inaccurately identified people". This behaviour looks suspicious. The suspicion being, that facial geometry is as unreliable today as it has always been, slightly worse than an unbiased coin, it can't do verification, and that's why DHS don't use facial geometry in US-VISIT.
So (1) I speak NPL biometrics, where "verification" means one-to-one, and (2) there's a background of suspicion, so that when I hear Mr Loudermilk say "facial recognition would be the killer application of biometrics. Unfortunately the algorithms do not exist to give the highly reliable verification required", or some such, I think bingo!, I've got a scoop. Someone's telling the truth at last facial geometry algorithms do not exist for reliable one-to-one, the FBI say so.
But Mr Loudermilk speaks FBI biometrics. And he was actually talking about identification. He was saying the same as Tony Mansfield and Marek Rejman-Greene. He said "verification", but he meant "identification", in NPLspeak. With that in mind, did I understand the qualification "with populations of over 100 million"? I obviously heard it because I've noted it down, see attached, but it didn't mean anything to me, it was noise, because I've got the NPL lesson drummed into me that "verification" means one-to-one.
"Surely the mention of '100 million' should have given you pause", you may say? "After all, the population in a one-to-one verification is one." Maybe, but it didn't. Until yesterday (Wednesday 20). Until thinking that things don't fit.
So, my humble apologies, Mr Loudermilk, I retract, without reservation, now things fit again, and I look forward to your findings, when you come to assess the current maturity of facial geometry algorithms.
Will NIST turn out to be right? "FRVT 2006 and ICE 2006 are the first technology evaluations that allowed iris recognition, still face recognition, and 3D face recognition performance to be compared. The results on the multi-biometric dataset show that the performance for all three biometrics is comparable." Is facial geometry just as good as fingerprinting these days?
Maybe. But it pays to be sceptical. Biometrics are guilty until proven innocent. FRVT2006 is another chimera trial. Just like NIST's 2004 report for US-VISIT, when they predicted a ½% false non-match rate for flat print fingerprinting using just two prints. Once the figures started coming in from the real world, with false non-matches running at more like 19%, NIST had to start begging DHS to use 10 prints. A sceptic would predict the same thing this time. And he certainly wouldn't waste a fortune, like some countries I could name, on smart gates and ePassports in advance of the verdict.
It's a long way of saying sorry, isn't it. But this is a letter of apology. To Mr Loudermilk.