Senior Biometrics Advisor
Home Office Scientific Development Branch (HOSDB)
c/o Direct Communications Unit
Mass consumer biometrics
Our correspondence began with my email to you dated 21 August 2008 . There followed a number of communications , ,  and . Then there was your letter dated 14 October 2008 , for which thank you, and my first answer dated 2 January 2009 .
You were kind enough to make several points in your letter. Let's start with the last, which is pretty well where I started six years ago:
I trust that these comments resolve your concerns. I am encouraged by your
interest in the potential of biometric technologies in the safeguarding
of people's identity and the privilege of
For six years I have sent proposals to UKPA (then UKPS, now IPS), NCIS (now SOCA), the Metropolitan Police and others, for a voluntary alternative  to the National Identity Scheme (NIS). The opening statement of my alternative proposal is: "The main objectives of the Home Office's ID cards scheme are beyond criticism". The methods chosen by the Home Office to achieve them are doomed to failure but there's nothing wrong with the objectives.
My early proposals took it for granted that mass consumer biometrics work. It took years for me to correct that mistake. But gradually, year on year, the evidence piled up , the 2004 UKPS biometrics enrolment trial was the final straw, and now it is embarrassing to imagine how naïve my faith in mass consumer biometrics must have looked to some of the recipients of those proposals.
To this mass consumer biometrics apostate, no, your comments do not resolve his concerns. You say:
You have also referred to the results of the Biometrics Enrolment Trial. Its objective, as set out in section 1.1.2, was 'to test the processes and record customer experience and attitude during the recording and verification of facial, iris and fingerprint biometrics, rather than test or develop the biometric technology itself. It was not a technology trial.' Performance figures have been published as part of the documentation, but it is misleading to use this data from an enrolment trial, especially in discussions on the verification performance to be expected from fully engineered modern systems.
Whether or not it was the objective to test the reliability of the biometrics used in the UKPS enrolment trial, it was tested. Tested and reported, at length, over 300 pages, in detail, in May 2005, by Atos Origin , the consultancy which managed the trial on behalf of UKPS.
Under the heading Verification Success Rates, we learn that:
· 31% of the able-bodied participants in the trial could not have their identity verified by the facial recognition technology being used. For the disabled participants, that figure rose to 52%. As far as the latter are concerned, everyone would do better to toss a coin than try to rely on biometrics based on facial recognition.
19% of the able-bodied,
and 20% of the disabled, could not have their identity verified
by their fingerprints. It follows that 19 or 20% of people could
have trouble proving their right to work 
4% of the able-bodied
and 9% of the disabled could not have their identity verified by
their irisprints. It is commonly held that there would be about
50 million ID cardholders at any one time if the
How will these people who have
no verifiable biometric identity survive in the world of the
Some people could suffer an even worse fate. Under the heading Enrolment Success Rates, we learn that:
10% of the able-bodied
could not register their irisprints in the first place and, for
the disabled, the figure was 39%. These people would not even exist
You say that it was not a reliability trial, it was a usability trial. Not just you, of course. It is the Home Office party line. But if that is the case, if it was a usability trial and not a reliability trial, then certain questions arise:
· Why are the verification and enrolment results above included at section 1.2 of the Management Summary in the 2005 Atos Origin report? There is something self-contradictory going on here.
· Why are they headed "Key Findings"? How can they be key findings if they formed no part of the objectives of the trial?
· Why are there hundreds more pages of performance figures in the report?
These are not rhetorical questions. I have a hypothesis to put to you by way of an answer – that usability and reliability cannot be separated. You cannot test one without testing the other. Usability, in the Atos Origin report, is discussed for 300 pages in terms of reliability. They stand or fall together. If the reliability results are misleading, so are the usability results. To say that the UKPS trial was valuable for its findings on usability but irrelevant for its findings on reliability is neither true nor false, it is meaningless. It is nonsense.
There are other instances which seem to support that hypothesis. The UKPS 2003-04 Annual Report and Accounts , for example:
A trial to determine the most appropriate biometric for future needs began in late 2003–04 (continuing into 2004–05), and will involve 10,000 volunteers providing facial, iris pattern and fingerprint data.
This trial will help inform the UKPS and partner organisations on implementing, if appropriate, a second biometric to passport documents. A second biometric will help deter and detect duplicate applications, and strengthen the link between passport/ID cards and the individual holder.
Of course UKPS must take people's experience of enrolment into account. But it's not as though UKPS would choose a biometric that people find congenial but which just happens also to be unreliable. It must be a reliability trial as well as a usability trial, otherwise it can't "determine the most appropriate biometric".
And it makes no sense to separate
enrolment into the
A further instance of my hypothesis occurred when the House of Commons Home Affairs Committee  asked why the start of the UKPS biometrics enrolment trial was delayed for so long (4 May 2004):
Q672 Mr Prosser: There has been some discussion this afternoon about the passport pilot project. That was some months late in starting, we think that it will have less coverage than was anticipated and will be over a shorter period. In fact, three attempts by this Committee to meet with that group have been postponed because of all of those delays. What is that an indication of? Is it an indication of the complexities or difficulties of the technology behind the ID card or something else?
Mr Blunkett: It is an indication, firstly, that there has been a change from previous methodology which was not as successful as the new programme. Secondly, that it is important to get it right rather than to get it quickly, which is what we have explored already. Thirdly, that it is the number in the pilot, not the time. I do not know where this differential of five or six months has come from. We want 10,000 people and, as was being indicated a moment ago by you, a lot of people are queuing up to be part of this right across the country, which is very encouraging, including parts of the country where there is not a pilot. We are enthused by that. It is the 10,000 pilot we are interested in rather than whether it takes five or six months. I hope very much that we can learn very rapidly from it. The whole point of the pilot, the whole point of this process, is to learn the lessons, and I do not mean just go through the motions but actually learn what it is, and the development partner and commercial consultation and the scrutiny by Parliament is all part of that same process.
Q673 Mr Prosser: How thorough will the testing be? For instance, will there be attempts to deceive the system or to feed in fake and false information?
Mr Blunkett: Yes. We already did that with the previous technology and one of the reasons why it is important to get this pilot right is that there has been enhancement having learnt the lessons from that. I think worldwide it has been known, and it is one of the reasons why it has taken until now to move to such a scheme, that it was possible in the past to be able to defraud the equipment. Do either of you want to say a word about this because you have been on top of it?
Katherine Courtney: I think it is important to say that while the pilot itself is not really about testing the robustness and scalability of the particular biometric technologies that are being deployed, it is about studying the enrolment process and the customer experience and being able to validate some of the assumptions that we have built into the business case around the time that it takes to enrol and the customer acceptability. I am pleased to say on the limited sample so far that is bearing out our assumptions. I am quite pleased about that. We will be attempting to re-register duplicate identities even with this technology, which is not being tested as the technology that we would expect to take forward, to gain some lessons from this experience about how robust this particular configuration might be. Also we are considering the security risks around the enrolment process, ie the environment in which people enrol, the process itself, how people arrive, how they go through having their fingerprints recorded, their irises scanned etc to ensure things are built into the system like the inability for somebody to replace themselves with somebody else half way through the process so that the application is actually reflecting more than one individual, that sort of thing. We are building those things into the pilot and gaining a lot of experience from that.
Q674 Mr Prosser: Finally from me, how will you make your final assessment? How will you say, "Yes, this works well enough to go ahead"? Will any of that assessment be done independently?
Mr Blunkett: Yes. Do you want to add to that?
Mr Browne: The contract for this particular piece of work is a number of what are called deliverable milestones. Payment will be made on the contract on the basis of people reaching those milestones and there will be an independent element in the assessment of those results. Tony Mansfield from the National Physical Laboratory is an independent assessor of progress and outcome of the trial.
For all that the trial was supposed
to be about usability only, it seems to have been delayed because
some of the elements weren't reliable enough – the "previous
methodology" was "not as successful as the new programme",
i.e. it was not as reliable. And it seems to have been designed
to check reliability as well as usability – the trial was to be
used to test the resilience of the system against spoofing, for
example, which would make the
Hypothetically (remember) you are reduced to talking nonsense if you try to separate usability from reliability. Not just you but Katherine Courtney, too: "We will be attempting to re-register duplicate identities even with this technology, which is not being tested as the technology that we would expect to take forward, to gain some lessons from this experience about how robust this particular configuration might be". Run that past me again ...
What is the point of testing the usability of a technology that you won't use? Why weren't "fully engineered modern systems", as you call them, used in the trial? Or were they? After all, why would anyone pass up the opportunity of a useful trial?
To be consistent, presumably the Home Office believe that some eminent experts who should know better are deliberately misleading the public. That seems unlikely:
"To be honest, I think it is a possibility that eventually we will conclude it isn't good enough or that the current systems we're using aren't good enough for a large scale public domain application such as an ID card," she said.
The findings of a biometrics enrolment trial conducted by Atos Origin on behalf of the UK Passport Service (UKPS) show that biometric technologies are still not foolproof and suggest that large-scale issuance of biometric identity and travel documents would inevitably run into some glitches ...
Facial recognition was the least successful identification technology ...
Among other things, further trials are needed, specifically targeted towards those disabled groups that have experienced enrolment difficulties due to environment design, biometric device design, or to specific group problems – for example, black participants and participants aged over 59 had lower iris enrolment success rates ...
A report released by the European Commission on 30 March 2005 warned that – on the technological side – there is currently a lack of independent empirical data. This means that there is an urgent need to conduct large-scale field trials to ensure the successful deployment of biometric systems.
Was the UKPS biometrics enrolment trial a technology trial, or wasn't it? The question was still being asked when the House of Commons Science and Technology Committee published their report  on identity card technologies in July 2006. They found that even a Home Office Minister had used the results as though it was a technology trial:
88 ... the Home Office has selectively used evidence from the biometrics enrolment trial to support its assertions. We believe that the Home Office has been inconsistent regarding the status of this trial and this has caused confusion in relation to the significance of the evidence gathered about biometric technologies.
Amidst the inconsistency and confusion
and unanswered questions, one point is clear. If it wasn't a technology
trial, as the Home Office contend, then people are being asked to
spend billions of pounds on the
91 … we are surprised by the Home Office's unscientific approach and suggest that rather than collating figures merely to provide information regarding performance, the Home Office admits that it cannot release details until it has completed trials.
93 We are surprised and concerned that the Home Office has already chosen the biometrics that it intends to use before finishing the process of gathering evidence. Given that the Identity Cards Act does not specify the biometrics to be used, we encourage the Home Office to be flexible about biometrics and to act on evidence rather than preference. We seek assurance that if there is no evidence that any particular biometric technology will enhance the overall performance of the system it will not be used.
95 We note the lack of explicit commitment from the Home Office to trialling the ICT solution and strongly recommend that it take advice from the ICT Assurance Committee on trialling. We seek an assurance that time pressure and political demands will not make the Home Office forgo a trial period or change the purpose of the scheme.
96 In written evidence the Home Office said it was not necessary to embark on publicly funded scientific research to improve the capabilities of biometrics. This claim was subsequently denied in oral evidence and the identity card team asserted that research was being undertaken into fingerprint biometric performance. Katherine Courtney said “I would not say that we have not commissioned research. We have commissioned research. We have a piece of research that the Home Office is funding right now into fingerprint biometric performance”. We regret the confusion at the Home Office regarding the research that it is funding and what research it requires.
Is there any reason to believe
that the proposed mass consumer biometrics are
reliable enough to make the
The answer given to the Committee to that last question is a May 2004 report  produced by the US National Institute of Standards and Technology (NIST):
This report discusses the flat-to-flat matching performance of the US-VISIT fingerprint matching system. Both one-to-many matching used to detect duplicate visa enrollments and one-to-one matching used to verify the identity of the visa holder are discussed. With the proper selection of an operating point, the one-to-many accuracy for a two-finger comparison against a database of 6,000,000 subjects is 95% with a false match rate of 0.08%. Using two fingers, the one-to-one matching accuracy is 99.5% with a false accept rate of 0.1% ...
4.3 Trading FRR for FAR
As thresholds are increased, TAR decreases (which is bad) while FAR decreases (which is good). Given the definitions of TAR and FAR used in this section, the false reject Rate (FRR) is defined as (1 – TAR), so as IDENT thresholds are increased, FRR increases while FAR decreases. This means there is an inverse relationship, or trade-off, between the achievable levels of these two types of system errors. This means there is an inverse relationship, or trade-off, between the achievable levels of these two types of system errors.
To help understand the difference between FRR and FAR, we can use the example of a watch list application. A false reject occurs when a person known to be on the watch list presents his fingerprints to the biometric system but is not correctly identified. There are two conditions under which this can happen. Either the system remains silent and does not return any candidates, or the candidates returned do not include the person’s mate. In the first case, when the system returns no candidates, the person will pass on through primary inspection. In the second case, the person is redirected to secondary inspection while the candidates reported are reviewed ...
NIST predicted in their report that the fingerprinting technology used for verification in US-VISIT  would be 99.5% accurate, i.e. there would be a false reject rate of 0.5%. Were they right?
The Office of the Inspector General (OIG) at the US Department of Justice reported, in December 2004 , that 118,000 people pass through US-VISIT each day, they are all subject to primary inspection, and 22,350 of them fail and are referred to secondary inspection. 22,350 is 19% of 118,000. Not 0.5%. NIST were wrong.
I know how they must have felt. Embarrassed by their earlier proposal after the OIG report was published. And let down by the technology they had previously trusted.
Of the 22,350 people detained every day for
secondary inspection, just 1,811 are subsequently refused entry
Mass consumer biometrics are a blunderbuss, a million miles away from the precision of the forensic use of DNA, for example, and the proper, traditional fingerprints taken by police experts. With traditional fingerprinting, if there is ever any doubt, independent experts are flown in from all over the world to resolve it. That is the mark of a proper science. Think of the Shirley McKie and David Asbury case .
No-one is going to fly experts in to adjudicate on a technology with a 19% error rate. We're not dealing with biometrics here so much as glorified photocopies of people's fingers.
19% is not only the false reject rate for the fingerprinting technology used in US-VISIT, it is also the false reject rate recorded by the UKPS biometrics enrolment trial, for able-bodied participants, using fingerprints to verify their identity. It is possible that, far from being misleading, the UKPS trial figures are an accurate measure of the reliability of mass consumer fingerprinting.
What about biometrics based on facial recognition? Does US-VISIT experience similar failure rates – 31% for able-bodied visitors and 52% for disabled visitors?
No. Because US-VISIT doesn't use biometrics based on facial recognition. They're too unreliable. According to NIST:
3. VERIFICATION PERFORMANCE
3.3 Comparison with Face Recognition
... Even under controlled illumination, which is not used in US-VISIT, the error rate of face is 50 times higher than the two-fingerprint results discussed here. If the case of uncontrolled illumination is considered, this factor would be 250 ...
There is no equivalent in US-VISIT
with which to compare the 31%/52% able-bodied/disabled false reject
rate for biometrics based on facial recognition. But perhaps there
is an equivalent closer to home, at the facial recognition trial
currently being conducted at
"Up until the point of the official launch, it was rejecting 30 per cent of those who tried to get through it," the UKBA worker said.
"We believe they had to recalibrate it – essentially make it easier to get through the system."
That allegation by an unnamed UKBA worker is not evidence, of course, just an allegation. But has it been denied? Do UKBA care what allegations are made about them? Has any proof been volunteered that the allegation is false? Has the equipment been re-calibrated? Has the false reject rate decreased? Has the false accept rate increased as a result? More unanswered questions ...
In the feasibility study  conducted by you and Tony Mansfield, you say:
52 (c) In the BWG [
"6 in 10" is a 60% false non-match rate (or false reject rate) and well worthy of the exclamation mark you awarded it. The UKPS biometrics enrolment trial results for facial recognition seem to be in line with other studies.
I put it to you, Marek, that the
reliability figures reported for the UKPS biometrics enrolment trial
are not misleading. What is misleading is to suggest that today's
mass consumer biometrics are reliable enough
to achieve the goals for the
In fact, the
And it's not just the
eBorders , the plan to protect the
1.2 ... We believe a new doctrine
is demanded, where controls begin offshore and where we use information,
intelligence and identity systems to allow scrutiny at key checkpoints
on the journey to and from the
1.3 Managing identity is fundamental
to delivering this new approach. Using biometric technology we can
permanently link people to a unique identity. We can check this
against other records that can reveal, for example, if someone poses
a security risk, has previously committed crimes in the
So does the Cabinet Office's plan for transformational government  (November 2005):
39 (7) Identity Management: government will create an holistic approach to identity management, based on a suite of identity management solutions that enable the public and private sectors to manage risk and provide cost-effective services trusted by customers and stakeholders. These will rationalise electronic gateways and citizen and business record numbers. They will converge towards biometric identity cards and the National Identity Register. This approach will also consider the practical and legal issues of making wider use of the national insurance number to index citizen records as a transition path towards an identity card.
If the biometrics chosen are unreliable,
it is not just the
And that is what I fear is happening. There will be fireworks  ...
You assert that biometric technology has improved since 2004:
Regarding one of the principal comments you make - that there is no reason to believe that face recognition 'technology now works better than it did five years ago' - I believe you are aware of the conclusions of the Face Recognition Vendor Test 2006  [FRVT2006], which demonstrated that there has been considerable improvement. These results confirmed the feasibility study's observation that this technology could be applied successfully in a one-to-one (verification) mode. These tests were conducted by a consortium led by the US National Institute of Standards and Technology.
Another NIST report. Another 56 pages of impressive-looking academic research. And very possibly just as wrong as their other report, where they predicted 0.5% and the outcome was 19%. More computer-based competitions whose results are never replicated in the field. Nothing in the outside world is "demonstrated" or "confirmed". More disappointment in store. It's not evidence that the technology has improved. As you say yourself:
Of course, it is only once this technology is tested in a specific context, with optimised algorithms and reference images, that we can be sure that such improvements translate into capabilities which can be deployed in a working system ...
So we agree. Up to now, IPS have had no reason to believe that mass consumer biometrics are capable of doing the job. They were just hoping ... they had their fingers crossed ... something will turn up – it's just not a responsible or scientific or businesslike approach, is it?
You and Tony Mansfield recommended
in your feasibility study that the Home Office set up a national
network of 2,000 biometric registration centres. In the event, they
have set up a network of only 69 centres (
NIST's FRVT2006 progress report concerns biometrics based on facial recognition and irisprints only. Presumably we may assume that biometrics based on fingerprints have made no progress and that the false reject rate is stuck on the 19% mark.
Mass consumer flat print fingerprinting is not like traditional rolled fingerprinting. Rolled prints are admissible as evidence in court. Flat prints aren't. They're too unreliable. Arguably, to give both technologies the same name, "fingerprinting", is a confidence trick .
Is that the explanation for the
inability of IPS to produce its invitations to tender (ITTs)
to begin the procurement process for biometric systems for the
You go on to say:
... Operational testing, e.g. in
You cannot see it from the typography, Marek, and so I have tell you that several days have passed since the previous full stop and the start of this sentence. If the Australian and Portuguese trials provide compelling evidence that computerised facial recognition now works reliably, then something very important has happened. Perhaps that faith in mass consumer biometrics from six years ago can be revived. You do not provide any references in your letter. Please provide some if you can. I have looked for evidence. I have not found any. Just this , from Australian IT (8 April 2008):
After several delays the
federal Government has given the go-ahead to the $62 million SmartGate
project and the biometric passport technology will be rolled out
nationally for Australian and
The system incorporates facial recognition technology, which matches a live image of the traveller against a digitised photo stored on a microchip embedded in the passport. If the images match, the traveller is cleared through the control point.
If not, the traveller will be referred to a customs officer for further examination ...
The ability of the system to accurately identify people was questioned in the early stages of the pilot of the technology in Sydney and Melbourne, which had false rejection rates of 6 to 8 per cent ...
Customs refused to disclose the rates at which the system inaccurately identified people ...
When the system was
being tested in
If the Australians and/or the
Portuguese have compelling evidence that mass consumer biometrics
are reliable enough to deliver the benefits sought from the
The Australians may have reservations
about publishing the evidence. There are no such constraints in
… Safeguards, openness, proportionality and common sense.
For the public to have confidence that we will protect them and protect their rights, it is our responsibility as a government to ensure that these standards apply even as technology evolves.
... I am equally clear that we have to measure these efforts [robust powers to tackle crime and disorder] against our standards for safeguards, openness, proportionality and common sense.
... I will continue to put safeguards and openness, a sense of proportion and above all common sense, at the heart of everything we do.
Openness is the order of the day
here in the
By the same token, HOSDB treated my 21 August 2008 email to you as a Freedom of Information request. And today's letter to you, as Head of the Home Office Biometrics Centre of Expertise (Science and Technology Committee report, para.4a), is an open letter, also available at http://dematerialisedid.com/bcsl/Marek%20Rejman-Greene%20002.html.
I would also like to take the opportunity to clear up one area of confusion regarding verification and identification accuracies. As you have observed, para 58 in our report does indeed state that 'For one-to-one identity verification, it is only necessary to use a single finger, a single iris or face recognition'. This is confirmed by the matching error rates as noted earlier in para 52, in the section on identification accuracy, and where we comment that such figures also show that face recognition on its own would indeed not be appropriate for 'one-to-many' matching in very large scale systems.
I would contend that under Tony Mansfield's expert, patient, tolerant, accidental and intermittent tutelage, I have actually got quite good at distinguishing verification and identification. And I've noticed something – you've had to give up on identification.
The politicians and the civil service are still talking about unique, fixed electronic identities :
Gordon Brown  (January 2008): biometrics "will make it possible to securely link an individual to a unique identity".
The Home Office  (December 2008): "ID cards will securely lock foreign nationals to one identity and help businesses crack down on illegal working".
But the mass consumer biometrics
community make it plain that identity, for them, is discretionary.
Set the threshold on the biometric equipment to (x1,y1)
and a match takes place, you are
That much has been clear for years. Statements such as those above by the Prime Minister and the Home Office are simply false.
Daugman said that even if the error rate was as low as one in a million, the 10 to the power of 15 comparisons needed to verify the IDs of 45 million people would result in one billion false matches.
He told silicon.com: "The use of fingerprints will cause deduplication to drown in false matches" ...
Even with a tiny error rate, one in a million, it is not practical to prove that each person is recorded on the NIR once and only once. Not with the millions of people involved. The mass consumer technology cannot deliver what the politicians and the civil service are promising.
And no-one in the mass consumer
biometrics community is pretending that it can. The terms of reference
for your feasibility study were to see if cheap, off-the-shelf biometric
systems could establish identity in a population of 50 million people.
They can't. When I suggested to Tony Mansfield that the important
test is to see if identity can be established in the world population
of 6 billion+, ... he wasn't very impressed.
But that's what we really need, isn't it, if the
There are still plenty of claims made for verification. No serious mass consumer biometrics expert has offered identification for years.
Marek, I made many of the points above to John Reid  when he was Home Secretary. Twice . And to Gordon Brown . It's a strange thing, but the message doesn't seem to be getting through. Politicians just carry on wasting resources on biometrics. That is understandable if I am wrong. But if I am right, then it is a scandal. Which is it, do you think? Is it a scandal?
 http://dematerialisedid.com/PDFs/Strategic_Action_Plan.pdf, Annex 1