Real World, Most Demanding
Biometric System Usage
or
“If you can get your system
to work here, it will work anywhere!”
PART II
Diplomacy
Disclaimer
Disclosure
1
Why our biometric application is“inverted”
from the typical biometric scenario?
Our liability from a false accept is
limited to <$100 per occurrence
Our liability from a false reject exceeds that of a false
accept due to our valuation of our guest relationships
We must process 300-350 presentations per hour, per
sensor for at least 3 hours per day on many days; so
throughput is paramount, but we still care about accuracy
2
What are our key operational
issues?
“Auto-enrollment” - enrollment must occur like
verification; no time for multiple presentations
Maximum device intuitiveness; no opportunity for training
or instruction (potential language barriers)
We define throughput as the average transaction time for
all transactions from insertion of ticket to time of next
insertion, including failures to acquire, repeat attempts, etc.
Device technology must be acceptable to the user
population
We felt we had set “the bar” for ourselves internally...
We tried to optimize…
Customizations:
Enclosure ruggedization for FL
environment
Platten material
improvements
Added second
alignment peg
“Christmas Tree” visual
feedback
3
Throughput improvements:
Around 9.0 seconds is where we had hoped to get to…
Seconds / Guest
35
30
30.4
25
19.0
20
16.0
15
14.7
11.5
Current Non-Biometric Turnstile Time (5.5)
10
5
0
Aug-95
Mar-96
May-96
Nov-96
Jan-97
Early Throughput Time History
WDW Hand Geometry System with trained
Users: EER ≈ 9%
FRR FAR
120.0%
100.0%
80.0%
Omni FRR
60.0%
FRR
FAR
40.0%
20.0%
10
0
15
0
20
0
25
0
30
0
35
0
40
0
45
0
50
0
55
0
60
0
65
0
70
0
75
0
80
0
85
0
90
0
15
00
20
00
25
00
30
00
35
00
>4
00
0
0
50
0.0%
Threshold
4
Testing in 2000
Our testing provided us with incomplete results due to a
lack of comprehensive interface standards at the time;
However, we did find out what we needed to know…
5
We still thought that finger-scanning technology offered
the fastest potential transaction possibilities
User
User throughput
throughput
System
Face
Fingerprint-Optical
Fingerprint-Chip
Hand
Iris
Vein
Voice
Transaction Time (Seconds)
Mean
15
9
19
10
12
18
12
Median
14
8
15
8
10
16
11
Minimum
10
2
9
4
4
11
10
Time includes
entry of PIN?
Excluded
Excluded
Excluded
Included
Included
Included
Excluded
User
User transaction
transaction times
times
Ref: Tony Mansfield, Et Al., “Biometric Product Testing”, Version 1.0
March 19, 2001, Center for Mathematics and Scientific Computing
(CMSC) of the National Physical Laboratory, U.K.
And this publication confirmed our instincts…
What have we done?
“Magic Your Way!”
6
“Magic Your Way!”
The
The environment
environment is
is the
the same;
same; however
however now….
now….
We
We require
require biometric
biometric verification
verification of
of all
all guests
guests age
age 10
10 or
or
older.
older.
This
This population
population still
still includes
includes elderly,
elderly, foreigners,
foreigners, adolescents,
adolescents,
disabled
disabled (and
(and we
we still
still allow
allow “opting-out”).
“opting-out”).
Our
Our guests
guests must
must still
still be
be able
able to
to present
present their
their biometric
biometric
feature
while
wearing
sunglasses,
hats,
suntan
feature while wearing sunglasses, hats, suntan lotion,
lotion, and
and
jewelry.
In
addition,
they
are
still
carrying
children,
jewelry. In addition, they are still carrying children, tote
tote
bags,
bags, food,
food, strollers,
strollers, etc.
etc.
Our
Our guests
guests are
are still
still often
often sweating,
sweating, tired,
tired, anxious,
anxious, and
and confused.
confused.
We approached this from a systems
engineering perspective…
•Appropriate
•Appropriate team
team members
members were
were put
put in
in place
place so
so that
that
decisions
decisions could
could be
be made
made quickly
quickly
•Project
•Project schedules
schedules were
were created
created and
and adhered
adhered to
to by
by both
both the
the
vendor
and
us
vendor and us
•Testing
•Testing requirements
requirements were
were established
established and
and milestones
milestones were
were
set
set in
in the
the schedule
schedule that
that allowed
allowed for
for “go/no-go”
“go/no-go” checkpoints
checkpoints
7
Our conceptual goals started with:
•Increase
•Increase our
our average
average throughput
throughput 25%
25% (get
(get back
back to
to 11.5
11.5 sec.)
sec.)
•Increase
•Increase our
our accuracy
accuracy to
to the
the neighborhood
neighborhood of
of 5%
5% FAR,
FAR, with
with
FTA
FTA as
as low
low as
as possible
possible <0.0%
<0.0% would
would be
be nice!>
nice!>
•Establish
•Establish the
the ability
ability to
to operate
operate in
in all
all existing
existing weather
weather
conditions,
including
all
sun
angles
conditions, including all sun angles
•Reduce
•Reduce our
our staffing
staffing requirement
requirement from
from 1:1
1:1 to
to 1:2
1:2 and
and allow
allow rerepositioning
to
enhance
the
greeting
process
positioning to enhance the greeting process
Some example RFP verbiage…
4.1.2.2.5
CCD sensors/imagers/cameras: CCD sensors shall be capable of capturing
the biometric feature in a single presentation requiring no movement or
adjustment of the biometric feature. i.e., rolling or swiping of a finger,
etc. State the matrix specifications of any CCD sensor being used and
pixel density
4.1.2.2.6
The biometric device shall be capable of acquiring 99.9% of all persons
presented to the device (FTA shall be lower then 0.1%). This parameter
must be valid for statically significant and diverse user populations.
Describe the threshold setting that this occurs at and the testing data that
supports this claim.
4.1.2.2.7
The biometric device shall either incorporate the processing algorithm onboard in a processing module or in a remote, server-side software module.
In either case, the vendor must provide proof of ownership of the
algorithm and licensing arrangements to allow the owner to store the
algorithm in any server-side software.
4.1.2.2.8
The biometric device is capable of transmitting threshold and score
information (if processing is done on-board the device) and receiving
threshold and score information (if done off-board the device).
4.1.2.2.9
The biometric device features visual and/or audible user feedback to assist
in positioning the user’s biometric feature for capture. Describe the type
of feedback provided.
4.1.2.2.10
The biometric device can capture a feature independent of feature
presentation angle or orientation. If there are any limitations on feature
presentation, list them here.
8
Degree of Compliance Legend
Degree Of
Compliance
Text
Definition
A
Available Commercially
Vendor’s commercially available
hardware and software meets the stated
requirement in its entirety.
B
Minor Modification
Vendor’s commercially available
hardware and software does not fully meet
the stated requirement, however with
minor programming changes to the source
code or cosmetic changes to the hardware,
vendor’s commercially available hardware
and software will be modified to meet the
stated requirement.
C
Substantial Modification
Vendor’s commercially available
hardware and software does not meet the
stated requirement. However the
hardware and software may be modified
(with changes/modifications to the
hardware at the board or system level
and/or the software source code) to meet
the stated requirement.
D
Development
Vendor’s hardware and software does not
meet the stated requirement. Vendor will
either be required develop new hardware
and/or software source code or
extensively modify the existing hardware
and/or software to meet the stated
requirement.
Some the Core Team’s questions:
•Park
•Park Operations:
Operations: Can
Can this
this system
system be
be successfully
successfully operated
operated and
and
owned
owned on
on aa day-to-day
day-to-day basis?
basis?
•I.T.
•I.T. Ticketing
Ticketing and
and Data
Data Communications:
Communications: Can
Can this
this system
system be
be
successfully
successfully administrated
administrated and
and maintained?
maintained? Can
Can we
we retrofit
retrofit this
this
new
new system
system into
into our
our existing
existing back-end?
back-end?
•Facility
•Facility Asset
Asset Management:
Management: Are
Are we
we staying
staying on
on schedule
schedule and
and
within
within budget?
budget? Are
Are we
we coordinating
coordinating our
our efforts
efforts and
and communicating
communicating
effectively?
effectively?
•Imagineering:
•Imagineering: Will
Will this
this be
be an
an acceptable
acceptable “part
“part of
of the
the Show”?
Show”?
What
What will
will this
this look
look like?
like? What
What signage
signage will
will be
be required?
required?
•Design
•Design &
& Engineering:
Engineering: Is
Is this
this going
going to
to work?
work? IfIf so,
so, for
for how
how long?
long?
9
So, we held the vendor responsible for all of the
technical development of the sensor package
itself, and we took ownership of the sensor
enclosure and ergonomic interface elements:
•Enclosure
•Enclosure mock-ups
mock-ups would
would be
be created
created for
for visual
visual acceptability
acceptability
•Ergonomic
•Ergonomic interface
interface mock-ups
mock-ups would
would be
be created
created and
and tested
tested to
to
compare
compare effectiveness
effectiveness
•Testing
•Testing requirements
requirements would
would be
be established
established and
and milestones
milestones would
would
be
be set
set in
in the
the schedule
schedule that
that allowed
allowed for
for “go/no-go”
“go/no-go” checkpoints
checkpoints
So, let’s take a look at some examples of the intensity
of the efforts that were involved…
In May of 2005, the vendor delivered ten (10) alpha
prototypes that were retrofitted into commercially
available optical TIR scanners
We used these to conduct a Phase I - Technology test, as well
as to study the human factors of what people would do when
instructions were minimal
10
The
The Phase
Phase II Technology
Technology testing
testing looked
looked very
very promising
promising and
and itit met
met
our
our Phase
Phase II goals:
goals:
Results:
“The test was successful and we recommended continuing with the next steps. We
believe that by utilizing the different spectral images, we obtain a very usable final
image, which improves our results dramatically over standard fingerprint type units.”
It was easy to tell from the beginning in Phase I
that we were already getting better results by
having multiple image planes to choose from…
11
In
In Sept.
Sept. of 2005, we received the first beta test unit
(it
(it was
was nicknamed
nicknamed OCAM:
OCAM: One
One CAMera)
CAMera)
and
and itit supposedly
supposedly came
came with
with full
full sunlight
sunlight capabilities…
capabilities…
The
The alpha
alpha units
units performed
performed like
like normal
normal TIR
TIR sensors
sensors against
against sunlight…
sunlight…
44
44 Ft/Cd
Ft/Cd
133
133 Ft/Cd
Ft/Cd
2000
2000 Ft/Cd
Ft/Cd
OCCAM
OCCAM performed
performed well
well at
at up
up to
to 6000
6000 Ft/Cd…and
Ft/Cd…and we
we agreed
agreed to
to proceed.
proceed.
12
Early
Early enclosure
enclosure mock-ups
mock-ups made
made from
from foam-core
foam-core to
to
match
match what
what we
we might
might end
end up
up with
with for
for an
an package…
package…
13
We also thought a lot about how this system
was going to be operated and maintained for
the next few years:
•How
•How would
would maintenance
maintenance be
be performed?
performed?
•Could
•Could we
we fully
fully mitigate
mitigate weather
weather and
and sun
sun impact
impact ?
?
•Could
•Could we
we implement
implement field
field upgradeable
upgradeable firmware
firmware (remote
(remote
flashing
flashing via
via network)?
network)?
So, let’s take a look at some examples of the
efforts that were involved here…
14
We
We did
did some
some “quiet”,
“quiet”, early
early testing
testing in
in Nov.
Nov. 2005
2005
with
guests
and
a
mocked-up
enclosure
with guests and a mocked-up enclosure and
and some
some
ergonomic
ergonomic interface
interface mock-ups
mock-ups to
to see
see ifif we
we could
could
learn
learn enough
enough about
about what
what worked
worked and
and what
what
didn’t…
didn’t…
Even during the brief OCCAM Beta Test, we were
able to simply gesture and have untrained users
successfully get authorized at a much faster
pace…
15
16
17
18
Preliminary data from internal Beta tests with untrained
staff as users: ERR < 3%
•We
•We worried
worried about
about enclosure
enclosure
details
details and
and survivability
survivability
•Were
•Were there
there places
places where
where
water
could
enter?
water could enter?
•How
•How many
many times
times could
could aa
mechanism
be
operated?
mechanism be operated?
19
20
So what was the end result of the project?
•Still
•Still aa one-to-one
one-to-one verification
verification system
system
•Verification
•Verification is
is still
still anonymous;
anonymous; we
we do
do not
not know
know the
the identity
identity
of
of the
the passholder
passholder at
at the
the turnstile
turnstile
•Our
•Our enrollment
enrollment is
is still
still transparent
transparent -- single
single feature
feature
presentation
with
truly
minimal
operator
presentation with truly minimal operator intervention
intervention or
or
feedback
feedback (now
(now operating
operating at
at 1:2
1:2 staffing
staffing on
on most
most days)
days)
•We
•We are
are now
now operating
operating at
at approx.
approx. 11.5
11.5 seconds
seconds average
average
per
transaction,
which
was
our
goal…and
per transaction, which was our goal…and our
our accuracy
accuracy is
is
is
rejection rate
rate of
of 1111is in
in the
the 5-6%
5-6% FAR
FAR range
range with
with aa total
total rejection
12%
12% and
and well
well below
below 1.0%
1.0% FTA
FTA
21
Don’t ask me – ask them…
49,000,000 x 70% =
34,000,000
st full year of new sensor operation we’ve
In our 1st
processed over twice the number of people than were
processed in the previous 10 years of operations!
<each sensor has had well over 200,000 touches so far…>
What have we done since
implementation?
We put them in at Hong Kong Disneyland!
But what’s wrong with this picture?
22
In conclusion:
Our lessons learned:
There is still no magic bullet device for all applications,
but we did build a better “mousetrap”
Implementing an enterprise-sized biometric system
upgrade with a proprietary data interface is both costly
and painful to accomplish
A change in biometric technology does not solve all
your problems, it exchanges them for other problems
Questions?
23