TEST AND VALIDATION PROCEDURES FOR
INSTALLATIONS CARRIED OUT WITH SURVISION
EQUIPMENT
1
Introduction..................................................................................................................................... 4
2
Testing-related issues and problems............................................................................................... 5
3
4
5
6
2.1
Issues ........................................................................................................................................5
2.2
Problems...................................................................................................................................5
Terminology and ratios evaluated................................................................................................... 7
3.1
Definition of ratios....................................................................................................................7
3.2
Eligible and non-eligible number plates ...................................................................................7
3.3
Some examples of non-eligible plates......................................................................................9
3.4
Performance guidelines..........................................................................................................11
Tools provided by SURVISION ....................................................................................................... 12
4.1
The VSS ...................................................................................................................................12
4.2
The PVT Recorder ...................................................................................................................12
4.3
The PVT Player ........................................................................................................................12
Test methodology.......................................................................................................................... 13
5.1
Selection of the site and of the camera .................................................................................13
5.2
Choice of test period ..............................................................................................................13
5.3
Filming of the sequences........................................................................................................13
5.4
Analysis of sequences.............................................................................................................13
5.5
Test report: global performance evaluation ..........................................................................14
Training and support ..................................................................................................................... 15
1
INTRODUCTION
This document aims to provide integrators and end customers with the methodological elements to
enable them to evaluate the performance of SURVISION solutions in the most objective and precise
way.
This evaluation is based on carrying out and subsequently analyzing video sequences which record
the passage of vehicles in a real situation.
Apart from the methodological elements, these evaluations are based on the use of tools (provided
by SURVISION) for camera adjustment, and for recording and analysis of video sequences. This guide
introduces these tools without going into detail about their use. There is a specific detailed user
guide for each of these tools.
The tests can be carried out by the customer or by a third-party using the tools, methodology and
training provided by SURVISION. SURVISION can also carry out these tests on behalf of the customer,
and provide him with all of the necessary verification elements.
2
TESTING-RELATED ISSUES AND PROBLEMS
2.1
Issues
In some cases, the projects implemented with the SURVISION technology are subject to an
evaluation stage which allows the end customer to verify the performance of the cameras provided
by SURVISION. This performance is expressed in a synthetic way as being the proportion, expressed
in percentage, of vehicles detected and/or correctly identified on a given sample.
This requirement of performance evaluation is often associated with large-scale projects, which are
subject to a call for tender, and for which the evaluated performance constitutes one of the criteria
of the attribution of the market. The performance may therefore be evaluated before or after the
market is obtained, by means of validation of a “sample”.
As the camera performances can vary significantly depending on the conditions of installation and of
use of the equipment, SURVISION can only guarantee performances after a precise evaluation of
each project. In the same way, the verification measures are only taken into account by SURVISION
under the following specific conditions:

that all of the general or specific setting and installation instructions have been respected by
the installer.

that the test methodology conforms to the specifications of this guide

that the operational test conditions conform to the usual expected deployment of the
cameras.
2.2
Problems
The validation of a number plate reading performance is therefore a useful, sometimes indispensible
exercise, but also one which is complex and difficult to carry out.
This evaluation should fulfil three contradictory aims:

It should be objective, in order to provide a result which satisfies both the provider and the
user of the technology. In fact, only the examination of images processed by the cameras is
really objective. All other methods (for example, the comparison of data from the number
plate recognition system with data provided by another electronic system or device such as a
ground loop, a laser or even another video surveillance camera) introduces a bias in the
analysis.

It should be exhaustive (for example, it should be carried out both night and day, and on both
light and heavy traffic) in order to provide an idea of the closest possible performances to
those in real deployment conditions.

It should be economic in technical and human resources, as the analysis of video sequences
may take days, weeks or months.
The methodology and the tools offered by SURVISION make it possible to fulfil these three aims.
3
TERMINOLOGY AND RATIOS EVALUATED
3.1
Definition of ratios
The performances are expressed in a synthetic way as a percentage of number plates identified or
read, correctly or not, in connection with a base of checked vehicles.
The ratios used can sometimes be calculated in different ways. This is why, in order to avoid all
ambiguity concerning the measurements carried out, these ratios should be precisely defined.
The measurements can apply to all or part of the following four ratios:

Gross Detection Rate or “GDR”). This relates the number of detected plates (i.e., read, with
or without a recognition error) to the total number of vehicles passing in front of the camera.

Net Detection Rate or “NDR”). This relates the number of detected plates (i.e. read, with or
without a recognition error) to the total number of vehicles passing in front of the camera,
after the number plates judged “non eligible” have been removed.

Gross Read Rate or “GRR”). This relates the number of correctly read plates (i.e. excluding
the plates read but with errors) to the total number of vehicles passing in front of the
camera.

Net Read Rate or “NRR”). This relates the number of correctly read plates (i.e. excluding the
plates read but with errors) to the total number of vehicles passing in front of the camera,
after the number plates judged “non eligible” have been removed.
The customer should be aware that for the GDR and the GRR, the total number of vehicles has to be
cleaned from double detections of false detections. These events are rare but must not be taken into
account (otherwise a strong bias is introduced in the calculation of ratios).
3.2
Eligible and non-eligible number plates
The idea of “non eligible” number plates also must be defined. This refers to plates with “anomalies”
or “characteristics” likely to be detrimental to the localization, segmentation or character-reading
algorhyme. It also includes the limits which may result from the installation position of the cameras
(if, for example, they are positioned directly over the lanes, at such an angle that the plates of lorry
trailers, often embedded in the body, are partially or totally hidden).
To be eligible, a plate should be:

Conform to the regulation of the country (this criteria often includes very precise restrictions
on number plates),

Clean and in good condition,

With a fixation system (screws, rivets, holding fixture) which does not cause any parasite
signal in the image (i.e., reflective if on the background, or non-reflective if on a character). A
space should also be left between the characters and the edge on all four sides when the
plate is maintained with a holding fixture,

Not be totally or partially masked,

Have a standard syntax and font,

Be of standard rectangular format,

Be reflective to infrared,

Be filmed from the front (or a specific « Rear » ratio must be defined taking into account the
fact that performances are always a little lower on rear plates),

Not have in their environment (present in the same image), plates which send a signal similar
to that of a number plate (for example, the driver’s name inscribed on a number plate placed
behind the windscreen).
These criteria can be negotiated or adapted according to the projects and the levels established for
the NDR and/or the NRR.
The installation must of course conform to the manufacturer’s instructions (SURVISION).
3.3
Some examples of non-eligible plates
Plate non-conform to the regulations (italic characters)
Plate non-reflective to infrared
Plate in bad condition
Dirty plate
Partially concealed plate
3.4
Performance guidelines
The ratios may vary significantly from one country or one installation to another. For example, the
characteristics of Belgian plates, or certain Eastern European plates or of North American plates
(Mexico, Canada, USA), or the plates of lorries can have a negative impact on performances. On the
other hand, the performances observed in the Netherlands and in certain Scandinavian countries, or
at access control installations where only a limited number of vehicles have access and are checked
(subscribers, or company car-parks, vehicle hire companies…) are above average. This makes it
difficult to establish common standards.
However, some there are some common guidelines for the main European countries (France,
Germany, Spain, United Kingdom, Italy):

For free-flow applications on motorways (installation directly above the lanes, with vehicles
travelling at high speed and possibly with a very significant vertical angle), the GDR may be
>95%, the NDR >97%, the GRR >90% and the NRR >92%.

For free-flow applications in urban zones (installation directly above the lanes or to the side)
and lower speeds, the GDR may be >96%, the NDR >98%, the GRR >91% and the NRR >93%.

For toll or parking type applications (vehicles which stop momentarily), the GDR may be
>98%, the NDR >99%, the GRR >93% and the NRR >95%.
4
TOOLS PROVIDED BY SURVISION
4.1
The VSS
The VSS (or VISIPAK SETTING SOFTWARE) is a software tool for camera setting and parametring.
These operations should be carried out conform to the instructions of SURVISION contained in the
VSS User Guide.
The installation files and the VSS User Guide are available on request. This tool is free, but it is not
provided without training on how to use it.
4.2
The PVT Recorder
The PVT Recorder (Performance Validation Tool) is a tool with which it is possible to record in a
continuous or cyclic manner video streams and logs coming from number plate recognition cameras.
These data are then used to evaluate the performance. These video streams are “controlled”; this
means that the luminosity parameters are regulated by the number plate recognition algorhymes.
The installation files and the User Guide of the PVT Recorder are available on request. This tool it is
not provided without training on how to use it.
4.3
The PVT Player
The PVT Player is the tool that is used to index and verify the sequences produced by the PVT
Recorder, in order to evaluate recognition performances. Indexation means the addition of metadata
possibly containing number plates which may be included in each image of the video stream. This
indexation is first of all automatic, and may then be manually corrected by an operator.
The installation files and the PVT Player User Guide are available on request. This tool is not provided
without training on how to use it.
5
TEST METHODOLOGY
The suggested test methodology is a proposed solution to the above mentioned problems.
5.1
Selection of the site and of the camera
The number and the type of cameras from which the sequences are carried out depend on the
variety of the installations intended for the site. If all the cameras are installed in the same way, it is
enough to carry out the tests on just one single camera. If the installations are carried out according
to two very different modes (one in a toll lane, the other in the middle of a lane on the motorway, for
example), a camera corresponding to each situation can be selected, etc…
The user should bear in mind that the precise analysis of the sequences following the methodology
described in this chapter will require approximately one hour for 400 vehicles.
5.2
Choice of test period
The performances can vary depending on luminosity conditions. However, the exhaustive analysis of
vehicles over 24 hours requires several days of data breakdown and study.
This is why SURVISION recommends that 24 sequences be carried out every hour using the PVT
Player in order to cover all the conditions, all the while consecrating a reasonable time to the analysis
procedure. The duration of each sequence can be of 5 minutes or more depending on the vehicle’s
flow density on the site and of the total number of vehicle that one wants to reach. The SURVISION
experience shows that above 400 vehicles, the performances do not vary anymore.
5.3
Filming of the sequences
The sequences are made and recorded with the PVT Recorder. For more information, see the User
Guide of this tool.
5.4
Analysis of sequences
The sequences are analysed using the PVT Player. See the User Guide of this tool for more
information.
5.5
Test report: global performance evaluation
Once all the sequences have been analysed and indexed, the test reports are automatically provided
by the PVT Player. They allow the above-mentioned ratios to be calculated.
It is essential that the performance analysis is carried out globally. This obviously means that a single
ratio should be calculated on all of the sequences, regardless of vehicles and luminosity conditions,
etc…
Effectively, only the global conditions reflect the reality of performances which the user will be
confronted with on site. To isolate particular performances (night-time, on lorries, on the number
plates of particular countries) could lead to significantly above or below average performances,
which do not give any indication about the real performances.
For example, there is no point in isolating and taking into account performances on a particular
country associated with exceptionally good or bad performances if the plates of this country never
appear in front of the cameras.
Taking a global performance into account means automatically taking into account how the vehicles
are naturally distributed among a range of different possible conditions.
6
TRAINING AND SUPPORT
SURVISION offers training sessions for users and integrators so that they can familiarize themselves
more quickly with how the SURVISION solutions function, notably concerning the methods and the
rules to be respected with respect to testing and validation.
Please contact SURVISION if you would like to participate in a training session.