Invitation to tender for a contract to deliver:
Verification rule sets for NBN Record Cleaner and recommendations on species
whose records should be treated as sensitive
Supported by the NBN/Defra Contract 2011-14
Background
NBN Record Cleaner is an automated validation and verification decision-support tool for recorders and
biodiversity data managers. It is designed to improve the efficiency of data flow and to ensure the quality
of datasets on the NBN Gateway, by enabling the automated checking of large datasets in a variety of
formats against validation and verification ‘rule sets’. Verification rule sets flag up any records of species
that fall outside the known temporal or spatial distribution of that species, as well as highlighting records of
species that are inherently difficult to identify. Plans are underway to develop Record Cleaner as a web
service to facilitate its integration with existing data management and online recording systems.
Verification rule sets have already been developed for birds, vascular plants, butterflies, macro moths,
hoverflies and some marine species. To maximise the usefulness of Record Cleaner within online data
entry portals and by managers of multi-taxa datasets, rule sets need to be developed for other taxonomic
groups.
The NBN Gateway and Indicia include access controls to protect records of sensitive species. Organisations
and individuals that provide data to the Gateway or to an online recording system need to be assured that
their understanding of ‘sensitive species’ is shared by those who access and use their data. It is therefore a
priority to develop national standards on sensitive species within each taxonomic group. Sensitive species
lists have already been produced for birds by the BTO, for plants by BSBI and for butterflies and moths by
Butterfly Conservation.
Contract Outputs
1. Specification of essential attribute fields for new records
For any taxonomic group there will be a set of attributes that should ideally be associated with a record.
The contractor will recommend a standardised set of attribute fields for new records of species within
their chosen taxonomic group, identifying those attributes which are essential and those which are
desirable for particular applications. The criteria may be different for different species within a group.
Consideration should be given to the existence of aggregate taxa records, and to whether there is a need
to define and publish consistent definitions for these recording aggregates.
Format: PDF document
2. Documented procedure for dealing with records highlighted by Record Cleaner
Record Cleaner is a decision-support tool. When a record is highlighted as falling outside the known range
of that species or as being inherently difficult to identify, the data manager needs a documented
procedure to follow in order to have the record verified, and the verification rules updated if necessary.
The contractor will establish and document a procedure for dealing with records of species within their
chosen taxonomic group that fall outside the verification rules. This will include information such as who
to contact for verification, and how decisions should be recorded. Further information is provided in the
NBN paper “Improving Wildlife Data Quality”. The procedure should take account of the current
recommended framework for defining the verification status of records, i.e.
o Correct
o Considered correct
o Requires confirmation
o Considered incorrect
o Incorrect
o Unchecked
Format: PDF document with numbered points or flow chart
1
3. Identification difficulty rule set
Record Cleaner can colour-code records on a 1-5 scale of identification difficulty. 1 and 2 are coloured
green (easier), 3 and 4 are coloured amber (intermediate) and 5 is coloured red (difficult).
The contractor will produce a classification system for their chosen taxonomic group according to the level
of difficulty / expertise needed to identify each species. A typical example of this would be:
1) Can be identified at sight in the field by anyone with a bit of experience. Species with which the
beginner rapidly becomes familiar. Usually identifiable from a photo. Records acceptable from most
sources.
2) Can be identified in the field with care and experience. Needs a good view or the netting of a
specimen to check, but the specimen can then be released. Beginners may need to take specimens
until they gain familiarity and experience. May be identifiable from a good photo. Records
acceptable from competent recorders.
3) Specimen needs checking under magnification and good lighting. The Recording Scheme would
accept records from experienced recorders without further question unless the date, region or
habitat was especially unusual. Beginners should get specimens checked at first until they gain
experience. Records accepted from known experienced recorders
4) Voucher specimen should be retained. The Recording Scheme would require confirmation in the
majority of cases, e.g. specimen having been checked by an acknowledged expert or brought along to
an identification seminar.
5) Even the most expert of recorders would seek a second opinion. Specimen may need to be submitted
to experts elsewhere in Europe for comparison with a wider range of material.
Each species should be assigned to a category. Justification for the categorisation should be provided
where relevant, e.g. reference to confusion taxa.
Format: One .txt file containing rules for all taxa presented as in the following example:
[Metadata]
TestType=IdentificationDifficulty
Organisation=Hoverfly Recording Scheme
LastChanged=20101019
[EndMetadata]
[INI]
1=Can be identified at sight in the field by anyone with a bit of experience. Species with which the beginner rapidly
becomes familiar. Usually identifiable from a photo.
2=Can be identified in the field with care and experience. Needs a good view or the netting of a specimen to check,
but the specimen can then be released. May be identifiable from a good photo, or series of photos.
3=Identification only accepted from known recorders or else needs confirmation from vice county recorder
4=Species needs confirmation from national expert
5=Voucher specimen required to be examined by national expert
[Data]
NBNSYS0000005502=3
NBNSYS0000006862=1
NBNSYS0000006865=4
NBNSYS0000006866=2...
Justification for the classification should be provided separately in an Excel spreadsheet.
4. Spatial distribution rule set
Record Cleaner will flag any records that fall outside the known spatial distribution of that species.
The contractor will produce a geographical baseline to define the known UK distribution of each species in
their chosen taxonomic group. The rules should be defined as a list of 10km squares within the UK where
each species would reasonably be expected to occur. It is expected that the rules will be based on
published atlases though some level of generalisation will probably be necessary as it is unlikely that every
new ten km square for a species needs to be flagged as unusual. The NBN have produced a tool which
buffers the currently accepted distribution of species (see http://forums.nbn.org.uk/rsgwa/). Further
assistance with the use of this tool can be provided by JNCC if required.
2
Format: One .txt file per taxon presented as in the following example:
[Metadata]
TestType=WithoutPolygon
Group=Hoverflies
ShortName=Volucella bombylans: check grid is in range
Description=Check grid is within known range of Volucella bombylans
ErrorMsg=Grid is outside known range of Volucella bombylans
DataFieldName=Species
DataRecordId=NBNSYS0000007094
LastChanged=20090615
[EndMetadata]
[10km_GB]
HY20
HY30
HY31
NB00
NB44...
5. Temporal rule set – seasonal range (if relevant for this group)
The contractor should provide a seasonal range rule set for any species within their chosen taxonomic
group which are recorded only during a certain period of the year, e.g. migratory animals or species which
can only be identified during a certain stage in their life cycle.
Format: One .txt file per taxon presented as in the following example for species whose temporal range is
within a year, e.g. February to December:
[Metadata]
TestType=PeriodWithinYear
Group=Hoverflies
ShortName=Volucella bombylans: check date is during flight period
Description=Check date is during the known flight period for Volucella bombylans
tvk=NBNSYS0000007094
ErrorMsg=Date is outside flight period for Volucella bombylans
DataFieldName=SpeciesStage
StartDate=0131
EndDate=1218
LastChanged=20090615
[EndMetadata]
[Data]
Stage=All
startDate=0131
endDate=1218
Stage=Adult,Adult Male,Adult Female,Male,Female
startDate=0131
endDate=1218
Alternative format for species whose temporal range overlaps two years, e.g. September to April:
[Metadata]
TestType=PeriodWithinYear
Group=Birds Flight Period
ShortName=Barnacle Goose
Description=Checks date is within most regular occurrence period for Barnacle Goose
ErrorMsg=Date is outside usual flight period of Barnacle Goose
Tvk=NBNSYS0000000214
DataFieldName=
StartDate=0426
EndDate=0914
ReverseRule=True
LastChanged=20110826
[EndMetadata]
3
6. Temporal rule set – year range (if relevant for this group)
The contractor should provide a year range rule set for species within their chosen taxonomic group which
are known to have gone extinct or which were not recorded in the British Isles before a particular date.
One .txt file per taxon presented as in the following example:
[Metadata]
TestType=Period
Group=Moths Recording Period
ShortName=Zygaena filipendulae
Description=Checks date is within known recording period of Zygaena filipendulae
ErrorMsg=Date is outside known recording period of Zygaena filipendulae
Tvk=NBNSYS0000005670
StartDate=17000101
EndDate=
LastChanged=20110328
[EndMetadata]
7. Identify criteria for sensitive records
In accordance with NBN Data Exchange Principles, access to biodiversity data may need to be controlled if
making the data available is likely to result in environmental harm. This particularly relates to species
which are vulnerable to persecution, collection or hunting and where releasing the location of records is
likely to increase the risk of this occurring. For this reason, the NBN Gateway and Indicia allow users to
flag records as ‘sensitive’ but there is currently a lack of consistency in how this ‘sensitive’ flag is applied.
The contractor will produce a standard list of species within their chosen taxonomic group whose records
should be treated as sensitive, i.e. not freely shared. Appropriate justification should be given to explain
the inclusion of each species on the list, and in some cases it may be necessary to include additional
criteria such as sensitive only during breeding season. The goal is to be as open as possible with records
and so the list derived should be as short as possible.
Format: Word table or Excel spreadsheet with the following fields:

Species name

Preferred taxonversionkey

Reason/s for inclusion

Additional criteria – e.g. only sensitive when breeding, only sensitive in a certain geographic area

Level of resolution at which species records are considered sensitive, e.g. finer than 1km, 5km,
10km
Terms of Reference

Tenders should be sent by e-mail to Paula Lightfoot at p.lightfoot@nbn.org.uk using the form on
the following page.

The deadline for tenders is the 31st May 2012. Contracts will be awarded no later than the 29th
June 2012.

All contract outputs must be complete by the 31st March 2013.

The data products generated through the work will be jointly owned by both the contractor and
the NBN Trust on the understanding that they will be made freely and openly available for use by
anyone.

Technical support for converting the rules into the required format (.txt files) is available from
the Biological Records Centre and the Marine Biological Association if required.
4
Response to tender
Applicant details:
Organisation name:
Organisation address:
Contact person:
Position in
organisation:
Telephone number:
E-mail address:
Names of other organisations/experts who will collaborate on the contract (if applicable)
Taxonomic group/s in which the contractor has expertise
Summary of the contractor’s skills and experience in this area (no more than 300 words)
Number of species for which the contractor will create verification rules
(if possible, please attach a complete species list)
Verification rule sets to be produced under this contract
(Please state the number of species for which each rule type will be produced)
Identification
difficulty rules
Spatial distribution
rules
Temporal rules:
seasonal range
Temporal rules:
year range
Number of species which the contractor will review in order to provide recommendations with
justifications as to whether their records should be treated sensitive
Cost of work carried out under this contract
Total cost:
£
Breakdown of
costs:
(e.g. staff hours,
expenses)
£
£
£
5