Experience from the data collection exercise for
update of the WFD priority substances 2011
D.PREUX and
B.FRIBOURG-BLANC, IOW
International Office for Water
Non profit making association (NGO with 150 members), State approved
Created in 1991 (20 years!)
Missions :
 international networking: INBO
 services in the water domain (Training centre, Information and data
Centre)
 European and international cooperation: twinning, CIS-SPI,
6th WWF Marseille 2012
http://www.oieau.org/
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 2
International Office for Water
A Motto: “Capacity building for better water management!”
THREE DEPARTMENTS
Information
management
Vocational training
Institutional
cooperation
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 3
WFD priority substances: the project
• assistance to DGENV on WFD priority substances
• One main activity: monitoring based prioritisation
• Prioritisation methodology (INERIS) + data collection (IOW)
 EU relevant, scientifically based and pragmatic
 From universe of substances to a manageable list (30-50
substances)
 Minimum data requirements
• Target: pollution of surface water compartments (Water,
sediment and biota)
(Adapted from
V.Bonnomet,
INERIS)
D. Preux,
B. Fribourg-Blanc,
Norman data exchange Workshop, 20-21/04/2011
slide 4
WFD PS: the data collection steps
1.Define the needs: minimum mandatory and
optional fields, voluntary basis
2. Create the End user tool (MS Access) for data
collection
3. Data collection + helpdesk
4. Gather data, analyse, define the corrections and
quality checks, test and implement
5. Import data in a central database
6. Treat data for prioritisation
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 5
Tools to support the collection
The Microsoft Access End user tool (+template)
The How to use guide
A Web page www.oieau.fr/WISE-end-user-tool and a helpdesk
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 6
WFD PS: Data collection 2008 and 2009
• A strictly defined template with 22 mandatory fields
• Flexibility introduced on 8 fields (longitude/latitude,
LoD/LoQ, type of station, date of sampling, biota species, laboratory name)
• A flexible data collection tool, and detailed
documentation for non experts, to:
 minimise reporting burden: except on some validity
checks, any fraction, any unit, any substance even with
missing mandatory fields is accepted
 allow decentralised distribution of the tool (experienced
by RO)
 maximise quantity of data collected
• Interaction between data provider and consultant
for
 Quality checking before selection of relevant dataset(s)
 correction of missing fields and coherence
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 7
WFD PS: Preliminary data treatment
• Only datasets for surface water
 River, Lake, Transitional, Coastal, Marine
• Elimination of :




data measured before year 2000
non relevant parameters (e.g. P(tot), Nitrates, etc.)
datasets for which neither LOD nor LOQ were provided
Insufficient precision of matrix or fraction (all ”other”)
• Correction





•
station location if missing
substance name (in national language) or CAS code if detected wrong
unit (in national language)
fraction when it can be gathered with existing fractions
date of sampling or date of analysis format or when one is missing
Harmonisation of the measurement units (130 units on the
1130 substances provided and 102 units on 21 Tin compounds)
 Water: µg/l, Sediment: µg/kg dw, Biota: µg/kg ww
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 8
WFD PS: Central Database creation
• A central database similar to the End User tool (Station, Sampling,
Analysis)
 structured database (PostGre/PostGIS, SQL Language)
 on-line management tools for import of datasets (SQL based interface)
 Maintenance tools for storing and saving (SQL calculation modules)
• All individual files imported (more than 50 files, up to 40
GBytes)
XML file
Internal
checks and
validation
export
Discarded (cat. 1
incomplete)
End User Tool :
one for each data
provider
Use for
prioritisation
Central database
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 9
WFD PS: Summary situation
• Data 2000-2008
• Surface water
• 26 Member States + CH and NO
• almost 20 000 stations
- 4 water body types
• 547 000 sampling
• 14,6 million analyses
• 1 150 substances or groups (PCB
and Tin compounds)
Source: GIS layer : Official WFD Districts
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 10
WFD PS: Apportionment of analyses
3500000
3000000
by year
2500000
2000000
1500000
1000000
500000
0
2000
2001
2002
2003
2004
2005
2006
2007
Coastal Water
9%
by water category
2008
Transitional Water
4%
Lake Water
6%
River Water
81%
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 11
WFD PS: Apportionment of analyses
Sediment
6,3%
Biota
0,5%
X>LoQ
100%
X>Lod but no LoQ
90%
X<=LoQ but no LoD
80%
LoD<X<=LoQ
X=LoD
70%
X<LoD
60%
Water
93,2%
50%
40%
by matrix
30%
20%
10%
Biota
Sediment
Water
0%
along analytical result
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 12
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
104-40-5
107-06-2
115-29-7
117-81-7
118-74-1
12002-48-1
120-12-7
122-34-9
127-18-4
140-66-9
1582-09-8
15972-60-8
1912-24-9
206-44-0
2921-88-2
32534-81-9
330-54-1
34123-59-6
470-90-6
50-29-3
56-23-5
608-73-1
608-93-5
67-66-3
71-43-2
7439-92-1
7439-97-6
7440-02-0
7440-43-9
75-09-2
79-01-6
85535-84-8
87-68-3
87-86-5
91-20-3
WFD PS: The data quality aspects
-Example: performance criteria set in Directive 2009/90/EC: LoQ < 0,3 x EQS
100%
90%
80%
70%
60%
LoQ <= 0,3xEQS
50%
LoQ > 0,3xEQS
no LoQ available
40%
30%
20%
10%
0%
slide 13
WFD PS: Conclusion
• The first data collection of regular chemical monitoring data
under WFD and a huge database
• A collaborative exercise involving EU and MS authorities
• A complete report on quality assessments (restricted, accessible to CIS
expert) concluded future data collection will require:
 Adaptation of the data collection template and tool
 additional collection rules (more mandatory fields, checks…)
 A checking mechanism on the data content (station location, …)
 more quality information (LoD and LoQ, accreditation…)
 Use of WISE and INSPIRE reference lists
• Possible development could include:
 Use of a shared EU wide structured water language (Reportnet)
 Use of shared reference lists (Reportnet Data Dictionary, ROD…)
 Use of shared scenarios
 for example the French system: SANDRE and
EDILABO
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 14
SANDRE, French data and metadata for water
A cornerstone of French water information system
Data interchange in an evolving context: WFD implementation, progress of
scientific knowledge… Numerous actors with different needs and tools
I do not understand the data sent
I can not read the format
Water Agencies
Municipalities
RBD authorities
SANDRE
Laboratories
Industries
I need to develop a specific mapping
for each data collection
Ministry of
Environment
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
He used codes I do not
know
slide 15
SANDRE, a collaborative system
for water professionals
From natural water cycle to anthropogenic water cycle: a number of themes
Drinking water
Surface water
Rainwater
Groundwater
Transitional and
coastal water
Waste water
From professional language to database interoperability
<<complexType>>
PRELEVEMENT : 5
<<complexType>>
DEMANDE : 4
+
+
+
+
+
+
+
+
+
+
Code de la demande chez le commanditaire
Code de la demande chez le prestataire
Type de demande
Contexte de codification
Date de la demande
Libellé de la demande
Date de début d'application de la demande
Date de fin d'application de la demande
Référence du marché
Commentaires sur la demande
:
:
:
:
:
:
:
:
:
:
IdentifierType
TextType
CodeType
CodeType
DateType
TextType
DateType
DateType
TextType
TextType
Caractère
Caractère
Caractère
Caractère
Date
Caractère
Date
Date
Caractère
Texte
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
0..*
Code du prélèvement
Numéro d'ordre du prélèvement
Référence du prélèvement chez le préleveur
Date du prélèvement
Délai de réalisation du prélèvement
Heure du prélèvement
Durée du prélèvement
Localisation exacte du prélèvement
Conformité du prélèvement
Accréditation du prélèvement
Prélèvement sous réserve
Commentaires sur le prélèvement
Risque du produit de prélèvement
Usage du produit de prélèvement
Nature du produit de prélèvement
Norme appliquée au produit de prélèvement
Finalité du prélèvement
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
IdentifierType
TextType
TextType
DateType
NumericType
TimeType
TextType
TextType
CodeType
CodeType
CodeType
TextType
TextType
CodeType
CodeType
CodeType
CodeType[]
Caractère
Caractère
Caractère
Date
Numérique
Heure
Caractère
Caractère
Caractère
Caractère
Caractère
Texte
Texte
Caractère
Caractère
Caractère
Caractère
0..*
0..*
Commemoratif
0..*
Commemoratif
<<complexType>>
COMMEMORATIF
+ Code du commémoratif
: IdentifierType Caractère
+ Libellé du commémoratif
: TextType
Caractère
+ Descriptif du commémoratif : TextType
Texte
+ Valeur du commémoratif
: TextType[]
Texte
1
2
0..*
Commemoratif
3
0..*
Commemoratif
0..*
<<complexType>>
ECHANTILLON : 3
+
+
+
+
+
+
+
+
Référence de l'échantillon chez le commanditaire
Référence de l'échantillon chez le préleveur
Référence de l'échantillon chez le laboratoire
Complétude de l'échantillon
Acceptabilité de l'échantillon
Date de réception de l'échantillon
Heure de réception de l'échantillon
Commentaires sur l'échantillon
:
:
:
:
:
:
:
:
IdentifierType
TextType
TextType
CodeType
CodeType
DateType
TimeType
TextType
Caractère
Caractère
Caractère
Caractère
Caractère
Date
Heure
Texte
0..*
<<complexType>>
ANALYSE : 2
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Experts needs
Numéro d'ordre d'analyse
Date de l'analyse
Heure de l'analyse
Résultat de l'analyse
Code remarque de l'analyse
Limite de détection
Limite de quantification
Limite de saturation
Accréditation de l'analyse
Confirmation du résultat d'analyse
Analyse sous réserve
Incertitude analytique
Résultat de référence
Analyse in situ / en laboratoire
Rendement de l'extraction
Commentaires sur l'analyse
Volume filtré
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
NumericType
DateType
TimeType
TextType
CodeType
NumericType
NumericType
NumericType
CodeType
CodeType
CodeType
NumericType
TextType
CodeType
NumericType
TextType
NumericType
Numérique
Date
Heure
Caractère
Caractère
Numérique
Numérique
Numérique
Caractère
Caractère
Caractère
Numérique
Caractère
Caractère
Numérique
Texte
Numérique
Conceptual model
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
Database
slide 16
SANDRE, the approach
1st STEP: A COMMON
SEMANTIC
2nd STEP: COMMON
REFERENCES
3rd
STEP: SCENARIOS
DATA DICTIONARIES
FOR EACH THEME
CODE LISTS,
REFERENCE MAP LAYERS
DATA EXCHANGE BY SCENARIOS
AND FORMATS
XML-XSD
All the SANDRE results are freely available
www.eaufrance.fr
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
Scenario
documentation
slide 17
Extract of data model
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 18
SANDRE: code list management
The principle: evolving code lists freely available to
water community to ease data interchange.
PARAMETER
1107
UNIT
133
µg/L
METHOD
132
T 90-121
MATRIX
3
WATER
FRACTION ANALYSED
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
raw23
water
http://www.sandre.eaufrance.fr
slide 19
EDILABO, a scenario for water analysis
3 main actors, electronic exchange of sectoral data.
Asks for analyses
Client – public
Laboratory
administration
or private operator
Analyses results
Sampler
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 20
EDILABO: secured data exchange in water field
• Started 2001, based on EDIFACT standard and SANDRE,
• Target water physico-chemical and microbiological analyses,
• Example:
<Analysis>
<Parameter>1107</Parameter>
<Matrix>3</Matrix>
<Fraction>23</Fraction>
<Result>0.72</Result>
<Unit>133</Unit>
<Method>132</Method>
</Analysis>
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 21
Main conclusions
Based on WFD PS first experience:
 the first voluntary exercise proved feasibility of collecting big datasets
 Tool known: can be used for future
Quality of data expected to improve:
 QA/QC Directive entered into force Aug. 09 and transposed Aug. 11
(art 4 minimum performance, Art 5 calculation of mean, Art 6 ISO 17025)
 EQS Directive
a need to improve robustness and reliability of data collections
 More mandatory fields
 a need for more automatic data interchanges
 French SANDRE proves the feasibility of automating the process
A collaborative work: crucial for the success !!!
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 22
Thanks for your attention…
d.preux@oieau.fr
D. Preux, B. Fribourg-Blanc, Norman data exchange Workshop, 20-21/04/2011
slide 23