Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
1/36
CARBONES portal description
Simplified version V0
Change History
Date
Author
Comment
11th Mar 2010
Mark Jackson
First Draft
14th
Mar 2010
Mark Jackson
Review comments (Chris Johnson, Christine McHugh)
30th Mar 2010
Mark Jackson
Comments by Noveltis, LSCE
30th Mar 2010
Mark Jackson
Reinstate analysis of engineering solutions
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
2/36
Contents
1
2
3
Summary ............................................................................................................................ 3
Introduction ......................................................................................................................... 5
Description of V0 portal website ......................................................................................... 7
3.1
Content ..................................................................................................................... 7
3.2
Web mapping ........................................................................................................... 7
4 Analysis of engineering solutions ..................................................................................... 10
4.1
Purpose of analysis of engineering solutions ......................................................... 10
4.2
Summary of conclusions ........................................................................................ 10
4.3
Technologies discussed ......................................................................................... 11
4.4
Case study ............................................................................................................. 12
4.5
Market penetration ................................................................................................. 12
4.6
INSPIRE compliance .............................................................................................. 13
4.7
Cost ........................................................................................................................ 13
4.8
Development support ............................................................................................. 14
4.9
Product longevity .................................................................................................... 14
4.10
Technical implications of requirements .................................................................. 15
4.10.1
4.10.2
4.10.3
4.10.4
4.10.5
4.10.6
4.10.7
4.10.8
4.10.9
4.10.10
4.10.11
5
Portal architecture ............................................................................................................ 21
5.1
Post-processing...................................................................................................... 21
5.1.1
5.1.2
5.1.3
5.2
5.2.1
5.2.2
6
Operating system ............................................................................................................ 15
Programming language ................................................................................................... 15
Graphs and charts ........................................................................................................... 15
Metadata ......................................................................................................................... 16
Download service ............................................................................................................ 16
Discovery service ............................................................................................................ 17
Data format ..................................................................................................................... 17
Interactive maps .............................................................................................................. 18
Content management system ......................................................................................... 19
Post-processing.......................................................................................................... 20
Database .................................................................................................................... 20
Inputs .............................................................................................................................. 21
Post-processing specification .......................................................................................... 23
Outputs ........................................................................................................................... 27
Portal server and portal web application ................................................................ 28
Inputs .............................................................................................................................. 28
Outputs ........................................................................................................................... 28
Further development after V0 .......................................................................................... 30
6.1
Development for the first annual meeting .............................................................. 30
6.2
Development after the first annual meeting ........................................................... 30
7 Test plan for the V0 portal ................................................................................................ 33
7.1
Post-processing...................................................................................................... 33
7.2
Test plan for web portal .......................................................................................... 35
8 Test report for the V0 portal ............................................................................................. 36
8.1
Post-processing...................................................................................................... 36
8.2
Test report for web portal ....................................................................................... 36
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
3/36
1 Summary
This report describes deliverable D520.1 in the CARBONES project: the V0 version of the
CARBONES web portal.
The deliverable D520.1 comprises:



The source code for V0 of the portal, including the post-processing and the web
portal.
A live portal.
o The portal web pages are at www.carbones.eu
o The portal web-mapping is at http://carbones.eu:8080/Carbones, to which a
link exists on the CARBONES home page. Figure 1 shows an image of the
portal web mapping. The web-mapping includes highly interactive maps with
zoom and pan functionality similar to that on Google Maps. It presents the
“raw” data set without post-processing as well as the results of postprocessing, including temporal filtering.
Detailed description of the portal (in this document), including:
o summary of architecture
o plans for further development
o test plan and test report for the V0 portal
Figure 1. The portal web mapping
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
4/36
The V0 portal uses 2 years of sample CCDAS output data provided by LSCE, under action
PM2-02 1. These data were provided so that data were available in time for the V0 postprocessing and portal. These preliminary data were provided so that data were available in
time for the V0 post-processing of the portal. These sample data are from 1 year of:




simulations done with the global version of the ORCHIDEE model (without
assimilation) for the biogenic fluxes and stocks of C
Takahashi climatology for the net C ocean flux
GFED (Global Fire Emissions Database version 2) for biomass burning
from EDGAR database for anthropogenic fossil fuel emissions
The data for the second year are the same as for the first year. The complete 20-year data
set, D510.1, is scheduled to be delivered at the same time as the V0 portal and, therefore,
could not be used in the V0 portal.
The objectives for V0 portal have been met and the portal is available for use displaying maps
of the preliminary data.
1
Action 2 Progress meeting 2, 27th Oct 2010
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
5/36
2 Introduction
The task WP520 covers the development of a tailored, user-friendly interface, allowing
climate modellers, other science communities, local or regional communities and the general
public to understand and easily visualize the living carbon cycle over the past 20 years. The
users of the CARBONES system will use the system via a web site in which spatial data will
be displayed on a dynamic map, similar to Google Maps. It has been developed so that users
can select the output according to area, time period and output variable. Web map elements
are implemented as a thin client compatible with all major browsers without requiring any
plug-in or specific setting (although JavaScript is required to view the maps).
This document describes the first version of the web portal (V0), deliverable D520.1 from
WP520, due at month 12. The Description of Work (DoW) mandates a second version of the
web portal (V1), deliverable D520.2, due at month 30.
The definition of the web portal has been developed through gathering and analysing user
requirements. Various partners and others have been involved (NOV, LSCE, TASF, METO,
CUAG, AEAT). The conclusions have been captured in deliverables (D120.1, D120.2, D210.1,
D220.1, and D720.1).
Figure 2 shows an overview of the entire CARBONES system, showing how the portal fits
into the system.
The post-processing step is required to transform the output of the CCDAS and other sources
of data (validation data) into C-cycle relevant diagnostics for climate users and policy makers.
The post-processing is performed off-line at CERC. The results of the post-processing are
presented through the web portal. Website users will be able to view the results of different
post-processing options through the web interface, but they will not be able to repeat the
actual post-processing calculations through the web interface.
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
6/36
Figure 2. CARBONES system overview (data flow)
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
7/36
3 Description of V0 portal website
3.1
Content
The content of the portal at www.carbones.eu explains the project objectives and lists the
project partners. It gives an overview of the project, explaining the motivation, objectives,
product characteristics and planned dissemination of the products. There is a news section
which is updated to give details of project progress. The website provides an email address
so that the public or other interested parties can contact the project team with any queries
they may have.
3.2
Web mapping
The portal web-mapping is demonstrated at http://carbones.eu:8080/Carbones to which a link
exists on the CARBONES home page. The web-mapping includes highly interactive maps
with zoom and pan functionality similar to that available in Google Maps. It presents the “raw”
data set without post-processing as well as the results of post-processing, including temporal
filtering. Flux and stock maps are presented.
The controls in the web application are shown in Figure 3. To use the web application:





the user selects the variable to display from the tree control at the left. Only one
variable can be displayed at a time, because the data cover the whole globe.
the user selects the temporal averaging period for the data from the drop-down on
the top left. The options include “3-hourly”, “Day”, “Month”, “Year” and “Period”.
Different options are available depending on the variable selected by the user in the
tree control.
the user selects the temporal filtering from the drop-down at the centre top. The
options are “Mean”, “Smooth trend”, “Smooth curve” and “Recentred parameter”.
Different options are available depending on the variable selected by the user in the
tree control.
the user selects a date and time for which the data will be displayed using the
controls at the top right. The user can either select a date-time directly using the
calendar control and the drop-down for the times, or, the user can step forwards and
backwards through the dates and times in order using the (arrows) buttons. The
interval between the dates and times depends on the temporal averaging period that
has been chosen, as shown in Table 1.
the user can select a region from the “Regions” tree control. This zooms the map to
the selected region and also hides areas outside the selected region.
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
8/36
Temporal
averaging
period
Interval between Example time steps
time steps
3-hourly
3 hours
1 Jan 1990 (00:00, 3:00, 6:00, 9:00, 12:00, 15:00, 18:00, 21:00),
2 Jan 1990 (00:00, 3:00, 6:00, 9:00, 12:00, 15:00, 18:00, 21:00),
…
Day
1 day
1 Jan 1990, 2 Jan 1990, 3 Jan 1990, …
Month
1 calendar month
Jan 1990, Feb 1990, Mar 1990, …
Year
1 year
1990, 1991
Period
Only one set of data, averaged over the whole period. User
would not be able to choose different time steps
Table 1. Temporal averaging periods and corresponding time steps
The web mapping application displays the variable values as a colour-shaded map. Continent
and country outlines are overlaid on the map to assist the user in relating the colour-shaded
map to particular regions of the globe. The colour legend and units used in the map displayed
are given at the bottom left of the web display.
The map controls allow the user to pan and zoom the map freely. Clicking the globe resets
the view to show the entire world. Note that the user can also pan the map by clicking and
dragging with the mouse, and can zoom the map with the mouse wheel (if their computer has
a mouse wheel). The user can also view particular regions of interest by selecting them from
the pre-defined list in the regions controls.
The web mapping application will be compatible with the following browsers:





Internet Explorer 7
Internet Explorer 8
Firefox 3
Safari 4
Chrome 10.
Internet Explorer 6 is not supported. There are well-known difficulties in supporting this
browser for any website. The development effort required would not be justified by Internet
Explorer 6’s usage share, which is small and rapidly decreasing (under 4% in Europe 2).
Microsoft has recently begun a public campaign 3 to persuade all users to move away from
Internet Explorer 6.
2
3
Data across all operating systems, dated 28 February 2011, cited on ie6countdown.com
The campaign home is the Microsoft website ie6countdown.com
Time period drop-down allows
the user to select the time
period for averaging. The
options are “3-hourly”, “Day”,
“Month”, “Year” and “Period”.
Filtering drop-down allows the
user to select the temporal
filtering. The options are “Mean”,
“Smooth trend”, “Smooth curve”
and “Recentred parameter”.
Calendar control and hour dropdown allow the user to choose
the date and time to display. The
buttons allow the user to step
forwards and backwards
through the time steps.
Tree control allows
the user to select a
variable to display in
the map.
Map controls
allow the user to
pan and zoom the
map.
Regions tree
control allows the
user to view
specific regions of
the world.
Legend gives the
colour shading
and units used in
the map to display
the variable.
Position indicator
shows the latitude
and longitude of the
current mouse
position
Figure 3. The portal web mapping – user controls
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
10/36
4 Analysis of engineering solutions
4.1
Purpose of analysis of engineering solutions
4.1.1
Post-processing of CCDAS output
The post-processing calculations presented some special considerations. The development
of the scientific specification of the post-processing calculation required iteration between
LSCE, Noveltis and CERC that is ongoing. This presented a risk to the project timelines. To
mitigate this risk, the solution for the V0 post-processing is based on a commercial package
(IDL) and proprietary code (CCGVU). This mitigates the timeline risk for several reasons:
these packages have been proven in practice in similar projects; and LSCE, Noveltis and
CERC had staff members available with experience with IDL. We may study the use of
alternative tools for the V1 portal.
4.1.2
Web portal
This remainder of section 4 explains the analysis and selection of engineering solutions that
have been used to create an INSPIRE-compliant portal for the CARBONES project.
The analysis concentrates on two approaches: a solution based on commercial packages and
an open source solution.

The commercial vendor solution uses the ESRI ArcGIS Server product to illustrate
this option.

The open source solution is based on MapServer, a platform developed in the mid1990s at the University of Minnesota with support from NASA. Use of MapServer
requires, in addition, the use of a number of other free, open source packages.
The following criteria have been used in the analysis:
4.2

Baseline functionality: INSPIRE-compliant discovery services, view services and
download services; web portal features such as interactive maps and interactive
graphs.

Ease of implementation: including the capability to implement new services by
developing an API/SDK.

Licence cost and support costs.

Market penetration. In general, the larger the user base, the less the risks associated
with adopting the technology and of the product being retired or not supported in the
near future. A large user base generally means better support from the supplier as
well as the wider (user) community.
Summary of conclusions
From our research which is reported in sections 4.3-4.10, we are confident that both the
commercial and open source solutions can provide the functionality required for the project.
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
11/36
With little separating the two options in terms of functionality, there is little justification for the
extra cost of the ESRI solution (a minimum of £14,275 over the 3 years of the CARBONES
project, with an ongoing annual support cost of £1,600 after the end of the project).
Therefore, an open source solution was selected to create the portal. The open-source
technologies listed in the table have been identified as suitable for use in the portal and have
been used in the V0 portal. The specific choice of individual technologies remains open and
may change for the V1 portal.
One of the strengths of the project architecture is that the components communicate using
standardised communication protocols, making it possible to change the implementation of
individual components to different technologies with minimum effect on the system as a whole.
In particular, the use of OGC standards such as CS-W, WCS and WMS would make it
possible to change to a different GIS server package such as ArcGIS Server either during the
project, or after the project is completed. The technology analysis indicates that this will not
be necessary, but the system architecture minimises risk by leaving the possibility open.
Functionality
Technology Recommendation
Discovery services and metadata (CS-W)
GeoNetwork
Download service (WCS)
MapServer
View service (WMS)
MapServer
Server operating system
Linux
Web server
Apache / Apache Tomcat
Programming language
Java
Content Management System
Alfresco
Interactive maps in the portal
OpenLayers JavaScript API
Interactive graphs and charts in the portal
JFreeChart
Post-processing
IDL
Table 2. Selected technologies recommendations for V0 of the web portal
GENESIS (GENeric European Sustainable Information Space for Environment), an EU 7th
Framework project, will provide INSPIRE-compliant Web services, portal components and
toolkits. CARBONES will use GENESIS products, where appropriate and according to
availability and ease of implementation, to provide CARBONES functionality. The GENESIS
products will be assessed for suitability in CARBONES after the V0 portal.
4.3
Technologies discussed
This analysis concentrates on two approaches to achieve an INSPIRE-compliant web
mapping solution: a commercial vendor solution and an open source solution.

The commercial solution used in this document to illustrate this option is ESRI’s
ArcGIS Server product. Much of the information used in this document is based on
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
12/36
version 9.3.1, but we also include changes addressed in the recently released
version 10. The ArcGIS Server package is capable of fulfilling many of the
functionality requirements (WMS, WCS, CS-W, and JavaScript API).

The open source solution considered is based on MapServer, although other options 4
were researched. MapServer is a GIS server platform developed in the 1990s at the
University of Minnesota. The portal would also require a number of other free open
source applications, e.g. GeoNetwork and OpenLayers.
One of the advantages of ESRI is that it potentially supplies a single product to fulfil all of our
obligations, whereas the open source solution would depend on a several different products.
However, the required open source products are commonly used together, so integration
problems should not present a significant risk.
4.4
Case study
This analysis will make use of the experiences of the British Geological Society (BGS), who
are active developers of INSPIRE-compliant web mapping solutions. BGS have used both
ESRI and MapServer in their work, including work on their own Geology of Britain viewer5
(using ESRI’s ArcGIS Server) and an international project, OneGeology 6 , 7 (using
MapServer8). BGS have also taken a lead in bringing INSPIRE and MapServer together,
with their work on Open Geoscience and in producing their “cookbook” 9. The cookbook is a
set of instructions on implementing INSPIRE with MapServer that was produced in
conjunction with the Scottish government.
4.5
Market penetration
In general, the risk associated with adopting a technology is lower if there are a large number
of users. This reduces the risk of the product being retired, and it generally means better
support, from the supplier as well as the wider community.
ESRI have the largest GIS user base in the world 10 (over 100,000 organisations in 2000).
There are many proven implementations of ArcGIS Server available, including INSPIRE@EC,
Croatia’s Multipurpose Spatial Information System Geoportal Interface, etc.
There are many proven MapServer implementations, such as the OneGeology site mentioned
above. It is difficult to precisely quantify the size of MapServer’s user base. There are no
central records of its usage because it is open source software. However, there are hundreds
of messages each month on the mailing lists, and each year at the popular FOSS4G “Free
and Open Source Software for Geospatial” international conference, MapServer is the subject
of many of the presentations and workshops11.
4
Geoserver was seen as the main competitor to MapServer. ncWMS was also considered.
http://maps.bgs.ac.uk/geologyviewer_google/googleviewer.html
6
http://portal.onegeology.org/
7
Cookbook on setting up a WMS server using MapServer and ESRI:
http://www.onegeology.org/wmsCookbook/1_1.html
8
Example implementation and templates: ftp://ftp.bgs.ac.uk/pubload/OneGeology/OneGeology-WMS-exemplardata.zip
9
http://www.scotland.gov.uk/Publications/2010/05/06161701/22
10
http://www.esriuk.com/services/survey210509/
11
http://foss4g.org
5
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
4.6
13/36
INSPIRE compliance
ESRI participated in the prototype edition of the INSPIRE geo-Portal12, were part of the
consortium awarded the INSPIRE@EC work 13 and are committed to providing INSPIREcompliant software. ESRI-Europe are an SDIC (Spatial Data Interest Community) involved
in INSPIRE. BGS raised concerns of ArcGIS Server 9.3 failing to deliver an INSPIREcompliant solution14, but in a private communication to CERC BGS said “The latest version
(v10) of ArcGIS Server really moved ESRI’s support of OGC standards forward and enabled
us to provide [WMS] 1.3.0 services”.
OneGeology – Europe is an example of an INSPIRE-compliant portal developed with
MapServer and GeoNetwork. OneGeology - Europe are an SDIC involved in INSPIRE, and
the Project Coordinator for OneGeology are BGS. BGS themselves are an LMO (Legally
Mandated Organisation) involved in INSPIRE. While BGS (and others) continue to develop
INSPIRE-compliant solutions using MapServer and GeoNetwork, we can be confident that
these packages will adhere to INSPIRE directives.
Both ArcGIS Server and MapServer/GeoNetwork are capable of providing an INSPIREcompliant solution for the CARBONES portal.
4.7
Cost
There will be no software costs for the end user in using the CARBONES portal, although
there is the possibility of charges to cover data download costs. The following figures are
the costs to the CARBONES project of the portal software.
CERC have identified free open source options for all required elements in this development.
ESRI is a commercial vendor and adopting their solutions incurs costs for development
licenses, server licenses and maintenance. The pricing structure is shown in Table 3.
Item
Details
Annual Cost
Purchase Cost
Development Tools
1 EDN (ESRI Developer Network)
license, including ArcGIS Server
£995
£0
ArcGIS
Server
License and Annual
Maintenance
Workgroup Edition
from £1,600
from £6,490
Enterprise Edition
from £6,490
from £25,960
Enterprise ASP Edition
about £13,000
about £50,000
Awaiting details
Awaiting details
At least £2,595
At least £6,490
ArcGIS GeoPortal
Extension
Total
(Workgroup Edition)
Table 3. Cost implications of ESRI ArcGIS Server
12
http://www.inspire.esri.com/gos
http://news.gislounge.com/2009/02/european-commission-awards-inspireec-contract-to-esri-technology-basedconsortium/
14
http://location.defra.gov.uk/wp-content/uploads/2009/12/BGS-Data-Providers-Seminar-20100323-v1-0.pdf
13
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
14/36
The ArcGIS Server Workgroup license should be sufficient for the CARBONES portal. The
Workgroup license comes with the limitation of using a file-based database (SQL Server
Express) and using a maximum of four processor cores. The Workgroup license can only be
used if the portal website does not charge the end user and does not show paid
advertisements, otherwise we would need to upgrade to the Enterprise license.
It is not clear whether an ArcGIS Server license could be transferred from CERC to another
organisation at the end of the project. CERC pursued this with ESRI but were unable to
obtain a definite reply.
The total cost of the ESRI solution over the three years of the CARBONES project would be
at least £6,490 for the server license and £6,185 for three years of annual support costs 15.
After the project there would be ongoing maintenance costs of at least £1,600 per year. There
may be additional costs for the GeoPortal extension in order to provide INSPIRE-compliant
services. CERC are awaiting a definite statement from ESRI on whether this extension is
necessary to achieve INSPIRE compliance in the CARBONES portal. These costs are based
on a portal website which does not charge end-users or show paid advertisements.
The cost of the ESRI solution means we have to find definite advantages over the open
source solution to justify this expenditure.
4.8
Development support
ArcGIS Server’s user forums 16 are widely used, with new posts and published code
examples17 every day. ESRI also provide support, although there are cost implications.
MapServer comes with good documentation, although following their installation guide
requires a certain level of web server familiarity. BGS and the Scottish Government have
created a “cookbook”, a set of instructions on creating INSPIRE-compliant solutions using
MapServer.
Both options have good documentation and active “community” forums.
4.9
Product longevity
From CERC’s experience of ESRI’s desktop development environment, ArcView, there is a
history of ESRI replacing software or development languages, without providing an easy
upgrade path. In the last 10 years ESRI have retired at least three programming languages
(Avenue, VBA and VB6) and one product line (the GIS server product ArcInfo).
The open source alternatives, MapServer, GeoNetwork and OpenLayers, would be more
likely to evolve rather than being replaced. The potential concern is that the packages
themselves could be overtaken by new open source solutions, reducing the level of support
for the older products.
However, all the technologies have good customer bases and there is no immediate threat to
any of them. Although there are no guarantees in this area of rapidly-evolving technologies,
15
Three years of EDN license and 2 years of maintenance (the first 12 months maintenance is covered by the initial
purchase)
16
http://forums.arcgis.com/
17
http://resources.esri.com/arcgisserver/apis/javascript/arcgis/index.cfm?fa=codegallery&page=2
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
15/36
we can be relatively confident that whichever path we choose there will be a supported
product for the lifetime of the project and beyond. Furthermore, one of the strengths of the
INSPIRE architecture is that the use of open standards makes it relatively straightforward to
replace individual components if the need arises.
4.10 Technical implications of requirements
4.10.1 Operating system
The two most popular and best supported operating systems for servers are Microsoft
Windows and Linux. Whilst the majority of desktop machines 18 and general servers19 run
Windows (which has the built-in Internet Information Services web server), the majority of
web servers20 run Linux (using Apache as the web server). Linux is also regarded to be as
reliable and secure as Windows.
There are a wide range of scientific packages available on Linux for manipulation of netCDF,
which may be useful in the post-processing. Not all of these packages are available for
Windows.
ArcGIS Server21 and MapServer can both run on Windows or Linux, and we have selected
Linux for the web server.
4.10.2 Programming language
The chosen development language must be able to create and consume SOAP web services,
provide multi-lingual pages (i.e. must have full Unicode support), offer database connectivity
and have the ability to interact with the geo-server.
Java is a cross-platform language which can work on both operating systems and is a very
widely-used language for web server development. Both MapServer and ArcGIS Server offer
good support for Java development.
On the client side, we will use JavaScript to program the interactive maps.
4.10.3 Graphs and charts
There are a number of free open source Java graph and chart libraries, e.g. JFreeChart.
JFreeChart provides a number of professional looking graph styles, includes line graphs,
scatter plots, multiple-axis, overlaid, box & whisker and candlestick graphs (similar to the
Carbon-Tracker combination bar chart and scatter plots with error bars 22).
The graphs are dynamically generated on the server. This makes it possible to allow the user
to “zoom” to different ranges on the axes, or select different series for inclusion in the graphs.
It’s difficult to precisely quantify the user base for JFreeChart, since it is free open source
software. There are at least 20 commercial products available that make use of JFreeChart23,
18
19
20
21
22
23
http://www.w3schools.com/browsers/browsers_os.asp
http://www.zdnet.com/blog/microsoft/behind-the-idc-data-windows-still-no-1-in-server-operating-systems/5408
https://secure1.securityspace.com/s_survey/data/201007/index.html
http://wikis.esri.com/wiki/display/ag93bsr/ArcGIS+Server+Supported+Platforms
http://www.esrl.noaa.gov/gmd/ccgg/carbontracker/fluxtimeseries.php
http://www.jfree.org/jfreechart/users.html
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
16/36
which gives confidence that there is a very large number of sites using the JFreeChart library
or one of the commercial libraries that incorporate JFreeChart.
Neither ArcGIS Server nor MapServer offer native methods of creating and publishing graphs.
Therefore the choice of charting software is independent from the choice of GIS server.
4.10.4 Metadata
INSPIRE’s approach to metadata is hampered by some confusion surrounding various ISO
standards/schemas (19115, 19119 and 19139). These schemas are defined at two different
locations: the ISO repository for official standards and the Open Geospatial Consortium
Schema Repository. Unfortunately, the schemas available at these two sites differ because
they use different versions of Geography Markup Language (GML).
Max Craglia of the Joint Research Centre of the European Community stated “As a result, the
INSPIRE implementing rules are short and only say what functionalities are required, leaving
the detailed implementation to nonbinding guideline documents. This has its drawbacks, as
we cannot guarantee that everyone will use the guidelines and that interoperability will be
achieved immediately. On the other hand, experience has shown that we are still making
small adjustments to the guidelines for metadata two years after their approval.” 24 Within
CARBONES, however, we can ensure all partners will adhere to the same guidelines.
We believe that the developers of ESRI ArcGIS, MapServer and GeoNetwork are committed
to support metadata standards used in INSPIRE.
4.10.5 Download service
The INSPIRE documentation is not definitive regarding what is and what is not allowed in a
download service. INSPIRE’s draft technical guidance document 25 covers two types of
download service:
1. Downloading a pre-defined dataset or part of a pre-defined dataset via HTTP GET
request.
2. Direct access download services including a query capability.
Both types must have a metadata record and be discoverable using an INSPIRE-compliant
discovery service.
In the context of the CARBONES portal we would propose two download services:
1. Download pre-defined datasets. This would be direct downloads of postprocessed CCDAS output in netCDF. INSPIRE’s guidelines for encoding of spatial
data document26 mentions netCDF as a format for coverage-based data, particularly
in meteorological themes. The OGC has recently proposed netCDF for adoption as
an OGC standard (using the Climate and Forecast convention)27
2. Direct access download services via WCS, the most appropriate OGC web service
for direct access of grid coverage data. The draft INSPRE technical guidance
mentions WCS.
24
http://www.esri.com/news/arcnews/spring10articles/building-inspire.html
25
http://inspire.jrc.ec.europa.eu/reports/ImplementingRules/network/Draft_Technical_Guidance_Download_Services_v
1.0.pdf
26
http://inspire.jrc.ec.europa.eu/reports/ImplementingRules/DataSpecifications/D2.7_v3.0.pdf
27
http://www.opengeospatial.org/standards/requests/71
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
17/36
The specific needs of the end users may generate further requirements for additional
download services, besides the services mandated by INSPIRE. These requirements will
develop through the ongoing consultations with the users through the Climate Users Advisory
Group. We do not anticipate any implications for the technology selection. However if
necessary, there will be an opportunity to iterate in the later stages of the project, after the V0
portal.
Both the open source and commercial options would use the same technology for the predefined download option. The netCDF files would be hosted on a web server and the HTTP
URL will be made available through a discovery service.
ESRI provide support for an INSPIRE-compliant download service using their GeoPortal
Extension add-in, or con terra’s sdi.suite 28 (including the “INSPIRE download service
extension for ArcGIS Server”). ESRI UK recommend con terra’s product over ESRI’s own
extension, but have also raised the possibility of providing INSPIRE-compliant services
without additional extensions. CERC pursued this with ESRI but were unable to obtain clear
guidance.
MapServer can provide an INSPIRE-compliant WCS download service.
4.10.6 Discovery service
INSPIRE requires us to provide a compliant CS-W service.
There is uncertainty about what would be required to create an INSPIRE-compliant CS-W
service using ESRI’s ArcGIS Server. ESRI’s websites suggest their GeoPortal Extension,
which supports a number of protocols including CS-W, and supports a number of profiles
including INSPIRE CSW 2.0.2 AP ISO29. However, ESRI UK have stated that GeoPortal
Extension is complicated, and instead they recommend using con terra’s sdi.suite. ESRI UK
have also mentioned that it might be possible to provide an INSPIRE-compliant CS-W service
without any extensions. CERC pursued these issues with ESRI, but were unable to obtain
clear guidance and a firm price quotation.
MapServer can publish a variety of OWS (WMS, WFS, WCS, etc) from which a CS-W can
harvest metadata and publish for discovery, but cannot act as a CS-W. GeoNetwork is the
most popular open source CS-W package. It offers OGC compliance: in fact OGC cite
GeoNetwork as the reference implementation for the CS-W standard30. GeoNetwork is used
by large organisations31, including the UK government, the United Nations and the European
Space Agency. The UK government is sponsoring ongoing development of GeoNetwork.
Both ESRI and GeoNetwork provide well-supported INSPIRE-compliant CS-W discovery
services.
4.10.7 Data format
The main input for the CARBONES portal will be provided in netCDF format, a standard
format for scientific array-based data, which can contain metadata as well as data. It would
28
http://www.isotc211.org/WorkshopSouthampton/Workshop_presentations/WS_09_Paredes_Hahn.pdf
http://webhelp.esri.com/geoportal_extension/9.3.1/index.htm#hrvst_use_tool.htm
30
http://cite.opengeospatial.org/reference
31
http://www.geosur.info/geosur/contents/GeoNetwork_and_ESRI_GIS_Portal_Toolkit_Comparison.pdf
http://metheny.esrin.esa.int/geonetwork/srv/en/main.home
29
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
18/36
be desirable to be able to use this format directly in our solution rather than converting,
although conversion remains an option, particularly as some post-processing of the output will
be necessary so the conversion routines will exist.
Both MapServer and ArcGIS Server can read netCDF files to produce WMS and WCS,
although MapServer only supports certain formats of netCDF and they advise “for stability it
may be necessary to convert the files into GeoTiff format”32.
In the V0 portal we have found it necessary to follow this advice and convert the netCDF files
into GeoTiff format, using the free open source package gdal_translate, in order to publish
WMS using MapServer. As part of the further development of the portal beyond V0, we will
investigate using the netCDF directly from MapServer. See section 6.2 below.
There appears to be a problem using the temporal element33 of netCDF in ArcGIS Server,
but in general, the ArcGIS netCDF support seems to be slightly more robust than
MapServer’s.
As well as producing WMS and WCS services, both ArcGIS Server and MapServer can
consume external services in WMS and WCS. This may be useful in consuming data from
other services, for example MACC and GeoLand2.
4.10.8 Interactive maps
An essential part of the CARBONES web portal is the interactive maps. The functionality
required is summarised below:







32
Support for different internet browsers including Internet Explorer, Safari and Firefox.
Display different base mapping: proprietary free maps such as Microsoft’s Bing or
Google’s layers, and map layers available through WMS.
Basic navigation: zoom and pan functionality
More advanced navigation: overview maps, place name searches
Interaction with CARBONES data, including clicking to show individual data or links to
time series graphs.
Temporal data support through a calendar control, time selection control, etc.
Support for animations would be desirable but is not available in any standard web
mapping API at the present time. There are several possible methods for including
animations in the portal. All of these options would use different technologies from the
WMS/JavaScript API maps and would not allow the user to zoom and pan when
viewing the animations. It is not necessary to make the final selection of animation
technology now.
o JavaScript could be used to animate static images by manipulating the
browser Document Object Model, as in CarboScope 34. This would not allow
the end user to download the animations for offline use.
o Animations could be created in QuickTime movie format, as in
CarbonTracker35. Animated GIF files or Flash animations are other common
methods of presenting animations in web pages.
o The Google Earth API could be used to embed Google Earth animations into
the web pages36.
http://mapserver.org/ogc/wcs_format.html
http://forums.arcgis.com/threads/6086-Using-data-from-netCDF-with-ArcGIS-Server-10?highlight=netCDF
34
http://www.carboscope.eu
35
http://www.esrl.noaa.gov/gmd/ccgg/carbontracker/weather_movie.html
36
http://code.google.com/apis/earth/
33
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page

19/36
Support for printing.
ArcGIS Server provides client-side mapping utilising JavaScript, based on AJAX.
ArcGIS
Server provides a wizard to enable development of a front-end with zooming/panning, an
overview map, searches, data interaction, editing, printing, queries and geoprocessing 37 .
This can be customised by using the ArcGIS API for JavaScript. ArcGIS Server 10 also
adds a time-slider to deal with temporal data. Support for Web Services includes KML,
WMS, WCS, WFS and WFS-T (Advanced only). REST and SOAP access are included with
all editions38. Base mapping can be added using any ArcGIS Server cached map service, or
an ArcGIS Online base map39. ArcGIS Server can provide a client-side mapping utility
fulfilling all of our requirements.
The open source mapping JavaScript API under consideration is OpenLayers 40 .
OpenLayers supports KML, WFS and WMS, but WCS is currently not supported. However it
would be possible for MapServer to consume any external WCS on the server, and publish it
as a WMS into the JavaScript API. This would allow us to achieve all the functionality needed
for the CARBONES portal. Time-slider controls have also been used in OpenLayers 41 .
Examples of web map portals created with OpenLayers include the Sustainable Energy
Authority of Ireland’s geothermal maps at http://maps.seai.ie/geothermal
One element of displaying maps that is not yet covered is map tile caching, which is
sometimes used to improve performance. There are open source tile cache products such as
Meta Carta Labs Tile Cache. These may be used in development of the portal if it proves
necessary. There are no fundamental implications on the basic choice of technologies.
4.10.9 Content management system
A web content management system (CMS) is a software system which provides website
authoring and administration tools designed to allow users with little knowledge of web
programming languages or markup languages to create and manage the site's content with
relative ease.
A CMS would allow some CARBONES partners to edit the portal web pages directly, without
having to go through CERC. We require a system which can offer the following features.




Multilingual support
Security
Allow web designers without programming skills to create templates that define the
appearance of web pages
Ability to create custom modules through a programming API:
o for graphs generated on the server
o for our interactive maps
There are a number of free open source CMS systems available fulfilling our requirements,
for example Alfresco. These could be used with ArcGIS or MapServer: there would be no
direct interaction between the GIS server and the CMS.
37
38
http://www.esri.com/library/brochures/pdfs/arcgis-server-functionality-matrix.pdf
http://www.esriuk.com/products/showproduct.asp?prodid=26&groupid=2&activetab=1&mode=keyfeatures
39
http://resources.esri.com/help/9.3/arcgisserver/apis/javascript/arcgis/help/jshelp_start.htm#jshelp/overview_api.htm
40
http://openlayers.org/
41
http://www.mail-archive.com/dev@openlayers.org/msg03685.html
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
20/36
4.10.10 Post-processing
It will be necessary to post-process the CCDAS output data, including the carbon flux and
carbon store data. This post-processing will require tools capable of manipulating the
netCDF files. Possible tools include:



Commercial packages like MatLab, IDL or ArcGIS Server’s built-in tools;
Free open source tools like Ferret, the Climate Data Analysis Tools (CDAT), the
NCAR Command Language (NCL) and the netCDF operators (NCO).
The temporal filtering package CCGVU developed at NOAA. This is proprietary.
The post-processing calculations presented some special considerations. The development
of the scientific specification of the post-processing calculation has required considerable
iteration between CERC, LSCE and Noveltis, and is still continuing now. Therefore, to
mitigate the risk presented to the V0 deliverable timeline, the solution for the post-processing
in V0 is based on a commercial package (IDL 8.0.1) and a proprietary temporal filtering
package developed by NOAA, CCGVU. Both these packages have been proven in practise in
similar projects at LSCE; and CERC, Noveltis and LSCE had staff available with experience
in IDL.
We may study the use of open-source replacements for the V1 portal (see the future
development section below).
4.10.11 Database
The CARBONES portal will store some data in a relational database system, for instance the
post-processing output data for display on graphs and charts. The full output will not be
loaded into a relational database, since using the post-processed output files directly offers
more flexibility.
The Workgroup license for ESRI ArcGIS Server would restrict the selection to Microsoft’s free
product SQL Server Express. SQL Server Express is a file-based database, designed for
small datasets, with limits 42 on CPU support, memory usage, database size and
administration tools. It would be adequate for the CARBONES portal.
The free open source solution would be to use MySQL or PostGIS: PostGIS is based on
PostGreSQL. Both of these are robust and widely used and can support larger databases
than SQL Server Express. The advantage of PostGIS is its support for spatial extensions,
which offer more power in manipulation of geographic data than MySQL.
SQL Server Express, MySQL and PostGIS are all suitable for the CARBONES portal and
cost-free.
In the V0 portal we have used PostGIS to store spatial data: some of the tile indices required
to serve WMS-T with MapServer.
42
http://www.itproportal.com/portal/news/article/2010/7/3/limitations-sql-server-express/
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
21/36
5 Portal architecture
This section describes the CARBONES portal architecture, building on D220.1-D230.1.
Figure 2 showed an overview of the CARBONES system data flow. This indicates how the
post-processing and the web portal fit into the overall CARBONES system. The later
sections of this document discuss these two components in more detail.
5.1
Post-processing
The post-processing component is run off-line at CERC. The post-processing algorithms will
be developed in IDL. These algorithms will make use of the C and IDL temporal filtering
programs CCGCRV and CCG_CCGU.PRO, which belongs to NOAA.
5.1.1
Inputs
The input for the post-processing component will be netCDF files containing the output from
CCDAS. The format is specified in D220.1 section 4. The PM2-02 deliverable to be displayed
in the V0 portal contains only 2 years of data.
Outstanding issues with the netCDF formatting are:
- compliance with the Climate and Forecast (CF) conventions for netCDF due to the
data packing method used
- use of the CMIP-5 conventions for variables that are not yet included in CF (minutes
of progress meeting 2).
Both issues are to be resolved after the V0 portal.
The format of the netCDF files was originally described in deliverable D220.1.
Correspondence and discussions between CERC, Noveltis and LSCE have developed a
more detailed description.
The sample 2 years dataset (PM2-02) supplied by LSCE/Noveltis consists of 10 carbon flux
files and 4 stock files, as shown in Table 4. A further 4 variables have been derived from
those supplied. The derived variables are shown in Table 5.
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
22/36
No.
Type
Variable name
Symbol
Units
1
3
4
5
6
7
8
9
10
11
12
Land C flux
Land C flux
Land C flux
Land C flux
Land C flux
Land C flux
Land C flux
Land C flux
Ocean C flux
Land C stocks
Land C stocks
Land C stocks
Fnee
Fgpp
Fresp
Fnee_for
Fnee_gras
Fnee_crop
Fbbur
Ffos
Foce
Babove
Bbelow
LAI
kgC/m2/hr
kgC/m2/hr
kgC/m2/hr
kgC/m2/hr
kgC/m2/hr
kgC/m2/hr
kgC/m2/hr
kgC/m2/hr
kgC/m2/hr
kgC/m2
kgC/m2
m2/m2
13
14
Land C stocks
Land
Net ecosystem flux
Photosynthetic flux
Respiration flux
Forest net ecosystem flux
Grass net ecosystem flux
Crop net ecosystem flux
Biomass burning flux
Fossil fuel emission
Net ocean flux
Total above ground biomass
Total below ground biomass
Leaf Area Index (equivalent to leaf
carbon stock)
Soil organic carbon
Latent heat flux
SOC
Fle
kgC/m2
W/m2
2
Table 4. Carbon flux and stock files to be supplied by LSCE/Noveltis for V0
No.
Type
Variable name
Symbol
Derived equation
Units
15
Natural carbon fluxes
Fnatural
Fnee + Foce + Fbbur
kgC/m2/hr
Total carbon fluxes
Ftot
Fnee+Foce+Ffos+Fbbur
kgC/m2/hr
17
Land & ocean
C flux
Land & ocean
C flux
Land C flux
Fgppresp
Fgpp + Fresp
kgC/m2/hr
18
Land C stocks
Total photosynthesis
and respiration fluxes
Total biomass
Btot
Babove + Bbelow
kgC/m2
16
Table 5. Carbon flux and stock files to be derived by CERC for V0
The netCDF files supplied from CCDAS are ‘packed’ or compressed. This reduces the
physical size of the file through a linear transformation to map the original 32-bit floating point
values to a 16-bit integer value. To unpack the netCDF file, the following calculation is
performed:
Unpacked data = (Packed data x scale factor) + Offset
All values of 0 in the packed file indicate that the value is not available (e.g. for a forest net
ecosystem flux value at a grid point that is in the ocean) and do not correspond to a valid,
packed value of zero.
It remains to be resolved whether the ‘packing’ is compliant with the CF conventions.
NetCDF3 format has been used. Migration toward netCDF4 will be considered in later stages
of the project.
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
23/36
The data supplied comprise:
 Longitude
-179.5° to 179.5° in 1° steps
 Title = longitude
 Units = ‘degrees_east’

Latitude


-89.5° to 89.5° in 1° steps
Title = latitude
Units = ‘degrees_north’

Time
Decimal date format in years. For the year 1990, this will range
from 1990.000171232877 to 1990.999828767123 in steps of 3 hours i.e. 2920 points
in a non-leap year. The times are the mid-points of the 3-hour periods.
 Title = time counter
 Units = ‘decimal date’

Date array
An array with a row for each time point. The year column will contain the year (e.g.
1990), the month column will take a value between 1 and 12 and the day column will
contain a value between 1 to 28/29/30/31. The hour column will be either 1, 4, 7,
10, 13, 16, 19, 22 and the minute column will be 30. Therefore, a column in the date
array that reads:
1990,
4,
27,
4,
30
will correspond to 4.30 am on the 27th April, 1990.
 Title = date array yy/mm/jj/hh/mm
 Units

Variable
Gridded data for the specific variable values. There is a value for
each longitude and latitude point. There is a data grid for each time step.
5.1.2
Post-processing specification
CERC will post-process the data provided by LSCE, temporally and spatially aggregating
them as shown in Figure 4.
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
24/36
Fig
ure 4. Post-processing to be performed by CERC. See Table 6 for definitions of P, T
and A.
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
25/36
For the 2-year dataset, the time filtering CCGVU program fits the data to a polynomial of
degree 2, with 3 harmonics. The residuals are filtered in the frequency space using a lowpass filter of 667 days and a high-pass filter of 80 days. The outputs that are then passed
on to the CARBONES web portal are: the smoothed trend (T); smoothed curve (S); recentred parameter A. To calculate A the period mean is subtracted from the original, raw,
data.
The output values from the temporal filtering process are summarised in Table 6.
CARBONES post-processing
output parameter
Description
Input values, P
The original input data (flux, stock)
Smoothed trend, T
The polynomial fit plus the long term filter of the residuals
Smoothed curve, SC
The fitted function plus the short term filter of the residuals
Re-centred parameter, A
The original data minus the period mean.
Re-centred
A_SC
smoothed
curve,
The smoothed curve (SC) minus the period mean. A_SC
is not included in V0 of the portal.
Table 6. Output parameters from the CCGVU_PRO program
The spatial aggregation uses region masks provided by LSCE. There are 2 netCDF files for
the region masks, namely regions_mask.nc and regions_mask_extend.nc. In both files,
there will be 52 regions, with the difference being that the ‘extended’ masks have expanded
boundaries where there are land-only or ocean-only regions. The 52 regions are shown in
Table 7. These regions are also used in the portal web mapping “regions” control. The
expanded boundaries of the regions_mask_extend.nc include adjacent ocean/land grid boxes
of 1x1 degree. The region file selection will depend on the region type. The file
regions_mask.nc is used for regions that are land and ocean, and regions_mask_extend.nc is
used for all other regions.
A third set of region masks was supplied to CERC by LSCE for use by the web portal in
creating maps of regions. This set masks the ocean regions for land only quantities and the
land regions for ocean only quantities. It is said to be more general than regions_mask.nc and
regions_mask_extend.nc and does not rely on the CCDAS particular definition of land/sea
regions.
The precise treatment of missing data (NaN values) in the original data remains to be
resolved. CERC, Noveltis and LSCE are working together to specify this treatment.
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
26/36
Region name
Land/
ocean
Region name
Land/
ocean
Global
Both
Southern sub-antarctic ocean
Ocean
Northern land and ocean
Both
Austral ocean
Ocean
Tropical land and ocean
Both
Boreal North America
Land
Southern land and ocean
Both
Temperate North America
Land
Global land
Land
Canada
Land
Northern land
Land
USA
Land
Tropical land
Land
Northern Europe
Land
Southern land
Land
Central mid-Europe
Land
Tundra
Land
Eastern Europe
Land
North America
Land
Mediterranean Europe
Land
Europe
Land
Boreal Asia
Land
North Asia
Land
Temperate Asia
Land
Tropical Asia
Land
Boreal Asian forest
Land
Southern America
Land
Boreal Asian steppe
Land
Africa
Land
China-Korea-Japan-Mongolia
Land
Australia and New Zealand
Land
Arabia
Land
Global ocean
Ocean
South Asia
Land
Northern ocean
Ocean
South east Asia
Land
Tropical ocean
Ocean
Equatorial Asia
Land
Southern ocean
Ocean
Australia
Land
North Atlantic ocean
Ocean
North Africa savannah
Land
North Pacific ocean
Ocean
Equatorial African forest
Land
Tropical Atlantic ocean
Ocean
South Africa
Land
Tropical Pacific ocean
Ocean
Tropical South American forest
Land
Tropical Indian ocean
Ocean
Tropical South American savannah
Land
Southern sub-tropical ocean
Ocean
South America temperate
Land
Table 7. The regions for the spatial aggregation and the portal web mapping
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
5.1.3
27/36
Outputs
The output parameter-averaging time combinations that will be passed on to the portal are
shown by the white boxes in Tables 8 and 9, for maps and time series respectively. A_SC is
the re-centred value of the smoothed curve and is not included in V0 of the portal.
Averaging C flux and latent heat flux
time
P
T
SC
A
C stocks including LAI
A_SC
P
T
SC
A_P
A_SC
3-hour
Daily
Monthly
Annual
20yr
Table 8. Parameter-averaging time combinations used for maps in the web portal
The gridded data for maps have been created in netCDF files in an identical format to the
supplied CCDAS data, except that the files are not packed. These files are the input to the
portal server for the generation of maps and direct-access data services through WMS and
WCS. They will be made available for direct download by the end-user through the web
portal but the V0 data sets have not been made available for download as they contain only
preliminary data.
Averaging C flux and latent heat flux
time
P
T
SC
A_P
C stocks including LAI
A_SC
P
T
SC
A_P
A_SC
3-hour
Daily
Monthly
Annual
20yr
Table 9. Parameter-averaging time combinations used for time series (regional
aggregates) in the web portal
The 1-D time series resulting from the spatial aggregation and temporal filtering have been
created in a simple ASCII CSV (comma separated variable) format. These are not in V0 but
will be made available in V1 of the portal. These data will be loaded into a database for use
by the web portal server to create graphs. We do not anticipate making the whole post-
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
28/36
processing output CSV files available for direct download by the user: instead the web portal
server will provide a CSV download service that queries the data in the database.
For V1 there will be animations, created off-line at CERC during the post-processing that can
be displayed on the web portal. Possible formats include KML/KMZ (for display in Google
Earth), Flash or QuickTime. Some of these formats are proprietary, closed, formats but they
have the advantage of wide user acceptance for deploying multimedia on the web. The end
user of the CARBONES animations would not be required to purchase special software.
There are alternative open standards but they have not seen significant user uptake. This
area of the architecture remains open, and will be determined at a later time.
5.2
Portal server and portal web application
Figure 5 shows the detailed architecture for the portal web server and web application.
The custom code on the portal web server has been developed in Java and the web
application in JavaScript / HTML / CSS. Some third party components have also been used.
The choice of components is listed in the analysis of engineering solutions, section 4.2.
5.2.1
Inputs
The inputs for the portal web server are the outputs of the post-processing, and the CCDAS
output. Both are described above.
5.2.2
Outputs
We are implementing the following web services from the portal web server.
 Discovery services: access to metadata served by OGC CS-W.
 View services: maps and map legends served by OGC WMS. These will be used
by the CARBONES portal web application and will also be available for use in
other services, e.g. other geoportals or other GMES services.
 Download services
o Downloading a pre-defined dataset. Download of the netCDF files created
in the post-processing and by CCDAS
o Direct access data services: direct programmatic access to the gridded
data in the netCDF files (post-processing output and CCDAS output),
served by OGC WCS.
The portal web mapping application is a JavaScript application which runs in the user’s web
browser when they visit the www.carbones.eu website. The maps are displayed using
OpenLayers, a JavaScript client for OGC WMS, connected to the OGC WMS service from the
portal web server. Dojo, a well-known open source JavaScript prototyping library, has been
used for some of the controls in the web mapping application.
The graphs will be generated dynamically on the server using JFreeChart. The server-side
code will access the post-processing output from a database on the server.
The download of graph data will use a dynamic service on the web server, which will query
the data from the database.
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
29/36
Page
CCDAS output, post-processing output
Portal Server
CMS content
Data for graphs
Metadata
netCDF
JFreeChart
CMS
MapServer
GeoNetwork
Chart Module
Module
Map Module
HTML
CSS
JavaScript
Images
Maps
WMS
Maps, direct
access to
data
WMS, WCS
Download
data files
netCDF
Metadata
CS-W
Web browser
carbones.eu
Maps,
data,
metadata
WMS, WCS,
CS-W, netCDF
OpenLayers
Custom code
Dojo
Interactive maps and
graphs
General public
Scientific users,
policy makers
Other core
services
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
30/36
6 Further development after V0
6.1
Development for the first annual meeting
Development of the web portal will continue after the delivery of the V0 version. Here is a list
of the features to be implemented, in priority order. We hope these features will be completed
by the Annual Meeting in May 2011.
Post-processing
1. Complete the specification of the spatial aggregation, including treatment of missing
values in calculations (NaN).
Post-process and integrate the 20-year dataset PM3-04
2. LSCE are to deliver 20-year fluxes to CERC by 15th of March43. These fluxes will
include: ORCHIDEE + oceanic fluxes for 20 years. The biospheric CO2 fluxes will
be optimized based on the PYVAR assimilation system. CERC will work to postprocess this data and integrate it into the portal by the Annual Meeting 44. It is
unlikely to be possible to integrate all the data by the Annual meeting.
Interactive time series graphs for stocks and fluxes
3. Line and bar graphs with labelled axes, and legends identifying the series
4. Ability for user to change the ranges on the axes
5. Download underlying graph data as CSV
Further features for the web mapping.
6. User-controlled transparency for the flux and stock maps, with satellite imagery as
background maps. We will offer the user a choice of background maps. There will
be some foreground layers to be shown as transparent overlays on top of the
CARBONES data layers, e.g. continent outlines and country outlines. We will
offer the user the facility to turn off these overlays.
Animations
7. Animations of the changes to carbon fluxes and stocks using the monthly values.
This will not be done in the interactive maps but separately, e.g. with an animated
GIF, QuickTime, Flash or Google Earth. The user should be able to download the
movie (D230.1 section 4.3.2.2) which would not be possible if the animations
were hosted in the interactive maps.
6.2
Development after the first annual meeting
There will be another phase of requirements analysis and design after the first annual
meeting. This will take into account various inputs.
 Partner feedback on the V0 portal
 EU reviewer feedback on the V0 portal
 Second CUAG meeting (D120.3 due end of September 2011)
 AEAT refined market analysis (updated D720.3 due end of March 2011)
Most of these inputs should be available in May 2011 for the first annual meeting. It may
be possible to begin the discussions at this point.
43
44
Action PM3-04 from progress meeting 3, 1st February 2011
Action PM3-06 from progress meeting 3, 1st February 2011
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
31/36
We also propose another version of the portal, V0.5, to be delivered at the end of
December 2011. This version is not required by the DoW but ensures progress made is
publicly available before the deliverables required by the DoW, V1 being required at month 30.


V0.5 will display the first CARBONES 20 years data set with assimilation of data (due
to be made available by LSCE at the annual meeting45).
Creating the V0.5 portal will allow some feedback from the further requirements
analysis phase to be realised in a live portal. This will help with further discussions
about features and priorities for the final portal, V1.
Here is a list of features to be considered for the V0.5 portal
Post-process and integrate the 20-year assimilated dataset (PM3-07) to be made available by
LSCE at the first annual meeting.
8. After the annual meeting. CERC will begin work to post-process this data and
integrate it into the portal.
The following features have been agreed but have not yet been prioritised for order of
implementation. They may be left until V1.
9. INSPIRE compliant services (CSW, WMS, WCS).
a. INSPIRE compliant WMS and WCS can be provided simply by configuring
MapServer appropriately.
b. INSPIRE compliant CSW will be provided by a cataloguing component.
CERC’s current selection is GeoNetwork but there is a possibility that we
may use the GENESIS cataloguing toolbox. This selection will be made after
the annual meeting.
10. Use a Content Management System so that website is easier to edit (for CERC and
some project partners)
11. Website is to be translatable
12. Portal to present analysis of sensitivity of regional carbon fluxes to climate change.
This feature needs to be defined more precisely.
13. Portal to present comparisons between model fluxes and other similar products. This
feature needs to be defined more precisely.
14. Portal to present uncertainties trend detection and vulnerability detection, sensible
heat fluxes, and other aspects that are only available in the V1 CCDAS output
D510.2. See D230.1 for details. This feature needs to be defined more precisely.
The features listed below are speculative and may be implemented in the V1 portal, or may
not be implemented at all. We have not yet decided the priority of these features, nor
assessed their practicability. Later discussions with project partners and others may suggest
additional desirable features that are not listed here.
15. Click on a point on the map to obtain:
a. the numeric value at the point
b. a time series graph for the point
16. Overview map
17. Printing maps / saving map images / creation of PDF
18. Printing graphs / saving graph images / creation of PDF
19. Export maps to Google Earth
45
Action PM3-07 from progress meeting 3, 1st February 2011
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
32/36
20. Add/remove data series to graphs interactively
21. Place name search in the maps
22. User-defined region of interest for spatial aggregation (may require post-processing
to be performed dynamically – impractical?)
23. Download data for the map currently being viewed (i.e. dynamically create a small
netCDF file containing a subset of the data).
24. In the V0 portal we have found it necessary to convert the netCDF files into GeoTiff
format, using the free open source package gdal_translate, in order to publish
WMS using MapServer. We will investigate using the netCDF directly from
MapServer. This is likely to require the netCDF files to follow the Climate and
Forecast conventions. We may make use of the open-source ADAGUC
modifications to GDAL and MapServer 46.
46
http://trac.osgeo.org/gdal/wiki/ADAGUC
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
33/36
7 Test plan for the V0 portal
7.1
Post-processing
Each stage of the IDL post-processing programme was tested and the outcome of each
testing stage recorded.
The code was temporally tested for one real grid point (110, 43) or (108, 56) and one missing
data (‘NaN’) grid point (66,121) for the following variables:
Variable
Variable type (Original/derived; land/ocean)
Fgpp
Original, land C flux
Foce
Original, ocean C flux
Babove
Original, land C stocks
Fnatural
Derived, land & ocean C flux
Ftot
Derived, land & ocean C flux
Table 10. Variables used for testing
For the spatial aggregation, the code was tested using one mask and one time point. Manual
spot checks were performed to ensure that the data values were read correctly from the input
netCDF file.
Note that in the testing files, there were slight differences in values due to IDL and C program
rounding errors. The percentage difference was used to determine whether values were a
match or not rather than an absolute difference.
The IDL code includes checks to ensure the data are in the expected format. If the data do
not fit the expected format the programme stops with an error message.
a. Checking that the variable data are read in correctly from netCDF files:
-
the dimension of the longitude should be 360
the dimension of the latitude should be 180
the dimension of the time variable should be 5840 (corresponding to 2 years of 3
hourly data)
the dimension of the variable should be 3 (i.e. that the variable has a value for each
longitude, latitude and time point)
the date dimension is as expected
the sum of the first half of the year entries in the date array should equal 1990
the sum of the second of the year entries in the date array should equal 1991
the months, days, hours and seconds in the date array, should start at 01:30 on 1 st
January 1990, be at 3 hourly intervals and contain 8 values per day
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
34/36
b. Checking that zero's are replaced by NaN's (missing data)
-
the number of real values added to the number of ‘NaN’ values after unpacking the
data should equal the unpacked number of data values
the number of zeros in the packed data should equal the number of ‘NaNs’ after
unpacking
Testing was carried out at each of the stages in the IDL program listed in Table 11.
Number
Description of IDL post-processing step tested
1
Mask data read in correctly from netCDF files
2
Raw derived data as expected
3
Spatial aggregation of 3 hourly data
4
Time averaging: daily means
5
Spatial aggregation of daily mean data
6
Time averaging: monthly means
7
Spatial aggregation of monthly mean data
8
Time averaging: yearly means
9
Spatial aggregation of yearly mean data
10
Time averaging: period mean
11
Calculation of daily A (‘re-centred mean’)
12
Spatial aggregation of daily A
13
Time averaging: monthly mean A
14
Spatial aggregation of monthly A
15
Time averaging: yearly mean A
16
Spatial aggregation of yearly A
17
Temporal filtering of daily means
18
Spatial aggregation of daily trend
19
Spatial aggregation of daily SC (smoothed curve)
20
Decimal date testing
21
Writing the netCDF files (raw_out)
Table 11. Program stages tested
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
7.2
35/36
Test plan for web portal
The following steps have been followed to test the web portal. The test steps will be repeated
with different browsers in order to test the browser compatibility of the website.
To test the content pages:
1. Visit each of the content pages from www.carbones.eu by clicking the links from the
home page, and check the appearance. The appearance should be checked with the
different accessibility options selected: normal font, large font, very large font, and
high contrast.
a. Home page
b. Project page
c. Partners page
d. Contact details
To test the web mapping application:
1. Load the web mapping application page. Check that the controls appear correctly.
The tree view list of variables should appear, with the variables expanded.
“Photosynthetic flux” should be selected. The map and legend should appear for
photosynthetic flux and the whole world should be visible. The drop-down selections
should be “3-hourly” and “Mean”, and the date-time selection should be the first
timestep 1/1/1990 00:00-3:00
2. Select different variables from the tree control. Check that the map and legend
change. Check that the tree control only permits one variable to be selected at a time
(“radio button” behaviour).
3. Select different options from the temporal averaging period drop-down and the
temporal filtering drop-down. Check that the map and legend change when these
selections are made. Check that the hour drop-down is disabled (greyed out) except
when the temporal averaging period is “3-hourly”. Select different variables in the tree
and check that the available options in the temporal averaging period and temporal
filtering drop-down change appropriately.
4. Select different time steps from the calendar control / hour drop-down and check that
the map and legend change appropriately. Check that the selection is restricted to the
valid range of dates and times: 1/1/1990 to 31/12/1991 (with the 2-year data set).
5. Select different regions from the region tree control and check that the map zooms to
the correct extent and that areas outside the selected region are hidden.
6. Test the use of the map controls.
a. Use the mouse to click the left, right, up and down arrows, and check that the
map view is panned.
b. Use the mouse to click the plus and minus buttons and check that the map
view is zoomed in and zoomed out.
c. Use the mouse to click the globe and check that the map view is returned to
the full extent (the whole world should be visible)
d. Use the mouse to pan the map by clicking the mouse button, and then
holding the button down while dragging the mouse.
e. Use the mouse wheel button (if available on the computer) to zoom the map
in and out by scrolling the mouse wheel button.
Deliverable D520.1
Ref
CARBONES-D520.1-REP-CERC-015-01-00
Date
31/03/2011
Page
36/36
8 Test report for the V0 portal
8.1
Post-processing
Stages 1-21 described in Table 11 were tested.
following comments or exceptions.
The results were as expected with the
[3] Spatial aggregation of 3 hourly data
For derived variables such as Fgppresp, the spatial aggregation tests give the correct results
when the constituent variables have NaNs at the same spatial location. However, when NaNs
occur at different locations the sum of the spatially aggregated variables cannot equal the
spatially aggregated derived variable. This is due to the way the derived variables are
created i.e. if one of the constituent variables has an NaN, then the derived variable is set as
NaN. The treatment of NaN is a matter still being discussed between the CARBONES
partners.
8.2
Test report for web portal
The testing steps described in the test plan were carried out in the web browsers listed in
Table 12. The web application behaved correctly in all cases.
Browser
Version
Operating system
Internet Explorer
7
Windows XP
Internet Explorer
8
Windows Vista
Firefox
3.5
Windows XP
Chrome
10
Windows XP
Safari
4
Macintosh OS X (Snow Leopard)
Table 12. Web browsers used for testing the web application