Private Public Partnership Project (PPP)
Large-scale Integrated Project (IP)
D.2.6.3: State of the Art Analysis – Emerging Technologies
Project acronym: FI-WARE
Project full title: Future Internet Core Platform
Contract No.: 285248
Strategic Objective: FI.ICT-2011.1.7 Technology foundation: Future Internet Core Platform
Project Document Number: ICT-2011-FI-285248-WP2-D.2.6.3
Project Document Date: 2013-06-27
Deliverable Type and Security: Public
Future Internet Core Platform
Author: FI-WARE Consortium
Contributors: FI-WARE Consortium
1.1
Executive Summary
The present document provides an analysis of the State of the art in the main domains of knowledge
that concern FI-WARE, namely:







Applications & Services
Cloud computing
Internet of things
Data Context management
Interface to Networks and Devices
Security
Middleware and Web User Interfaces
There are external references for each one of the chapters to provide our data sources and also to
supply pointers to those interested in expanding their knowledge in a concrete area beyond the
information reflected in the present document.
The First issue (D2.6.1) was skipped with the explicit approval of the EC and the present one is coded as
D.2.6.2 consistently with the delivery schedule.
D.2.6.3 FI-WARE State of the Art
Page 2
Future Internet Core Platform
1.2
About This Document
This document consolidates the inputs of the technical chapters of FI-WARE that combined their efforts
in order to put together a comprehensive view of the landscape in each area of the Future Internet. The
document delves into the most recent technological advances as well as innovative market disruptions
and user trends that may impact or are already impacting the Future Internet.
All the partners have contributed to reflect the State of the Art in their knowledge domains but the
academia has put a relatively higher effort, as planned.
1.3
Intended Audience
The audience of this document is very wide. It targets anyone interested in the innovation areas of the
Future Internet. As such, it comprises the FI-WARE project partners, the Use Case Projects, the FI-PPP
Architecture Board and the EC.
Those outside the PPP interested on the technology foundation of the Future Internet services in
different vertical sectors may find also helpful references and ideas. This therefore also targets decision
makers within companies, who consider the adoption of FI-WARE technologies, investors in companies
targeting implementation of FI-WARE technologies, market researchers and analysts trying to evaluate
the potential of FI-WARE technologies
1.4
Structure of this Document
Unlike most of the deliverables in FI-WARE, this document is not part of the wiki and therefore it is not
generated from wiki pages. It follows however the same template as the wiki generated documents.
The structure is very straightforward and there is a separate section per chapter plus the usual
introductory block.
1.5
Typographical Conventions
Starting with October 2012 the FI-WARE project improved the quality and streamlined the submission
process for deliverables, generated out of the public and private FI-WARE wiki. The project is currently
working on the migration of as many deliverables as possible towards the new system.
This document was edited on Microsoft Word following the common template that is shared with the
semi-automatic scripts out of a MediaWiki system operated by the FI-WARE consortium.
D.2.6.3 FI-WARE State of the Art
Page 3
Future Internet Core Platform
1.6
Acknowledgements
The current document has been elaborated using a number of collaborative tools, with the participation
of Work Package Leaders and Architects as well as those partners in their teams they have decided to
involve.
The R&D centres played a significant role in the creation of the body of this document given their
expertise in the different fields of our interest within the project scope.
1.7
Keyword list
FI-WARE, PPP, Architecture Board, Living Lab, OIL, Steering Board, Roadmap, Reference Architecture,
Generic Enabler, Open Specifications, I2ND, Cloud, IoT, Data/Context Management, State of the Art,
Applications/Services Ecosystem, Delivery Framework , Security, Developers Community and Tools , ICT,
es.Internet, Latin American Platforms, Cloud Edge, Cloud Proxy.
1.8
Changes History
Release
V0.1
Major changes description
First draft of WP13 contributiion
Date
2014-03-13
Editor
WP13
V0.2
V1.0
1.9
Table of Contents
1.1
Executive Summary ......................................................................................................................... 2
1.2
About This Document...................................................................................................................... 3
1.3
Intended Audience .......................................................................................................................... 3
1.4
Structure of this Document ............................................................................................................. 3
1.5
Typographical Conventions ............................................................................................................. 3
1.6
Acknowledgements ......................................................................................................................... 4
1.7
Keyword list ..................................................................................................................................... 4
1.8
Changes History ............................................................................................................................... 4
D.2.6.3 FI-WARE State of the Art
Page 4
Future Internet Core Platform
1.9
2
Table of Contents ............................................................................................................................ 4
Middleware and Web User Interface Technologies ............................................................................... 6
2.1
Advanced Middleware technologies ............................................................................................... 6
2.2
Web User Interface Browser Core Technologies ............................................................................ 7
2.3
Web User Interface Server Core Technologies ............................................................................... 8
2.4
Web User Interface Supporting Services......................................................................................... 9
2.5
Web User Interface Application-Oriented Technologies .............................................................. 10
2.6
References ..................................................................................................................................... 11
D.2.6.3 FI-WARE State of the Art
Page 5
Future Internet Core Platform
2
Middleware and Web User Interface Technologies
In order to become widely visible and adopted by end users, Future Internet technologies must not only
offer server functionality but must also offer much improved user experiences. The Advanced Webbased User Interfaces are based on an extended version of HTML-5. The objective is to significantly
improve the user experience for the Future Internet by adding new user input and interaction
capabilities, such as interactive 3D graphics, immersive interaction with the real and virtual world via
Augmented Reality, virtualizing the display by driving them over the network, and many more. To build
highly efficient and secure distributed applications, it is required to support a wide range of
communication scenarios. Future Internet Advanced Middleware technologies are built to support these
functionalities.
2.1
Advanced Middleware technologies
There has been a long history of middleware for supporting distributed applications that goes back to the
original ARPANET almost 40 years ago and has developed into different form of middleware [Schantz86,
Bernstein96, Myerson02, Emmerich00]. [Schantz02] provides a nice overview. Relevant middleware for
remote procedure calls (RPC) or the related remote method invocation (RMI) includes ONC-RPC [ONCRPC] designed for use with the NFS file system, DCE RPC [DCE] for early distributed applications, CORBA
[CORBA] as a full object-oriented message broker with an extensive infrastructure, DCOM [MS-DCOM]
and Java RMI [RMI] as approaches for remote method invocation built into their respective runtimes,
XML-RPC as the predecessor for Web services [WS], and many others.
A key role in the context of distributed applications today is played by the Web Service standards, here in
particular SOAP [SOAP] and REST [REST, Fiel00]. The key idea is that all data in messages transmitted is
structured and self-describing using XML-based formats. SOAP (originally Simple Object Access Protocol,
successor to XML-RPC) defined an XML-based message envelope into which the payload data is encoded
also as XML. All data is therefore transmitted in text form (while options for binary encoding of
structured data exist, they are not commonly used). SOAP usually relies on an Application Layer protocol
for message negotiation and transmission, most often the Hypertext Transfer Protocol (HTTP) but Simple
Mail Transfer Protocol (SMTP) is also specified. They are all based on TCP/IP.
RESTful services [REST, Fiel00,FT02] typically access resources via the HTTP protocol and thus have not
been specifically designed for method call or streaming communication, but they can be used to emulate
them with some overhead. REST does not define a separate protocol but essentially reuses the HTTP
envelope and its operations or “verbs” (get, put, post, delete) and allows essentially arbitrary, embedded
payload formats. Typically XML- or [JSON]-formats over HTTP are being used for Restful services.
Despite the long tradition and large number of systems as well as the dominance of SOAP or RESTful
Web services today, several new communication middleware frameworks appeared over the last few
years. In particular the needs of large players like Google and Facebook for efficient data processing on
cloud clusters (e.g. large databases, real time response) have led to new requirements and new solutions
D.2.6.3 FI-WARE State of the Art
Page 6
Future Internet Core Platform
like Protocol Buffers: Google’s Protocol Buffers [GP], Apache Thrift [Thrift], Cisco Etch, Apache Avro
[AVRO], ICE [ICE] and ZeroMQ [0MQ].
Another alternative is the "publish-subscribe" Data Distribution Service for Real-Time systems [DDS]
created in 2004 by the Object Management Group (OMG). DDS is based on the DDS is mainly used in
very specific industries like Military or Aerospace. It offers the basic messaging services needed in all
middleware communication systems and other layers can be implemented on top of DDS. Some
commercial [RTI-DDS][CoreDX] and open source implementations [OpenDDS][OpenSplice] of DDS are
available. The publish-subscribe programming model is very flexible, but needs some familiarization by
the developers. Therefore OMG has started 2012 the standardization process for RPC over DDS [RPCDDS], which makes the standard RPC client-server paradigm available for DDS.
All of these recently designed systems address specific problems and offer clear advantages over
previous systems. However, each one of them still lacks important features and there is no general
architecture for distributed systems. Here are a few of the drawbacks that apply to some or all of the
above projects: they are not backwards compatible to Web Services (with the exception of Etch), they
cannot dynamically adapt to the requirements of peers (with the exception of Avro), they can generate
fast de/marshalling code only at compile time, they do not offer a data access layer that allows for
efficient access to application data structures, they mostly do not support specialized transport
mechanisms like shared memory and others, there is often no real security mechanism, the systems
cannot adapt to the (changing) requirements and QoS levels of the application and its environment. The
serialization formats vary significantly, as documented in [Serial], which provides a detailed comparison
of many different serialization formats used in these and other systems.
FI-WARE Advanced Middleware addresses the shortcomings of the existing solutions on multiple layers.
It will (1) improve the interface between application and middleware by providing a dynamic declarative
mapping between application data-structures and transport formats and simultaneously optimizing the
data flow by minimizing the copy operations. Beside the more (2) performing binary data encoding it will
also (3) support traditional text-based Webservices. Runtime Negotiation (4) will allow choosing the
optimal transport mechanism and we will also provide (5) integration into modern software defined
networking [SDN] infrastructure. For the first time in the area of distributed systems, especially in the
web services domain, FI-WARE advanced middleware investigates the use of a (6) design conformant to
Common Criteria for IT-Security Evaluation [CC] rules for all security aspects of a complete, technically
involved system and will address additional security features like field encryption, and data
anonymisation.
Currently OMG is also developing the DDS security standard [DDSSecurity], which supports pluggable
security modules for authentication, access control, cryptography, key managment logging and data
tagging.
2.2
Web User Interface Browser Core Technologies
2D-UI: A key contribution in FI-WARE has been the use of early versions of the W3C WebComponent
specification to create object-based descriptions of higher-levels entities in 3D worlds. Additionally,
D.2.6.3 FI-WARE State of the Art
Page 7
Future Internet Core Platform
initial work has been done on providing abstracted user input for the wide range of modern (3D) input
devices.
Upcoming research will allow to build on top of that and evolve FI-WARE technologies by (i) making
major advances in better abstracting input devices as well as their usage in 3D-Web applications and (ii)
extend the usage of WebComponents also for input specifications and other use-cases.
3D-UI: Originally HTML was limited to text, images, and simple layout. 2D and 3D vector graphics were
only available as plug-in (e.g. Flash, SVG, X3D, Unity). Only with HTML5 [W3C 12] and WebGL [Khronos
11] the capabilities of the HTML canvas were extended by a low-level graphics API. However, HTML is a
declarative scene graph approach and does not mix well with a low-level API. One approach are scene
graph APIs like Three.js [mrdoob.github.com/three.js]. In contrast, declarative, DOM-based approaches
[Behr 09, Sons 10] extend HTML5 by new HTML elements for 3D graphics and are much better aligned
with HTML and its developer skills.
FI-WARE technology is based on XML3D [Sons 10] and will in future extend the current state-of-the-art
by (i) adding a much extended content pipeline for integrating 3D content, (ii) a 3D asset data base
including versioning and different, Web-optimized 3D data formats (working with the glTF initiative at
Khronos), (iii) improved resource management for adaptive, on-demand loading and unloading of scene
content, (iv) improved developer tools for distributed and synchronized Web applications, and (v)
significantly improving the access to hardware resources for computational expensive tasks like AR,
animation, and others. The 3D-UI technologies will closely tie together with the Interface Designer
technology that will provide all of the features mentioned above in an interactive context directly to
users.
2.3
Web User Interface Server Core Technologies
Synchronization: Today, real-time synchronization of 3D scenes is limited to specialized applications such
as SecondLife (secondlife.com) and games engines like Unreal and others. The protocols in use in these
engines are typically rather specific to gaming and do not offer generic solutions. On the research side,
two efforts stand out: The Sirikata/Meru system [ECPF12] developed at Stanford and the Virtual World
Infrastructure Project at Intel Labs [Liu 12]. Both are focused only on the server side and use different
strategies to achieve scalability of virtual worlds. Other research activities are Open Cobalt
(opencobalt.org, previously Open Croquet) and others.
FI-WARE technology is currently based on realXtend (realXtend.com) [Dahl 13]. Extensions beyond the
current state-of-the-art can be done by including (i) automatically partitioned spatial domains for
different aspects of a scene (scripting, physics, etc.) using scalable synchronization algorithms like those
from [Liu 12], (ii) advanced interest management to better determine which synchronization data needs
to be prioritized [Vatjus-Anttila 13], (iii) a network-efficient communication layer, ideally based on the
Advanced Middleware of FI-WARE.
D.2.6.3 FI-WARE State of the Art
Page 8
Future Internet Core Platform
2.4
Web User Interface Supporting Services
Cloud Rendering: Real-time rendering in the cloud has become a major factor in the industry by now,
ranging from GPUs optimized for network delivery of the rendered images with very low latency
provided by many well-known companies, like Autodesk, RTT, and others. However, most if not all of
these services are proprietary and are tightly connected with specific products.
Future Internet technologies will add open technology to this, such that it can easily be reused to build
novel services. In particular, will allow integrating advanced rendering features, like ray tracing and
exploiting novel portable material descriptions.
Display as a Service: In the past displays had to be connected using special HW connections that are
expensive for non-standard (1:1) setting. Software solutions like SAGE [Brown 10], Apple iPlay, or WiDi
only address very specific parts of the application spectrum, like only one-to-one connections or only
scientific applications. Display as a Service acts as a general, virtual SW video crossbar and covers most
application scenarios. It is fully synchronized and virtualizes not only the displays but also their physical
placement, resolution, etc.
Upcoming RTD activities will continue to evolve the Display as a Service technology to go beyond this by
including (i) also a back-channel for abstract user interface streams and by also (ii) integrating hardware
support for the main pipeline elements, including video de/encoding, pixel processing, and others.
GIS Data Provider and POI Data Provider: These technologies mainly build on top of existing services
(W3DS) and formats (Point Of Interest, POI) defined by the Open Geospatial Consortium (OGC)
standardization organization. FI-WARE has started to work with OGC in order to extend these
specifications to meet the advanced requirements for AR and related applications.
FI-WARE RTD activities continue this work and push for standardization of the enhanced features within
the on-going standardization processes. Work is still required in 3D navigation techniques including
indoor navigation, working towards a fully 3D navigation application using openly available 3D
information on the open Web.
2D-3D Capture: The key contribution in this area is the simplification of capturing 2D images and video
particularly in a mobile, Web-based context and to interface with 3D reconstruction services that create
3D models and others out of the captured material. There have been a few such services, mostly in the
research area, e.g. [Vergauwen 06], but commercial services are also becoming available (e.g. Autodesk’s
123D-Catch app, or www.my3dscanner.com), which are increasingly driven by the trend towards 3D
printing.
Fi-WARE RTD activities will help to evolve these technologies to go beyond these isolated applications
and offer a unified interface to 3D reconstruction services. In particular, to (i) more tightly integrate with
streaming services for simplifying the capturing process, (ii) extend “streaming” to cover more than just
video data, towards i.e. streaming of continuous sensor and other data, (iii) include other (meta-) data
such as position and orientation information, and (iv) coordinate more closely with the Real-VirtualInteraction technologies with which it shares some commonalities.
D.2.6.3 FI-WARE State of the Art
Page 9
Future Internet Core Platform
2.5
Web User Interface Application-Oriented Technologies
Augmented Reality: AR has been around for very long with a long history in research [Milgram 94,
Azuma 01]. One can distinguish between coarse AR, which provides location based services based on the
rough position and orientation of the user (e.g. layar.com, wikitude.com), and the traditional notion of
AR with image-accurate real-time features tracking (e.g. metaio.com). Well-known image descriptors
used in AR include SIFT [Lowe 04], SURF [Bay 08], MSER [Forssen 07] and others. In addition geometry
must be estimated from these features, e.g. using Bundle Adjustment [Engels 06] or other, more efficient
methods like Relative Bundle Adjustment [Holmes 09]. The goal is to create a complete, high-level, highperformance AR solution for the 3D-Web.
Thus, evolving FI-WARE technologies will allow to (i) create a first Web-based solution that solves the
performance and hardware access issues and (ii) offer tools that make it easier to integrate AR with
interactive 3D scenes, repositories, and shared 3D spaces. Additional advanced features, such as, (iii)
image marker recognition, marker-less recognition and face recognition will also be added as features.
Real Virtual Interaction: Augmented Reality has always been about the interaction with a mixed reality,
combining the real and the virtual world. At the same time the Internet of Things (IoT) developed as a
technology to interact with sensors and actuators. However, so far few approaches have taken a
comprehensive approach to combine IoT with AR.
FI-WARE technologies will allow to go beyond the state-of-the-art here by (i) creating a comprehensive
approach that connects IoT services directly to 3D representations in the 3D-UI and (ii) allow users to
interact with these services via virtual representatives/proxies. Additionally, (iii) it will allow to combine
AR and feature detection and tracking with the ability to locate real world objects via video capturing in
mobile devices, (iv) link them directly to their virtual counterparts via extended POI specifications and
suitable dynamic databases and (v) allow interaction with the real devices through their virtual
counterparts.
Virtual Characters: Embodied agents have the potential to make human-computer interaction more
intuitive, engaging and accessible and bring virtual worlds to life, starting with frameworks like [Perlin
96] and many novel versions have been created, including advanced techniques like physically-based
animation, ergonomics, motion synthesis via e.g. motion graphs, and others. Higher level frameworks
like the Behaviour Markup Language (BML) [Vilhjalmsson 07] are particularly well suited for simplified
but domain-specific limited animations.
Is is still desireable to advance existing FI-WARE technologies on top of that by creating (i) a Web-based
character editor and (ii) the ability to manage the character and animation assets in a flexible way using
distributed, Internet-based repositories, thus creating a market for such assets. It might also (iii) add
advanced animation control using some combination of physics-based animation, inverse kinematics,
transition management and blending, cloth and rag-doll simulation. More over, (iv) the editor will allow
greatly shortened deployment time by implementing automatic rigging features, which allow existing
skeletal animations to be matched with any given mesh, given that the shape of the skeleton and the
shape of the mesh [Baran 2007]
D.2.6.3 FI-WARE State of the Art
Page 10
Future Internet Core Platform
Interface Designer: A key element for the interface designer is services for uploading, storing, and
accessing 3D models. The largest such site is Trimble (ex. Google) Warehouse
(http://sketchup.google.com/3dwarehouse/). Others repository services have been SketchFab
(sketchfab.com), ShareCG (sharecg.com), Verold Studio (studio.verold.com), OurBrick (ourbrick.com),
and others. None of these offer support for versioning or collaboration, interaction are not supported,
and only OurBrick offers a simple Web service API for accessing models. On the editing side, besides the
CAD and animation packages, on the Web we have Flash 3D and Unity3D, but major CAD companies, like
Autodesk and Dassault Systems, are beginning to develop Web/cloud-based solutions but they are
mostly closed solution. A few Web-besed 3D editors have become available including the CL3VER
product (www.bevelity.com), kuda (code.google.com/p/kuda/), and ThreeFab
(blackjk3.github.com/threefab/).
Future technolgies will improve upon this by (i) making a generic, in-browser editor that is not a standalone application but can be added to any 3D-UI context simply by adding a single Javascript resource.
Using a hot-key or API call immediately activates the editor and allows the user to change the paused or
fully active scene interactively. While only simple editing operations are supported in FI-WARE, it is
possible to (ii) add more advanced editing features, including animation editing, advanced materials,
more complex interactions with the base app, easy content publishing tools (integration with network
storage) and others.
2.6
References
[AVRO]
Apache Software Foundation, The AVRO web site, http://avro.apache.org
[Baldauf 07]
Baldauf, M., Dustdar, S. and Rosenberg, F. (2007) "A survey on context-aware
systems". International Journal of Ad Hoc and Ubiquitous Computing, vol. 2, num.
4, pp. 263-277, 2007. ISSN 1743-8225.
[Bernstein96]
Middleware: a model for distributed system services. Communications of the ACM,
pages 86 – 98, 1996.
[Brewer 00]
Brewer, E.A. (2000) "Towards robust distributed systems".In Proceedings of the
Annual ACM Symposium on Principles of Distributed Computing, vol. 19, pp. 710, 2000.
[CC]
Common Criteria for Information Technology Security, Version 3.1, Rev. 3,
volume 1-3, July 2009. Available: http://www.commoncriteriaportal.org/cc
[CORBA]
Commont Request Broker Architecture, OMG Standard,
http://www.omg.org/spec/CORBA/
[DCE]
Distributed Computing Environment, http://www.opengroup.org/dce/
D.2.6.3 FI-WARE State of the Art
Page 11
Future Internet Core Platform
[MS-DCOM]
Microsoft Distributed Component Object Model Remote Protocol Specification,
http://msdn.microsoft.com/library/cc201989.aspx
[DDS]
Data Distribution Service, http://portals.omg.org/dds/
[DTD]
Document Type Definition, SGML Standard,
http://en.wikipedia.org/wiki/Document_Type_Declaration
[Emmerich00]
W. Emmerich. Software engineering and middleware: A roadmap. Communications
of the ACM, pages 117-129, 2000.
[FT02]
R. Fielding; R. Taylor (2002), Principled Design of the Modern Web Architecture,
ACM Transactions on Internet Technology (TOIT) ACM, 2 (2): 115–150, 2002.
[ICE]
Internet Communications Engine,
http://en.wikipedia.org/wiki/Internet_Communications_Engine
[JSON]
JavaScript Object Notation, IETF Standard RFC4627 (2006),
http://tools.ietf.org/html/rfc4627
[Myerson02]
Judith M. Myerson, Identifying enterprise network vulnerabilities, Int. Journal of
Network Management 12(3):135-144 (2002)
[ONC-RPC]
Open Network Compute Remote Procedure Calls, IETF Standard RFC5531.
http://tools.ietf.org/html/rfc5531
[OpenDDS]
Open Source DDS implementation, http://www.opendds.org
[OpenSplice]
Prismtech commercial DDS implementation http://www.prismtech.com/opensplice
[REST]
Representational State Transfer,
http://en.wikipedia.org/wiki/Representational_state_transfer
[RMI]
Java Remote Method Invocation,
http://www.oracle.com/technetwork/java/javase/tech/index-jsp-136424.html
[RPC-DDS]
RPC over DDS, OMG Proposed Specification (2012), http://www.omg.org/cgibin/doc?mars/2012-06-29
[RTI-DDS]
Commercial implementation of DDS, http://www.rti.com/products/dds/
[Schantz02]
Richard E. Schantz and Douglas C. Schmidt, Research Advances in Middleware for
Distributed Systems: State of the Art, 2002.
[Schantz86]
Schantz, R., Thomas R., Bono G., “The Architecture of the Cronus, Distributed
Operating System”, Proceedings of the 6th IEEE International Conference on
Distributed Computing Systems (ICDCS-6), 1996.
[SDN]
Software Defined Networking, http://en.wikipedia.org/wiki/Softwaredefined_networking
[SOAP]
Simple Object Access Protocol, W3C specification, SOAP Version 1.2 (2007),
D.2.6.3 FI-WARE State of the Art
Page 12
Future Internet Core Platform
available at http://www.w3.org/TR/soap/
[Thrift]
The Apache Software Foundation (2010) Apache Thrift web site,
http://thrift.apache.org
[XML]
Extensible Markup Language, W3C Recommendation (2008),
http://www.w3.org/TR/xml
[XPath]
XML Path Language, W3C Recommendation (1999), http://www.w3.org/TR/xpath
[XSL]
Extensible Stylesheet Language, W3C Recommendation (2006),
http://www.w3.org/TR/xsl
[XSLT]
XSL Transformations, W3C Recommendation (2007), http://www.w3.org/TR/xslt20
[XQuery]
XML Query Language, W3C Recommendation (2010),
http://www.w3.org/TR/xquery
[ZeroMQ]
ZeroMQ (0MQ), Transport Library, http://www.zeromq.org
D.2.6.3 FI-WARE State of the Art
Page 13