8th IFAC Conference on Control Applications in Marine Systems
Rostock-Warnemünde, Germany
September 15-17, 2010
Virtual and augmented reality for the maritime sector –
applications and requirements
Uwe Freiherr von Lukas
Fraunhofer IGD, Competence Center Maritime Graphics
18059 Rostock, Germany
(Tel. +49 381 4024 150, e-mail: uwe.von.lukas@igd-r.fraunhofer.de )
Abstract: The maritime sector with its wide spectrum of ancillary conditions can benefit from advanced
computer graphics in many ways. The paper presents several applications from marketing over design to
training and maintenance that are based on virtual or augmented reality technology. Some of the
technical challenges that arise from the ambitious environment are sketched and an outlook on future
research and development is given.
Keywords: Virtual reality, computer graphics, interactive programs, simulators.
1.
MOTIVATION
2.1
The maritime sector offers a broad variety of applications for
advanced computer graphics technology. Ranging from
marketing and design over manufacturing support to
familiarization, training and maintenance assistance, there is
no phase in the lifecycle of a ship or seaborne structure that
would not profit from 3D modelling, simulation,
virtual/augmented reality or computer vision.
Marketing
Not only for the direct customer but also for informing
citizens about future installations such as offshore wind
parks, computer graphics plays an important role. Often in
this phase, free interaction is not required but the virtual
product is presented by predefined settings of viewpoints,
lights and some animations. Figure 1 shows an example of
the early days of offshore wind-parks in the Baltic Sea, where
animations have been used to inform the public about these
new installations.
Compared to other industrial sectors we find some unique
requirements and conditions that are exceptionally
challenging. To name just a few of them we have to deal with
limited-lot production, extremely complex products (> 1 Mio
parts), operation in quite harsh environments and high
economical and ecological risks – e.g. in the area of cruise
liners or oil platforms.
The paper uses several examples to illustrate the specific
challenges and solutions for using virtual and augmented
reality in the maritime sector. By this gives an overview over
this vivid field of interdisciplinary applied research of naval
architects, engineers, computer scientists and usability
experts.
2.
SELECTED APPLICATIONS
Fig 1: Visualization of offshore installation procedures
Ships and offshore installations are typical product in the
maritime sector. This paragraph presents selected
applications following the typical lifecycle of a product from
idea over design to training and maintenance.
978-3-902661-88-3/10/$20.00 © 2010 IFAC
2.2
Design review
Some of the leading shipyards in Germany are now in the
process of integrating Virtual Reality into their standard
design procedures (Mesing et al. 2008; Nedeß et al. 2009).
Supported by national German research projects, they focus
on using VR as a tool for review where participants of
196
10.3182/20100915-3-DE-3008.00045
CAMS 2010
Rostock-Warnemünde, Germany, Sept 15-17, 2010
various disciplines and/or stakeholders have a natural
environment to inspect the current model and discuss several
aspects of the design (ref. Figure 2).
high degree of automation where the virtual world is directly
derived from the CAD data.
2.4
Simulation-based training for operators
Simulation based training is state of the art for pilots of
aircrafts as well as nautical officers. It is even more important
for a completely new kind of vessel such as a wing-in-ground
craft which is quite difficult to handle and operates with high
speed. Wing in ground crafts (WIG) are a good example for
the type of vessels. Based on a MATLAB simulation and the
mixed reality framework instantReality (Fellner et al. 2009),
a scalable solution was implemented that mimics the
behaviour of an 140 hm/h WIG craft (ref. Figure 4). One of
the challenges here is to replace the complex equations of
motion by approximations that are accurate enough but
solvable in real-time.
Fig 2: Virtual reality-based design review at Fraunhofer
IGD’s Maritime Graphics lab in Rostock
Research aspects in this context include the handling if the
huge data sets in realtime, the efficient augmentation of the
geometry data with meta data from external systems or
functional behavior.
2.3
Game-based training for maritime security
The gaming industry does not only influence industrial
applications with affordable high-end graphics boards but
also with powerful software platforms. The game engines
combine handling and rendering of 3D objects with an
efficient way to describe interaction and behaviour. This
specific mixture of gaming and “serious” simulation offers
interesting perspectives for interactive training (Wolfe and
Crookall, 1998).
Fig 4: WIG simulator with tiled display
Such a Human-in-the-Loop approach (Smid and Cheok 1998)
does not only serve for training purposes but also allows the
optimization of the simulated vessel (Johnson and Fontaine
2001).
Another challenge, where it is crucial to have well-trained
personnel is for underwater vehicle operators (ref. Figure 5).
Like in the case of WIG crafts, the real-time graphics and
simulators can offer an extremely efficient and cheap
approach for practicing the handling of a complex technical
object (Ridao et al. 2004).
Fig 3: Virtual fire-fighting application
A serious game approach introduces new media in the
training of ship crews as illustrated in Figure 3. It shows a
PC-based shipboard virtual fire-fighting application that is
part of a blended learning course for basic fire-fighting
(Deistung et al. 2008).
The authoring of such a training environment typically
comprises a lot of manual work. Current research aims for a
Fig 5: UV Simulator NEPTUNE (Source: University of
Girona)
197
CAMS 2010
Rostock-Warnemünde, Germany, Sept 15-17, 2010
Beside the training aspect, those simulators play an important
role in AUV or ROV development. They support the
engineers in reviewing the behavior (such as
maneuverability) of the designed vehicle. Some of them even
serve as a basis for hardware in the loop, where virtual
objects can be mixed with real objects (Song et al.2001).
2.5
Maintenance support
Augmented reality denotes a technology, where digital
content is combined with real objects – e.g. by mixing
computer generated content with a live video stream of a
scene (Azuma 1997). This approach can be extremely useful
to support a ship crew with its limited resources and
competencies by an assistance system for repair or
maintenance operations. The system presents tools and/or
procedures as an overlay to a real pump or filter to be
repaired.
Fig 7: Three-stage process for interactive graphics
3.1
In each and every phase of the lifecycle of a ship we find
different 3D models. It starts with a concept model that
roughly describes the later product and has a strong focus on
marketing. It is followed by a detailed CAD model which
forms the basis for the production. A third model is
sometimes produced for explaining the product and training
purposes. All those models are produced separately with
specific tools and fulfill a specific task. Obviously this lack of
a single universal model is an example of redundant work
that could be eliminated by technical and organizational
improvements.
Fig 6: Adjustment of a ship engine governor with augmented
reality (governor by courtesy of MAN Diesel)
3.2
In Figure 6 we see an example application where the engineer
uses a head-worn display for hands-free operation. Even for
clean and well-lighted situations, augmented reality is still in
a research phase. The robustness of the tracking which
computes the position of the user’s head by computer vision
techniques as well as a natural way of mixing real objects and
virtual objects with correct occlusion and shadows are
currently in the focus of visual computing research.
3.
Lack of universal 3D models
Lightweight digitalization of real objects
The digitalization of existing objects (plants, ships, platforms
etc.) is feasible but expensive due to the size of those objects.
The typical approach is semi-automatic and based on
intermediate data of scanners or produced by
photogrammetric systems. Those data models still need a
high amount of manual work to deliver models that provide a
good structure as well as a suitable level of detail and
accuracy. Especially for large structures such as ships, this is
a quite expensive approach.
SUMMARIZING TECHNICAL CHALLENGES
The technical challenges for implementing applications in the
maritime sector are directly derived from the following
common three-stage process for graphics applications (ref.
Figure 7):
3.3
Efficient authoring
Even if there is a good 3D model available, this is only one
side of the coin. An interactive 3D environment also needs
additional specifications including textures, animation paths,
meta information (e.g. materials or weight) and a story book
that guides the user through a scenario. Up to now, there is no
“silver bullet” how to do this additional authoring in a
standardized way.
198
CAMS 2010
Rostock-Warnemünde, Germany, Sept 15-17, 2010
3.4
standards, interfaces and processes of the maritime sector to
really meet the demands of this important industry.
Integrating 3D objects with behavior
However, applications of virtual and augmented reality are
not limited to harbours, ships and platforms. The next
generation technologies will also be able to support the user
in underwater scenarios. This makes especially computer
vision and human computer interaction even more
challenging.
Inspecting the geometry of virtual objects is not enough to
satisfy the end users anymore. Today it is also important to
cover selected aspects of the behavior of those objects. This
can be done by coupling the virtual environment with an
external simulator which is in charge of computing (e.g.
physical)
behavior.
However,
finding
adequate
correspondences of objects in both virtual worlds (graphical
representation and simulation) and additionally, meeting the
real-time requirements of interactive graphics needs
sophisticated concepts for the integration.
ACKNOWLEDGEMENT
The research work presented here was partly funded by
Federal Ministry of Economics and Technology (BMWi) in
context of the projects USE-VR, POWER-VR and
MARSPEED. The author wishes to thank Eik Deistung,
Benjamin Mesing, and Matthias Vahl for their
implementation work.
A radically new approach is presented for example in
(Havemann et al. 2008) where geometry and
semantics/behavior is not modeled and stored separately but
in one code such as the Generative Modeling Language. This
approach overcomes many of the limitations that current
solution of semantics enrichment suffer from.
3.5
REFERENCES
Scalability of applications
Azuma, R. T. (1997) A Survey of Augmented Reality. In:
Presence: Teleoperators and Virtual Environments, Vol.
6(4).
During maintenance operation onboard the ship the worker
needs a small portable device, but large display setups are
required when performing a design review of whole sections.
Those divergent needs are addressed by different display
devices like smartphones on the one end and high-end virtual
reality environments such as a CAVE or high-resolution
power wall (e.g. HEyeWall, ref. Kresse et al. 2003) on the
other end. instantReality, our platform for mixed reality
applications is able to support this broad range of devices and
helps a company to efficiently spread interactive 3D content
in different departments and usage scenarios.
3.6
Deistung, E.; Lukas, U. von; Sedlacek, D.; Kucharzewski, H.
(2008). Game-based training for individualized
shipboard fire-fighting. In International Maritime
Simulator Forum. Proceedings. CD-ROM: RostockWarnemünde/Germany
Fellner, D.W., Behr, J., Bockholt, U. (2009). Instantreality - a
framework for industrial augmented and virtual reality
applications . In 2nd Sino-German Workshop "Virtual
Reality & Augmented Reality in Industry" 2009. Invited
Paper Proceedings. Participants Edition: 16.-17. April
2009, Shanghai, P.R. China. Shanghai: Shanghai Jiao
Tong University.
Interaction in 3D environments
Not only the big variety of output devices but also the
differences of the end users experiences and preferences have
led to a plentitude of interaction devices and metaphors: FlyStick, game controller, 3D mouse and data glove are just a
few examples of approaches to support 3D navigation and
control in virtual environments. Current research is now
dealing with usability aspects in these very specific setups
where the specific requirements of maritime applications are
taken into account.
4.
Haveman, S.; Settgast, V.; Berndt, R. Eide, O.; Fellner, D.
(2008) The Arrigo Showcase Reloaded –towards a
sustainable link between 3D and semantics. In The 9th
International
Symposium
on
Virtual
Reality,
Archaeology and Cultural Heritage VAST..
Johnson, E.; Fontaine, S (2001). Use of flight simulation to
complement flight testing of low-cost UAVs: AIAA
Modeling and Simulation Technologies Conference.
SUMMARY AND OUTLOOK
Kresse, W., Reiners, D., and Knöpfle, C. (2003). Color
consistency for digital multi-projector stereo display
systems: the HEyeWall and the Digital CAVE. In
Proceedings of the Workshop on Virtual Environments
2003 EGVE '03, vol. 39. ACM, New York, .
The given selection of application scenarios demonstrates
that interactive graphics can support the maritime sector in
many ways. Compared to other industry sectors we are still in
a immature stage where additional research is necessary to
gain the full potential of the technology. Additionally, the
tools and processes must be optimized in terms of efficiency
to really convince the end user.
Mesing, B.; Vahl, M.; Lukas, U. von. (2008). The USE-VR
platform - a framework for interoperability among
different VR solutions. In Proceedings: 2nd
International Workshop Virtual Manufacturing, VirMan
08, Torino, Italy.
Various specific working fields have been highlighted. In all
these areas we have to combine profound knowledge of
computer graphics technology with the specific IT systems,
199
CAMS 2010
Rostock-Warnemünde, Germany, Sept 15-17, 2010
Nedeß, Chr.; Friedewald, A.; Schäfer, Chr.; Schleusener, S.
(2009). Deploying Virtual Reality (VR) Technology in
Shipbuilding by using a Construction Kit for VR
Applications Focused on the Business Processes In:
Bertram, V. (Ed.): Proceedings 8th International
Conference on Computer and IT Applications in the
Maritime Industries (COMPIT '09), Budapest.
Ridao, P.; Batlle, E.; Ribas, D.; Carreras, M (2004) Neptune:
a hil simulator for multiple UUVs. In OCEANS '04.
MTTS/IEEE TECHNO-OCEAN '04, Vol. 1.
Smid, G.E.; Cheok, K.C. (1998) Human integration in
simulation. In Advanced Motion Control, 1998. AMC
'98-Coimbra.
Song, F. Folleco, A.; An, E. (2001) High fidelity hardwarein-the-loop simulation development for an autonomous
underwater vehicle. In OCEANS, 2001. MTS/IEEE
Conference and Exhibition.
Wolfe, J.; Crookall, D. (1998): Developing a scientific
knowledge of simulation/gaming. In: Simulation &
Gaming: An International Journal of Theory, Design and
Research, 29(1)
200