3-D Visualization of Simulink Physics Models
Using Unreal Engine
Elise R. Haley, David J. Coe, and Jeffrey H. Kulick
Department of Electrical and Computer Engineering
University of Alabama in Huntsville, Huntsville, Alabama USA
Abstract - Model-based design (MBD) tools have
evolved that enable simulation of complex cyberphysical systems. Improved visualization techniques
are needed to help engineers better understand the
results of these simulations. We present a toolkit that
facilitates interactive visualization of Simulink models
using the advanced graphics capabilities of the Unreal
Development Kit. Our toolkit allows Simulink models
to create and control actors within a UDK virtual
world and allows transformations and events that
occur within the UDK to be communicated back to the
Simulink models for processing.
This two-way
communication allows the modeler to distribute the
simulation between the two tools to take advantage of
each tool’s capabilities as needed. For demonstration,
we present a Simulink model that controls the orbit
and rotation of the solar system within a UDK virtual
world.
Keywords: Simulink, Unreal, cyber-physical, 3D visualization
1
Introduction
The decreasing cost of digital computation has
facilitated the integration of programmable computing
elements into a wide range of physical systems, and
networking
technologies
have
enabled
the
interconnection of these systems to create ever more
complex cyber-physical systems. Engineers of these
multi-domain, cyber-physical systems need better
tools to help them design, integrate, and verify these
increasingly complex systems. Model-based design
(MBD) tools provide engineers the opportunity to
evaluate the design of complex cyber-physical
systems prior to implementation. MBD tools are
particularly attractive since they facilitate exploration
of alternative designs and removal of defects early in
the lifecycle both at relatively low cost.
A challenge for any modeling and simulation tool
is making the results accessible and understandable.
In this paper we present a toolkit that simplifies the
integration of Mathworks’ Simulink MBD tool with
Epic Game’s Unreal Development Kit 3 (UDK) to aid
in the visualization of simulation results. We also
present a proof-of-concept application developed
using our simulation-visualization framework.
2
2.1
Background
What is Simulink?
For many engineering disciplines, Simulink is
the de facto standard MBD tool. Simulink is a
graphical modeling and simulation tool that allows an
engineer to assemble a model of a cyber-physical
system by connecting a series of predefined subsystem
blocks using virtual wires (signals) [1]. Blocks may
represent fundamental model elements such as
constants, gain blocks, and queues, or more
complicated algorithms and subsystems such as Fast
Fourier Transforms or Kalman filters. It is important
to note that rather than being interconnected function
stubs, the block models themselves are fully
executable with actual outputs computed from the
applied inputs. Abstraction via nesting of subsystem
models into custom blocks may used to simplify
modeling of large complex cyber-physical systems.
Figure 1 below shows a Simulink model of a standard
PID controller assembled from a combination of gain,
summation, integration, and differentiation blocks.
The availability of domain-specific sets of
modeling blocks (toolboxes) for control system
design, digital signal and image processing, and other
domains eliminates the need to implement
fundamental domain elements and operations making
Simulink a particularly attractive option for MBD.
Figure 1 – Simulink model of a Proportional-Integral-Derivative (PID) Controller
For example, rather than assembling a PID controller
from basic elements as in Figure 1, a control system
designer can just use the predefined PID controller
block which includes an auto-tune button to assist in
the selection of appropriate gain coefficients. These
toolboxes allow a domain expert to quickly model and
simulate complex systems. When no appropriate
predefined block exists, an engineer may develop a
custom block or write a custom function using either
C or the MATLAB language and integrate the code
with the block model. Simulink also includes the
ability to synthesize C-language or Hardware
Description Language (HDL) implementations
directly from the model, and the ability to conduct
hardware-in-the-loop simulations.
2.2
Why interface Simulink with UDK?
MathWorks has developed a 3D Animation
toolbox for Simulink which is based on the Virtual
Reality Modeling Language (VRML) standard [1], but
its feature set and graphics quality are not yet
competitive with that of an industry standard game
development tool such as UDK which allows
developers to create immersive, first person, 3D
games with high quality graphics and animation [2].
The UDK’s built-in networking support allows
multiple-players to interact within the virtual world
and also provides a convenient means of interaction
with multiple Simulink models.
The versatility and advanced graphics
capabilities of the UDK have already been extensively
used in a variety of first-person games and nonentertainment applications. Militaries, for example,
have explored the use of the UDK-based games for
recruiting and training [3] and as platforms for
exploring small unit infantry tactics [4]. UDK-based
training aids have been developed for nuclear power
plant workers [5] and to teach foreign languages and
culture-specific gestures and body language [6]. The
UDK has also been used to recreate crime scenes as
part of a forensic investigation [7].
Notably, Mathis et al. presented a method of
interfacing Unreal Tournament (UT) with MATLAB,
Simulink, and USARSim to visualize operation of the
SAMURAI nano air vehicle [8]. Vehicle dynamics
modeled within Simulink were used to control the
motion of the vehicle within UT. USARSim was used
to gather image data for the user. MATLAB code was
used to interface MATLAB and Simulink to
USARSIM and UT via TCP/IP communication.
Our toolkit expands on previous work by
enabling two-way interactions between Simulink
models and UDK virtual worlds, which allows the
modeler to distribute the simulation across Simulink
and the UDK to take advantage of the strengths of
each tool. Simulink, for example, would be used to
model advanced algorithms, physics, and hardware
that govern the behavior of actors (objects) within the
virtual world, and the UDK would use its GPU
processing capabilities to reposition UDK objects as
needed, detect collisions between actors, apply
appropriate lighting, etc.
With our toolkit, the
Simulink model can
•
Dynamically create instances of actors from the
UDK actor library
Figure 2 – Block diagram of Simulink-UDK toolkit.
•
•
•
•
•
Modify the location, size, roll, pitch, yaw and
other properties of the UDK actors
Move objects via instantaneous translation of
actors in the UDK model
Move objects via interpolated motion of actors in
the UDK model
Respond to UDK events and actor interactions
within the UDK that are reported back to Simulink
as events, event streams, or actor property updates
including
o single event reports which aggregate
multiple events associated with an object
into a single, most recent event and
o a stream of reports that report every
interaction of an object with other actors
in the scene via a vector of interaction
events
Modify behavior of a UDK actor via user inputs
funneled from a Simulink/MATLAB GUI to the
Simulink model
By allowing the Simulink model to completely control
the building and animation of the UDK model, we
hope to minimize the learning curve for the Simulink
modeler. Below we provide a brief overview of the
toolkit and present a proof-of-concept application.
3
Overview of toolkit
As shown in the block diagram appearing in
Figure 2 above, our toolkit makes use of TCP for twoway communication and interaction between Simulink
models and the UDK virtual world. To interact with
an actor in the UDK, the Simulink model creates a
TCP socket connection. Multiple socket connections
can be created so that multiple Simulink models (or
interactive users if desired) can interact with the UDK
virtual world similar to the actions of a multi-person
game. One advantage of this method is that
computationally intensive Simulink models may be
executed on separate hardware from the UDK to speed
execution. Moreover, the UDK can perform basic
tasks to maintain the virtual world, such as actor
collision detection, allowing the Simulink model to
focus on its computation until it is notified by the
UDK of a collision or other critical event.
On the left side of the block diagram, you will
see “Custom TCP Client 1”, for example, listed within
the Simulink block.
The Simulink-side TCP
send/receive code has two parts: a standard TCP stack
and a message creation part which must specify the
actor to affect as well as the operation that the UDK is
to perform. Examples of operations include the
“spawn” command which directs the UDK to create
an instance of an actor predefined within the UDK
actor library, the “set location” operation that directs
the UDK move an actor instantaneously to a specified
location, and the “move” operation that smoothly
nudges an actor according to the provided velocity
vector.
Similarly, the Unreal TCP communication code
is also generic code that is customized for interfacing
with specific classes of actors. The TCP server actor
spawns one child connection handler per connection,
to handle the actual send/receive operations. The
connection handler can access any actor class you
declare an instance for in the child's code. To spawn
or manipulate a specific UDK actor object, you must
instantiate that actor within its associated connection
handler. At the start of the UDK/simulation run, all
Figure 3 – Simulink orbit model of solar system with details of Mercury orbit.
actors that may be instantiated by the Simulink model
in the UDK application have to be predefined. The
number, location, and properties of the actors can be
changed on the fly by the Simulink model. UDK
function delegates are used to forward UDK events
such as collisions back to Simulink for processing by
the relevant model.
Timing for the interactions between the Simulink
models and the UDK model may be synchronous or
asynchronous. In the asynchronous case, the Simulink
models run as fast as they can and update the data on
the UDK model as soon as possible. The UDK engine
attempts to keep a "real time" view so that time
progresses continuously. If synchronous interaction
with the Simulink model is wanted, then both the
UDK model and Simulink model can progress time
step by time step in lock step. In cases where the
Simulink model runs substantially slower than real
time and a real time playback is desired, the control
events from the Simulink model can be recorded in a
file with time stamps. These files can be played back
in real time after the simulation is over.
The toolkit provides for interaction with one or
more human users via the UDK or via the Simulink
model. Various input/output devices are supported
including keyboard, mouse, and force reflective
joystick. In the case of the force reflective joystick,
the force feedback values are computed by the
Simulink model and presented back to the user via the
standard Simulink support for force reflective devices.
4
Proof-of-concept application
At present we are developing a simple proof-ofconcept astrophysics application. Users can generate
solar systems with varying numbers of planets, with
varying orbits, masses, etc. Our goal is to develop a
model that allows users to explore the concept of
launch windows for space exploration.
We are
examining other models that are more amenable to the
use of force reflective devices such as a pool table
simulation as further demonstrations of our toolkit’s
capabilities.
In our proof-of-concept application, the Simulink
model configures the rotation settings for Mercury and
Figure 4 – Screenshot of UDK virtual solar system driven by Simulink model.
Figure 5 – Simulink rotation model with MATLAB GUI for controlling direction of rotation.
Sun actors and implements the orbit of Mercury about
the Sun. The UDK interface can process spawn, set
rotation rate, and set position commands for all the
actors. Figure 3 above contain a screenshot of the
Simulink orbit model. Figure 4 below shows a
screenshot of the virtual solar system as it appears
within the UDK. Finally, Figure 5 shows a screenshot
of the Simulink rotation model for a solar system
actor, which includes a MATLAB GUI that is used to
vary direction and rotational speed parameters of the
Simulink model. One could use a similar technique to
create “instructor” and “student” interfaces to the
same simulation. For example, through the UDK
interface, the workers undergoing training explore and
interact with a virtual world (say a nuclear power
station or chemical processing plant) to achieve a
specific objective.
Meanwhile, the instructor,
monitoring the workers’ progress via the UDK
interface, can use the MATLAB GUI to trigger
specific training scenarios that will require an
appropriate response from the workers. The full
physics model implemented within Simulink would
provide the workers the ability to view via the UDK
the consequences of their various decisions.
5
Conclusions and future work
As cyber-physical systems become more
complex, it will be increasingly important to
understand our models of these systems so that we can
make better design and implementation choices. Our
toolkit allows engineers to link their Simulink models
to the UDK for first person, interactive visualization
of simulation results. We extend previous work by
facilitating two-way communication and interaction
between Simulink and the UDK, and this allows for
partitioning of the simulation between the two tools to
take advantage of each tool’s strengths and
capabilities. As a result, each Simulink model may
control spawning and manipulation of actors within
the UDK, and relevant events occurring within the
UDK virtual world can be forwarded back to Simulink
for processing by the Simulink model. The use of
TCP communications also allows multiple Simulink
models to interact within the UDK while one or more
human users interact with the virtual world through
the UDK itself, as would first person gamers.
Future work includes the development of
applications which can exploit the Simulink force
reflective device interface, the investigation of
mechanisms for switching on-the-fly between low
quality and high quality visualization, and an
exploration of synchronization mechanisms for timing
critical simulations distributed between Simulink and
the UDK.
6
References
[1] Simulink/Matlab by Mathworks, Inc.,
www.mathworks.com
[2] Unreal Development Kit 3.0 by Epic Games,
http://udk.com
[3] Michael Zyda, John Hiles, Alex Mayberry, Casey
Wardynski, Michael Capps, Brian Osborn, Russell
Shilling, Martin Robaszewski, and Margaret Davis,
“Entertainment R&D for Defense,” IEEE Computer
Graphics and Applications, January/February 2003,
pp. 28-36.
[4] Jun Lai, Wei Tang, and Yihui He, “Team Tactics
in Military Serious Game,” 2011 Fourth Int.
Symposium on Computational Intelligence and
Design, pp. 75-78.
[5] Z. Kriz, R. Prochaska, C.A. Morrow, C. Vasquez,
H. Wu, and Rizwan-uddin, “Unreal III Based 3-D
Virtual Models for Training at Nuclear Power Plants,”
Proc. 1st Int. Nuclear and Renewable Energy
Conference (INREC10), Amman, Jordan, March 2124, 2010, pp. INREC10-1 – INREC10-5.
[6] Danna Voth, “Gaming Technology Helps Troops
Learn Language,” IEEE Intelligent Systems,
September/October 2004, pp. 4-6.
[7] Xu Feng, Shan Daguo, and Yang Hongchen,
“Simulation Research of Crime Scene Based on
UDK,” Information Science and Engineering
(ICISE), 2010 2nd International Conference on, 4-6
Dec. 2010, pp. 1-4.
[8] Allison Mathis, Kingsley Fregene, and Brian
Satterfield, “Creating High Quality Interactive
Simulations Using MATLAB and USARSim”,
https://robotics.ucmerced.edu/Robotics/wspapers/IRO
S_USARSimWS_AMathis_Final.pdf