Supervisory Control by Using Active Virtual 3D Models in-the-loop
T. Reichenbach, D. Miklić, Z. Kovačić
Faculty of Electrical Engineering and Computing
Department of Automatic Control, Zagreb, Croatia
tomislav.reichenbach@fer.hr damjan.miklic@fer.hr zdenko.kovacic@fer.hr
Inputs
(trajectories,
references)
Virtual Outputs
(states, events
positions,
velocities, etc)
I. I NTRODUCTION
If an assumption is made that virtual models are as close
as possible to the real ones, events in the virtual world
should reflect those from the real world. The concurrence
of real and virtual systems allows the use of virtual sensors
instead of real ones. Ordered by increasing complexity,
virtual environment (VE) can be used for visualization, collision detection and collision avoidance. Real-time collision
avoidance strategy assumes that a feedback from a collision
detection algorithm executed in VE must exist [1]. Regarding
the implementation of effective collision avoidance strategies, the main usability factor of such virtual modeling
and collision detection engine is its ability to interact with
other applications. Herein we demonstrate the concept of
supervisory control using active 3D models in-the-loop that
are effectively interacting with the Matlab+SimulinkTM simulation environment thanks to the Matlab’s built-in Component Object Model (COM) support. The examples presented
in this paper demonstrate the seamless integration of the
developed virtual modeling engine with a supplementary
Simulink models.
The paper is organized in the following way. First we
describe the concept of supervisory control using active
3D models in-the-loop. Then we describe integration of the
virtual modeling engine called L.I.S.A. and Matlab using
COM. Two examples demonstrating the applicability of the
concept are presented; implementation of a virtual sensor,
and introduction of the virtual reality feedback in the closed
control loop.
II. V IRTUAL MODELS IN - THE - LOOP CONCEPT
The overview of the proposed concept is shown in Fig.
1. The reference or desired behavior is both input into the
real system and into the virtual models of the real system.
Real
World
Corrections
Abstract— Virtual 3D models play important role in a today
factory layout design, physical modeling, control synthesis,
performance analysis, dynamic simulation and visualization
of real systems. In this paper, we go one step further and
besides visualization, we monitor all measurable states in the
real system and use active 3D models in-the-loop for real-time
supervisory control. Such concept enables an event detection
not possible with conventional sensors, including the ability
to predict possible outcomes in the system operation. A virtual
environment simulator and the way it can be connected to other
simulators (e.g. Matlab/Simulink), is presented. The examples
of virtual sensors, and collision free-based trajectory planning
for robotic manipulators are demonstrated.
Fig. 1.
Outputs
(states, events
positions,
velocities, etc)
Virtual
World
A general schematic of the virtual models in-the-loop concept
Outputs of the real systems are, however, inputs into the
virtual system enabling tracking of processes and events
happening in a real system. The virtual system output serves
as a feedback for the real system, allowing calculations made
in virtual systems to regulate the behavior of the real ones.
This feedback loop enables use of virtual sensors and the
ability to predict possible outcomes in system operation,
generally used in a collision avoidance.
III. C OLLISION DETECTION
The basis of a virtual environment is the ability to detect
interactions between objects, namely collisions. Collision
detection is a part of interference detection, which can be
divided into three portions: collision detection that detects
whether objects are in collision, collision determination that
finds the exact collision point, and finally, collision avoidance
that determines what actions should be taken in response to
the collision. There are numerous approaches to a collision
detection problem [2], which can be mainly grouped into ;
space-time volume intersection, swept volume interference,
multiple interference detection, and trajectory parametrization. A collision detection algorithm used in this paper
belongs to a multiple interference detection category. This
implies the algorithm reduces a general collision detection
problem to multiple calls to the static interference tests
focused on detecting intersections among simple geometrical entities; triangles and oriented bounding boxes (OBB)
belonging to the objects being tested. As the algorithm is
static, i.e. collision detection occurs only at discrete times,
it is fast enough and effective from the computational point
of view, thus it can provide real-time collision detection in
very complex (high polygon count) virtual environments [3].
An intersection between two OBBs, A and B is is based
on the separating axis theorem presented in [4] and [5].
According to the separating axis theorem it is sufficient to
find one axis that separates objects A and B to determine
they do not overlap. After the intersection between the
OBBs is determined an exact collision point is found with
triangle/triangle intersection test. The algorithm used for
triangle/triangle intersection test is the algorithm from in
ERIT package [6].
The collision point and the normal to the collision surface
are further used for a collision avoidance algorithm [1],
where automatically the collision-free path is generated only
by specifying the desired start and the end trajectory point.
IV. A PPLICATION I NTEGRATION U SING COM
This section specifically focuses on integration of virtual
modeling engine L.I.S.A. with Matlab/Simulink using COM
[7], an application integration framework provided by the
Microsoft Windows operating systems.
A. A COM Interface for Collision Avoidance
To enable other programs to use the virtual reality modeling engine, it was written as a COM local server. Its
functionality was encapsulated inside a COM object and
exposed through the object’s interface. The object was placed
inside an out-of-process COM server that handles the tasks
involved in providing the object to clients upon their request
(see Fig. 2).
Client application
(Simulink)
Fig. 2.
Interface
pointer
(handle)
Interface
- Method1()
- Method2()
-...
COM Object
(VR engine)
COM Out-ofProcess Server
Basic COM topology for application integration
The COM interface exposes the object’s functionality
through a set of public methods. The methods can be roughly
grouped together in four groups, by their functionality:
•
•
•
•
scene object selection
scene object manipulation
collision detection
collision free trajectory planning
Most methods come in get/set pairs. By calling the COM
object’s methods, clients can exercise complete control over
the virtual model and get feedback information on it’s states.
This enables client programs to use virtual reality models for
various control tasks, ranging from simple visualization to
soft sensors and virtual reality in-the-loop applications.
B. Integration with Simulink Models
Integrating COM objects into Simulink models is fairly
straightforward, since Matlab features built-in COM support.
The COM object must provide a type library which is a
binary description of its interface. The object’s functionality
is accessible from the Matlab command line or in m-code
through Matlab’s built-in ”COM” object.
To enable a more user friendly integration with Simulink
models, a pair of s-function blocks has been built around
the COM object, encapsulating scene manipulation and trajectory planning functionality respectively. The functionality
has been split in two blocks to reflect the fact that one part
of it deals with geometrical representations and relationships
between scene objects thus belonging to the physical description of a system, whereas the other part provides trajectory
planning capabilities and belongs to the control logic. The
usage of the blocks in simulation models can be seen in the
figures accompanying the experiments.
V. E XAMPLE 1 - COLLISION DETECTION
This example demonstrates the integration of the virtual
reality engine within a Simulink model. Simulink is used
for modeling system dynamics while a virtual reality scene
provides spatial representations and interactions between
objects. Component Object Model is the underlying technology that handles the communication between the two
applications.
The basic idea in the first example is to simulate dynamics
of a ball in simple “pong” type game (A minimalistic depiction of two player indoor soccer match). The objective of the
game is to keep the ball inside a confined environment by
controlling two slabs, representing players, positioned at the
opposite sides of the playing field. The field is constrained
in y-axis direction, bouncing the ball back; furthermore, the
playing field is cluttered with various objects that can change
the course of the ball. To keep matters simple, we assume
that the ball movement is confined to the x-y plane only and
that materials are frictionless. A discrete simulation step of
the simulation should be sufficiently small so that a collision
will be detected before the object passed completely through
the obstacle.
To control the ball, one must control the two slabs that
can move only vertically, requiring two control variables
y1ref and y2ref , representing respectively set-points of the
slabs. Additional parameters used are m1 , m2 , mball the slab
and the ball mass, v = {vx , vy } the ball velocity, and ci ,
coefficients of restitution for the i-th object.1
The block schematics in Simulink, showing system structure, are presented in Fig. 3 and the actual virtual environment in Fig. 4. The dynamics of moving objects and control
laws are modeled by ordinary differential equations and object positions are sent to the 3D scene model through custom
s-function blocks. A feedback from the 3D model block
provides collision information for the dynamical model.
1A
measure of amount of energy lost in the process of collision.
(x,y)
(x,y)
(x’,y’)
sfcn_lisa
COLLISION
BALL DYNAMICS
BRICK_1
POSITION CONTROL
sfcn_lisa
BRICK_1 VR MODEL
BRICK_1 DYNAMICS
PI
Fig. 3.
(x,y)
(x,y)
(x’,y’)
PI
BRICK_2
POSITION REFERENCE
(USER INPUT)
BALL VR MODEL
BRICK_2
POSITION CONTROL
(x,y)
(x’,y’)
(x,y)
sfcn_lisa
BRICK_2 VR MODEL
BRICK_2 DYNAMICS
“pong” type game simulation schematic in Matlab’s Simulink connected with a virtual environment simulator
3D Virtual environment
simulator
Sensor failure
detection
object position [mm]
Fig. 4.
“pong” type game simulation in a virtual environment simulator
VI. E XAMPLE 2 - VIRTUAL SENSOR
The simple example is given in Fig. 5 where the speed
of real conveyer belt is passed through RS232 connection to
the virtual conveyer belt, modeled in 3D environment. An
infrared LED sensor, at the end of the belt, signals that the
object has reached the end of the belt, and accordingly the
belt can be stopped. The analogy in virtual environments
is a virtual sensor, that detects collisions between a line,
positioned at the same place as the LED sensor, and objects
moving on a conveyer belt. Ideally, both should signal at
the same time, although some small time margin is allowed,
providing redundancy and exemption from a sensor failure.
The friction between an object and the belt is high, consequently the premise is that an object is moving with the
same speed as the belt. However, additional camera detects
if the object is moving or not because of the obstacle in its
path. Sensor fusion between the camera and conveyer belt
position sensor is used to determine object position. If the
redundancy is not a requisite, only the virtual sensors can be
RS 232
TCP/IP
Belt speed [cm/s]
Camera
CamSensor CS4
Conveyer belt
Driver
Fig. 5.
sensor failure detection
used, thus lowering the complete cost of the system.
VII. E XAMPLE 3 - COLLISION AVOIDANCE
The Rhino XR3 robot has several predefined trajectories
that are constantly traversed back and forth. The robot is
traversing the same trajectory until it encounters an obstacle
(a box). If an obstacle is hit the robot starts another trajectory,
higher or lower in the z-plane, depending on the collision
point. The Simulink schematics of the system is shown in
Fig. 6 and the corresponding virtual environment for the
simulation is shown in Fig. 7.
The z-position of the box is changed interactively during
the simulation (see top graph in Fig. 8) so it becomes the
obstacle in the traversed trajectory of the robot. First, the
collision occurs around 7s after the start (bottom graph in
Fig. 8), forcing robot to another preset trajectory, afterwards
after 12s robot is again forced upward to another trajectory.
0.65
x
Scope1
0
sfcn_lisa
y
Box
XY Graph1
0.4
Scope5
z
Scope
boolean
Data Type
Conversion
Collision signal
U U(E)
−1
Z
sfcn_lisa
Integer Delay
Rhino
x_y_z
Selector
XY Graph
Trajectory references
Scope6
Fig. 6.
Simulink schematics for the robot evading a box example
box z−pos. [m]
0.06
0.04
0.02
0
0
5000
10000
15000
0
5000
10000
15000
0
5000
10000
15000
tool y−pos. [m]
0.4
0.3
0.2
0.1
0
collision
1
0.5
0
time [ms]
Fig. 8. Box z-position, manipulator’s tool y-position, and collision state
between the manipulator and the box
Fig. 7.
Virtual reality environment for the robot evading a box example
The middle graph in Fig. 8 represents robot’s tool y-position,
and it can be observed how it has increased after each
collision. The complete tool trajectory in X-Y space is
presented in Fig. 9.
VIII. CONCLUSIONS AND FUTURE WORKS
The significance of virtual 3D models in physical modeling, control synthesis, performance analysis, dynamic simulation and visualization of real systems is very high. On the
other hand, virtual modeling tools and related applications
use passive 3D object models (like the figures in the chess).
For example, a data flow from a Simulink dynamic model of
a process to the Matlab’s Virtual Reality Toolbox is always
unidirectional. The potential of virtual models could be much
higher if their passive role on the virtual scene would be
replaced with the active one. Setting this as a design goal,
a 3D modeling engine L.I.S.A. has been developed, whose
features like collision detection and trajectory planning have
enabled closing the feedback, and thus making L.I.S.A. the
part of a closed control loop. In other words, the existence
of virtual 3D models and collision detection in L.I.S.A.
enabled implementation of a so called ”3D model in-theloop” supervisory control. ”Virtual model in-the-loop” is a
step further from the passive visualization, as we still monitor
all measurable states in the real system and visualize them,
but we also detect events and predict possible outcomes in the
system operation, turning passive 3D models into the active
ones. The paper describes control concept implementation
details on the examples of collision free-based trajectory
planning for robotic manipulators and implementation of
virtual sensors. Attention has been paid to integration of
a virtual environment simulator and the process simulation
model created in Matlab/Simulink. Regarding a future work
on the systems controlled by the ”3D model in-the-loop”
supervisory control, effective on-line collision avoidance algorithms must be developed and implemented. Performance
in dynamically reconfigurable virtual scenes and scenes with
0.25
tool y−pos. [m]
0.2
0.15
0.1
0.05
0
Fig. 9.
−0.25
−0.2
−0.15
−0.1
−0.05
0
0.05
tool x−pos. [m]
0.1
0.15
0.2
0.25
XY position of the tool (for the robot evading a box example)
stochastic movements of 3D objects (identified for example
by stereo-vision and virtual scene reconstruction) should be
thoroughly analyzed and optimal solutions should be found.
IX. ACKNOWLEDGMENTS
This work was fully supported by a grant from the Croatian Ministry of Science, Education and Sports (the project
title 0036044 ”Integrated Control of Robotized Plants”).
R EFERENCES
[1] T. Reichenbach and Z. Kovačić, Cutting Edge Robotics. Advanced
Robotic Systems (ARS), 2005, ch. Collision-free path planning in robot
cells using virtual 3D collision sensors, pp. 683–704.
[2] P. Jiménez, F. Thomas, and C. Torras, “3D collision detection:
a survey,” Computers and Graphics, vol. 25, no. 2, pp. 269–
285, Apr. 2001. [Online]. Available: http://www.elsevier.nl/gejng/10/13/20/57/29/34/article.pdf
[3] T. Reichenbach and Z. Kovačić, “Collision avoidance in virtual environment based on usage of kinematics derived from a 3d robot model,” in
Proceedings of MED2003 CD, DNBI 85/-766/5-0-/, June 2003, Rhodes,
Greece.
[4] T. Möller and E. Haines, Real-Time Rendering. A. K. Peters, 1999,
mÖL t 02:1 1.Ex.
[5] S. Gottschalk, “Collision queries using oriented bounding boxes,” Ph.D.
dissertation, University of North Carolina at Chapel Hill, 2000.
[6] M. Held, “ERIT: A collection of efficient and reliable intersection
tests,” Journal of Graphics Tools: JGT, vol. 2, no. 4, pp. 25–44, 1997.
[Online]. Available: http://www.acm.org/jgt/papers/Held97/
[7] Matlab, The External Interface Guide, Matworks.