Title
A full-immersive CAVE-based VR simulation system of forklift
truck operations
Advisor(s)
Choi, SH
Author(s)
Yuen, Ka-kei.; 源嘉祈.
Citation
Issued Date
URL
Rights
Yuen, K. [源嘉祈]. (2011). A full-immersive CAVE-based VR
simulation system of forklift truck operations. (Thesis).
University of Hong Kong, Pokfulam, Hong Kong SAR. Retrieved
from http://dx.doi.org/10.5353/th_b4697447.
2011
http://hdl.handle.net/10722/144249
The author retains all proprietary rights, (such as patent rights)
and the right to use in future works.
A Full-immersive CAVE-based
VR Simulation System of Forklift Truck Operations
by
Ryan Ka Kei YUEN
B.Eng. HKU
A thesis submitted in partial fulfilment of the requirement for
the Degree of Master of Philosophy
at The University of Hong Kong
August 2011
Abstract of the thesis entitled
A Full-immersive CAVE-based
VR simulation system of forklift truck operations
Submitted by
Ryan Ka Kei YUEN
for the degree of Master of Philosophy
at the University of Hong Kong in August 2011
For decades, forklift trucks have been playing a pivotal role in warehouse
management. Their high flexibility facilitates material handling operations in a timely
and cost-effective way. However, mishandling a forklift truck often results in serious
damages or fatalities.
Although certificate courses are provided by driving institutes or government bodies
to train forklift truck operators, the result is not significant as the main cause behind
forklift truck accidents are not about skills acquired by the safety awareness of
operators. To address the issue, Virtual Reality (VR) technology was applied to
simulate the forklift truck operations. The proposed simulation system was built in a
CAVE-based VR environment where drivers can fully immerse to practise forklift
truck manoeuvres and pallet operations. They can easily review and rectify their
driving techniques, and more importantly, several common accident scenarios were
created and visualised to enhance safety awareness without any real dangers. As a
result, forklift truck drivers would become more vigilant and skilful in handling
adverse conditions.
Powered by the built-in Physics Engine of Virtools, several common forklift truck
accidents scenarios, for instance skidding, overturn and falling of pallet loads were
mocked up. An in-depth analysis was performed to study the variation of turning
i
radius of a forklift truck with respect to different turning velocities. By capturing the
drivers‟ performance in simulation, their driving misbehaviours can be evaluated and
rectified. Passive Infra-red (IR) marker system was chosen in the project to create an
intuitive interface for users to interact. This IR motion tracking technique was applied
on head tracking for additional driving views to enhance the degree of realism in user
experience and hand tracking for menu manipulation using simple hand gestures
respectively.
Although the proposed simulation prototype cannot yet mimic all possible accident
scenarios and physics behaviours of forklift truck operations in full, it does provide a
general concept on how forklift truck operations can be simulated in a full-immersive
CAVE-based VR system for safety analysis.
ii
With man this is impossible,
but with God all things are possible.
Mt 19:26
iii
DECLARATION
I declare that this thesis represents my own work, except where due acknowledgment
is made, and that it has not been previously included in a thesis, dissertation or report
submitted to this University or to any other institution for a degree, diploma or other
qualification.
Ryan Ka Kei YUEN
iv
ACKNOWLEDGEMENTS
Praises be to GOD for sending HIS one and only son Jesus Christ as my Savior who
rescues me from sin and death, and make me one of HIS servants to preach the good
news to the campus young souls. Thank GOD for taking up my infirmities and grant
me the faith and strength to walk through each darkest valley. May the name of Jesus
be lifted high and glorified!
Thanks to my supervisor Dr. S.H. Choi for his kindness, patience and encouragement
especially in the tough times when I was recovering from knee surgery. Thanks are
also expressed to Mr. Yang XiBei, all the staff and colleagues in the Department of
Industrial and Manufacturing Systems Engineering who have devoted their precious
time and efforts to the project.
I would like to thank my Father John Yuen and Mother Dolores Yuen for nourishing
me in the past quarter-century, sacrificing whatever possible to bring me up from a
rebellious boy to a good man. Last but not least, thank you Shepherdess Rebekah and
all brothers and sisters in Christ for their constant prayers and supports.
v
TABLE OF CONTENTS
ABSTRACT .................................................................................................................. i
DECLARATION ....................................................................................................... iv
ACKNOWLEDGEMENTS ....................................................................................... v
TABLE OF CONTENTS .......................................................................................... vi
LIST OF FIGURES ................................................................................................... ix
LIST OF TABLES ................................................................................................... xiii
CHAPTER 1
Introduction .............................................................................. 1-1
1.1 Basic Concepts in Forklift Truck Operation ................................................ 1-3
1.1.1 Turning Radius ................................................................................. 1-3
1.1.2 Stability of Forklift Trucks ............................................................... 1-4
1.1.3 Cases of Forklift Truck accidents ..................................................... 1-7
1.2 What is Virtual Reality .............................................................................. 1-10
1.2.1 Types of VR Systems ..................................................................... 1-11
1.2.2 Full-immersive CAVE-based VR systems ..................................... 1-13
1.3 Research Objectives ................................................................................... 1-14
1.4 Outline of thesis ......................................................................................... 1-15
1.5 Summary of contributions ......................................................................... 1-17
CHAPTER 2
Literature Review .................................................................... 2-1
2.1 Introduction .................................................................................................. 2-1
2.2 Vehicle Simulators ....................................................................................... 2-1
2.3 Forklift Truck Simulators ............................................................................ 2-1
2.3.1 Desktop VR systems ......................................................................... 2-2
2.3.2 Immersive VR Systems (Semi-/Full-) .............................................. 2-4
2.4 Physical Behaviours ..................................................................................... 2-8
2.4.1 Examples of Physics Engines ......................................................... 2-10
2.5 VR User Interfaces..................................................................................... 2-12
vi
2.5.1 Generic Input Devices .................................................................... 2-14
2.5.2 3D Mouse ........................................................................................ 2-14
2.5.3 Motion Tracking Devices ............................................................... 2-16
2.6 Summary .................................................................................................... 2-20
CHAPTER 3
Implementation ........................................................................ 3-1
3.1 Introduction .................................................................................................. 3-1
3.2 CAD Model Preparation .............................................................................. 3-1
3.2.1 Forklift Truck Vehicle ...................................................................... 3-1
3.2.2 Texture mapping ............................................................................... 3-3
3.3 Developments Phase .................................................................................... 3-6
3.3.1 Virtools Dev 4.0 ................................................................................ 3-6
3.4 Implementation and Testing ........................................................................ 3-7
3.4.1 ImseCAVE ........................................................................................ 3-8
3.4.2 Infrared Tracking Tools .................................................................... 3-9
3.4.3 Motion Feedback Wheel ................................................................. 3-10
3.4.4 VR Distribution Concept ................................................................ 3-11
CHAPTER 4
Physics Behaviours in Virtools ............................................... 4-1
4.1 Introduction .................................................................................................. 4-1
4.2 Getting objects physicalized ........................................................................ 4-1
4.2.1 Convex Hull ...................................................................................... 4-1
4.2.2 Concave Hull .................................................................................... 4-2
4.2.3 Physicalized Building Block (BB) .................................................... 4-3
4.3 Phase I – Building Forklift Truck Movement .............................................. 4-4
4.3.1 Physics Car BB ................................................................................. 4-5
4.4 Phase II – Mimicking the Pallet Operations ................................................ 4-8
4.4.1 First Approach – “Unphysicalize BB”.............................................. 4-8
4.4.2 Sensor Detection System (SDS) ....................................................... 4-9
4.4.3 Second Approach – “Beam Physics Object BB” ............................ 4-11
vii
4.5 Phase III - Integration of “Motion Controller BB” .................................... 4-13
CHAPTER 5
Integration of IR Tracking System ........................................ 5-1
5.1 Introduction .................................................................................................. 5-1
5.2 The IR Tracking System in ImseCAVE ...................................................... 5-1
5.2.1 Principles of IR Tracking .................................................................. 5-3
5.3 Head Tracking.............................................................................................. 5-3
5.4 Hand Tracking ............................................................................................. 5-6
5.4.1 3D Rotatable Option Menu ............................................................... 5-7
CHAPTER 6
Case Analysis of Forklift Truck Accidents ............................ 6-1
6.1 Introduction .................................................................................................. 6-1
6.2 Common accidents scenarios ....................................................................... 6-1
6.2.1 Skidding or overturn ......................................................................... 6-1
6.2.2 Slipping or falling of pallet loads ..................................................... 6-3
6.2.3 Tipping over ...................................................................................... 6-3
6.3 Analysis of forklift truck overturns ............................................................. 6-5
6.4 Summary .................................................................................................... 6-10
CHAPTER 7
Conclusion and Recommendations ........................................ 7-1
7.1 Conclusion ................................................................................................... 7-1
7.2 Recommendations ........................................................................................ 7-3
REFERENCES........................................................................................................ R-1
viii
LIST OF FIGURES
Figure 1.1
A Typical Physical flow of Materials in a Supply Chain ..................... 1-1
Figure 1.2
Various Logistics Activities in a Warehouse (source: Geodis Wilson) 1-2
Figure 1.3
Turning Radii of an Automobile and a Folklift Truck ......................... 1-3
Figure 1.4
Load Centre .......................................................................................... 1-4
Figure 1.5
Load Chart of a Forklift Truck ............................................................. 1-5
Figure 1.6
Manufacturer's data plate ...................................................................... 1-5
Figure 1.7
Stability Triangle of a forklift truck ..................................................... 1-6
Figure 1.8
Sudden break; Turn over ...................................................................... 1-7
Figure 1.9
Accidents Scenario 1 - Struck by a truck ............................................. 1-8
Figure 1.10 Accident Scenario 2 - Fall from height ................................................ 1-9
Figure 1.11 A Non-immersive VR systems in Gaming Kinect from Microsoft; PS
Move from Sony ................................................................................. 1-12
Figure 1.12 HMD prototype design by Sony features 720p OLED display with 5.1
surrounded audio simulated ................................................................ 1-13
Figure 2.1
Full Flight Simulators from Aeronautical System Engineering (AES) 2-2
Figure 2.2
A web-based simulator - Forklift Madness .......................................... 2-2
Figure 2.3
Cover page and screen capture from Forklift Truck Simulator 2009 ... 2-3
Figure 2.4
Some Screen Captures from NSC SafetyworkTM Lift Truck ............. 2-4
Figure 2.5
Structure of driving cockpit of Motion-based forklift simulator by
Bergamasco et al. (2006) ...................................................................... 2-5
Figure 2.6
Examples of vehichle simulators with high field of regard (FOR)
.............................................................................................................. 2-6
Figure 2.7
The Forklift Simulator developed by ITCL .......................................... 2-7
Figure 2.8
Distribution of released game titles with respect to Physics Engines used
............................................................................................................ 2-12
Figure 2.9
Interactions among users and Virtual Environment (VE) .................. 2-13
Figure 2.10 Handheld Bluetooth Wireless Keyboard ............................................ 2-14
Figure 2.11 Wii Remote paired up with Motion-Plus; Characteristics of Wii Remote
- Motion, Pointing, Rumble, Audio .................................................... 2-15
ix
Figure 2.12 Kinect Sensor; Depth Map taken by a Kinect Sensor ........................ 2-17
Figure 2.13 Skeleton joint positions relative to the human body; Active tracking for
two players .......................................................................................... 2-18
Figure 2.14 Screen shot of drawing a gesture “S”;
Defined gestures and
corresponding actions ......................................................................... 2-19
Figure 2.15 An early prototype of pose-traker ....................................................... 2-19
Figure 3.1
Electric Moto Narrow Aisle Truck; Rough Terrain Forklift Truck...... 3-2
Figure 3.2
Major body parts of a counterbalanced forklift truck ........................... 3-2
Figure 3.3
CAD Models of Movable parts of forklift truck................................... 3-3
Figure 3.4
A CAD Model of an ambulance truck with and without textures ........ 3-4
Figure 3.5
UV mapping (a) Forklift truck model in 3dsMax; (b) UVW map of
forklift truck; (c) Texture rendered for Forklift Truck; (d) Texture
rendered for Driving Wheels ................................................................ 3-5
Figure 3.6
User Interface in Virtools Dev 4.0........................................................ 3-7
Figure 3.7
(a) Stereoscopic Images through 3D polarized glasses; (b) CAVE
system with four projection screens; (c) Computer Network of
ImseCAVE............................................................................................ 3-9
Figure 3.8
MOMO Racing Force Feedback Wheel as driving panel for forklift . 3-10
Figure 3.9
Cause Distribution .............................................................................. 3-11
Figure 3.10 Effect Distribution .............................................................................. 3-12
Figure 4.1
Examples of Convex Hull..................................................................... 4-2
Figure 4.2
Examples of Concave Hull ................................................................... 4-2
Figure 4.3
Convex Hull on Concave Objects ........................................................ 4-2
Figure 4.4
Difference between Convex Hull and Concave Hull on a storage rack4-3
Figure 4.5
Dialog box of parameters in a “Physicalized BB” ............................... 4-4
Figure 4.6
Reference axes of car system using “Physics Car BB”; Reversed Axes
used for Forklift Truck Model .............................................................. 4-5
Figure 4.7
Standard I/O parameters of “Physics Car BB” ..................................... 4-6
Figure 4.8
Parameters in Data Arrays of “Physics Car BB” (Body, Wheel, Engine,
Skidding Output) .................................................................................. 4-7
Figure 4.9
A programme for driving wheel alignment .......................................... 4-7
x
Figure 4.10 Three additional blocks used in detecting the forks motion ................. 4-9
Figure 4.11 Senor Detection System (SDS)........................................................... 4-10
Figure 4.12 Sensor detection region in SDS; Plane view on horizontal position
testing.................................................................................................. 4-10
Figure 4.13 Pallet attached to the forks after passing both horizontal and vertical test
............................................................................................................ 4-11
Figure 4.14 A physicalized block lifted by a fork; A programme for “Beam Physics
Object BB” ......................................................................................... 4-12
Figure 4.15 A testing fork loops with the physicalized block. .............................. 4-12
Figure 4.16 Standard I/O parameters of “Physics Torque BB” and “Physics Impulse
BB”; Snowboard simulation using “Physics Torque BB”.................. 4-13
Figure 4.17 Dialog box of parameters in a “Motion Controller BB” .................... 4-14
Figure 4.18 3d frame – plane; Lifting and rotation of forks .................................. 4-15
Figure 4.19 A programme that controls the forks operations using “Motion
Controller BB” .................................................................................... 4-15
Figure 4.20 Lift cylinder with physicalized or without physicalized .................... 4-16
Figure 4.21 Overview of a virtual warehouse in Virtools...................................... 4-17
Figure 4.22 (a) Addition view for pallet picking; (b) driving up a ramp; .............. 4-17
Figure 4.23 (c) dropping off at storage rack; (d) loading into container ............... 4-18
Figure 5.1
(a) IR camera of OptiTrack; (b) Network layout of IR cameras; (c)
Setup of IR cameras in ImseCAVE; (d) Camera arrangement and
TrackCoverage in ImseCAVE .............................................................. 5-2
Figure 5.2
(a) IR markers; (b) A IR marker is being tracked; (c) Image captured by
an IR camera ......................................................................................... 5-3
Figure 5.3
“Rigid-body” in reality and TrackingTools; Positions of IR markers in
head tracking......................................................................................... 5-4
Figure 5.4
Cameras Positions in CAVE-based VR system: Static; Mobile........... 5-5
Figure 5.5
Simulated views of user operating a forklift truck ............................... 5-6
Figure 5.6
Spatial arrangement of gestures; “grab”; “drop” .................................. 5-7
Figure 5.7
A 3D rotatable option menu ................................................................. 5-8
Figure 5.8
The concept of “Focus Zone” ............................................................... 5-9
xi
Figure 5.9
Command controls of the option menu ................................................ 5-9
Figure 5.10 Hand tracking for menu command control ......................................... 5-10
Figure 6.1
Forklift Overturn between reality and simulation ................................ 6-2
Figure 6.2
Colour Indicator of overturn during simulation ................................... 6-2
Figure 6.3
OSHA's Examples of Tipping Over Potential ...................................... 6-3
Figure 6.4
Failure in simulating the tipping over of forklift truck ......................... 6-4
Figure 6.5
Forklift truck turning on level surface .................................................. 6-5
Figure 6.6
Top view of “Zone A” and “Zone B” ................................................... 6-7
Figure 6.7
A programme for testing turning radius ............................................... 6-7
Figure 6.8
Output text file generated by the testing programme ........................... 6-8
Figure 6.9
Turning tracks of forklift truck at various velocities ............................ 6-9
Figure 6.10 Other possible case scenarios for future analysis ............................... 6-10
xii
LIST OF TABLES
Table 2.1
A general comparison between HMD and CAVE ............................ 2-8
Table 2.2
Physical behavioural criteria of forklift trucks versus different forklift
truck simulators ........................................................................... 2-9
Table 2.3
Common examples of Physics Engines ........................................ 2-10
Table 3.1
Technical Specification of Computers using in ImseCAVE .............. 3-8
Table 4.1
Strengths and weaknesses of tested building blocks ....................... 4-18
Table 6.1
Summary of overall performance in each case scenarios ................ 6-10
xiii
CHAPTER 1 Introduction
Huge advancements over the past decades in computer- and information-related
technologies, like RFID and automation systems, have enabled the logistics industry
to shorten lead times and to improve warehouse management much better than ever,
facilitating realization of just-in-time and error-free deliveries. However, overall
success would be possible only when the supply chain operations are carried out
timely and precisely from the point where raw materials are received to the point
where finished goods are delivered to the end users.
Figure 1.1 A Typical Physical flow of Materials in a Supply Chain
As illustrated in Figure 1.1, a warehouse may be considered as a mid-way storage
point of the supply chain connecting both ends of demand and supply. By storing
bulk quantity of inventory in a systematic way and making them available
conveniently when needed, warehouses play a pivotal role in maintaining the
continuity of goods flow in the supply chain. To ensure a warehouse is running at
high efficiency, material handling operations within should be carefully implemented
and monitored. There are various kinds of material handling operations in a
warehouse, as shown in Figure 1.2, including storage of raw materials or finished
goods, pick-and-pack operations and order picking for shipment. These operations
often require the use of forklift trucks. Forklift trucks most commonly used to lift,
transport, and load/unload goods and materials to and from the warehouse. They are
indeed powerful and versatile industrial vehicles that can hugely alleviate dependence
on manual labour thus save human resources and speed up the material handling
process.
1-1
Figure 1.2 Various Logistics Activities in a Warehouse (source: Geodis Wilson)
However, if a forklift truck is not properly handled, accidents can be triggered.
Countless forklift trucks accidents have been reported all over the world. Each year,
forklift trucks are involved in nearly 80 fatalities and 90,000 injuries in the United
States. According to National Safety Council of US, the annual cost of occupational
injuries in 2009 cost U.S. workplaces $168.9 billion, exceeding the total profits
reaped by the 50 largest “Fortune 500” that year. Forklift accidents do not only incur
workers injuries, property damages, insurance claims, medical and rehabilitation
expenses, they also interrupt the warehouse operations and lead to a breakdown of the
entire supply chain through domino effect. Once an accident happens, the warehouse
has to be shut down for clearance and investigation, calling for an immediate halt to
all related material handling operations. As a result, production schedule may be
delayed and stock-out may occur. In the long run, they will cost losses on both sales
revenue and creditability of the firm. Therefore, safety of forklift trucks is an issue to
be highlighted and addressed by all logistics industries.
1-2
1.1 Basic Concepts in Forklift Truck Operation
Many people may argue that operating a forklift truck is more or less similar to
driving a passenger car. This is not true because a forklift truck and a car are quite
different in nature. Firstly, a forklift trick has a narrower wheelbase and a higher
centre of gravity (COG), making it more difficult to maintain stability than in other
automobiles. We may consider the balancing of a forklift truck as a teeter-totter. A
counterweight is installed at the rear of the truck to balance the weights added by the
loads during operation. Therefore, a forklift truck can also be called a counterweight
balanced truck. As the counterweight is fixed and cannot move, there is a significant
weight imbalance even when the forklift truck is empty or unloaded. Forklift truck
drivers are always reminded that the vehicle carries its loads outside, while a
passenger car usually carries loads inside. This explains why counterweight trucks
usually have a higher risk of turning over than any other vehicles during operation.
1.1.1 Turning Radius
As shown in Figure 1.3, a noticeable difference between automobiles and forklift
trucks is their turning radii. Forklift trucks are rear-wheel steering, and they require
a much smaller turning radius than that in automobiles; this allows the operators a
greater degree of rotation in confined workplaces like tight corners. In addition, while
steering, the wheels on the rear axle turns back and forth which means no extra
steering force is required to maintain the turning moment. Thus, controlling a forklift
differs from the drivers‟ experiences on most wheeled vehicles.
Figure 1.3 Turning Radii of an automobile and a forklift truck
1-3
1.1.2 Stability of Forklift Trucks
Beside structural issues, there are key principles governing stability of forklift trucks,
particularly the Load Centre and the Stability Triangle.
Load Centre
As we all know the COG, or the centre of gravity, refers to the mean point where the
gravitation force pull on an object. As usual, the COG of a forklift truck lies at the
centre of the vehicle. Yet, in practice, this is not the case. During operation, the COG
of truck may shift to either direction due to the change of load centre and inertia
caused by truck movements.
Load centre, illustrated in Figure 1.4, is the distance from the inside edge of the forks
or platform to the centre of the load. In standard calculation, load centre is measured
at 60 cm above the forks and 60 cm forward from the mast. And the maximum load
centre or capacity of a forklift truck is identified on the manufacturer‟s data plate
printed on the truck body.
Figure 1.4 Load Centre
Since loads usually come with difference sizes, mass and weight distributions,
operators must be trained to identify the nature of the load being handled and locate
the load centre precisely. Sometimes even the load is within capacity, a forklift truck
may still not be able to handle the load safely if its load centre is greater than 60cm.
Moreover, when the load is titled forward or backward, the COG will moves along.
As the COG shifts, the maximum load capacity will decrease. The full capacity of a
forklift truck can be reduced drastically to about 50% when the load is raised and
1-4
titled forward. Therefore, forklift truck operators are always encouraged to keep their
loads as low as possible to reduce risks. Examples of Load Chart and Manufacturer‟s
data plate of a forklift truck are shown in Figure 1.5 and Figure 1.6, respectively.
Figure 1.5 Load Chart of a Forklift Truck
Figure 1.6 Manufacturer's data plate
For example, if the maximum capacity of a forklift truck is 1,800kg, as shown in the
diagram, measured at the load centre of 60 cm away the mast, the cm-kg capacity of
the truck is then equal to (60X1,800=) 108,000 cm-kg. Now we suppose have a new
load centre at 10cm away from the previous one (60cm), the maximum capacity will
1-5
then be decreased from 1,800kg to (108,000/(60+10)=) 1,543 kg, assuming the height
of forks B remains unchanged.
Stability Triangle
As previously mentioned, other than the load centre, inertia may also account for
shifting of the COG. Most forklift trucks are of a three-point suspension system,
instead of four wheels used in cars. Two front wheels and the centre of the rear axle
provide a triangular suspension. In Figure 1.7, a stability triangle is illustrated by a
black dot (COG) located inside a triangle which is joined up by the imaginary points
at the three suspension points. A forklift truck is only stable when the black dot lies
within the stability triangle. This is so when the forklift truck is in steady and
unloaded state, but the black dot shifts once the truck moves.
B
COG of Vehicle
(Unloaded)
COG of Vehicle
(Maximum Load)
A
C
Stability Triangle
(Safety Zone)
Figure 1.7 Stability Triangle of a forklift truck
When the forklift truck experiences a sudden break, the inertia will shift the COG
forwards to the line BC, as shown in Figure 1.8. As long as the black dots remains in
the stability triangle (safety zone), the forklift remains well-balanced. However, on
skidding or making sharp turns, the COG may shift beyond the zone, resulting in tipover or overturn.
1-6
B
A
C
B
A
C
Figure 1.8 (top) Sudden break; (bottom) Turn over
1.1.3 Cases of Forklift Truck accidents
Forklift truck accidents often involve in tip overs, driver ejection, falling objects, offthe-dock or crushing into pedestrians. These accidents could usually happen in
scenarios like turning at speed, travelling with the load raised, crossing over uneven
surfaces, sudden brakes or backing up a slope. Whenever the forklift truck‟s inherent
balance is disrupted, there is a possibility of having an accident. In the latest release
of top 10 most citied safety violations (Oct, 2010) from OSHA, forklift trucks ranked
8th with 2,993 violations. Accordingly to a casebook published of the Labour
Department of Hong Kong in 2006 that currently provides most up-to-date
information on occupational fatalities in the container handling and storage sector,
several scenarios of workplace accidents can be featured and shared to provide
lessons to both managers and workers in material handling industries. Two of them
have been chosen and discussed.
1-7
Case 1: A warehouse keeper got struck by a forklift truck when stacking goods
in a container.
Figure 1.9 Accidents Scenario 1 - Struck by a truck
Scenario: Goods were to be loaded into the container depot. The duty of the deceased
person was to assure goods were loaded correctly into the correspondingly container.
Yet, the deceased person noticed the doors of container could not close after the
commodities were stacked. He then arranged a forklift truck with a pushing block
installed at the front trying to push the goods inward. During the action, he got
trapped between the pushing block and the goods accidentally, as illustrated in Figure
1.9.
Lessons to learn: After adding the pushing block attachments, the operator‟s vision
will be blocked. He should have tried beeping or any other means to make sure the
driving path is clear and unobstructed. If he is unable to do so, a signaller is suggested
to assist him in this particular operation.
1-8
Case 2: A worker fell from a height while trying to release a jammed container
door.
Figure 1.10 Accident Scenario 2 - Fall from height
Scenario: The deceased person was about to unload goods from a 20-feet container
truck parked in the dock. He discovered the locking mechanism of the left container
door malfunctioned and intended to fix it with a jack. He stood on a wooden pallet
and requested his co-worker to lift him up. When he tried to restore the locking
mechanism with the jack, the jack suddenly displaced. As a result, the deceased
person lost balance and fell to the ground, as illustrated in Figure 1.10.
Lessons to learn: No person should work on any wooden pallet rested on a forklift
truck. Even when working at heights is necessary, safety measure and adequate
precautions like life beat or other fall arresting equipment must be taken. Also, all
failure parts should only be repaired by experienced professional mechanics.
The seriousness of forklift truck accidents has long been recognized. It is discovered
that most of such accidents were not caused by mechanical defects of forklift trucks,
but by the carelessness as well as the inability of operators to take appropriate risk
assessment and control. According to Occupational Safety & Health Administration
1-9
(OSHA), about 70% of forklift truck accidents could be actually prevented with
proper safety training.
Nobody sets off with the innate skills and abilities to operate a forklift safely. To be
qualified to operate a forklift vehicle, one should first complete the basic training
lessons offered by its local accredited training schools or institutes before acquiring a
formal permit for driving a forklift truck. Apart from regular training programmes,
OSHA regulations also offer remedial policy for operators involved in accidents.
Without a doubt, traditional training courses are vital to enhance or polish the skills
of forklift truck operators. But on the other hand, they may not be adequate, in terms
of effectiveness, in reducing accidents that mostly owe to the lack of safety awareness
and alertness of the drivers. Veteran forklift truck drivers tend to operate the vehicles
merely by their experience, overlooking the hidden hazards in their workplace. An
effective way to improve the driving mentality of forklift truck operators is to bring
them into real danger and let them to learn by experience. However, it is almost
impossible to do so in reality as it costs human lives.
For this reason, the use of virtual reality (VR) is proposed. Through VR simulations,
drivers can immerse in various sorts of life-threatening scenarios where they learn to
react, give response and receive feedback instantaneously. As a result, they will
become more watchful of the latent dangers, and accumulate personal experience in
tackling those adverse situations. Some of their poor driving habits may also be get
rid of through trial and error.
1.2 What is Virtual Reality
Virtual reality (VR) was first developed in mid-1980s. It refers to a computer
technology for creation and simulation of a scenario that appears to our senses in such
a way indistinguishable from what we perceive the real or “physical” world. Apart
from eyesight, which is the dominant perceptual sense or mean to receive information,
other senses like sound and touch have also been exploited in the past decades. There
1-10
are different VR hardware configurations based upon head-mounted displays
(HMDs), workstations, large screen display systems, cave automation virtual
environments (CAVEs), and most recently three-dimensional (3D) television.
1.2.1 Types of VR Systems
Although it may not be easy to provide a complete categorisation, most VR systems
tend to fall into three main categories: non-immersive (desktop), full-immersive
(presence), and in-between the semi-immersive.
Non-immersive (desktop) VR Systems
A non-immersive VR system, as the name tells, mimics a virtual environment
through a standard high resolution monitor. Interactions can be made by means of
keyboards, mice, joysticks or any other 3D interaction devices. Compared with other
VR systems, a non-immersive VR system is less demanding on hardware
configuration. Its setup cost is comparatively low and can easily be implemented on
any up-to-date personal computers. Yet, its performance is considered as the biggest
shortcoming. Due to the hardware constraint, it is unable to produce the same level
of VR experiences as those immersive ones. But the emergence of Virtual Reality
Modelling Reality Language (VRML) has kept the system alive. VRML is a textbased language that incorporates the information of vertices, edges, colours, textures,
etc. of graphical components to be used particularly in the World Wide Web for
modelling objects and environment. By interpreting the VRML descriptions, Internet
browsers are allowed to produce 3D environment locally for animation interaction
and navigation. Recently, many game developers have successfully launched their
VR gaming systems into home platform, as illustrated in Figure 1.11. With the use of
certain sensory cameras or motion tracking devices, body movements are captured for
interactive gameplays inside the virtual world.
1-11
Figure 1.11 A Non-immersive VR systems in Gaming (left) Kinect from
Microsoft; (right) PS Move from Sony
Immersive (presence) VR System
Different from non-immersive systems, an immersive VR system allows users to get
into the virtual world for interactions hence provides a higher degree of immersive
feeling and a greater sense of presence. Quite often, while immersive VR is being
discussed, head-mounted display (HMD) is brought up. An HMD is a helmet-like
device put on the user‟s head. It comprises a small monitor positioned in front of each
eye. The display unit may be made up of CRT, LCD or even OLED. Most of the
HMDs are packed with speakers or headphones so as to produce both visual and
audio outputs. To achieve the full potential of HMDs, tracking devices are normally
added in to track the head movements of the user during simulation. With such input
data like positions and orientations, the VR applications can then render the
corresponding images instantly, allowing the user to look around inside the virtual
environment by moving his/her head as if it happens in reality. However, this
hardware has some drawbacks, such as its bulk size and weight, limited field of view
(FOV) (at about 60 degrees, when compared to 180 degrees of human eyes), and
motion illness resulted from prolonged usage. Indeed, these issues become less
significant nowadays as the technology becomes more mature. Figure 1.12 shows a
newly designed HMD prototype by Sony in 2011.
1-12
Figure 1.12 HMD prototype design by Sony features 720p OLED display
with 5.1 surrounded audio simulated (Sources: IGN Entertainment)
1.2.2 Full-immersive CAVE-based VR systems
Cave Automatic Virtual Environment (CAVE) was originally conceived by CruzNeira at al. in 1993. It was a 10-ft cubic structure with multiple rear projected screens
of stereoscopic graphics. The goal of CAVE is to overcome the shortcomings of other
VR systems with low image resolution, isolation from the real world and inability to
share the full-immersive experiences with other users. Instead of projecting on a
single large screen, a CAVE system is designed with a large projection screen or wall
at each side, and literally fully surrounded by six screens in 360 degrees. Yet, due to
the high setup cost and area constraint, only a few sides are used in most cases. In
practice, four sides which include the right, left, front and bottom will be sufficient to
mimic a semi to full immersive virtual environment.
With the immersive experience provided by CAVE, users are no longer required look
from outside but able to experience the virtual world as if they are physically
presence and become part of it. As the FOV of CAVE is wider than other VR systems,
models presented can be larger and more details can be shown. The other main
advantage of using CAVE is its possibility to allow not only one but up to 5 to 6
people to get involved in the VR simulation, facilitating ideas sharing and
interactions. Hence, it is often regarded as the most appropriate VR platform for
design communication and evaluation. A vast number of CAVE-based vehicles
1-13
simulators, flight simulators have emerged to fulfil both education and training
purposes.
1.3 Research Objectives
The project aims at developing a full immersive VR simulation system for supporting
safety training of forklift truck operations.
By developing and implementing a
forklift truck simulation system in a CAVE-based VR environment, this system
provides stereoscopic graphical visualisation of forklift operations in a virtual
warehouse for a driver to practise all sorts of manoeuvring and pick-and-drop
operations of pallet loads. Incorporating with some tracking tools like head & hand
tracking devices, a forklift truck operator can react and receive feedback from the
virtual environment instantaneously. In addition, certain “virtual accidents” scenarios
were created with the impacts and damages analysed and visualised accordingly. In
doing so, it will be easier for forklift truck operators to review their driving habits,
discover their mistakes and correct them in trials. Hence the safety awareness and
mentality of forklift truck drivers could be enhanced. As a result, workplace accidents
would be minimised, and both warehousing management and the logistics industry
will then be benefited. The objectives of the research are summarised as follows:
1
To develop a prototype system for forklift truck driving simulation in a full
immersive CAVE-based VR environment
1.1
To prepare the 3D models required to build up a warehouse environment
with docks and containers in which a forklift truck can operate.
1.2
To design the interactions among objects and apply physics engine for
manipulation of objects according to their physics behaviours in reality.
1.3
To integrate the prototype system with CAVE for simulating various
operations of a forklift truck inside a virtual warehouse.
2
To implement head & hand tracking input interface
2.1
To incorporate an IR motion capture system for making commands instead
of using typical input devices like keyboards and joysticks.
1-14
2.2
To develop a 3D rotatory menu to allow users to change the parameters of
the scenarios by making a simple hand gesture during simulation.
3
To design and launch certain accident-like scenarios to validate the effectiveness
of the system
3.1
To identify causes of accidents to be studied and develop case scenario
respectively.
3.2
To highlight specific factors like fiction or speed in contributing to the
occurrence of accidents.
3.3
To consolidate the above modules into a versatile VR simulation system
used for training forklift truck operators.
1.4 Outline of thesis
The chapters of the thesis are organized in the following order:
Chapter 2 reviews the recent developments and limitations related to forklift truck
simulation, and justifies the research gap based on past achievements. Two kinds of
VR simulators, both desktop and full immersive systems, are discussed. In particular,
full immersive CAVE system is considered as a versatile VR platform that allows the
most realistic driving experience to be simulated. Since most forklift truck simulators
do not incorporate much physics behaviours, a few common physics engines that
render physics behaviours in VR simulation are reviewed. Also, various VR user
interfaces are compared on their functionalities and usability in the environment of
CAVE-based VR system.
Chapter 3 presents the design and implementations of the proposed simulation system.
All objects to appear in the virtual environment are prepared in CAD and mapped
with colour textures. They are then converted and imported into a software kernel of
CAVE. With the 3D life platform provided, imported identities can be interacted with
each other in the virtual world by defining relationship and constraints accordingly.
Likewise, a physics behaviour model is developed to power the virtual forklift truck
to behave as it does in reality. Through several testing and debugging, VR
1-15
distribution scripts are added to pinpoint the position of the user and projection of
each screen in order to get all PC clusters in CAVE synchronized and run in parallel.
Chapter 4 describes the functionalities of the built-in Physics Library empowered by
Havok of Virtools, and its efficiency in mocking the operation of a forklift truck. The
design paradigm of physics behaviours of the proposed simulator is presented. There
are three phases in total. In Phase I, the forklift truck body is physicalized for normal
manipulation. In Phase II, Sensor Detection System (SDS) is tested in mimicking fork
physics in pallet operations. However, it fails to mimic the operations in full.
Therefore, a new design paradigm using “Motion controller BB” is introduced.
Chapter 5 discusses the concepts of IR motion tracking system and how it works with
the CAVE system. A head tracking system is developed to trace user‟s head
movement and produce corresponding images in a real-time basis. This IR tracking
approach is also applied in hand tracking. An intuitive hand input-based interface is
built to allow users to command the system by a simple hand gesture like “grab” or
“drop”. To validate the result, a 3D-rotatory menu is created to be used in the
simulation system for functions calling.
Chapter 6 covers the studies on simulation of certain accident-like scenarios of
forklift trucks, such as skidding and falling of pallet loads. Tipping over is not
simulated due to the physics constraints of the built-in Physics Library of Virtools.
Nonetheless, an in-depth analysis of forklift truck overturns was performed and the
result validated the proposed forklift truck simulator in evaluating the manoeuvring
skills of operators. Limitations and suggestions are included later in the chapter.
Chapter 7 summarises the topics mentioned above, re-emphasises the strengths of the
proposed system, and followed by several future recommendations.
1-16
1.5 Summary of contributions
Forklift accidents have been recognised as a headache problem to the front offices of
many logistics companies. They not only cause workplace injuries and fatalities, but
also result in enormous amount of compensation claims. Although remedial training
programmes are highlighted and reinforced to refresh the skills and knowledge of
truck operators, their safety awareness can only be effectively enhanced by
experiencing the danger themselves. Therefore, the project aims to address the issue.
A fully immersive CAVE-based VR simulation system for forklift truck operation is
developed by integrating three main modules: (1) a physical behaviour model for
simulating daily operations of forklift trucks; (2) a virtual warehouse with common
features located in a warehouse, like ramp, dock, container truck and pallet racks; and
(3) an IR motion tracking system modified to be used in CAVE system for head &
hand tracking. The simulation prototype system built in the research provides an
innovative, interactive and immersive 3D experience on operating a forklift truck in a
virtual warehouse, where the following hazardous scenarios can be simulated.
1. To acquire first-hand experience on tackling adverse situations
In many cases, even the forklift truck operators are well-trained, they do not have a
proper concept on when and how accidents are going to happen in their workplaces.
For example, they are taught that driving too fast will cause the forklift truck turnover
however they may have different interpretations on the term “fast”. Rather, they tend
to drive according to their own experience which is often found undependable and
eventually leads to accidents. With the proposed simulation system, forklift truck
drivers can now visualize the influence on the truck‟s stability as speed increases
gradually. Similarly, other adverse conditions, such as skidding and oil spillage, can
be simulated with the system. As such, drivers will be able to learn bit by bit a better
or safer way to handle their vehicles by the first-hand walk-in experience earned from
the VR simulation. In the long run, both senses of emergency and alertness to the
hidden workplace hazards could be enhanced.
1-17
2. To rectify mistakes or driving habits developed over time
As interaction is one of the core values of VR, the proposed system is designed to
provide feedback to users in terms of visual display and audio output during
simulation. Specific warning signals or driving tips will be triggered as simulation
progresses. When the lift truck is operated over the speed limit allowed, a message
box will pop out onto the screen to notify the users. In addition, a real-time turnover
indicator is developed to show the stability of the truck. It will continuously monitor
the inertia force acting on the truck body and present it in a colour indicator. The
higher the level displays, the easier the truck will turnover. Trainees are then able to
identity which operation is more risky than the others. Therefore it provides rooms
for them to rectify their mistakes or poor driving habits developed unconsciously over
time. Moreover, their performance during simulations can be recorded for review
and follow-ups. Forklift instructors are also encouraged to provide guidance aside
during simulation in order to make full use of the CAVE environment.
3. To provide a general view on launching a forklift truck simulator into CAVEbased VR system for safety purpose
To the author‟s knowledge, no report has been found on any existing VR driving
simulators that were specifically designed to study the accidents scenarios by
manoeuvring the physics behaviours of a forklift truck in a CAVE-based VR
environment. This research may indeed be considered as an initial endeavour in this
direction. Even the proposed system may not fulfil everything needed in a training kit
to be used in driving schools or institutes, it provides some insights into the feasibility
of using such VR system in aid of forklift safety. Evaluations and suggestions are
included in the later chapters. Moreover, with the addition of the head tracking
system as well as the hand-based input interface, CAVE-based VR driving experience
is brought to a new level of immersion. The proposed system may be adapted for
other industrial mobiles like truck crane, stacker, bulldozer, excavator for similar
study in safety issues.
In summary, this research aims to develop a VR simulation prototype of forklift truck
1-18
operations as a supporting tool to improve forklift safety. The simulation system is
built in a full-immersive CAVE-based VR environment to train drivers by getting
them involved in the accidents virtually. By visualising the impact and damages
caused, drivers will become more vigilant and sensitive to the dangers hidden in their
workplaces. In addition, their poor driving habits developed over time can be
effectively rectified by learning their own mistakes during simulations. With drivers‟
safety mentality and skills enhanced to deal with emergency enriched, accidents rate
can be hopefully reduced, and warehouses will become a safer and effective
workplace that would significantly benefit the entire logistics industry and supply
chains.
1-19
CHAPTER 2 Literature Review
2.1
Introduction
This chapter discusses some recent developments and limitations related to forklift
truck simulation. Simulation models are first reviewed based on their level of VR
immersion, functionalities, and user experiences provided. After that, the physics
behaviours of forklift truck simulators are discussed and a few common physics
engines are presented. Lastly, VR user interfaces like 3D mouse, tracking system are
listed and compared in areas where they are used and performed best.
2.2 Vehicle Simulators
Vehicle simulation is one of the core values of VR applications. Apart from the wellknown virtual flight simulator for training pilots, we now have simulators for training
fire-fighters, parachutists, surgeons and so on. Many car manufacturers employ
vehicle simulators as a supporting tool to streamline their product design and
validation process. Engineers and investors are encouraged to participate in the
design process by trying out the expected performance of the new car in a simulator
to solicit feedbacks and exchange of design ideas. Nonetheless, skills training and
behaviour analysis of drivers are the primary goals of most vehicle simulators. With
the ability to mock up real-life scenarios under a controlled virtual environment, VR
simulators provide a more economical, versatile and safer option for practical training.
Figure 2.1 shows the overview of the Full Flight Simulator (FFS) developed by AES.
It meets the highest demand level (D) by the US Federal Aviation Administration by
having all six DOFs, a visual system of the outside world horizontal FOV of at least
150 degrees together with other sound and motion effects.
2.3 Forklift Truck Simulators
Although not many forklift truck simulators have been maturely developed, they can
be categorized into two groups based on their level of VR immersion, which can be
desktop-based or immersive-based. The higher the immersive level, the better the
user experience a driver can perceive. Generally, full-immersive simulators are
2-1
considered technically more sophisticated and serve better for education or training
purpose.
Figure 2.1 Full Flight Simulators from Aeronautical System Engineering (AES)
Figure 2.2 A web-based simulator - Forklift Madness
2.3.1 Desktop VR systems
As mentioned in section 1.2.1, desktop VR systems create a virtual environment by
displaying scenes on a single computer monitor. Compared to full-immersive systems,
the costs of setting up desktop VR systems are minimal. They can be easily applied
on home-used computers without installing additional expensive equipment.
Therefore they are often found in gaming industries. “Forklift Madness”, as shown in
Figure 2.2, is a web-based simulation flash application created by a small team of
2-2
Austrian developers, called SB-games. They launched a number of levels for
computer users to get familiar with the basic movements of a forklift truck.
Another example of desktop VR simulators is the Forklift Truck Simulator 2009, as
shown in Figure 2.3. It was developed by Astragon Software GmbH of Germany. By
incorporating with high end graphics, the Gabelstapler Simulator software provides
interactive experiences of operating of a forklift truck inside a warehouse-like
environment. With the aid of a mini-map system, users are guided to pick the pallet
load from a certain location before reaching the destination to get the task done.
However, the application does not cover any safety issues as the forklift truck itself is
not designed to flip or cause a turnover during simulation. Yet, it does give some
insights into the design of both warehouse scenarios and graphic interfaces of the
project.
Figure 2.3 Cover page (left) and screen capture (right) from Forklift Truck
Simulator 2009
On top of Forklift Truck Simulator 2009, a more in-depth, safety-related forklift truck
training tool has been jointly developed by Etcetera Edutainment and the National
Safety Council (NSC). NSC SafetyworkTM Lift Truck which based on OSHA
standards and best practices for forklift safety is built to provide hands-on forklift
truck operations experience to trainees in a virtual warehouse environment. This
system is guided with a series of task-based lessons focused on certain key concepts
of forklift truck safety. Other than simulating the full control of a forklift truck and
movements for warehouse operations, it gives prompt feedbacks to the trainees with
2-3
pop-out messages during simulation to remind them of the code of practice, the
mistakes they have made, and give suggestions right away. For example, if the
forklift truck is operated beyond the indoor speed limit, an alert message will appear
to warn the trainees. As a result, certain poor driving habits will be corrected as the
safety awareness of the trainees is enhanced. Some screenshots of the demo are
shown in Figure 2.4.
Figure 2.4 Some Screen Captures from NSC SafetyworkTM Lift Truck
2.3.2 Immersive VR Systems (Semi-/Full-)
Although desktop VR applications are favourable in spreading ideas as they are
informative and ready to use on most personal computers, yet the 2D images
delivered fail to mimic the reality in a sense of immersiveness. Especially in driving
which requires different viewing angles to support decision making, users should not
be only allowed to observe from outside through a computer screen. Instead, they
should be involved in the virtual scenario and become part of it for interactions. To
get over the hurdle, thousands researchers have poured continuing efforts in making it
ever possible.
2-4
Figure 2.5 Structure of driving cockpit of Motion-based forklift simulator by
Bergamasco et al. (2006)
An innovative research of the first motion-based forklift simulator by Bergamasco et
al. (2006) reaffirmed that the causes of forklift truck accidents were generally resulted
from the wrong manoeuvres of drivers. Therefore, the goal of their project was to
polish the driving skills of operators. As shown in Figure 2.5, a forklift truck
simulator was developed for training demonstration in industrial environment. In
order to provide users with physical feelings as it does in reality, the simulator was
built on the cockpit of a real forklift truck of a weight of 1360 kg. It was then fixed on
a parallel kinematics robot called Steward Platform with up to six degree of freedom.
For visual display, both Head Mounted Displays and LCD Wide Screen Monitors
have been taken into consideration yet they were later withdrawn due to the motion
sickness caused and incompatibility to the system. Instead, retro-projection system
was used. Two high luminosity LCD projectors were installed under the cabinet to
2-5
project the corresponding images, front and rear, against the mirrors mounted at a
distance of 1.5 meter far from the cockpit at both ends.
In VR training simulators, physics or dynamic behaviours is always one of the key
issues to be addressed. In Bergamasco et al.‟s work, the Physical Based Model (PBM)
was built with the use of an open-source library (ODE, Open Dynamic Engine) which
was open up by Russel Smith (http://www.ode.org/) to simulate right body dynamics.
Three different scenarios of an Italy packaging factory were mimicked and a sample
group of six workers was asked to complete the task of uploading waste paper bale
from one location to another.
Although positive feedbacks have been received on the user experience of the model,
still there are rooms for improvement or further development. Above all, driving
simulators should always provide users with wide viewing scope, instead of a limited
one on a single flat projection screen. The degree of space to be filled up with the
virtual world can be expressed in term of field of regard (FOR). In this case, the FOR
is limited, less than 100%, because the virtual scenarios cannot be displayed outside
the projection screens. However, in practice, forklift truck divers often require extra
views like side view in making the right judgement in times of loading or unloading
process. Therefore, one of the solutions is to modify the forklift simulator in a way
that its cabinet is surrounded by a circular wide screen. Nevertheless, this may take
up more room but little modification can be done. Two examples of vehicle
simulators with high FOR are shown in Figure 2.6.
Figure 2.6 Examples of vehichle simulators with high field of regard (FOR)
2-6
To address the viewing problem, apart from applying a wide screen monitor, CastillaLeon Institute of Technology (ITCL) (2008) put HMD into considerations. Similar to
the work of Bergamasco et al., ITCL developed the simulator on a real forklift truck
cabin which attached on a movable platform for motion feedback, as illustrated in
Figure 2.7. However, an HMD was used in place of the projection screens to generate
stereoscopic visualization by tracking the wearer‟s head position and orientation. This
intuitive interface design allows users to have a normal vision as in the reality under
“look left see left” principle.
Recalling from section 1.2.1, motion or simulator sickness and limited FOV are two
common problems associated with HMD. Motion sickness can be induced by a
number of factors like position tracking error, latency or flicker. Also, prolonged
usage will also lead to muscle fatigue and strains on neck and eyes, causing not only
a loss of motivations but also certain inappropriate user behaviours during simulation
due to discomfort experience.
Figure 2.7 The Forklift Simulator developed by ITCL
Although there were many research works comparing the degree of motion sickness
verses various type of simulators (fixed/motion based, HMD/CAVE), there is still a
huge debate on which one predominant over the others. Users may perceive and react
differently even they are put under an identical circumstance for the same duration.
But in general, the higher the degree of immersiveness of the simulator, the higher the
2-7
chance of motion sickness will be induced. Therefore, compromises and choices are
made according to the nature and needs of the simulators. For example, motion-based
design may show greater importance in flight simulators than others, while field of
view (FOV) is a deeper concern in forklift truck simulator than a train simulator.
Table 2.1 is the comparison of the general aspects between HMD and CAVE.
Field of
Resolution
view
(FOV)
Ease of
Body
setup
attachment
Motion
Interactions
Sickness
CAVE
HMD
with System:
Good
Fair
(1280x1024
(60-120
NVIS)
degree)
Excellent
(4096x2400
VisCube)
Excellent
Light
Easy
Good
Subjective
with Peers:
(depends on
Poor
calibrations
with System:
and
Lighter
Good
duration of
(Most wireless)
with Peers:
usage)
(wired/wireless
helmets)
Fair
(Unlimited)
Good
Table 2.1 A general comparison between HMD and CAVE
Although the HMD has several advantages over the CAVE, such as being cheaper,
more compact and transportable, HMD is not a complete solution. Due to HMD‟s
nature that keeps the user‟s vision inside the computer generating world without any
connections with the physical world, users cannot even locate their hands in sights.
This may create discrepancy in user experience, leading to unexpected misbehaviours
throughout the simulation. On the other hand, CAVE allows users to participate in
groups, promoting effective interactions and spread of ideas among peers.
2.4 Physical Behaviours
Traditionally, the motion of a ball bouncing against the floor can be simulated
virtually by drawing hundreds of frames and displaying them frame-by-frame in a
particular sequence and time interval. However, motion behaviours in VR simulations
are not pre-defined. Rather, they all run in real-time. Objects appearing inside the
2-8
virtual environment have nothing to do with physics, not even having a mass or
obeying the Newton‟s Laws of Motion unless they are programmed to do so.
Therefore, physics attributes like the mass, gravity, initial falling height, motion
directions, and elasticity of a ball need to be set precisely as each of these attributes
can influence the final simulation result. Therefore, it is always a challenge for the
developers of VR simulators to evaluate what physics behaviours are required and
how they are going to be mimicked in their simulation systems.
Among VR vehicles simulators, certain physics behaviours, such as acceleration/
deceleration, turning motion and collision detection, are commonly found. They can
be mimicked by programming certain formulas and rules to govern them. But to get a
forklift truck operating in a virtual warehouse, a lot more physics behaviours are
involved. For example, in simulating the pallet-loading process, the mass of pallet
loads, upwards impulse applied by the forks, the centre of gravity of the forklift truck
body are all indispensable factors in achieving a sensible result. To verify whether
certain physics behaviours are used in the VR forklift truck simulators mentioned in
section 2.1.1, they are put under testing and the results are listed in Table 2.2.
1. Simulation of
Overturn/tip over
2. Moment of Inertia (e.g.
skidding)
3. Weight variation of the
pallet loads
4. Will the pallet loads fall
off the forklift trucks?
Forklift
Forklift Truck
NSC Safetywork
Madness
Simulator 2009
Lift Truck












Table 2.2 Physical behavioural criteria of forklift trucks versus different forklift
truck simulators
2-9
To highlight the physics behaviours of forklift trucks in time of adverse conditions,
four essential criteria are set to determine if one can fulfil the requirements. These
include the simulation of overturn and tip over, skidding, weight variation of the
pallet loads and the possibility of the pallet loads falling off the forklift truck. From
the result, although some of the VR simulators allow the pallet loads to slip off the
lifting forks during operations, they cannot simulate common physics behaviours
neither like flipping the car sideward nor forward to cause overturn and tip over.
2.4.1 Examples of Physics Engines
In practice, formulas and rules of physics behaviours are seldom programmed for one
particular use. Instead, they are often grouped into a library of codes called “Physics
Engine”. A physics engine is a general solution to simulate objects in a virtual
environment as if they are rigid bodies that obey the Laws of Newton in reality.
Gravity, elasticity, moment of inertia and collision detection are some common basics
supported by a general physics engine. There are various physics engines available in
the market and can be categorized into two main groups: open source and closed
source, as illustrated in Table 2.3.
Open Sources
Box2D, Bullet, Newton Game Dynamics, Open Dynamic Engine
(ODE), Tokamak Physics Engine
Closed Sources
PhysX, Havok (open free for non-commercial), Vortex
Table 2.3 Common examples of Physics Engines
(Sources: http://web.cs.wpi.edu/~rich/courses/imgd4000/lectures/C-Physics.pdf)
Each physics engine supports different features and performs in its own characteristic
way. A simulation with the same inputs like mass, weight, and heights in systems
with different physics engines may generate possibly different results. Some physics
engines may show better performance in joint-based physics while others can render
collisions at a higher degree of precision. Here are some examples of common
physics engines:
2-10
PhysX (Previously known as NovodeX): It is owned by NVIDIA and is the only
solution in the market that fully supports NVIDIA GPU/PPU acceleration for
graphics.
Havok: A reputed physics solution not only supports real-time collision, dynamics of
rigid bodies, but dedicated tools for character animations. Recently, its SDK library
has been available to all developers for non-commercial uses.
ODE (Open Dynamics Engine): Developed in C/C++ by Russel Smith in 2001 and
has been ported to many platforms. It includes a rigid body dynamics simulation
engine and a collision detection engine.
Newton Game Dynamics: An open source engine available for Windows, MAC and
Linux under zlib license. Current version is reported to support the use of multi-core
CPUs and GPUs.
Bullet: Another highly favoured physics engine used in both games and movies. It is
free for commercial use. Latest version 2.77 was released on September 22, 2010.
A Boeing (2007) suggested there are number of factors affecting the performance of a
physics engine, such as the simulation paradigm, collision detection and response, air
resistance, etc. As a result, it is hard to tell which physics engine gives the best
performance in simulating the operation of a forklift truck vehicle. ODE (Open
Dynamic Engine) has been used by Bergamasco et al. (2006) in INDICA simulator.
However little has been mentioned on the study of the engine itself. And again, to
author‟s knowledge, no physics behaviours of forklift trucks accident have been
demonstrated in his work.
Instead of benchmarking the physics engines, the project aims at investigating a
simple and realistic way to get the forklift trucks operating in the virtual environment
for accidents review. Among the available 3D platforms to be used in CAVE for VR
simulations, Virtools has proven to be a practicable, quick-fit solution. Virtools 4 is a
3D platform to develop and deploy virtual experience on personal computers, game
2-11
consoles and other web applications. With the VR Library add-on, developers can
easily overcome the complexity inherent in distributed computing on PC clusters
during the configuration of CAVE. Moreover, Virtools allow developers to create
realistic physics by integrating Havok Physics Engine it its Physics Library. In
addition to typical dynamics like gravity, mass, fiction, elasticity, collision detection
behaviours, other physics constraints like hinges, springs, and joints can also be
mimicked in great simplicity. With the opportunity of changing all physics
simulations parameters real-time in Virtools, both the diversity and functionality of
the scenarios built could be enhanced. Step-wise procedures on the paradigm of
applying such Physics Library in the project are reviewed later in chapters 3 and 4.
Figure 2.8 shows the distribution of game title released with respect to different
physics engine from 2006 to 2009.
Figure 2.8 Distribution of released game titles with respect to Physics Engines
used (source: http://physxinfo.com/articles/?page_id=154)
2.5 VR User Interfaces
In addition to a high immersive level of VR display backed up by a right-fit physics
rendering engine, user interfaces of VR simulations share equal importance in
2-12
bringing forth the best virtual experience. As shown in Figure 2.9, by connecting the
end users and the VR simulation system with the most common motion tracking
techniques, users are allowed to navigate and interact with the virtual world through
simple movement or hand gestures. The more intuitive and easy-to-learn the design
of the interface, the more favourable virtual experience the simulator brings about.
Virtual
Environment (VE)
Virtual Objects
& Other Attributes
Multimodal
Displays
User / Input
Interfaces
Users Presence
&
Virtual Experience
Figure 2.9 Interactions among users and Virtual Environment (VE)
There are various types of user interfaces being used in VR simulations, ranging from
typical keyboards and mice for desktop level simulators, motion or gesture tracking
systems to high-end emerging techniques like brain-computer interface. To choose
an appropriate user interface that suits best for a particular application, some key
factors like functionalities, physical interaction techniques required, complexity to
learn and control, ergonomics and universal usability need to be examined because a
particular user interface may suit well in one application but entirely incompatible
with another. For example, keyboards and mice which have long been recognized the
best combination of input interfaces for desktop computers may not be appropriate in
some immersive virtual environments wh/ere users are supposed to stand. In contrast,
a complex 6-DOF tracking device may only be fully utilized in a pre-defined room
space. In general, user interface devices can be classified, based on their
functionalities and configurations, into the following categories:

Generic Input Devices

3D Mouse

Tracking Devices
2-13
2.5.1 Generic Input Devices
Keyboards and mice are classic examples of desktop input devices due to their
universality and user familiarity. Even nowadays, with the emerging of touch screen
technologies for tablets PC, the typical “keyboard + mouse” interface is irreplaceable
in near future. Because there is no other input devices work as effective and precise
as this combo do. However, many Virtual Environments are featured to allow users to
move around physically. The users may then find inconvenient to carry along the
devices. One way to overcome the obstacle is to eliminate the connection chord and
reduce the size of the device as compact as possible, as illustrated in Figure 2.10. Yet
holding the device will hinder the user from further interacting with the VEs.
Figure 2.10 Handheld Bluetooth Wireless Keyboard
Trackballs and joysticks are also common input devices to be used frequently in
tradition driving and flight simulator or other CAD applications. They can be
redesigned or modified to fulfil the requirements of certain VR applications.
2.5.2 3D Mouse
Traditionally, 2D mouse offers the function to move and to select on a coordinate
plane. But for 3D mouse, they are specifically designed to interact with 3D objects by
detecting movements in x, y and z directions. One of the most recent breakthroughs
was the launching of Wii Remote in late 2006 by Nintendo, as shown in Figure 2.11.
Wii Remote is the game controller designated to be operated with Wii game console.
It is a hand-held VR controller with built-in accelerometers to sense the orientation
changes, both directions and acceleration, at a 3D space defined for signal
2-14
transmission through Bluetooth. Besides, it incorporates an IR light emitter together
with an infrared camera placed at a distance of a few meters far for pointing and
navigation on screen. The Wii remote has also multi-buttons, a speaker and a rumble
device for further enhancements of user experience. Apart from gaming, Wii Remote
has been transplanted by third party applications to replace generic input devices, like
keyboard, mouse or joystick.
Figure 2.11 Wii Remote paired up with Motion-Plus (left); Characteristics of
Wii Remote - Motion, Pointing, Rumble, Audio (right)
Sreedharan et al. (2007) determined the key functions required by selection common
gesture performed by avatars in a VR application - Second Life to be mimicked by
five gestural controls on the Wii Remote. Ten participants from various backgrounds
were invited to take part in testing and validation. And 70% of them would rather use
Wii Remote interface than the keyboard interface. Wingrave et al. (2010) related Wii
Remote as a Spatially Convenient Device to be used by 3D User Interfaces
developers to use or modify it to meet their needs without substantially investing in
hardware. With the add-on of Motion-Plus, which is consisted of two gyroscopes
reporting the angular velocity, the Wii Remote is much more of a general 3D tracker
for a wide range of applications.
Smith and Ecrison (2009) developed a VR user interface similar to Wii Remote into a
fire-safety training simulation for children in CAVE. By using a 6-DOF wand with
tracker reporting its orientation, a ray will be generated to the direction the wand
pointing. It is an intuitive design to be learned and used quickly by children. However,
2-15
beside a wand, an extra gamepad is required for users to navigate through the virtual
environment. This may isolate users from the virtual environment. Functionality and
design of the interface remain the major concerns of 3D mouse solution.
2.5.3 Motion Tracking Devices
For most VR applications, it is vital for the user interface to be able to track the user‟s
physical position or orientation in the given 3D space. This can be achieved by
acquiring the information of body movement or gesture from the target person.
Currently motion tracking techniques in VR can be divided into Gesture (Non-marker)
Tracking and Marker Tracking.
Gesture (Non-marker) Tracking
In Gesture Tracking, the user‟s bare motion is recognized by the system through
cameras. As it requires a plane view for detection, gesture tracking is often used in
desktop level applications instead of projection screen system like CAVE. In general,
a well-illuminated environment with good contrasts between the user and the
background will be more recognizable to the tracker, thus giving more speedy and
accurate result.
Dhawale et al. (2006) introduced a simple, inexpensive way to capture bare hand
motion through a single camera. By defining certain hand gestures like
horizontal/vertical flip and fist, the system is able to recognize them at a recognition
rate of more than 90% in most cases. The interface was then put into two desktop
applications running on Windows to validate the results. However, the proposed
system will only be able roughly track the shape of hand and associate it with the predefined patterns instead of tracking the movement of the hand. Song et al. (2008)
develop a tracking interface which supports 3 DOFs of a single finger to interact with
a desktop AR gaming application. But the accuracy of finger detection algorithm is
easily affected by the shapes surrounded and comparatively less reliable to that of a
3D mouse. Also, it does not support multiple hands of fingers tracking.
2-16
To get over the hurdle, Microsoft integrated the detections of both 3D position and
body movement or gestures in its latest add-on peripheral for Xbox360. Kinect, the
portmanteau of “kinetics” and “connect”, is a motion sensing input device designed
to completely replace the traditional game pads through a natural human interfaces
using gestures and voice commands. As shown in Figure 2.12 (left), there is a Kinect
Sensor which is indeed a combination of an RGB Camera, a pair of 3D Depth
Sensors, four multi-array microphones and a motorized tilt. To track the human body
from the background, one of the 3D Depth Sensors emits millions of Infrared dots on
all the objects in the 3D space while the other sensor collects the Infrared lights
coming back. By recognizing the distribution of the Infrared dots, the sensor system
will then be able to give a full 3D depth view on the detection region which is 1.2 to
3.5 m far from the sensors. Figure 2.12 (right) shows the depth map generated by the
3D Depth Sensors using colour gradients from red (near) to purple (far).
Figure 2.12 (top) Kinect Sensor; (bottom) Depth Map taken by a Kinect Sensor
As illustrated in Figure 2.13, after tracking the 3D position of user‟s body, Kinect
will automatically pair up the nearest skeletal structure from its library to the person
with a feature extraction of twenty joints. By doing so, Kinect will be able to create
2-17
unique avatar for each specific user for motion tracking later in game applications.
Meanwhile, Kinect Sensor can track up to six people at a time including two players
with motion tracking. Apart from body motion, users can trigger the menu or make
selection through simple hand gestures and voice command. As Kinect is still a piece
of raw jewellery to be exploited, there is little publication of its applications. But with
the release of beta version of Kinect for Windows SDK in June 2011, Kinect will be
no longer a peripheral to gaming console, but also a simple, intuitive and effortless
3D user interface to be used in lots of VR applications.
Figure 2.13 (left) Skeleton joint positions relative to the human body; (right)
Active tracking for two players
Marker Tracking
Marker tracking is another tracking method which makes use of passive markers. By
attaching these markers on user‟s hand, one‟s motion or gesture can be tracked by the
marker recognition system easily with high accuracy. This can be done by using
either ultraviolet light (Kim and Fellner 2004) or InfraRed light, because these lights
are visible to sensors but not to human eyes. In comparison, IR tracking is more
preferred as it is less interfered by the surrounding light sources than that in UV
tracking. Rahman et al. (2009) proposed a motion-path based gesture recognition
technique and applied it in a smart home environment. An IR camera was used to
track the position of the user‟s hand glove to produce a series of motion-points which
are then analysed to match the intended hand gesture with that defined in the systems.
Fifteen users were invited to evaluate the gesture recognition process and 97.25% of
accuracy has been achieved.
2-18
Figure 2.14 (left) Screen shot of drawing a gesture “S”; (right) Defined gestures
and corresponding actions
However, this recognition technique, as shown in Figure 2.14, did not support
multiple hands, and the marker tracking was limited to movements on a 2D screen.
In order to track the orientation of a target object in a 3D virtual environment, instead
of using single marker, a network of four infra-red markers was introduced in the
work of Pintaric and Kaufmann (2007). As illustrated in Figure 2.15, four IR cameras
were used to track markers in a room-sized space. By fixing the distance and angle
between markers, a unique pose will be formed. This recognisable pose will then be
used to identify the target object, as well as tracking its real-time position and
orientation. This technique enhances the speed and accuracy of IR tracking and can
be applied to large virtual environment like CAVE.
Figure 2.15 An early prototype of pose-traker
2-19
Other than 3D mouse and motion tracking, there are biological tracking systems like
brain-computer interface (BCI) of Touyama and Hirose (2007), and eye-gaze tracking
of Steptoe et al. (2008). Yet they remain experimental and are limited to specific
purposes.
2.6 Summary
This chapter reviews several forklift trucks simulators and discusses their limitations.
In general, simulators with full VR immersive applications bring about better user
experiences by immersing them into the virtual environment for more realistic
interactions. From the comparison between two popular immersive displays HMD
and CAVE, HMD is much cheaper and transportable than CAVE. However, HMD is
not feasible in driving simulators because of its limited FOV and discrepancy in user
experience throughout the simulation.
This chapter also points out that most of the forklift truck simulators under review are
only able to mock up graphic simulation of forklift truck operations instead of
mimicking the physics behaviours of forklift trucks accidental conditions like
overturn and skidding. To let a virtual forklift truck perform according to the
dynamics in reality, a physics engine is needed. The characteristics of a few physics
engines are presented. Some may show better performance in certain physics
behaviours than others do, and vice versa. There is no silver bullet to all physics
matters.
The proposed project uses CAVE as the platform to study the feasibility and
effectiveness of simulating accidental behaviours of forklift truck operations. The
Havok physics engine embedded in Virtools Physics Library will be used. Some
accidentals scenarios are built for testing and being reviewed. For hardware, a real
mock-up of forklift truck cockpit may bring forth the best use experience. Yet, in this
simulation prototype, a gaming driving wheel from Logitech will be used instead for
testing purpose.
2-20
Apart from operating the virtual forklift truck, user is proposed to interact with the
system, like pause, restart or call off simulation programme, alter current virtual
environments or change physics parameters of forklift truck body or pallet loads. In
view of this, VR user interfaces like generic input devices, 3D mouse and tracking
devices of both gesture and marker have also been reviewed. Although Kinect from
Microsoft offers high accuracy and is easy to learn, it is not considered feasible in the
CAVE system. Instead, the marker system is proposed. Based on the idea proposed
by Pintaric and Kaufmann (2007), IR tracking is used for both head-tracking and
hand-tracking in CAVE to provide a cutting edge driving experience in forklift truck
operations.
The methodology, implementation and analyse of the proposed simulation model will
be discussed in detail in the following chapters.
2-21
CHAPTER 3 Implementation
3.1
Introduction
This chapter demonstrates the design paradigm of the proposed forklift truck
simulator to be used in the virtual environment of a full immersive CAVE system. To
begin with, the hardware configuration of the system is reviewed. It is then followed
by the procedures of setting up such a system with the aid of CAD development
programmes and Virtools Dev platform. The implementation of the simulator is
carried out in three phases: CAD model preparation, development in Virtools, and
implementation and testing.
3.2 CAD Model Preparation
Same as the production in 3D animation, each object perceived virtually by the user
is first mocked up with CAD models. The more complex the virtual interaction
required, the more efforts of both time and money developers need to pour into CAD
design. In this project, not all but those major components involved in forklift truck
operations are drawn. These include the body parts of the forklift truck, pallets, a
virtual warehouse, pallet racks, etc. They are drawn in scale with reference to the
factual dimensions released by the manufacturers listed on data sheets.
3.2.1 Forklift Truck Vehicle
There are various types and brands of forklift trucks, as illustrated in Figure 3.1,
which are sometimes called powered industrial trucks. The Industrial Truck
Association has identified them into eight classes, including “Electric Motor Narrow
Aisle Trucks” (Class II) that are specially designed to be used in warehouses with
narrow aisle and confined workspace and “Rough Terrain Forklift Trucks” (Class VII)
that are usually found in outdoor areas like construction sites and container terminals.
But in comparison, “Counterbalanced forklift trucks” (Class VI, V) are most versatile
and used widely nowadays. These trucks contain forks with adjustable distances, a tilt
mast, and most importantly a counterweight fixed at the back of the vehicles for
weight balance during operations. Among various models of counterbalanced forklift
3-1
trucks, Toyota Forklift 8-series 8FG(C) U is chosen as the prototype model to build
with for the VR simulators. Its specification was extracted from the official website
of Toyota. Figure 3.2 shows the major components of a forklift truck.
Figure 3.1 (left) Electric Moto Narrow Aisle Truck; (right) Rough Terrain
Forklift Truck
Control lever
Steering wheel
Overhead guard
Mast
Headlights
Lift Cylinder
Counterweight
Back rest
Adjustable
Forks
Driving wheels
Steering/ rear wheels
Figure 3.2 Major body parts of a counterbalanced forklift truck
SolidWorks is initially used as a development tool in building CAD models of the
forklift truck. It is developed by Dassault Systems SolidWorks Corp in 1993. It
adopts a feature-based approach to model construction. Simply by selecting the
desired feature and inputting the corresponding parameters, CAD models can be
3-2
easily built up or assembled. However, one should bear in mind that once a CAD
model is built and imported in the VR world, it is regarded as a single object or
identity. In other words, although its size can be changed, its shape is fixed and not
supposed to break down by any means. For examples, if the four driving wheels of
forklift truck are mimicked to rotate with respect to the truck movement, they should
be built separately as four identities instead of attaching to the body as a whole.
Likewise for simulation of fork motions on pallet operations, the models of mast,
back rest and lift cylinder are prepared respectively, as shown in Figure 3.3. It is
important to consider all movable parts needed for interactions in virtual environment
before creating the models.
Figure 3.3 CAD Models of Movable parts of forklift truck
3.2.2 Texture mapping
Before inputting into VR for interactions setup, all the CAD models are converted
into certain formats (*.3ds / *.asc / *.wrl) supported by Virtools Dev 4.0, which is the
software kernel of CAVE-based VR system for launching the simulators. However,
in times of conversions, unexpected results may appear. The CAD model of forklift
truck imported may either collapsed or become single-coloured. These matters are
commonly found in conversions between CAD objects as data stored for the model
like shininess, emissiveness, transparency and other material properties are likely lost
or modified during conversion due to format discrepancy. Therefore, instead of using
the format converter plugins, colourings in the project is made within 3ds Max of
3-3
Autodesk. All CAD models from SolidWorks are imported into 3ds Max for texture
mapping.
Texture mapping is a shading technique that has often been used in adding details,
texture or creating particular visual effect on a CAD model. It enhances the visual
experience of users like depth and lighting without increasing complexity of the
geometry or mesh of the model. Even a low-polygon CAD model like the ambulance
in Figure 3.4, which is made up of only 545 polygons, 603 vertices can give a
relatively high quality of visual effect. Nowadays, many games and VR applications
apply this technique in polishing the surfaces of models used in their virtual
environments.
Figure 3.4 A CAD Model of an ambulance truck with and without textures
To project a 2D photo or texture onto a 3D model, UV mapping is used. There are
several ways in UV mapping: planar, box, spherical and cylindrical. For planar
mapping, it works like a projector that projects the texture directly onto the 3D
surface from a single direction while the other three mappings project images
correspondingly on boxes, planets and pillars. For more complex models with
irregular shapes like the ambulance truck, basic UV mapping techniques mentioned
may not be compatible. Instead, UV maps are unwrapped and exported as an image
file before further editing using graphics editing programmes like Adobe Photoshop
or Illustrators.
The 2D image used in texture mapping is indeed a square image, most commonly be
(*.jpg) or (*.tga) format, with different resolutions. Texture data of various parts of
3-4
the model are packed closely inside the square in the best way that details of model
may be presented. In the project, beside the forklift truck body, all other CAD models
used in the simulators like pallets and cylinders are built and texture-mapped in this
way. Figure 3.5 shows the textures prepared in mapping the forklift truck body.
(a)
(b)
(c)
(d)
Figure 3.5 UV mapping (a) Forklift truck model in 3dsMax; (b) UVW map of
forklift truck; (c) Texture rendered for Forklift Truck; (d) Texture rendered for
Driving Wheels
3-5
Apart from the issue in texture, some of the CAD models built from SolidWorks were
later found too complex for use in the virtual environment. Instead of rebuilding, they
are optimized by “Nugraf” from Okino. It is a fast and powerful tool in processing a
CAD model by reducing its number of vertices or faces by nearly half without
sacrificing much in details.
3.3 Developments Phase
After preparing the CAD models, they are then imported into a VR platform for 3D
model rendering as well as coordinating the hardware infrastructures of VR before
the simulator can perform in action in CAVE-based VR systems.
3.3.1 Virtools Dev 4.0
Virtools, owned by 3DVIA, is an extensive real-time platform that facilitates
prototyping and robust development of highly interactive 3D experience through web,
game consoles, PC clusters or any other immersive virtual environments. It is also
commonly adopted as the software kernel of CAVE-based VR systems. By rendering
all 3D models and coordinating the required hardware infrastructures in an optimized
way, Virtools opens rooms for game developers or researchers to explore various VR
applications.
In addition, Virtools works as a behavioural engine (CK2) by processing various
behaviours assigned onto objects or entities. A behaviour is actually a description of
the ways certain elements act inside the given environment. In Virtools, there are
collections of ready-to-use behaviours in the form of Behaviour Building Blocks
(BBs) for users to create contents through the schematic interface. Each BB is a
packed C++ object containing I/O input, parameter I/O for certain function in 3D
Transformations, Collisions, Physics, Camera Sounds, etc. By combing the functions
BBs in a logical sequence, developers can create motion behaviours without the need
of mastering a new programing language. For advanced developers, they are built
their customized BB using the “Virtools Script Language (VSL)” or modify the
existing one.
3-6
Figure 3.6 shows the User Interface in Virtools Dev 4.0. Each imported CAD model
or 3D Object in Virtools is shown in the 3D layout, top left corner, with respect to its
3D coordinate. It can also be positioned, resized or rotated by changing the World
Matrix in 3D Object Setup. On the top right corner of Figure 3.6, there is a library of
pre-set Behaviours BBs to be called out and used in Schematics view. Also, an
attribute manager and a hierarchy manager are included for defining attributes and
hierarchy of 3D objects. The design details of the physics behaviours in the forklift
truck model will be discussed in the next chapter.
Figure 3.6 User Interface in Virtools Dev 4.0
3.4 Implementation and Testing
After all required VR interactions of forklift truck operations are set and debugged in
Virtools at desktop level, they can be imported into the ImseCAVE for final delivery.
3-7
Virtools not only render all 3D models or objects but also controls a set of four virtual
cameras that mimic the human user‟s view to generate stereo images for projection
on the four screens (front, floor, left and right) of the ImseCAVE, based on the
cameras‟ locations.
3.4.1 ImseCAVE
The ImseCAVE at the University of Hong Kong is a typical example of CAVE-based
VR systems. It is a fully immersive and interactive visualization system for users to
participate in a cube-like structure (10x10 feet) with walls for VR simulation. It
provides a low cost platform for design, analysis and evaluating the complex
engineering systems or operations. Figure 3.7 and Table 3.1 show the schematic
layout and the specification of the control computers of the ImseCAVE. It consists of
eight computers (PCs) with medium-to-high configuration, shown in Figure 8, in both
graphical and processing units linked together. One of them is the Master (PC 03)
while all others are Slaves. Eight corresponding projectors are installed in pairs to
project respective views on four screens: front, left, right and floor, separating the
L/R views of human eyeballs. Due to inter-pupillary distance, each eye captures a
slightly different perspective image. Virtools is the graphics kernel used for
generation of these images.
To view a stereoscopic 3D image with both eyes, we
should first place a linear polarizing filter in front of each projector in order to omit
either the vertical or the horizontal light rays. After that, we can spot the image with
depth by using a pair of 3D polarized glasses that merely allows single side of the
linear lights to pass through, resulting in a virtual 3D image effect.
Technical Specification of ImseCAVE (identical for all PC clusters)
AMD Athlon 64 3200+
CPU
1 GB DDR PC3200 RAM
Memory
NVIDIA GeForce 8800 GTX
Display
Realtek AC‟97
Sound
80.0 GB WD
Hard disk
Epson EMP-820@1024x768
Projectors
Table 3.1 Technical Specification of Computers using in ImseCAVE
3-8
(a)
Front
Screen
Right
Screen
Floor
Screen
Left
Screen
(b)
(c)
Figure 3.7 (a) Stereoscopic Images through 3D polarized glasses; (b) CAVE
system with four projection screens; (c) Computer Network of ImseCAVE
3.4.2 Infrared Tracking Tools
With the aid of Virtual-Reality Peripheral Network (VRPN) from VR Library of
Virtools, which is a class library defining the rules governing VR peripherals like
tracker, haptic devices are used in VR systems, we are able to transfer data, like
position and orientation, from Infrared tracking tools, assigning them to a virtual
object in Virtools on which all the virtual cameras attach. Therefore, whenever the
user is moving around in the ImseCAVE, his or her movements can be captured and
transferred into Virtools, which makes
corresponding actions on screen
spontaneously. Implementation and applications of Infrared Tracking technique in
both hand tracking and head tracking will be further explained and demonstrated in
chapter 5.
3-9
3.4.3 Motion Feedback Wheel
For most desktop-level simulators we have discussed in chapter 2, they are solely
controlled by the combination of keyboard and mouse. However, it is considered
infeasible to carry the combo inside the ImseCAVE system. Also it would be a poor
user experience of forklift truck operations. Using the mock-up of a real forklift truck
vehicle without a doubt will brings the best experience ever. Yet the cost will be too
high and it does not fit the multi-purpose of the ImseCAVE system. Therefore, a
mobile racing wheel is used instead. Though it is primarily designed for PC gaming,
its functions and quality are sufficient to meet the basic requirements of performing
forklift truck operations.
There are lots of brands or models of driving wheels. After reviewing several models,
a mid-range model – Logitech MOMO Racing Force Feedback Wheel, as shown in
Figure 3.8, is chosen for the simulation prototype. The racing wheel includes a
driving panel attached with a switchable gear shift lever and a flat no-slip foot pedal
base. Six programmable buttons are also installed within easy reach on the wheel.
These buttons are used as the secondary control of the proposed simulator, backing
up the hand tracking tools. To get people at different height be involved in the
simulator, a customized wheel stand with adjustable joints was used to integrate the
wheel panel and pedal as a whole. The setup can be folded up and carried easily with
one hand. Owing to its good quality and durability with such a compact size, it can be
used in most CAVE-based VR systems.
Figure 3.8 MOMO Racing Force Feedback Wheel as driving panel for forklift
3-10
3.4.4 VR Distribution Concept
As mentioned previously, one of the core values of the Virtools is its high flexibility
in launching the Virtools composition files (*.vmo / *.cmo) into complex VR systems
that run on PC clusters using stereoscopic images or other tracking devices.
According to the documentation of Virtools VR Library, there are mainly two ways
to distribute and synchronize a composition file among PC clusters in ImseCAVE.
While having one central computer as a distributor (master, Id=0), other hosts act as
slaves.
Cause Distribution
The first approach as illustrated in Figure 3.9 is called cause distribution: the master
will first identify the signals received from the input devices before pushing it to each
slave. As soon as the salves received the state of signal, they will send an
acknowledge reception respectively to the host. This shared state remains until it is
shared and hold among all, then they will execute their own frame, based on the same
shared causes. Time and random events are also aligned to ensure each of the cluster
hosts behaves in the same way, creating identical results.
(32.45, 46.79,-94.99)
F
Computation
(32.45, 46.79,-94.99)
Signal
Transmission
Master
Input Letter
Slaves
(32.45, 46.79,-94.99)
Figure 3.9 Cause Distribution
Effect Distribution
Another approach as illustrated in Figure 3.10 is effect distribution, instead of
distributing the state of a signal to all slaves for computation, the master will first
compute its own module, generate the result, and distribute it out afterwards. In doing
so, the result is not only identical among all but is also synchronized at the same time.
3-11
F
Computation
(32.45, 46.79,-94.99)
Master
Input Letter
Signal
Transmission
Slaves
Figure 3.10 Effect Distribution
Let‟s take an example of moving the forklift truck forward in the virtual environment.
Assume a letter “F” is set to trigger the motion. In the case of cause distribution, the
input signal “F” will be synchronized and distributed to each slave by the master. By
doing so, each slave has its own computation and moves forward the forklift truck
correspondingly. But for effect distribution, once the letter “F” command is received
by the master, it will compute the new 3D coordinates of the forklift truck before
synchronizing the position and orientation, or world matrix, with each of the slaves.
Theoretically, two approaches should deliver the same result. But in effect, there is
always latency between the master and the slaves. Even the latency are just a few
microseconds, the deviation is noticeable in the first case. Therefore, effect
distribution is always preferable than cause distribution.
However, effect distribution requires a higher workload from the VR network than
cause distribution. When the number of data sets using effect distribution exceeds
twenty, the system becomes unstable. In the proposed simulator, other than
positioning which requires both high precision and consistency, other functions like
switching the camera view or sounding the horn are distributed by the first approach
in order to get the simulation run smooth.
In the next chapter, the design paradigm and validation of the physics behaviours of
the forklift truck model will be presented.
3-12
CHAPTER 4 Physics Behaviours in Virtools
4.1
Introduction
This chapter reviews the functionalities of the built-in Physics Library empowered by
Havok of Virtools, and its efficiency in mocking the operation of a forklift truck. To
begin with, the ways objects physicalized in Virtools are demonstrated. After that, the
design paradigm of physics behaviours of the simulator is presented. It can broadly be
divided into three phases. In Phase I, only the forklift truck body itself is physicalized.
In Phase II, the research scope is extended to study the effect and difference between
various methods, including Sensor Detection System (SDS) and “Beam Physics
Object BB”, in mimicking the fork physics in pallet picking and release. Since they
are not able to mimic pallet operations in full, a new design paradigm using “Motion
controller BB” is introduced in Phase III. The feasibility of the proposed model is
discussed at the end of this chapter.
4.2 Getting objects physicalized
Apparently, imported CAD models looks visually alike to what we look at in the
reality. In fact they do not inherit any physics properties. In other words, they are
solely visually, but not physically, realistic. They do not have a mass; they cannot
collide with each other, and are therefore not governed by any Newton‟s laws of
motion. With reference to the documentation of Physics Library, it is necessary to
create a mesh, which is the physics surface of an object, to get the object physicalized.
In Virtools, a mesh can be represented in two ways: Convex Hull or Concave Hull. In
either way, it remains in effect once it is created, unless “rebuild surface” is called by
the user.
4.2.1 Convex Hull
A hull is a volume estimated by the built-in physics engine to the volume inhabited
by an imported CAD model. As shown in Figure 4.1, convex hull represents objects
4-1
without holes or hollows. For example, if a rubber band being stretched around an
object can touch its surface at all points, the object is convex.
Figure 4.1 Examples of Convex Hull
4.2.2 Concave Hull
On the contrary, in Figure 4.2, concave hull represents objects with holes or hollows.
Recalling the previous example of stretching a rubber band, it would not be able
touch the surface of the object at all points this time round.
Figure 4.2 Examples of Concave Hull
By comparison, convex objects always require less processing time than concave
objects. Therefore, convex hull is more preferred especially when lots of physicalized
objects are involved in interactions. In general, convex hull representation will be
sufficient to provide good approximation on the volume of the original mesh even on
a concave objects shown in Figure 4.3.
Red:
Original Shape
Black:
Convex hull
representation
Figure 4.3 Convex Hull on Concave Objects
4-2
To better illustrate how mesh representation affect the simulation results, Physics
Debug Rendering is used. Figure 4.4 shows the results between convex and concave
hull representation on a storage rack under Physics Debug Rendering. Objects
coloured in green are those physicalized in the virtual environment. In this project,
both topologies were used to represent the geometry of all physicalized objects.
Figure 4.4 Difference between Convex Hull (left) and Concave Hull (right) on a
storage rack
4.2.3 Physicalized Building Block (BB)
Within hundreds of Building Blocks (BB) provided by Virtools, there is a category
named Physics. There are five sub-categories under Physics: Creation, Destruction,
Modification, Control and Special. “Physicalized BB” which located under Creation
is the BB responsible for making an object part of the physics environment. When it
is added to an object, a mesh will be created automatically for the object using the
convex hull (default) representation. The physicalized object will then behave
realistically to interact with other physicalized objects under the governance of the
physics engine. As shown in Figure 4.5, there are
different categories in a
“Physicalized BB”, for instance the friction, elasticity, mass, collision groups, mass
centre, damping speeds for linear or rotation movement, etc. The values stored in
these categories can be modified easily to fulfil particular requirements at a real-time
basis during simulation.
4-3
According to the technical information of “Physicalized BB”, standard values for
physical properties are suggested for in each category. Let us take friction as an
example, the standard values 0 = frictionless, 0.1 = ice, 3 = very rough are suggested.
Since these values are not friction coefficient that ranges from 0 to 1, estimations are
made based on the given standard values. For example, we assume the friction of oil
or lubricant lies between 0.1 (ice) and 3 (rough surface). However, the sliding motion
between object A and object B will depend on the friction values of both. The optimal
value is often achieved by a number of trials, instead of calculations. Figure 4.5
shows the dialog box of a “Physicalized BB” for inputting parameters.
Figure 4.5 Dialog box of parameters in a “Physicalized BB”
4.3 Phase I – Building Forklift Truck Movement
As mentioned early in this chapter, there are three phases in building physics
behaviours of a forklift truck. In Phase I, the main objective is to physicalize the
forklift truck model, so that it can operate as a normal vehicle. As usual, mimicking a
vehicle often requires a lot of physics and equations. However, with the use “Physics
Car BB”, this can be achieved in a whole lot easier way.
4-4
4.3.1 Physics Car BB
“Physics Car BB” is another built-in BB found under the sub-category Special of
Physics. It is a tailor-made BB used in creating a car system. By simply assigning
names to the vehicle body and its four driving wheels (“FR”, “FL”, “BR”, “BL”)
accordingly to the predefined rules, Virtools will be able to recognize them and mock
up the car system accordingly.
However, there are a few criteria to satisfy:

The axes of both the car and wheels should be set equal to that shown in Figure
4.6.

The wheels should be put under the name of the car in the hierarchy manager

All the parts should be imported separately, instead of putting in a single
assembly file
Figure 4.6 (left) Reference axes of car system using “Physics Car BB”; (right)
Reversed Axes used for Forklift Truck Model
4-5
Four Data Arrays
Basic movement
control of a car
Figure 4.7 Standard I/O parameters of “Physics Car BB”
The “Physics car BB” is fine-tuned for typical front-wheel steering automobiles, but
the forklift trucks are rear-wheel steering, as described in section 1.1.1. Therefore,
the axes of both the forklift truck body and driving wheels need be modified as in
Figure 4.6. When all the models are prepared, the forklift truck body will then be
connected directly to the “Physics Car BB”.
The basic movement of the forklift truck vehicle is manipulated by “Forward” and
“Backward”, “Turn Left”, and “Turn Right”, as highlighted in the yellow region of
Figure 4.7. “Handbrake” and “Booster” are used to trigger brake or acceleration
instantaneously. Besides, there are four set of data arrays governing the physics
properties of the forklift truck to be mimicked: “Body”, “Wheel – Suspension”,
“Engine – Steering”, and “Skidding Output”. Under each set of array, there are quite
a number of parameters, listed in Figure 4.8, to be modified in order to fit any car
system. Factual data available such as body mass in the specification sheet collected
is input into each parameter accordingly, while the rest are estimated and modified
4-6
one at a time until the forklift truck operates naturally as it does in reality.
Nonetheless, it is always better to input as much factual data as possible.
Body
Wheel - Suspension
Engine - Steering
Skidding Output
Body Mass
Wheel Mass
Max Steering
Skid Value
Body Friction
Wheel Friction
Max Speed Steering
Skid Position
Body Elasticity
Wheel Elasticity
Steering Velocity
Skid Normal
Body Speed Damp
Wheel Speed Damp
Engine Power
Floor Friction
Rotation Damp
Wheel Rotation Damp
Min Engine Rpm
Body Rotation Inertia
Wheel Rotation Inertia
Max Engine Rpm
Shift Mass Center
Suspension Constant
Axle Torque Ratio
Counter Torque Factor Suspension Damp
Max Speed Steering
Extra Gravity
Suspension Compression Front Brake Deceleration
Down Force
Max Body Force
Rear Brake Deceleration
Down Force Offset
Stabilizer Constant
Gear Ratios (1/2/3/4/5)
Figure 4.8 Parameters in Data Arrays of “Physics Car BB” (Body, Wheel,
Engine, Skidding Output)
“Physics Car BB” can be controlled either by generic input devices like “keyboard +
mouse” combo or by a joystick controller. In the development phase, the simulator
was tested by a keyboard and a mouse. “Switch On Key BB” was applied to define
each corresponding key in trigger the functions inside the yellow region of Figure 4.7.
Later, on launching into ImseCAVE, a MOMO Racing Force Feedback Wheel will
be used as the primary manipulation devices. As illustrated in Figure 4.9, a
programme is added to track the orientation of the driving wheel continuously,
making corresponding changes on the virtual driving wheel in the virtual
environment.
Get real-time orientation
from MOMO driving wheel
Virtual Wheel
Alignment
Figure 4.9 A programme for driving wheel alignment
4-7
4.4 Phase II – Mimicking the Pallet Operations
Other than mimicking the vehicle system of the forklift truck, a great challenge in
building physics behaviours would be the pallet picking operation. It takes up huge
efforts in looking for the best approach to get it done realistically in Virtools. Due to
the physics constraint of the built-in Physics Library of Virtools, once an object is
physicalized, there is no way for it to be positioned or orientated by standard 3D
Transformation BBs like “Translate BB”, “Move to BB” or “Set Position BB”.
Instead, Virtools suggests developers to use either “Unphysicalize BB” or “Beam
Physics Object BB”.
In the case of “Unphysicalize BB”, physicalized objects are temporarily removed
from the physical environment by calling “Unphysicalize BB”. After standard 3D
transformation, they will then be physicalized back to their original state. The
advantage of using such a method is that the simulation result is precise and
predictable. However, in exchange, objects which are initially physicalized will no
longer be governed by any physics law for at least as a short duration of time. The
other way of moving physicalized objects is making use of “Beam Physics Object
BB” which pauses the physics engine processing while moving. The design paradigm
and simulation results of both solutions will be discussed explicitly in the following
sections.
4.4.1 First Approach – “Unphysicalize BB”
Because physicalized objects cannot move, it is necessary to unphysicalize a pallet
before it can be lifted up by the forks in the virtual environment, and only until it has
reached the destination should it be allowed to physicalize again. In doing so, we will
have to consider what would be the right moment to unphysicalize or physicalize a
pallet. This makes the detection of both picking and release of pallets a huge
challenge.
To deal with it, an initial solution is to make use of three additional blocks, which are
not physicalized. These blocks are transparent and penetrable to help detect whether
4-8
the pair of forks have inserted fully into a pallet. As illustrated in Figure 4.10, two of
them were placed at the entrances of one side while the other long block was put
inside the pallet at roughly 80% of the length of the inserted forks. Whenever the
forks of the lift truck are inserted, they will be detected by the blocks and only when
all of them collide with the forks will the pallets be able to lift up.
Additional Blocks
Figure 4.10 Three additional blocks used in detecting the forks motion
Indeed, “Collision Detection” BB is assigned to all the three blocks for testing
whether any object collides with them. If only the blocks at entrance collide, it is
considered that the pair of forks has not yet inserted fully, and thus the pallet remains
intact. Only until collision of the forks with the long block is detected will the pallets
be unphysicalized by the “Unphysicalize BB” and attached under the hierarchy of the
forks. Wherever the forks are moved, the pallet will follow as a whole, keeping the
same orientation and speed.
In theory, this method sounds good. However, it will inevitably increase the
complexity of the scripts, especially when 12 different blocks (3 blocks X 4sides) are
required for a single pallet.
4.4.2 Sensor Detection System (SDS)
Based on the proposed idea in section 4.4.1, a simplified model called Sensor
Detection System (SDS), as illustrated in Figure 4.11, was developed. Instead of
building virtual blocks for collision detection, four sensors are fixed on the forks of
the forklift trucks.
4-9
Figure 4.11 Senor Detection System (SDS)
These sensors are called 3d frames in Virtools. A 3d frame is a dummy object which
is transparent and not occupying any volume. It is often used as a tracker or a point of
reference to other 3D objects in the virtual environment. As we can see in Figure 4.12
(left), the four sensors used in SDS work in pairs diagonally forming a virtual region
for detection. A programme is written to detect the both X and Z (horizontal)
coordinates of the nearest pallets continuously unit it lays within the detection region,
the horizontal position test is passed. During trials, the sensors have once been set to
work in linear instead of diagonally. The result was however not satisfactory.
Z
S1
P
S2
S1
P
S2
P: pallet
S: sensor
X
Figure 4.12 (left) Sensor detection region in SDS; (right) Plane view on
horizontal position testing
4-10
Once the horizontal position test is passed, the vertical position of the nearest pallet
will be tested whether it is located within the pre-defined allowance. If it does, the
pallet will then be unphysicalized and attached under the hierarchy of the forks as a
whole to move along during the process of pallet picking, as illustrated in Figure 4.13.
Figure 4.13 Pallet attached to the forks after passing both horizontal and
vertical test
Different from pallet picking, there is no clue whether the pallet is ready to drop off
because in reality the forklift truck operators can drop it off anywhere, anytime by
any means. Therefore, in the SDS built, releasing a pallet requires the user to control
manually. By typing a certain key on the keyboard, the pallet will be detached from
the forks and re-physicalized a few seconds later automatically.
4.4.3 Second Approach – “Beam Physics Object BB”
“Beam Physics BB” is the other solution suggested by Virtools to moving a
physicalized object to a new position or orientation without processing the physics
behaviours in between. It is often called after a 3D transformation, such as “Set
Position BB” or “Translate BB”. To validate the functionality of “Beam Physics
Object BB”, a simple programme is made, as shown in Figure 4.14, to lift up a
physicalized block with a single fork. The result is favourable at start but becomes
unstable once the fork collides with the block. As illustrated in Figure 4.15, the fork
starts looping unexpectedly with the block after collision and the entire physics
4-11
system crashed afterwards. Even after fine tuning the speed, physical properties of
both physicalized objects, the result is not stable enough to be used in the proposed
simulation model.
Figure 4.14 (left) A physicalized block lifted by a fork; (right) A programme for
“Beam Physics Object BB”
Figure 4.15 A testing fork loops with the physicalized block.
Both approaches mentioned above have been put under testing. For the case of
“Unphysicalize BB”, three blocks were first used for forks insertion detection in trial
before a more complex SDS system is introduced. Although the pallet picking
operations have been successfully mimicked through horizontal and vertical testing,
release of a pallet remains unsolved. Moreover, since the approach of unphysicalizing
objects will bring certain objects out of the physics environment at a time. This will
inevitably affect the simulation result by hindering some common accidents like tip
over of a forklift truck from happening. For the second approach - “Beam Physics
4-12
Object BB”, it supports simple physicalized movement, like translation or position
and performs well on a single object. However it is not preferred in a complex system
which requires much detection of collisions with other physicalized objects.
Besides, there are indeed two more BBs in Virtools supporting the movement of
physicalized objects: “Physics Impulse BB” and “Physics Torque BB”, as illustrated
in Figure 4.16. “Physics Impulse BB” provides a linear impulse, while “Physics
Torque BB” provides a rotation impulse. They are excellent for simple simulation
applications, such as snowboarding which requires low precision of the resulting
positioning of the object. However, in mocking the forks movement during pallet
picking and release operations, relative high precision and stable physics behaviour
are needed. Therefore, they are considered not for use in the project. In the next
section, we will introduce “Motion Controller BB” and demonstrates the way it is
incorporated with the forklift truck model built in Phase I.
Figure 4.16 (left) Standard I/O parameters of “Physics Torque BB” and
“Physics Impulse BB”; (right) Snowboard simulation using “Physics Torque
BB”
4.5 Phase III - Integration of “Motion Controller BB”
Compared with the previous approaches, “Motion Controller BB” provides a greater
degree of control over physicalized objects. Different from “Beam Physics Objects
BB”, no objects will be unphysicalized in this case, keeping the objects perfectly
coherent with other physics simulations. “Motion Controller BB” manipulates a
4-13
physicalized object by forcing it to follow the movement of another 3D object, such
as a 3d frame. However, the position of physicalized object may not be always equal
to that of the object to be followed. For example, if there is a wall or other blockages
in the path of motion, the physicalized object will stop even when the object to be
followed are positioned elsewhere.
Figure 4.17 Dialog box of parameters in a “Motion Controller BB”
Figure 4.17 shows the dialog box of “Motion Controller BB”. “Entity to follow”
refers to the object to be followed. In the proposed simulation model, a 3d frame
named “plane” is created as a reference. In addition, there are several parameters for
monitoring the motion of the physicalized object. The greater the maximum
translation or rotation force, the greater the impulse will be generated. There are also
other factors like “Force Factor” and “Damp Factor” to control the position and speed
of the object in motion.
To apply “Motion Controller BB”, a 3d frame, which is a dummy object, is first built
as a reference at same position as the forks, as illustrated in Figure 4.18. It is set
under the hierarchy of the forklift truck system so as to remains static with respect
position to the forklift truck during movement. On the other hand, a programme is
prepared to manipulate the movement of the 3d frame. For an authentic forklift truck,
the lifting forks will move along with the mast and the lift cylinder, while lifting
4-14
provided with a certain degree of back and forth tilt. Therefore, the sub-programme
in area A of Figure 4.19 was made with “Translate BB” and “Rotate BB” for
mocking vertical and rotation motion respectively. Whenever the 3d frame is moved
or rotated, the physicalized mast and the lift cylinder will follow its motion as if they
are directly manipulated by the user.
Figure 4.18 (left) 3d frame – plane; (right) Lifting and rotation of forks
B
A
Figure 4.19 A programme that controls the forks operations using “Motion
Controller BB”
4-15
Figure 4.20 Lift cylinder with physicalized (left) or without physicalized (right)
To constrain the forks movements, allowances must be set. Therefore, a subprogramme illustrated in area B of Figure 4.19 is made to ensure both the height and
the angle of the lifting peripherals are within the range of limits given by the forklift
truck‟s manufacturer.
Ideally, the entire forklift truck body, including the lift cylinder and the lift forks,
should be physicalized as illustrated in Figure 4.18 (left). However, on testing and
debugging, physicalized objects perform badly when they are closely positioned at 1
to 2cm apart. Trembling and other misbehaviours, such as slipping and bouncing
happen. To resolve it, the lift cylinder is set unphysicalized instead, as illustrated in
Figure 4.20. Because of the geometry and position, the lift cylinder is not involved in
most physics simulation. Nevertheless, it does not affect the physics simulation of
normal forklift truck operations very much.
To validate the feasibility of the proposed model using “Motion Controller BB”, two
routine tasks of forklift truck operations are tested. In the first test, the user should
operate the virtual forklift to position A of Figure 4.21 and pick one pallet load of
four 200ml cylinders. Then, the user is instructed to unload the pallet at a certain
partition of the storage rack at position B. To assist the user in making judgements
during forks insertion, an extra camera is added at the front of the forklift truck. This
camera can be called out anytime. Although it does not exist in reality, it can help
user to evaluate their ability in space perception. For the second test, the user is
4-16
required to pick the pallet from the storage rack at position B and load it into the
container truck at position C. Through these tests, physics behaviours involved in
basic forklift truck operations, such as picking or dropping pallets, driving up a ramp,
making turns at narrow aisle and backing, can be simulated. Some screen captures of
the simulation are demonstrated in Figure 4.22 and Figure 4.23.
B
A
C
Figure 4.21Overview of a virtual warehouse in Virtools
a
b
Figure 4.22 Screens captured from simulation (a) Addition view for pallet
picking; (b) driving up a ramp;
4-17
d
c
Figure 4.23 (c) dropping off at storage rack; (d) loading into container
Physics
Building Blocks
Behaviours
Physics Car BB
Forklift
Truck
Manipulation
Forks
Motion in
Pallet
Operations
Unphysicalize BB
(SDS)
Beam Physics
Object BB
Physics Impulse
BB/ Physics
Torque BB
Motion Controller
BB
Strengths (S) & Weaknesses (W)
S: easy to use in mimicking car systems without
involving complex calculations.
W: modification needed for forklift truck
vehicle which is back-wheels steered.
S: supports movement that requires precise
position and timing.
W: isolate objects from physics environment
hindering accidents from happening.
S: favourable for single object.
W: trembles at collisions.
S: provides impulse for simple VR interactions.
W: uncontrollable results.
S: relatively stable and precise in movement
W: much complicated than other BBs.
Table 4.1 Strengths and weaknesses of tested building blocks
Table 4.1 summarizes the strengths and weaknesses of each Building Block described
above for mimicking the physics behaviours of forklift truck operations. For forklift
truck manipulation, Physics Car BB is used. By modifying physics parameters of data
array each at a time; a virtual forklift truck system was successfully mimicked. It
reacts properly in some accident scenarios like overturn. An in-depth analysis and
discussion of forklift truck performance verse accident-like scenarios will be covered
in chapter 6.
4-18
Moreover, several BBs are used in imitating the forks motion on pallet operations,
“unphysicalize BB” performs better in object positioning while “Beam Physics
Object BB” can move physicalized objects without affecting its physical state.
However, only “Motion Controller BB” is capable of both. Two tests are conducted
to validate the result.
In the next chapter, the design, characteristics and applications of infrared marker
system in building VR interfaces on both hand tracking and head tracking will be
presented.
4-19
CHAPTER 5 Integration of IR Tracking System
5.1
Introduction
This chapter describes the design and application of an intuitive VR interface
developed in the proposed simulator using infra-red (IR) tracking technique. As
mentioned in chapter 2, most of the generic input devices like keyboards and mice
combo are considered unfeasible in CAVE-based VR applications. Therefore, a
marker system is chosen in the project to create a natural and intuitive interface for
users to interact with the virtual environment. By capturing the 3D positions of
passive markers with IR cameras, the position and orientation of a user can be
correctly tracked. This IR tracking technique is applied on head tracking for
additional driving view to enhance the degree of realism in user experience. Likewise,
the proposed design is modified to support simple gesture recognition in hand
tracking. The user‟s hand gesture can be detected and identified by the IR tracking
system to execute the corresponding commands. A 3D rotatable option menu is built
in the project incorporating with hand tracking for functions calling during driving
simulation.
Hardware configuration and design infrastructure of such tracking
system are discussed in the chapter.
5.2 The IR Tracking System in ImseCAVE
The IR Tracking System adopted in the project is jointly developed for another MPhil
project in the Department (Yang, 2010; Yang et al., 2010). It is based on the
hardware configuration of Motion Capture System from OptiTrack. A total of six IR
cameras are mounted at the top edges (8 feet) of ImseCAVE. They are linked up with
a master hub connected to dedicated computers for processing in high speed USB
cable.
Each camera supports variable frame rates from 25 fps to 100fps at a
resolution of 640 x480 (VGA), attaining a high accuracy to sub-millimetre with less
than 10ms of latency. The positions and orientations of the IR cameras are optimized
to track over 70% of the cubic space in ImseCAVE. Figure 5.1(c) and (d) show the
5-1
cameras arrangements and graphic presentation of the tracking coverage in
ImseCAVE.
(a)
(b)
IR cameras
IR cameras
Tracking Coverage
(c)
(d)
Figure 5.1 (a) IR camera of OptiTrack; (b) Network layout of IR cameras; (c)
Setup of IR cameras in ImseCAVE; (d) Camera arrangement and
TrackCoverage in ImseCAVE
TrackingTools is a software package provided by OptiTrack to track and render the
position of the target object in the virtual space. In the project, the rendered
information of a user‟s position is transmitted to Virtools via local area Ethernet
network under the Virtual Reality Peripheral Network (VRPN) protocol. In both
TrackingTools and Virtools, a virtual cubic space is mocked up in the exact scale of
the physical one in ImseCAVE. By synchronizing their spatial arrangements, the
system will then be able to track the position and movement of the user and render its
avatar in the virtual space accordingly for 3D interactions.
5-2
5.2.1 Principles of IR Tracking
Twenty six high efficient illuminated LEDs are built in each IR camera emitting IR
ray, which is not sensible to human eyes but IR reflective coating. Under the IR lights,
the IR marker surfaced with IR reflective coating, illustrated in Figure 5.2, appears to
be shiny and distinguishable to the IR cameras. Each IR camera will be able to track
only the 2D position of the marker. These data from six cameras will then be gathered
in TrackingTools to calculate the corresponding 3D coordinates of the marker. The
IR tracking system used in the project can track IR markers of size of 1cm in
diameter or larger. In addition, the inter-distance between IR markers should not be
less than 3 cm.
(a)
IR Ray from LEDs
Reflection Lights
(b)
(c)
Figure 5.2 (a) IR markers; (b) A IR marker is being tracked; (c) Image captured
by an IR camera
5.3 Head Tracking
Based on the principle stated above, the 3D position of an object in the physical space
can be recognized by tracking its IR marker. However, to track both position and
5-3
orientation of an object, “rigid-body” tracking is required. By fixing three or more IR
markers at a certain distance in-between, a virtual “rigid-body” is formed. In Figure
5.3 (left), a triangular “rigid-body” coloured in red is bounded by three IR markers,
which are attached on one‟s hand.
A “rigid-body” serves as an identification device (ID) for an object to be tracked.
Instead of tracking the shape of an object, the IR tracking system tracks the pattern of
the “rigid body” of IR markers to distinguish an object from another. Different
objects may be represented by the unique pattern of a “rigid-body”. For examples, a
square pattern is for hand position, and a triangle pattern for head position. These
patterns should be first defined in TrackingTools as virtual trackers before the
tracking system is able to recognize them correctly. Also, the IR markers used in
forming the “rigid-body” should always remain in position relative to each other, or
else the patterns will vary and no longer be recognizable.
Rigid-body
Reality
TrackingTools
Figure 5.3 (left) “Rigid-body” in reality and TrackingTools; (right) Positions of
IR markers in head tracking
As shown in Figure 5.3 (right), three IR markers are fixed on top of a pair of
polarized glasses forming an orange “rigid-body”. This “rigid-body” is defined in
TackingTools as a virtual tracker to track the position and orientation of the user‟s
head. Generally, head tracking may not be required in driving simulators. However,
the experience of operating a forklift truck differs from other vehicles, especially in
times of pallet operations. Forklift truck operators are often required to tilt his or her
head slightly to gain a better viewing angle to get the task done smoothly. Meanwhile,
all forklift trucks simulators being reviewed in chapter 2 have not come across in this
5-4
matter. Instead, the driving views of most driving simulators are static and have no
interactions with the head movement of operators. Undoubtedly, this will affect the
decision-making of forklift truck operators. Because if the pallet loads to be handled
can only be viewed from a fixed angle, it would be hard for an operator without
having good spatial perception to insert the lifting forks or load the pallets onto the
storage rack precisely. Therefore, in the proposed model, hand tracking is integrated
to track user‟s head position and orientation in order to render images of
corresponding viewing angle on screen simultaneously.
Head tracking incorporated in CAVE-based VR systems was originally used in
rendering better stereoscopic images with respect to the user location in the cubic
space of 3m x 3m x 2.5m. To generate the corresponding images on all projection
walls in ImseCAVE (Front, Floor, Left, Right), a set of four imaginary cameras are
used. In practice, they will be positioned at 3D coordinates (0, 0, 0) which is the
cubic centre of CAVE-based VR system. Figure 5.4 shows the top view of
ImseCAVE, even the user is found by head tracking at the left side of the system, the
position of the camera set will always remain at centre. Instead, the position of
cameras is often manipulated by a generic input device, such as a joystick.
Front Walls
Side Walls
User
Figure 5.4 Cameras Positions in CAVE-based VR system: (left) Static; (right)
Mobile
In contrast, the camera set in the proposed model is mobile and subjected to the
tracked position of users. Referring to the previous example, if the user‟s head
position is tracked at left side of the CAVE-based VR system, the camera will rotate
5-5
slightly and keeps content displays in focus. Apart from providing additional side
views of content displays ahead at the front wall, the design enables the camera to
follow the movement of user in ImseCAVE. Therefore, if the user approaches the
front wall, objects projected on screen will grow larger, and vice versa.
In the proposed model, the user‟s position in ImseCAVE is recognized by tracking
the “rigid-body” formed by the markers attached to a pair of polarized glasses that the
user wears. By synchronizing the position and orientation between the user and the
camera set, the resulting images projected on screen will be aligned with the head
movements of the user. Simulated views of a driver operating a forklift truck are
demonstrated in Figure 5.5.
Side view
Side view
User Position
Figure 5.5 Simulated views of user operating a forklift truck
5.4 Hand Tracking
As mentioned, a “rigid-body” should remain in shape in order to be identified and
tracked continuously.
For hand tracking, an extra IR marker was added to the
existing triangular pattern for tracking both position and orientation. The fourth IR
5-6
marker associates with two of the existing IR markers to form a new “rigid-body” for
gesture tracking.
Figure 5.6 shows the shapes of the new “rigid-bodies”. In addition to the existing
“rigid-body” which is in yellow, there is a new “rigid-body” formed in green. Since
the fourth IR marker is attached on the finder tip of the middle finger, the shape of the
“rigid-body” will alter as the hand gesture changes from grab (fist) to drop (release).
Therefore, two different “rigid-bodies”, green for grab and red for release, are formed
to recognize the hand gesture of a user in order to triggers interactive events in virtual
environment. Different from head tracking where IR markers are fixed at the top of
polarized glasses, the position of the fourth marker in hand tracking varies for all
users due to the size difference in hand. Therefore, personal calibration in
TrackingTools is required for the first time of each new user.
Figure 5.6 Spatial arrangement of gestures; (left) “grab”; (right) “drop”
5.4.1 3D Rotatable Option Menu
It has been stated in the previous discussion that generic input devices like keyboards
and mice are considered infeasible in the environment of CAVE-based VR system.
Therefore, a VR interface system based on the hand tracking technique is developed
to select various options from the design menu. Four IR markers are attached on the
glove to be worn by user. By making hand movement combined with simple gesture,
user can browse the options available on the menu and select the preferred one
intuitively. A 3D rotatable option menu was built to demonstrate how hand tracking
is applied in menu command control.
5-7
Figure 5.7 A 3D rotatable option menu
As illustrated in Figure 5.7, a 3D rotatable option menu was built using 3ds Max and
imported into Virtools afterwards. A total of 12 functions, 3 rows x 4 columns, are
available in the menu. The user can swap driving scenarios, vary the weight of pallet
loads, change the camera views or reload the forklift truck if necessary anytime
during simulation. By default, the option menu is minimized and set to be invisible.
To active it, user simply reach out his or her hand to the up right corner, make a
“grab” gesture and drag it to the front. The menu will then automatically emerges and
slides to the centre of the screen. Once the menu is activated, a column with three
option buttons of same colour will be highlighted and brought to the front of user,
illustrated in Figure 5.8. According to the user‟s eye level, which can tracked by head
tracking, the space between the user and the column in focus can be divided into two
zones: “Focus Zone” of “Area #1”, “Area #2”, “Area #3” and “Non-focus Zone”
which is the area out the “Focus Zone”. Figure 5.9 reveals the main command
controls of the option menu including navigation, selection and operation.
5-8
Figure 5.8 The concept of “Focus Zone”
Figure 5.9 Command controls of the option menu
5-9
Menu Navigation:
Based on the “Focus Zone” concept, a 3D rotatable option menu interaction is
developed mimicking the focusing behaviour of human‟s eye by tracking the hand
position. Whenever the user‟s hand reaches a menu area, “Area #”, intuitively, a sport
light will be triggered to light up the corresponding menu button, while other options
buttons will be dimmed.
Menu Selection:
To make a selection, all user requires is simply making a simple hand gesture of
“grab”. The IR tracking system will then recognize the change in the pattern of
“rigid-body” and pop out the selected option button right away. A yellow frame box
is built to surround the selected option button as a selection indicator. In reverse, user
can cancel the selection and return back to menu navigation by making a hand gesture
of “drop”.
Menu Operation:
Once an option button is selected, user can execute the function by lowering his or
her hand to reach a virtual Activation Zone. Also, user is able to withdraw his or her
selection any time by making a “drop” hand gesture before the function is being
executed.
Figure 5.10 Hand tracking for menu command control
In summary, this chapter presents the design and application of IR tracking system in
the proposed simulator. IR tracking was used in both head tracking and hand tracking.
5-10
For head tracking, user‟s head position and orientation are continuously tracked to
render images from corresponding viewing angle. This design does not solely
enhance the visual perception, but also bring forth the most intuitive driving
experience of a forklift truck vehicle ever in CAVE-based VR system. In addition,
hand tracking is incorporated with a 3D rotatable option menu to provide an intuitive
and effortless way to call functions for simulation by a simple hand gesture, as
illustrated in Figure 5.10.
In the next chapter, the proposed forklift truck simulator model is evaluated in several
case scenarios of accidents. Performance analysis and limitations are discussed.
5-11
CHAPTER 6 Case Analysis of Forklift Truck
Accidents
6.1
Introduction
In chapter 4, the design paradigm of physics behaviours of forklift truck in the
proposed simulator using the built-in Physics Library of Virtools was demonstrated.
However, accidents scenarios have not been covered so far. This chapter further
investigates several case scenarios of common forklift truck accidents mocked up in
the simulator. Also, an in-depth analysis has been performed on overturn accident.
The virtual forklift truck is programmed to turn an angle at various speeds. By
tracking the turning path of the forklift truck, the correlation between velocities of a
forklift truck against its turning radii is established. Lastly, the overall performance of
the proposed simulator is reviewed.
6.2 Common accidents scenarios
There are numerous causes of forklift truck accidents such as slipping off, crushing
pedestrians, tipping over, driver ejection, off-the-dock, etc. In the proposed project,
there are several accident-like scenarios built to allow user to percept and manoeuver
the warehouse operations of forklift truck. These include skidding or overturn,
slipping of pallets loads, tipping over.
6.2.1 Skidding or overturn
Skidding, or overturn, is considered to be the leading cause in forklift truck accidents
by OSHA, representing about 25% of the forklift-related deaths. It occurs in many
possible ways for instance travelling on steep slopes, operating over slippery ground
like oil spills, crossing soft or uneven surface, being unevenly loaded or overloaded
and making sharp turns at speed. Under the concept of Stability Triangle discussed in
section 1.1.2, the forklift truck begins to skid or overturn when the centre of gravity
6-1
(COG) of the forklift truck body shifts beyond the triangular safety zone. Figure 6.1
shows the result of overturn in the proposed simulator.
Figure 6.1 Forklift Overturn between (left) reality and (right) simulation
To allow users to monitor the stability of the forklift truck during simulation, a colour
scale indicator with floating pointer has been created and attached to the left side of
driving view showing the real-time rotation speed of the forklift truck vehicle. As
illustrated in Figure 6.2, the pointer rises from green to yellow as the speed gets
higher. When the pointer reaches the red zone, the forklift truck will possibly get
overturned. With this colour indicator, the user will then be able to review and rectify
their manoeuvring skills during simulations. Overturn scenario like oil spillage is
mimicked by varying the value of ground friction of the affected area.
Figure 6.2 Colour Indicator of overturn during simulation
6-2
6.2.2 Slipping or falling of pallet loads
A pallet load usually falls off either when its weight is unevenly distributed or the
forklift truck is experiencing a sudden break or sharp turns. For safety reasons, the
forklift truck operators are always advised to drive slowly and stay the lifting forks
low, a few inches from the ground, when loaded. Favourable result was simulated in
this scenario.
6.2.3 Tipping over
There are two ways for a forklift truck to tip over, either forward or sideward. For
tipping forward, the longitudinal stability is considered. It is calculated by measuring
the moment on the forklift truck and that of the pallet load. When the load moment is
greater than that forklift truck moment, the forklift truck will tip forward. Other than
longitudinal stability, lateral stability is also used in determining the stability of a
forklift truck.
Figure 6.3 OSHA's Examples of Tipping Over Potential
Note the illustration provided by OSHA in Figure 6.3, the tilted forklift truck remains
stable as long as the vertical stability line of the combined centre of gravity of the
forklift truck and the loads falls within the stability triangle. However, the forklift
6-3
truck becomes unstable or tip over if the vertical stability line falls out of the stability
triangle. The higher or heavier the loads are elevated, the higher chance the forklift
truck gets tipped.
To simulate such a scenario in the proposed simulation model, a programme is
created to manually increase the weight of the pallet load. Ideally, the forklift truck
will tip over once the moment of pallet load exceeds that of the forklift truck body.
However, it can be noted in Figure 6.4 that only the pallet mast has tipped forward
but the forklift truck remains stable on the ground regardless. It is because the built-in
Physics Engine of Virtools has considered the lifting mast and the forklift truck body
as two different physics identity in physics rendering. As a result, their physics
behaviours are independent of each other creating huge obstacle in mimicking forklift
truck accidents that involves the combined centre of gravity of forklift truck body and
pallet loads such as tipping over. In an attempt to solve the issue, physics joints and
constraints have been added to integrate the lifting mast into part of the forklift truck
system. Yet, it did not work either. It is believed that the bottleneck lies on the
constraint of the “Physics Car BB” used in mocking the virtual forklift truck. Once
the “Physics Car BB” is activated, the physics properties including the centre of
gravity of the forklift truck are not subjected to change. Since modification of
“Physics Car BB” is not allowed in Virtools, no feasible solution has come up so far
in the project. Yet, alternatives are suggested later in the chapter summary.
Figure 6.4 Failure in simulating the tipping over of forklift truck
6-4
6.3 Analysis of forklift truck overturns
To validate the results simulated by the proposed simulator, an in-depth analysis of
forklift truck overturns has been carried out to demonstrate how data can be extracted
from the simulator and interpreted for the sake of forklift truck safety. As mentioned
in 6.2, there are many possible ways causing a forklift truck overturn. Among all, the
most common one is making turns at speed.
When a forklift truck turns at level surfaces, a centripetal force ( Fc 
mv 2
), or
r
named inertia is required to keep the forklift truck moving in a circular path. Ideally,
this force can be provided by the lateral frictional force ( Ff ) between the inner and
outer pair of tyres of forklift truck. When the lateral friction force is greater than the
required centripetal force ( Ff  Fc ), the forklift truck will remains on track.
However, as the velocity of the forklift truck increases, the required centripetal force
will increase until the lateral friction force no longer be enough to provide the
required force ( Ff  Fc ), the forklift truck will skids, or overturns.
R1
R2
mg
F1
F2
Figure 6.5 Forklift truck turning on level surface
As shown in Figure 6.5, the weight of forklift truck „mg‟ is turning on a level surface.
R1 and R2 are the forces of normal reaction of the ground on the tyres. They are
acting vertically upward as the surface is levelled. Hence,
6-5
R1  R2  mg
Let F1 and F2 be the lateral friction forces between tyres and ground towards the
centre. Then, we have
F1  R1
and
F2  R2
where μis the coefficient of friction between tyres and ground.
Assume „v‟ is the velocity of the forklift truck and „r‟ is the turning radius, the
centripetal force required will be (mv2/r). If the friction force is sufficient to provide
the centripetal force, therefore ( Fc  Ff )
mv2
 ( F1  F2 )
r
mv2
  ( R1  R2 )
r
mv2
  (mg )
r
v 2  gr
v  gr
By simple calculations, we get the maximum velocity which the forklift truck can
safely turn at a level surface without skidding is
gr m/s. If the coefficient of
friction remains constant, we can conclude that the higher the turning velocity of the
forklift truck, the greater the turning radius will be.
A programme was prepared in Virtools to visualize the correlation between velocities
of forklift truck against its turning radii on the proposed simulator. The forklift truck
is programmed to accelerate to certain speed before making turns. In Figure 6.6,
“Zone A” is provided for the acceleration of forklift truck. Its velocity is continuously
monitored by the system to make sure it has met the target maximum speed before
reaching “Zone B”. Once the forklift truck enters “Zone B”, the rear wheels will be
6-6
set to turn left. Meanwhile, the positions of forklift truck will be tracked at time
intervals of milliseconds. Eventually, the data collected will be output to a text file.
Zone B
Zone A
Figure 6.6 Top view of “Zone A” and “Zone B”
(a)
(c)
(b)
(e)
(d)
(f)
Figure 6.7 A programme for testing turning radius
Figure 6.7 shows the main procedures in the simulation programme:
(a) Test continuously the position of the forklift truck in Virtools until it has reached
the trigger point of X-Z coordinate of (0, 0).
(b) “Chrono BB” is used together with “Counter BB” in order to keep track on the
time elapsed in the simulation.
6-7
(c) Five “Set Cell BB” are used to assign corresponding parameters into the data
array. (Time elapsed, Velocity, z-Position, x-coordinate, states of forklift truck)
(d) A “Delayer BB” is used to define the time interval between each position tracked,
200ms is chosen for velocities at 12m/s or below. For higher speed, 15m/s and
20m/s, 100ms is used instead for more accuracy result. The choice of time
interval here is independent of the simulation results.
(e) All the parameters recorded in the data array of Virtools is output to a text file
which will later be imported into Microsoft Excel for data analysis, as illustrated
in Figure 6.8.
(f) A “Key event BB” is used to monitor the states of the forklift truck. Once the
forklift truck overturns, the state will change from FALSE to TRUE to indicate an
accident at that particular time interval.
Figure 6.8 Output text file generated by the testing programme
As we can see from the text file generated in Figure 6.8, there are five sets of data
collected from Virtools each time for a particular velocity. They include time elapsed
in zone B, velocity of the forklift truck, z-coordinate of forklift truck, x-coordinate of
forklift truck and a Boolean variable stating the forklift truck is tipped or not. A total
of 50 rows of data will be collected for each run. Ten velocity samples have been
chosen from 1m/s to 20m/s as the target maximum velocity allowed in zone A.
Finally, these data are imported into Microsoft Excel for graphical analysis.
6-8
20 m/s
Turnover
2-4m/s
Figure 6.9 Turning tracks of forklift truck at various velocities
Ten turning tracks of forklift trucks are plotted with respect to their velocities in
Figure 6.9. Broadly speaking, the velocity of forklift truck is directly proportional to
its turning radius. The faster the forklift truck in making turns, the longer the circular
path will the forklift truck needed to travel. At 2-4 m/s, the turning path is steady
because the lateral friction force is sufficient to provide the centripetal force required.
However, as the turning velocity increases gradually from 7m/s to 15m/s, the forklift
truck begins to skid. At 20m/s, the forklift truck experiences an overturn at the red
dot indicated in the figure. According to OSHA, the speed of forklift truck should
never exceed 5 mph (~2.24 m/s) during operations. Indeed, this limit can be validated
by the simulation results obtained.
Through the test, not only can we estimate the safety speed limit of forklift truck in
making turns, but also the driving paths of the forklift truck on simulation can be
tracked. As such, the driving misbehaviours of user can be evaluated and rectified. By
and large, the overall objective of the proposed simulator prototype is achieved.
6-9
Similar studies can also be conducted on other physics behaviours such as the
correlation between weight variations in pallet loads to turning radius and stopping
distance. Figure 6.10 shows some other possible case scenarios to be studied in future.
Figure 6.10 Other possible case scenarios for future analysis (Sources:
http://www2.illinoisbiz.biz/osha/PDF/Books/ForliftFullManual_ENGandSPN.pdf)
6.4 Summary
In this chapter, several case scenarios of forklift truck accidents built in the proposed
simulator are presented. Their performances can be summarized in Table 6.1.
Type of accident
Result
Remarks
Skidding/
Overturn
Pass
Favourable result is achieved. A real-time
indicator was added to monitor the lateral balance
of forklift truck. Also, an in-depth analysis was
conducted on the correlation between velocity and
turning radius on making turns.
Falling objects
Pass
Pallet loads slip or fall off in experiencing a
sudden break or a sharp turn.
Tipping Over
Fail
Due to the constraint of built-in “Physics Car BB”,
the forklift truck and the lifting mast are
considered as two separate physics identities.
Tipping over is not mimicked in the project.
Crushing
pedestrians/
driver ejection
N/A
Not attempted in the project. More interactive
elements such as Artificial Intelligence (AI)
characters are required. One of the possible
extensions in future.
Table 6.1 Summary of overall performance in each case scenario
6-10
Most accident scenarios like skidding and falling objects are successfully simulated
in the proposed simulator except the case of tipping over. Due to the constraint of the
“Physics Car BB”, physics properties of the forklift truck cannot be modified once
the simulation runs. To solve the issue, we can either reconstruct a new physics
system for manoeuvring the dynamics of forklift truck operations or replace the builtin physics engine with other open-source engines mentioned in chapter 2.
Seugling and Rölin (2006) evaluates several physics engines including Open
Dynamic Engine (ODE), AGEIA Novodex (PHYSX), Newton Game Dynamics,
Bullet, etc. in Virtools. Various tests were conducted on frictions, scalability and
stability, ODE and Newtown Game Dynamics stand out from the rest. Nonetheless,
more research work is required on the feasibility of replacing the built-in Havok
Engine of Virtools.
6-11
CHAPTER 7 Conclusion and Recommendations
7.1
Conclusion
The success of supply chain management requires all frontline logistics operations to
be carried out safely on schedule. Forklift trucks are often involved in daily
operations in warehouses or docks to transport pallet loads from one place to another.
Although all forklift truck operators are trained before they are qualified to drive,
countless accidents happen each year, causing serious injuries and deaths due to the
lack of safety awareness and mentality of operators.
To deal with the issue, simulations using Virtual Reality (VR) is considered to
supplement and enhance practical training. With the use of VR, forklift truck drivers
are then allowed to participate into various sorts of accident-like scenarios where they
practise their reactions and learn from their mistakes afterwards. In times of trials,
they can be more vigilant and experienced in handling adverse situations like oil spill
and forklift trucks overturn.
This research project therefore attempts to develop a prototype of forklift truck
operations to be simulated in a full-immersive CAVE-based VR environment. Since
most the forklift truck simulators reviewed have not incorporated with any Physics
Engine, they can barely simulate physics behaviours. In the proposed simulator, the
Havok Engine embedded in Virtools is integrated to mimic all physics behaviours of
forklift truck like pallet picking and dropping.
Since typical input devices like the “keyboard + mouse” combo are considered not
feasible in CAVE-based VR system, passive Infrared (IR) marker motion tracking
system is introduced. By recognizing the pre-defined rigid-body structure formed by
the passive IR markers, the IR motion tracking system is able to track the 3D position
of an object in the cubic space of ImseCAVE. This technique was applied on head
tracking for additional driving view to enhance the degree of realism in user
7-1
experience. Likewise, the proposed design is modified to support simple gesture
recognition, “grab” and “drop” in hand tracking. While operating the forklift truck,
users can execute commands like swapping accident scenarios, varying weight of
loads and changing camera views by making a simple hand gesture intuitively.
Several accidents scenarios such as skidding, overturn, falling objects are built and
tested. Most of them are successfully simulated as they are in reality except tipping
over. The simulated forklift truck is unable to tip forward even when weight of the
pallet load exceeds its maximum loading limit. This is due to the constraint of built-in
Physics Library of Virtools.
Other than mimicking the physics behaviours of a forklift truck, an in-depth analysis
of forklift truck overturns is carried out to demonstrate how data can be extracted
from the proposed system and interpreted real-time for the sake of forklift truck
safety. As the velocity of forklift truck increases, its turning radius will increases.
Until the frictional force is no longer enough to provide the centripetal force, the
forklift truck will skid or overturn. A programme is prepared to track the positions of
the forklift truck on making turns and the data collected are plotted on a graph for
analysis. As such, the system can estimate the safety speed limit in making turns and
review the manoeuvring skills of forklift truck operators.
As a prototype of forklift truck simulator, the project provides a general concept on
how forklift truck operations can be mimicked and launched into a full-immersive
CAVE-based VR system for safety analysis. In addition, with the use of IR motion
tracking system in both hand and head tracking, the proposed simulator brings forth
the most intuitive driving experience of a forklift truck vehicle ever. However, due to
time and resources constraints, not all possible accident scenarios and forklift truck
operations are mocked up in full. Once in a while, some unexpected physics
behaviours may cause system instability. Yet, it can be solved by resetting the
simulation. Modification and improvements are required to fine-tune and enhance the
7-2
proposed simulator to a more practicable learning kit for education or training
purposes.
7.2 Recommendations
The proposed simulation system is a feasibility study and an initial attempt to develop
a forklift truck simulator in full-immersive CAVE-based VR environment. Currently,
no industrial data of forklift truck engines or warehouses have been incorporated. For
in-depth study on the dynamic behaviours in accidents scenarios, factual information
like friction, weight, inertia, elasticity, damping, and suspension etc. should be
gathered. In addition, more interactive elements like Artificial Intelligence (AI)
characters of pedestrians and in-house workers may be added in simulating complex
accidents scenarios where human play an important factor.
The simulation model presented in the project, due to the constraint of the Physics
Library of Virtools, has not mimicked the forklift truck operations in full. The issue
of forklift truck tip-over remains untouched. Therefore, an essential technical
development is to explore the possibilities of incorporating other open source physics
engines, such as Open Dynamics Engine (ODE) which was suggested in chapter 6.
In the project, an in-depth analysis of forklift truck overturn accidents was
demonstrated. The relation between turning velocity and turning radius was studied.
Similar studies can also be conducted in future on other physics behaviours such as
the correlation between weight variations in pallet loads to turning radius and
stopping distance. With the issue of physics engine solved, tests can be performed on
tip-over of forklift truck as well. By adding small weight increments on the pallet
loads each time, the operators will then be able to percept the variations on forklift
truck performance and learn how to avoid the forklift truck from tipping over.
In the long run, the simulation system can be further developed into a practical tool
for training forklift truck operators; with standardized training rules or guidelines.
Sound effects, map guiding system, driving tips and warning signals can be
7-3
consolidated to get users better involved in the virtual scenarios as if they are working
in reality. Instead of letting users to drive aimlessly, tasks with certain objectives
should be set to be completed. By reviewing drivers‟ performance on simulation,
their manoeuvring skills can then be rectified and polished.
Indeed, the system together with the behaviour model can be ported into other similar
VR simulations of other industrial vehicles, like truck cranes, stackers, bulldozers,
and excavators for safety studies which eventually may save human lives.
7-4
REFERENCES
Autodesk 3ds Max, Autodesk, Inc. http://usa.autodesk.com/3ds-max/.
Boeing, A. and Bräunl T. 2007. Evaluation of real-time physics simulation systems.
In Proceedings of the 5th international conference on Computer graphics and
interactive techniques in Australia and Southeast Asia (GRAPHITE '07). ACM, New
York, NY, USA, 281-288.
Bergamasco, M., Perotti, S., Avizzano, C. A., Angerilli, M., Carrozzino and M.,
Ruffaldi, E. Fork-lift truck simulator for training in industrial environment, Emerging
Technologies and Factory Automation, 2005. ETFA 2005. 10th IEEE Conference on ,
vol.1, no., pp.5 pp.-693, 19-22 Sept. 2005
Cruz-Neira, C., Sandin, D. J.
and DeFanti, T. A.
1993. Surround-screen
projection-based virtual reality: the design and implementation of the CAVE,
Proceedings of the 20th Annual Conference on Computer Graphics and Interactive
Techniques, ACM, New York, NY, USA, 135-42.
Dhawale, P., Masoodian, M. and Rogers, B. 2006. Bare-hand 3D gesture input to
interactive systems, Proceedings of the 7th ACM SIGCHI New Zealand chapter's
international conference on Computer-human interaction: design centered HCI, ACM,
New York, NY, USA, 25-32.
Forklift madness, ShockBreak Games.
http://www.shockbreak.com/index.php/forklift-madness.
Forklift Safety Sim, Etcetera Edutainment.
http://www.etceteraedutainment.com/forklift-safety.php.
R-1
FORTUNE 500, Cable News Network (CNN).
http://money.cnn.com/magazines/fortune/fortune500/2009/full_list/.
Full Flight Simulator (FFS), Aeronautical Systems Engineering (AES).
http://www.aerosyseng.com/.
Gabelstapler-Simulator 2009, Astragon.
http://www.astragon.de/produktdetails/article/gabelstapler-simulator-2009.html.
ImseCAVE, Industrial and Manufacturing System Engineering Department, The
University of Hong Kong, Hong Kong.
http://www.imse.hku.hk/intellisyslab/facilities/imseCAVE.htm.
The Industrial Truck Association of US. http://www.indtrk.org/.
Kim, H. and Fellner, D. W. 2004. Interaction with hand gesture for a back-projection
wall, Proceedings of the Computer Graphics International, IEEE Computer Society,
Washington, D.C., USA, 395-402.
Kinect for Windows, Microsoft Research.
http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/.
Logitech MOMO Racing Force Feedback Wheel, Logitech.
http://www.logitech.com/en-us/441/320.
Nugraf , Okino Computer Graphics. http://www.okino.com/default.htm.
Occupational injuries in 2009, National Safety Council (NSC).
http://www.nsc.org/safetyhealth/Pages/Waronregulationsmisguided_2.17.11.aspx.
OptiTrack from Naturalpoint. http://www.naturalpoint.com/optitrack/.
R-2
Pintaric, T. and Kaufmann, H. 2007. Affordable Infrared-Optical Pose Tracking for
Virtual and Augmented Reality, Vortrag: IEEE Virtual Reality 2007, Charlotte, NC
(USA); 14.03.2007 - 17.03.2007
PlayStation Move, Sony . http://us.playstation.com/ps3/playstation-move/.
Rahman, ASM., Hossain, M. A., Parra, J. and El Saddik, A. 2009. Motion-path based
gesture interaction with smart home services. In Proceedings of the 17th ACM
international conference on Multimedia (MM '09). ACM, New York, NY, USA, 761764.
Seugling A, Rölin M. 2006. Evaluation of Physics Engines and Implementation of a
Physics Module in a 3d-Authoring Tool. Umea University.
Simulator for training fork lift truck and load handling machinery operators, Castilla
and Leon Institute of Technology ITCL (2008).
http://www.itclimasd.org/Virtual-Reality/Simulators/.
Smith, R. 2001. Open Dynamic Engine. http://www.ode.org/.
Smith, S. and Ecrison, E. 2009. Using immersive game-based virtual reality to teach
fire-safety skills to children, Virtual Reality, 13(2), Springer-Verlag, London, UK,
87-99
SolidWorks, Dassault Systèmes. http://www.solidworks.com/.
Song, P., Yu, H. and Winkler, S. 2008. Vision-based 3D finger interactions for mixed
reality games with physics simulation, Proceedings of the 7th ACM SIGGRAPH
International Conference on Virtual-Reality Continuum and Its Applications in
Industry, ACM, New York, NY, USA.
R-3
Sreedharan, S., Zurita, E. S., and Plimmer, B. 2007. 3D input for 3D worlds,
Proceedings of the 19th Australasian conference on Computer-Human Interaction:
Entertaining User Interfaces, ACM, New York, NY, USA, 227-30.
Steptoe, W., Wolff, R., Murgia, A., Guimaraes, E., Rae, J., Sharkey, P., Roberts, D.
and Steed, A. 2008. Eye-tracking for avatar eye-gaze and interactional analysis in
immersive collaborative virtual environments, Proceedings of the 2008 ACM
conference on Computer supported cooperative work, ACM, New York, NY, USA,
197-200.
Touyama, H. and Hirose, M. 2007. Steady-state VEPs in cave for walking around the
virtual world, Proceedings of the 4th international conference on Universal access in
human-computer interaction, ambient interaction, Springer-Verlag, Berlin/Heidelberg,
Germany, 715-17.
Toyota Industrial Equipment. http://www.toyotaforklift.com/.
Virtools, Dassault Systèmes. http://www.3ds.com/products/3dvia/3dvia-virtools/.
Wii, Nintendo. http://www.nintendo.com/wii.
Wingrave, C. A., Williamson, B., Varcholik, P. D., Rose, J., Miller, A., Charbonneau,
E., Bott, J. and LaViola Jr., J. J. The Wiimote and Beyond: Spatially Convenient
Devices for 3D User Interfaces, IEEE Computer Graphics and Applications, pp. 7185, March/April, 2010
Yang, X. B. 2010. A Hand Input-based Approach to Intuitive Human-Computer
Interactions in Virtual Reality. Mphil thesis. The University of Hong Kong, Hong
Kong.
R-4
Yang, X. B., Choi, S. H., Yuen, K. K. and Chan, K. Y. 2010. An intuitive HumanComputer Interface for Large Display Virtual Reality Applications, Computer-Aided
Design and Applications, 2010, v. 7 n. 2, p. 269-278
R-5