Yanfei Liu
Dept. of Electrical and Computer Engineering
Clemson University, SC
U N I V E R S I T Y

• Motivation for this research
– Current status of vision in industrial workcells
– A novel industrial workcell with continuous visual guidance
• Work that has been done
– Our prototype: camera network based industrial workcell
– A new generic timing model for vision-based robotic systems
– Dynamic intercept and manipulation of objects under semi-structured motion
– Grasping research using a novel flexible pneumatic endeffector
Clemson University

• Current industrial workcells
– No vision or a single snapshot in certain locations
– Disadvantages
• Cannot deal with flexible parts
• Cannot deal with uncertainty
Clemson University

• Our novel dynamic workcell design
– Manipulation is integrated with visual sensing
– Applications ( reduce fixtures, handle objects on the ship)
Clemson University

– A set of cameras embedded into the workcell
– An industrial manipulator with its conventional controller
Clemson University

• Our prototype
– Staubli RX130 manipulator with its conventional controller
– Six cameras, wired to two PC-RGB framegrabbers mounted in a
Compaq Proliant 8500 computer
– V+ Operating systems and language
– Alter command to accomplish real time motion
Clemson University

Clemson University

A new generic timing model for vision-based robotic system
Clemson University

desired position
+
e control
power amplifiers robot camera
• Classical visual servoing structure
– eye-in-hand systems
• Corke (1996), an eye-in-hand manipulator to fixate on a thrown ping-pong ball
• Gangloff (2002), a 6-DOF manipulator to follow unknown but structured 3-D profiles.
– part-in-hand systems
• Stanvnitzky (2000), align a metal part with another fixed part
– mobile robot systems
• Kim (2000), a mobile robot system to play soccer
Clemson University

camera desired position
+
e control joint controller robot encoders
• Vision guided control structure
– Allen (1993), a PUMA-560 tracking and grasping a moving model train which moved around a circular railway.
– Nakai (1998), a robot system to play volleyball with human beings.
– Miyazaki (2002), a robot accomplished a ping pong task based on virtual targets
Clemson University

• Three common problems in visual systems
– Maximum possible rate for complex visual sensing and processing is much slower than the minimum required rate for mechanical control.
– Complex visual processing introduces a significant lag
( processing lag ) between when reality is sensed and when the result from processing a measurement of the object state is available .
– A lag ( motion lag ) is produced when the mechanical system takes time to complete the desired motion.
Clemson University

Work
Corke, Good
Stavnitzky, Capson
Kim et . al .
Allen et . al .
Nakai et . al .
this work
Image processing rate
(HZ)
50
30
30
10
60
23
Control rate
(HZ)
70
1000
30
50
500
250
Processing lag
(ms)
48
--
90
100
--
151
Motion lag
(ms)
--
--
--
--
--
130
• the first two of the three problems have been addressed to some extent in previous works. All of these works neglect the motion time ( motion lag ) of the robot.
• Corke and Kim, presented timing diagram to describe time delay, used discrete time models to model the systems and simplified these asynchronous systems to single-rate systems.
Clemson University

 s
1
 s
2 sensing image processing synchronizing tracking
 q
 u
1
 k
 c
1
…  c
N controlling
 u
2 finishing motion processing lag
 f motion lag
Clemson University

• Inherent values (obtained by analysis/measurement)
–  s = 33ms
 u = 19+30+14 = 63ms
–  w m
= 39ms
 w f
= (5+16+27)/3 = 16ms
–  l =
 s +
 u +
 w =151ms
 f = 130ms
 w = 39+16 = 55ms
• User-variable values
–  c = 4ms
 q = 40ms
Clemson University

• Problem description
– The most recently measured position and velocity of the object is where the object was (  l+  k ) ms before, x t-
 l-
 k, v t-
 l-
 k
– The current position, x
 x t

 x t
  l
  k
 t
 v t
  l
  k
(
 l
  k )
– N,  d ?
Clemson University

• Solutions
 x t
  i
      t t
(
 c

(
 n
 x t
  i
 

 x t
  q
   
 d

1 )
  f )
 n i
 
 



(
 f
  c ) d
 v t

 

 c
 v t
 x t i
 

 x t
  q
 




Constraint :
 v t
 d
 c
 d
 i
 

 x t
  i
 

 x t
  q
 
N
Clemson University

• Setup
– A small cylindrical object is dragged by a string tied to a belt moving at a constant velocity.
– The robot will lunge and cover the object on the table with a modified end-effector, a small plastic bowl.
Clemson University

Clemson University

• Experiment description
– We set  q to two different values, 40 and 80, in these two sets of experiments. We let the object move at three different velocities.
For each velocity, we ran the experiment ten times.
• Results
Velocity range(mm/s)
84.4 – 97.4
129.8-146.7
177.6 – 195.1
 q=40
Stdev range
1.3 –3.8
1.7 - 3.2
0.5 – 2.6
Catch percentage
100%
100%
100%
Velocity range(mm/s)
85.9 – 95.1
126.1 – 137.7
175.8 – 192.8
 q=80
Stdev range
2.5 - 3.7
1.7 – 3.3
1.1 – 2.7
Catch percentage
100%
100%
100%
Clemson University

Clemson University

Dynamic Intercept and Manipulation of Objects under
Semi-Structured Motion
Clemson University

Clemson University

robot x x t
, y t
: object position at time t v x
, v y
: object velocity at time t x r
, y r
: initial robot position x f
, y f
: final impact position
Unknown variables: y f ,
 i y
Closed loop
Start tracking
Make prediction ( t )
Open loop
Impact ( t+
 i )
Clemson University

• Solutions x
 f x r y t
 v y
 i
 y f x t
 v x
 i
 x f
 i
 y f
 x r
 x y t v x
 v y v x t
 x r
 x t
 m

 i
  f
 c
• Object unsensed time
– Time between the last instant when reality is sensed and the final impact time
– Delay between visual sensing and manipulation
Clemson University

 t processing lag(
 l) +
 k synchronizing tracking
 q
 c
1
…
 c
N controlling
…
 q
 c
1
…
 c
N finishing motion closed loop m 20 motion lag (  f) open loop
 t =
 l +
 k + 4m+
 f m < N = 10,
 k < 30 + 14 = 44ms
 t = 151 + ( 40 + 44 ) / 2 +115 = 308ms
Clemson University

Clemson University
20 alters
10 alters impact point
10 alters
20 alters y z impact point

• Solutions x
f x f
 x i
 x i
 v i
  t
 v
  t
v i
 v
 f
 x f
 t v i
 v
 w
2

308
• Implementation
– Predict the maximum acceleration of the object motion that the robot still can achieve a successful catch
– Calculate the size of the end-effector in order to overcome the maximum acceleration of the moving objects
Clemson University

• Setup
– Two types of end-effector (bowl, two scoopers with different width).
– Three types of interference (wind, bump, ramp)
• Results
– With wind interference
Bowl
Catch
Miss
Too fast
Total failure
Clemson University
Catch
88.6%
1.4%
7.1%
2.8%
Miss
1.4%
1.4%
Scooper1
Catch Miss
84.3%
2.9%
2.9%
4.3%
5.1%
5.8% scooper2
Catch miss
95.7%
0.0%
0.0%
0.0%
4.3%
0.0%

– with bump interference, weighted corner
Catch
Miss
Too fast
Total failure
Catch
Bowl
Miss
2.9%
0.0%
0.0%
84.3%
12.9%
0.0%
– with bump interference, balanced
Scooper1
Catch Miss
2.9%
0.0%
7.1%
80.0%
10.0%
7.1% scooper2
Catch miss
1.4%
2.9%
4.3%
85.7%
5.7%
7.2%
Catch
Miss
Too fast
Total failure
Catch
Bowl
Miss
37.1%
1.4%
7.1%
45.7%
8.6%
8.5%
Scooper1
Catch Miss
31.4%
2.9%
0.0%
52.9%
12.9%
2.9% scooper2
Catch miss
50.0%
0.0%
2.9%
45.7%
1.4%
2.9%
Clemson University

– with ramp interference, weighted corner
Catch
Miss
Too fast
Total failure
Catch
Bowl
Miss
0.0%
2.9%
4.3%
82.9%
10.0%
7.2%
– with ramp interference, balanced
Scooper1
Catch Miss
0.0%
4.3%
8.6%
78.6%
8.6%
12.9% scooper2
Catch miss
1.4%
2.9%
8.6%
77.1%
10.0%
11.5%
Catch
Miss
Too fast
Total failure
Catch
Bowl
Miss
60.0%
4.3%
Scooper1
Catch Miss
7.1% 50.0% 5.7%
21.4% 14.3% 30.0%
7.1%
11.4%
0.0%
20.0% scooper2
Catch miss
68.6%
4.3%
8.6%
11.4%
7.1%
12.9%
Clemson University

A Novel Pneumatic Three-finger Robot Hand
Clemson University

• Three different types of robot hands
– Electric motor powered hands, for example:
• A. Ramos et. al . Goldfinger
• C. Lovchik et. al . The robonaut hand
• J. Butterfa  et. al . DLR-Hand
• Barrett hand
– Pneumatically driven hands:
• S. Jacobsen et. al . UTAH/M.I.T. hand
– Hydraulically driven hands:
• D. Schmidt et. al . Hydraulically actuated finger
• Vision-based robot hand research
– A. Morales et. al.
presented a vision-based strategy for computing threefinger grasp on unknown planar objects
– A. Hauck et. al. Determine 3D grasps on unknown, non-polyhedral objects using a parallel jaw gripper
Clemson University

• Disadvantages of current robot hands
– Most robot hands are heavy
– Even with visual guidance, the robot hand can only grasp stationary objects
• Novel hand architecture
– build-in pneumatic line in
Staubli RX130
– Paper tube, music steel wire embedded inside
– Camera mount adjusting
“finger” spread angle
– 120 degrees between each other
Clemson University

• Close position
Open position
• Our research here is to demonstrate that we use a novel idea to built a flexible end effector and it can grasp semi-randomly moving objects. This is not a new type of complex research tool-type robot hands.
Clemson University

• Problem statement ball track robot y initial hand position
Clemson University x
 final hand position

• Position prediction
– Same as the method in the second part work of this research
• Orientation adjustment
– Line fitting to get the final “roll” angle
– equations y
 a
 bx a
 b

 n i

1 y i n

 i n i

1 n

1 x i
2 x i
2


(

 n i

1 i n

1 x i x i

)
2 i n

1 n
 i n x i n

1
 i n

1 y i x i
2



( i n

1
 n i x i

1
 x i
) i n

1
2 y i x i y i
 
b
Clemson University

Clemson University

• A generic timing model for a robotic system using visual sensing, where the camera provides the desired position to the robot controller.
• We demonstrate how to obtain the values of the parameters in the model, using our camera network workcell as an example.
• Implementation to let our industrial manipulator intercept a moving object.
• Experimental results indicate that our model is highly effective, and generalizable .
Clemson University

• Based on the timing model, we present a novel generic and simple theory to quantify the dynamic intercept ability of vision based robotic systems.
• We validate the theory by designing 15 sets of experiments
(1050 runs), using two different end effectors under three different interference.
• The experimental results demonstrate that our theory is effective.
Clemson University

• A novel pneumatic three-finger hand is designed and demonstrated.
• It is simple, light and effective.
• Experimental results demonstrate that this novel pneumatic hand can grasp semi-randomly moving objects.
• Advantages
– The compliance from pneumatics will allow the three-finger hand to manipulate more delicate and fragile objects.
– In the experiments of grasping moving objects, unlike the traditional gripper, the contact position for this continuous finger is not very critical, which leaves more room for sensing error.
Clemson University

• The South Carolina Commission on Higher Education
• The Staubli Corporation
• The U.S. Office of Naval Research
Clemson University

Clemson University

Manipulation name
Scoop
Catch
Basic robot motion
Impact time
Changing position and yaw orientation
Half way of scoop motion
Changing position Finishing whole motion
Success definition
Scoop the object out of table
Grab the object
Trap
Ensnare
Changing position Half way of trap motion
Changing position and roll orientation
Finishing whole motion
Cover the object under the bowl
Trap and then grasp the object
Clemson University

bump interference ramp interference
The distribution of |v i
– v avg
| in the balance ramp and bump cases.
Clemson University

• An external camera to observe operation
• A conveyor moving in a fixed path at a constant velocity
• A light bulb as a tracking object
• A laser mounted in the end effector of the robot
Clemson University