Autonomous Human-Robot Interactive Skills
A Collaboration Project between KAIST and Stanford University
Principle Investigators
Prof. Ju-Jang Lee (KAIST) and Prof. Oussama Khatib (Stanford University)
Participating Researches
KAIST
Stanford University
Dr. Sudath R. Munasinghe
Mr. Jaeheung Park
Mr. Chang-Mok Oh
Mr. James Warren
1. Introduction
Objectives: The objective of this project is to develop basic autonomous skill primitives
for robots to be able to inhabit and corporately behave in human-populated environments.
These primitives will be organized in a task behavior library (TBL), allowing for intuitive
task-level robot programming. TBL will provide a set of standard task templates that can
be used in various robot behaviors by instantiating them with the appropriate control
primitives, parameters, and properties.
Motivation: Humans have a variety of sensing and perception capabilities including,
visual, auditory, and tactile, which allow them to outperform machines and compensate
for the generic lack of control resolution. There is an increased interest in exploring
methodologies to mimic, or actually transfer human skills to robotic systems, and to
make those systems capable of performing complex tasks.
Methodologies: The transferred human skills would be essential for robotic systems to
behave in human-populated environments, and also to perform complex tasks that are
ascribed to human capabilities. Proper behavior in human-populated environments is
particularly essential for service robots, where the capability to perform complex humanlike tasks is particularly essential for robotic substitutes for humans.
We have concentrated on two major approaches to implement advanced robotic
capabilities; (1) model-based approach, and (2) human demonstration. Model-based
approach takes an insight into the specific tasks(s), and tries to describe and model the
skilled human behavior precisely. Using these models, human skills can me implemented
on robots with required strategies and parameters. Human demonstration is a direct skill
transfer based on sensory-motor relationships, and motion patterns of human motions
such as hand manipulation, gait-skills etc. Human demonstration can motion pattern can
be easily recorded by a motion capture facility.
2. Model-based Approach
The major difficulty with the extension of a robotic task to human-populated
environments lies in the various uncertainties of the real world. Our approach to deal with
these constrained motion uncertainties is to rely on sensor-based compliant strategies. In
assembly task for example, contacts can be recognized when they occur and take the
advantage of them in compliant way to guide the object in the assembly task. Sensor data
can be used during motion to adapt the path to the actual geometry of the moving objects
(e.g. sliding along the surfaces). Sensing can also be used to decide when to stop a
motion and then to select the next move if the goal has not been achieved. The
manipulation task primitives are parameterized by a compliance frame, operational point,
and other task related parameters. By selecting these parameters appropriately, we can
instantiate the basic skill primitives in many different ways to adapt to the needs of a
specific task. We have implemented such strategies for several simple tasks including
insertion, object stacking, and surface following. In this project, we are focusing on new
strategies for compliant motion tasks in order to develop models for advanced primitives
that allow complex tasks to be specified at a higher level of abstraction.
2.1 Advanced Skill Primitives
2.1.1 Mobile Manipulation
Obstacle Avoidance and Real-Time Path Planning:
Most motion planning algorithms presumes that the environment is completely known at
planning time [1]. Most of them actually assume that the world will not change in the
future, implying that all obstacles are static. Some of them allow for moving obstacles,
with known or predictable motion functions. To generate a motion those algorithms build
a representation of the obstacles in the configuration space of the robot. The
configuration space is the space describing all possible spatial positions and arm
configurations of the robot. If a particular configuration, described by the spatial position
of the robot and its arm configuration is in collision with the environment, it does not
correspond to a physically possible configuration. If we are dealing with robots with
many joints, the configuration space is high-dimensional and the computation of
configuration space obstacles, or even approximations to it, is computationally very
costly and can take from multiple seconds to hours. Planning algorithms that also allow
for obstacles moving on a known trajectory augment the configuration space with another
dimension: time.
Motion planning is performed by finding a continuous curve connecting the initial and
the final configuration of the robot. Sounds simple, but it can get pretty complex. This all
works fine until an unforeseen obstacle appears, or a known obstacle does not move as it
was assumed during planning. Now the motion generated by the motion planning
algorithm might not be valid any more. The environment has changed and the
assumptions that lead to that particular motion do not hold any more. The solution is to
plan a new motion from scratch, given the new information about the environment. If a
robot is supposed to accomplish a task in a space that changes frequently, however, this is
not practical any more.
For a robot to be able to inhabit human-populated environments, it is required that the
robot be able to cope with unpredictable changes in the environment. To allow the
execution of motion plans in dynamic environments the global planning process can be
augmented with a fast, reactive obstacle avoidance component. With this augmentation,
an initial motion plan can be generated under the assumption that all obstacles are known
[2]. During execution the robot deviates from the planned path, guided by potential fields
caused by previously unknown or moving obstacles. The planner attempts to rejoin the
original path at a later point that remains unaffected by changes in the environment.
Deviation from the pre-planned path, however, can result in local minima that require
replanning [3]. To overcome this problem, the concept of elastic bands was introduced
[4] in that a path is represented as a curve, called “elastic band”, in configuration space,
and it is incrementally modified according to the position of obstacles in the environment
to maintain a smooth and collision-free path. Figure 1 illustrates the elastic path being
planed for PUMA robot on a mobile base.
Fig. 1 Elastic path-planning for obstacle avoidance [Stanford mobile platform]
At each workspace location of the mobile manipulator, collision-free space around each
rigid link is determined by integrating the collision free bubbles along the spine of that
rigid link. These collision-free hyperspaces make the protective hull [5], and by covering
entire elastic band with overlapping protective hulls we can guarantee that the path it
represents is collision-free. However, as the number of degrees of freedom of the robot
increases, the estimate of the local free space around a configuration would require
consideration of an excessive amount of bubbles, which affects real-time performance.
By defining bubbles in robot workspace, considering the distances to of the obstacles will
greatly reduce this drawback. Let ρ (p) be the function that computes the minimum
distance from a point p to any obstacle, then the workspace bubble of free space around p
{
}
is defined as B (p) = q : p − q < ρ (p) . Considering p along the spine of a rigid body
(a single link or the mobile base), a set of free space bubbles can be determined for that
rigid body. This set of bubbles Pqb determines the protective hull for that rigid body in
that configuration. The collection of these hulls PqR =
UP
b
q
determined the protective
b∈R
hull of the robot R at configuration q. An elastic stripe S TR = (q1 , q 2 ...q n ) is a sequence
of configurations q i on the trajectory T of the robot R. The elastic stripe determined in
this way is shown in Figure 2.
Fig. 2 Protective hulls and elastic stripe along the trajectory with modification to an
incoming obstacle
The continuity of the protective hulls along the trajectory, and the existence of a safe
trajectory has been verifies in [6]. When the elastic stripe undergoes modification at ith
configuration, an internal force generates at p ij , the origin of jth link co-ordinate frame
⎛ d ij−1
⎞
i
i
i +1
(p ij+1 − p ij−1 ) − (p ij − p ij−1 ) ⎟ , where d j = p j − p j is the
⎜ d i −1 + d i
⎟
j
⎝ j
⎠
as given by Fiint
⎜
, j = kc
distance in the initial original trajectory. These forces cause the elastic stripe to
contract and to maintain a constant ratio of distances between every three
consecutive configurations. Meanwhile, the obstacles cause external forces due to
⎧k (d − d (p)) 2
their potential fields Vext (p) = ⎨ r 0
0
⎩
if d (p) < d 0
otherwise
where d(p) is the
distance from p to the closest obstacle, and d 0 defines the region of influence
around the obstacle. The repulsive external force acting at point p is therefore
Fpext = −∇Vext = k r (d 0 − d (p))
d
where d is the vector between p and closest
d
point on the obstacle. Under the influence of external forces, each configuration
along the trajectory undergoes modifications. Modifications of configuration has
to come from joint torques, thus, it is required to map the external forces to joint
torques. This mapping under absence of task requirements is given by Γ = J pT Fp ,
where J p is the Jacobian at p. The 9 degree of freedom Stanford mobile platform
consists of a PUMA 560 manipulator mounted on a holonomic mobile base. Figure 3
shows the motion of the robot while responding to an incoming obstacle.
Fig. 3 Elastic stripe for a 9-dof robot incrementally modified by a moving obstacle
The approaching obstacle deforms the elastic stripe to avoid collision. As the obstacle
moves away, internal forces cause the elastic stripe to assume the original straight line
trajectory.
For mobile manipulation, some degrees of freedom are used for task execution
while others can be used to achieve task-independent motion behavior. This is
achieved by unifying motion and force control behaviors of redundant
manupulatiors
[7].
Redundant
Γ = J T (q)F + [I − J T (q) J −T (q)]Γ0
manipulator
dynamics
is
given
by
where the first term on RHS is the torques
due to the forces action at the end-effector, whereas the second term on RHS
affects internal motion. This decomposition can be used to achieve advanced
motion behaviors such as mobile manipulation. To ensure the execution of a task
specified in a particular task frame f, the internal and external forces are mapped
into the null space of the Jacobian Jf associated with the task frame. This
corresponds to the sets of tasks where the end-effector is required to move on a
certain trajectory and the redundant degrees of freedom can be used for obstacle
avoidance. We adapt the adapt the operational space formulation [8] to realize mobile
manipulation, in that the position and orientation of the end-effector is controlled
independent of the base motion and other redundant joints, which can be used for
obstacle avoidance. However, task execution has to be suspended temporarily when
and while task-consistent motion behavior is infeasible due to kinematic
constraints or change in the environment. Therefore, it is necessary to develop a
criterion
to
determine
task
suspension
and
resumption.
Let
N T ( J (q)) = [ I − J T (q) J −T (q)] be the dynamically consistent nullspace mapping
of the Jacobian J(q) associated with the task. The coefficient c =
N T ( J (q))Γ0
Γ0
corresponds to the magnitude ratio of the torque vector Γ0 before and after
mapping into the nullspace, thus indicates how well the behavior be implemented
within the nullspace of the task. We use an experimental lower limit c < c s at
which it becomes desirable to suspend the task in favor of the behavior. Task
suspension is gradually, as a transition. The reverse transition takes place to
resume the task at an upper value c > c r . Unnecessary transitions can be avoided
by introducing a dead-band c s < c r . The mobile manipulation experiment with
the Stanford mobile platform is shown in Fig. 4, in that the end-effector maintains
its straight-line position profile (which is the specified task) within the entire
motion, while the mobile base adapts repulsive curvatures to avoid an incoming
obstacle.
Fig. 4 Mobile manipulation: keeping end-effector along a straight-line (task) while
avoiding collision with unforeseen obstacles
Figure 5 shows (1) Real-time obstacle avoidance without specified end-effector task, (2)
Mobile manipulation with end-effector task (without suspension), and (3) Mobile
manipulation with task suspension
Fig. 5 Mobile manipulation: Task suspension and resumption (3rd trial)
First trial shows only obstacle avoidance under no task specified at the end-effector.
Econd trial has a task specified to keep the end-effector along a straight line. The obstacle
doesn’t come extremely close, therefore the end-effector task is maintained witout
suspension. However, in the third trial, the task has to be suspended for a while until the
obstacle moves away.
The task and posture decoupling can be used to realize skilled behavior as illustrated in
Fig. 6
Fig. 6 PUMA 560 in vacuuming the floor, opening a door, and ironing a fabric.
These skilled behaviors involve a specific task defined at the end-effector in terms
of position, orientation and contact force. The task is satisfied independent of the
overall posture of the arm.
2.1.2 Human-Robot Cooperative Skills
In human-Robot cooperative skills, our investigation is to develop protocols for tactile
communication and guided motion skills for human-robot cooperative tasks. Guided
motion involves tight cooperation achieved through compliant motion actions, or looser
free-space motion commands. The robot for instance, may support a load while being
guided by the human to an attachment. These motion skills rely on the interaction skills
and the guided-motion primitives. The issues involved in human-robot cooperation have
similarities with those associated with multi-robot cooperation. The investigation of
guided motion primitives is influenced by the decentralized control behaviors we have
developed for cooperative robots. In decentralized corporation of two robots, each robot
relies on a model based on an augmented load that takes into account the inertial
properties and the dynamics of the other robot. Relying on the interaction skills and using
a simplified model of the human arm inertial properties, we are going to investigate a
similar approach to human-robot cooperation. This approach will provide the basis for
the development of effective human-robot guided-motion primitives.
Our approach is based on the concepts of virtual linkage [9] and augmented object [10].
The virtual linkage characterizes the internal forces of the cooperative task, whereas
augmented object describes the closed-chain dynamics of the system. The two concepts
are briefly described as follows:
Virtual linkage and augmented objects:
The model for internal forces for multi-grasp manipulation is illustrated in Fig. 7 below.
Fig. 7 Virtual linkage concept for multi-robot cooperation in three-grasp manipulation
task
In this model, grasp points are connected by a closed, non-intersecting set of virtual links.
For N-grasp manipulation task, the virtual linkage model is a 6(N-1) degree of freedom
mechanism that has 3(N-1) linearly actuated members and N spherically actuated joints.
By applying forces an torques at the grasp points we can independently specify internal
forces in the 3(N-2) linear members and 3N internal moments at the spherical joints. The
internal forces of the object are then characterized by these forces and torques in a
physically meaningful way. The relationship between applied forces, their resultants, and
⎡ f1 ⎤
⎡Fres ⎤
internal forces is ⎢
= G ⎢⎢ M ⎥⎥ , where Fres is the resultant force at the operational
⎥
⎣ Fint ⎦
⎢⎣f N ⎥⎦
point, Fint is the internal forces, f i is the forces applied at the ith grasp point, and
G is the grasp description matrix. The inverse of the grasp description matrix
provides the forces required at the grasp points to produce the resultant and
⎡ f1 ⎤
⎡F ⎤
internal forces ⎢⎢ M ⎥⎥ = G −1 ⎢ res ⎥ acting on the object. The resultant force on the
⎣ Fint ⎦
⎢⎣f N ⎥⎦
object describes the required motion behavior, whereas internal forces refer to the
specified task at the object. Once these two quantities are specified, the individual
grasp-point forces can be determined using G −1 . Further, the grasp point forces
can be decomposed into corresponding motion and task components for each
robot as f i = f i ,motion + f i ,task and implemented using a decentralized control
structure as shown below
Fig. 8 Decentralized control for autonomous robots in cooperative tasks
Figure 9 illustrates the cooperative behavior of two PUMA 560 robots based on
the augmented object and virtual linkage concepts. The decoupled task and
posture (motion) control is also included in the decentralized control structure.
Fig. 9 Two PUMA 560 robots in a cooperative task
The decoupled motion and force (task) control together with the decentralized
behavior make the skill primitive for a robot to b able to cooperate with other
robots and human operators in a highly skilled manner [11]
3.
Direct Skill Transfer
Direct skill transfer can be generally applied to a variety of human skills. It does not
require an insight, or a model of the particular skill. Human demonstrated data can be
learned by a neural network, and then implemented on robots. In this approach, we view
human skills in terms of spacio-temporal motion patterns. We could define human skills
as proper combination of joint motion profiles.
Our approach here is to capture the kinematics of human demonstration and analyze it to
reveal the underlying organization of the skill human motions. We are first considering
how a human leaves a cup on a table quite easily, without damaging both of the objects,
and also in no time of thinking and planning. On the other hand, it is a very difficult task
for a robot to perform without the three-dimensional vision information, and advanced
contact force sensory system that the humans are equipped with. Object manipulation
under constrained environments requires compliance motion, in that the contact
information itself is used as important information in guiding the object towards the
target position. Therefore, the motion-capture profiles should be supplemented with
vision, and contact information, and then investigate the high level organization of human
skilled behaviors. This part of the project is still being carried out at KAIST.
4. Strengthening KAIST-Stanford Relationship
The project indeed has been a very successful venture for both KAIST and Stanford
University. Both sides visited each other twice in the duration of two years. Our visiting
of each other actually strengthened our personnel relationships, and mutual
understanding. In addition to the project work, we have learned a lot about what is going
on the other side. It enlightened us about the cutting-edge research in robotics. After
working on this project, now we have strong relationship and better grounds for
promoting advanced future collaborations.
5. Summary and Conclusion
This project has been dedicated for developing basic autonomous capabilities for robots
to be able to inhabit human-populated environments. We have particularly concentrated
on skill primitives on (1) Mobile manipulation and (2) Cooperative behavior. With regard
to mobile manipulation skills, we have developed methods for (1a) Decoupling force and
motion control by null space mapping, (1b) Real-time obstacle avoidance by elastic strip
and potential fields. With regard to cooperative skills, we have developed (2a)
Compliance control with virtual linkages and augmented object, and (2b) Decentralized
control structure for autonomous behavior.
Another parallel thread of this project has been devoted to revealing the organizational
structure of skilled human behavior by way of analyzing captured motion of human
demonstration, augmented with contact and vision information. However, this work is
still in progress, and we are mot is a position to publish any results on that as of yet.
The primitives that we have developed for mobile manipulation and cooperative skills
can be stored in a task behavior library (TBL) and the complex behaviors that a robot
would need to inhabit human-populated environment can be instantiated from this TBL.
6. Acknowledgement
This collaboration work was funded under the Brain Korea 21 project between KAIST
and Stanford University.
References
[1] J. C. Latombe, “Robot Motion Planning” Kluwer Academic Publisher, Boston 1991.
[2] J.P.H. Steel, G. P. Starr, “mobile robot path planning in dynamic environments”, in
Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, vol. 2,
pp. 922-925, 1988.
[3] W. Choi, J .C. Latombe, “A reactive architecture for replanning and executing robot motions
with incomplete knowledge,” in Proceedings of the IEEE/RSJ International workshop on
Intelligent Robots and Systems, vol. 1, PP. 24-29, 1991
[4] S. Quinlan and O. Khatib, “Elastic bands: Connecting path planning and control,” in
Proceedings of the IEEE International Conference on Robotics and Automation, vol. 2 , pp.
802-807, 1993
[5] S. Quinlan, “Real-time modifications of collision free paths”, Ph.D. thesis, Stanford
University, 1994
[6] O. Broke, O. Khatib, and S. Viji, “Mobile manipulation: Collision-free path modification and
motion coordination,” in Proc. Proceedings of the 2nd International Conference on
Computational Engineering in Systems Applications, vol. 4, pp. 839-845, 1998.
[7] O. Khatib, “A unified approach for motion and force control of robot manipulators: The
operational space approach”, IEEE Journal of Robotics and Autonation, vol. RA-3, no. 1, pp.
43-53, Feb., 1987
[8] O. Khatib, “A unified approach motion and force control of robot manipulators: The
operational space formulation,” International Journal of Robotics and Automation, 3(1):4353, 1987
[9] O. Khatib, “Object manipulation in a multi-effector robot system” in R. Bolls and B. Roth
(Eds.), Robotics Research 4, pp 137-144, MIT Press
[10] D. Williams and O. Khatib, “The virtual linkage: a model for internel forces in multi-grasp
manipulation”, in Proceedings International Conference on Robotics and Automation, vol. 1,
pp. 1025-1030
[11] O. Khatib, K. Yokai, O. Nrock, K. Chang, and A. Casal, “Robots in human environments:
Basic autonomous capabilities”, in archives Robotics Laboratory, Department of computer
science, Stanford University, Stanford, CA 94086, USA