

Publication Details
Reference Type  Conference Proceedings 
Author(s)  Peters, J.;Schaal, S. 
Year  2007 
Title  Reinforcement learning for operational space control 
Journal/Conference/Book Title  International Conference on Robotics and Automation (ICRA2007) 
Keywords  operational space control, reinforcement learning, weighted regression, EMAlgorithm 
Abstract  While operational space control is of essential importance
for robotics and wellunderstood from an analytical
point of view, it can be prohibitively hard to achieve accurate
control in face of modeling errors, which are inevitable in
complex robots, e.g., humanoid robots. In such cases, learning
control methods can offer an interesting alternative to analytical
control algorithms. However, the resulting supervised learning
problem is illdefined as it requires to learn an inverse mapping
of a usually redundant system, which is well known to suffer
from the property of nonconvexity of the solution space, i.e.,
the learning system could generate motor commands that try
to steer the robot into physically impossible configurations. The
important insight that many operational space control algorithms
can be reformulated as optimal control problems, however, allows
addressing this inverse learning problem in the framework of
reinforcement learning. However, few of the known optimization
or reinforcement learning algorithms can be used in online
learning control for robots, as they are either prohibitively
slow, do not scale to interesting domains of complex robots,
or require trying out policies generated by random search,
which are infeasible for a physical system. Using a generalization
of the EMbased reinforcement learning framework suggested
by Dayan & Hinton, we reduce the problem of learning with
immediate rewards to a rewardweighted regression problem
with an adaptive, integrated reward transformation for faster
convergence. The resulting algorithm is efficient, learns smoothly
without dangerous jumps in solution space, and works well in
applications of complex high degreeoffreedom robots. 
Place Published  Rome, Italy, April 1014 
Pages  21112116 
Short Title  Reinforcement learning for operational space control 
URL(s)  http://wwwclmc.usc.edu/publications/P/petersICRA2007.pdf


