

Publication Details
Reference Type  Journal Article 
Author(s)  Braun, D. A., Ortega, P. A., Theodorou, E., Schaal, S. 
Year  2011 
Title  Path Integral Control and Bounded Rationality 
Journal/Conference/Book Title  To appear in Approximate Dynamic Programming and Reinforcement Learnig 
Keywords  Bounded Rationality, Path Integral Control 
Abstract  Path integral methods [7], [15],[1] have recently been shown to be applicable to a very general class of optimal control problems. Here we examine the path integral formalism from a decisiontheoretic point of view, since an optimal controller can always be regarded as an instance of a perfectly rational decisionmaker that chooses its actions so as to maximize its expected utility [8]. The problem with perfect rationality is, however, that finding optimal actions is often very difficult due to prohibitive computational resource costs that are not taken into account. In contrast, a bounded rational decisionmaker has only limited resources and therefore needs to strike some compromise between the desired utility and the required resource costs [14]. In particular, we suggest an informationtheoretic measure of resource costs that can be derived axiomatically [11]. As a consequence we obtain a variational principle for choice probabilities that trades off maximizing a given utility criterion and avoiding resource costs that arise due to deviating from initially given default choice probabilities. The resulting bounded rational policies are in general probabilistic. We show that the solutions found by the path integral formalism are such bounded rational policies. Furthermore, we show that the same formalism generalizes to discrete control problems, leading to linearly solvable bounded rational control policies in the case of Markov systems. Importantly, Bellman’s optimality principle is not presupposed by this variational principle, but it can be derived as a limit case. This suggests that the information theoretic formalization of bounded rationality might serve as a general principle in control design that unifies a number of recently reported approximate optimal control methods both in the continuous and discrete domain. 
Link to PDF  http://wwwclmc.usc.edu/publications//D/DanielADPRL2011.pdf 

