Evolutionary Development of Hierarchical Learning Structures

Show full item record

Please use this identifier to cite or link to this item: http://hdl.handle.net/1853/37947

Title: Evolutionary Development of Hierarchical Learning Structures
Author: Elfwing, Stefan ; Uchibe, Eiji ; Doya, Kenji ; Christensen, Henrik I.
Abstract: Hierarchical reinforcement learning (RL) algorithms can learn a policy faster than standard RL algorithms. However, the applicability of hierarchical RL algorithms is limited by the fact that the task decomposition has to be performed in advance by the human designer. We propose a Lamarckian evolutionary approach for automatic development of the learning structure in hierarchical RL. The proposed method combines the MAXQ hierarchical RL method and genetic programming (GP). In the MAXQ framework, a subtask can optimize the policy independently of its parent task's policy, which makes it possible to reuse learned policies of the subtasks. In the proposed method, the MAXQ method learns the policy based on the task hierarchies obtained by GP, while the GP explores the appropriate hierarchies using the result of the MAXQ method. To show the validity of the proposed method, we have performed simulation experiments for a foraging task in three different environmental settings. The results show strong interconnection between the obtained learning structures and the given task environments. The main conclusion of the experiments is that the GP can find a minimal strategy, i.e., a hierarchy that minimizes the number of primitive subtasks that can be executed for each type of situation. The experimental results for the most challenging environment also show that the policies of the subtasks can continue to improve, even after the structure of the hierarchy has been evolutionary stabilized, as an effect of Lamarckian mechanisms.
Description: (c) 2007 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works. Digital Object Identifier: 10.1109/TEVC.2006.890270
Type: Article
URI: http://hdl.handle.net/1853/37947
ISSN: 1089-778X
Citation: Elfwing, S., Uchibe, E., Doya, K., and Christensen, H. I. Evolutionary development of hierarchical learning structures. IEEE Trans on Evolutionary Computing 11, 2 (apr 2007), 249-264
Date: 2007-04
Contributor: Georgia Institute of Technology. College of Computing
Okinawa Institute of Science and Technology
Georgia Institute of Technology. Center for Robotics and Intelligent Machines
Kungl. Tekniska Högskolan. Centrum för Autonoma System
Publisher: Georgia Institute of Technology
Institute of Electrical and Electronics Engineers
Subject: Autonomous development
Genetic programming
Hierarchical reinforcement learning
Lamarckian evolution

All materials in SMARTech are protected under U.S. Copyright Law and all rights are reserved, unless otherwise specifically indicated on or in the materials.

Files in this item

Files Size Format View
04141056.pdf 1.991Mb PDF View/ Open

This item appears in the following Collection(s)

Show full item record