Tracking Emotions: Intrinsic Motivation Grounded on Multi - Level Prediction Error Dynamics (original) (raw)

Intrinsic Motivation For Reinforcement Learning Systems

2005

Motivation is a key factor in human learning. We learn best when we are highly motivated to learn. Psychologists distinguish between extrinsically-motivated behavior, which is behavior undertaken to achieve some externally supplied reward, such as a prize, a high grade, or a high-paying job, and intrinsically-motivated behavior, which is behavior done for its own sake. Is there an analogous distinction for machine learning systems? Can we say of a machine learning system that it is motivated to learn, and if so, can it be meaningful to distinguish between extrinsic and intrinsic motivation? In this paper, we argue that the answer to both questions is “yes,” and we describe some computational experiments that explore these ideas within the framework of computational reinforcement learning. In particular, we describe an approach by which artificial agents can learn hierarchies of reusable skills through a computational analog of intrinsic motivation.

Intrinsically motivated model learning for a developing curious agent

2012

ABSTRACT Reinforcement Learning (RL) agents could benefit society by learning tasks that require learning and adaptation. However, learning these tasks efficiently typically requires a wellengineered reward function. Intrinsic motivation can be used to drive an agent to learn useful models of domains with limited or no external reward function. The agent can later plan on its learned model to perform tasks in the domain if given a reward function.

Intrinsic Motivations for Forming Actions and Producing Goal Directed Behaviour

In classical reinforcement learning framework, an external, handcrafted reward typically drives the learning process. Intrinsically motivated systems, on the other hand, can guide their learning process autonomously by computing the interest they have in each task they can engage in. We explore how intrinsic motivation could be implemented in the iCub platform on a learning task that was used previously with infants and monkeys, with a focus on discriminating between task of varying difficulty, and observing how their interest towards the tasks change as their knowledge of them progresses. Two main different approaches were taken : one where the reinforcement learning framework was adapted to an intrinsic reward, and another where the focus was put on a goaloriented architecture. Two experiments settings were used, one with a console proposing buttons that activated boxes, and another proposing an interaction with rods : both experiments exhibited two tasks, one easy, and one difficult to learn. In each experiment, the system is able to successfully focus on learning the easier task earlier than the difficult one.

Intrinsic and Extrinsic Motivation in Intelligent Systems

2020

There are two ways that systems, human or machine, can get ”motivated” to take action in problem solving. One, they can be given goals by some external entity. In some instances, they might have no capability other than to work towards the goals provided by that entity. Two, they can have their own, internal goals, and work towards those goals. If given a goal by an outside entity, they can then try to figure out whether, and how, the external goal might align with their internal goals. In that case, the agent might be said to be acting in a ”self-supervised” manner. There are, of course, cases where both intrinsic and extrinsic motivation come into play. This paper will argue that many machine learning systems, as well as human organizations, put too much emphasis on extrinsic motivation, and have not fully taken advantage of the potential of intrinsic motivation. Reinforcement learning systems, for example, have a ”reward signal” that is the sole extrinsic motivating factor. It is...

Intrinsically motivated machines

50 years of artificial intelligence, 2007

Children seem intrinsically motivated to manipulate, to explore, to test, to learn and they look for activities and situations that provide such learning opportunities. Inspired by research in developmental psychology and neuroscience, some researchers have started to address the problem of designing intrinsic motivation systems. A robot controlled by such systems is able to autonomously explore its environment not to fulfil predefined tasks but driven by an incentive to search for situations where learning happens efficiently. In this paper, we present the origins of these intrinsically motivated machines, our own research in this novel field and we argue that intrinsic motivation might be a crucial step towards machines capable of life-long learning and open-ended development.

What is intrinsic motivation? a typology of computational approaches

Frontiers in Neurorobotics, 2007

Intrinsic motivation, centrally involved in spontaneous exploration and curiosity, is a crucial concept in developmental psychology. It has been argued to be a crucial mechanism for open-ended cognitive development in humans, and as such has gathered a growing interest from developmental roboticists in the recent years. The goal of this paper is threefold. First, it provides a synthesis of the different approaches of intrinsic motivation in psychology. Second, by interpreting these approaches in a computational reinforcement learning framework, we argue that they are not operational and even sometimes inconsistent. Third, we set the ground for a systematic operational study of intrinsic motivation by presenting a formal typology of possible computational approaches. This typology is partly based on existing computational models, but also presents new ways of conceptualizing intrinsic motivation. We argue that this kind of computational typology might be useful for opening new avenues for research both in psychology and developmental robotics.

GRAIL: A Goal-Discovering Robotic Architecture for Intrinsically-Motivated Learning

IEEE Transactions on Cognitive and Developmental Systems, 2016

In this paper, we present goal-discovering robotic architecture for intrisically-motivated learning (GRAIL), a fourlevel architecture that is able to autonomously: 1) discover changes in the environment; 2) form representations of the goals corresponding to those changes; 3) select the goal to pursue on the basis of intrinsic motivations (IMs); 4) select suitable computational resources to achieve the selected goal; 5) monitor the achievement of the selected goal; and 6) self-generate a learning signal when the selected goal is successfully achieved. Building on previous research, GRAIL exploits the power of goals and competence-based IMs to autonomously explore the world and learn different skills that allow the robot to modify the environment. To highlight the features of GRAIL, we implement it in a simulated iCub robot and test the system in four different experimental scenarios where the agent has to perform reaching tasks within a 3-D environment.

Information-driven intrinsic motivation in reinforcement learning

2013

One of the main challenges in the field of embodied artificial intelligence is the open-ended autonomous learning of complex behaviours. Our approach is to use task-independent, information-driven intrinsic motivation(s) to support task-dependent learning. The work presented here is a preliminary step in which we investigate the predictive information (the mutual information of the past and future of the sensor stream) as an intrinsic drive, ideally supporting any kind of task acquisition. Previous experiments have shown that the predictive information (PI) is a good candidate to support autonomous, open-ended learning of complex behaviours, because a maximisation of the PI corresponds to an exploration of morphology-and environment-dependent behavioural regularities. The idea is that these regularities can then be exploited in order to solve any given task. Three different experiments are presented and their results lead to the conclusion that the linear combination of the one-step PI with an external reward function is not generally recommended in an episodic policy gradient setting. Only for hard tasks a great speed-up can be achieved at the cost of an asymptotic performance lost. Keywords: information-driven self-organisation, predictive information, reinforcement learning, embodied artificial intelligence, embodied machine learning controller . A similar learning rule was obtained from the principle of Homeokinesis . In both cases a gradient information was derived to pursue local optimisation. For the integration of external goals a set of methods have been proposed Martius and Herrmann [2012], which however cannot deal with the standard reinforcement setting of arbitrary time-delayed rewards that we study here. Prokopenko et al. [2006] used the PI, estimated on the spatio-temporal phase-space of an embodied system, as part of fitness function in an artificial evolution setting. It was shown that the resulting locomotion behaviour of a snake-bot was more robust, compared to the setting, in which only the travelled distance determined the fitness.