Learning to Prognostically Forage in a Neural Network Model of the Interactions between Neuromodulators and Prefrontal Cortex (original) (raw)
Related papers
Neural Mechanisms of Human Decision-Making
Cognitive, Affective, & Behavioral Neuroscience, 2021
We present a theory and neural network model of the neural mechanisms underlying human decision-making. We propose a detailed model of the interaction between brain regions, under a proposer-predictor-actor-critic framework. This theory is based on detailed animal data and theories of action-selection. Those theories are adapted to serial operation to bridge levels of analysis and explain human decision-making. Task-relevant areas of cortex propose a candidate plan using fast, model-free, parallel neural computations. Other areas of cortex and medial temporal lobe can then predict likely outcomes of that plan in this situation. This optional prediction-(or model-) based computation can produce better accuracy and generalization, at the expense of speed. Next, linked regions of basal ganglia act to accept or reject the proposed plan based on its reward history in similar contexts. If that plan is rejected, the process repeats to consider a new option. The reward-prediction system acts as a critic to determine the value of the outcome relative to expectations and produce dopamine as a training signal for cortex and basal ganglia. By operating sequentially and hierarchically, the same mechanisms previously proposed for animal action-selection could explain the most complex human plans and decisions. We discuss explanations of model-based decisions, habitization, and risky behavior based on the computational model.
Cortical substrates for exploratory decisions in humans
Nature, 2006
Decision making in an uncertain environment poses a conflict between the opposing demands of gathering and exploiting information. In a classic illustration of this 'exploration-exploitation' dilemma 1 , a gambler choosing between multiple slot machines balances the desire to select what seems, on the basis of accumulated experience, the richest option, against the desire to choose a less familiar option that might turn out more advantageous (and thereby provide information for improving future decisions). Far from representing idle curiosity, such exploration is often critical for organisms to discover how best to harvest resources such as food and water. In appetitive choice, substantial experimental evidence, underpinned by computational reinforcement learning 2 (RL) theory, indicates that a dopaminergic 3,4 , striatal 5-9 and medial prefrontal network mediates learning to exploit. In contrast, although exploration has been well studied from both theoretical 1 and ethological 10 perspectives, its neural substrates are much less clear. Here we show, in a gambling task, that human subjects' choices can be characterized by a computationally well-regarded strategy for addressing the explore/exploit dilemma. Furthermore, using this characterization to classify decisions as exploratory or exploitative, we employ functional magnetic resonance imaging to show that the frontopolar cortex and intraparietal sulcus are preferentially active during exploratory decisions. In contrast, regions of striatum and ventromedial prefrontal cortex exhibit activity characteristic of an involvement in value-based exploitative decision making. The results suggest a model of action selection under uncertainty that involves switching between exploratory and exploitative behavioural modes, and provide a computationally precise characterization of the contribution of key decision-related brain systems to each of these functions.
2013
Selective attention in real-world environments enables subjects to direct information gathering systems towards relevant stimuli in the presence of distractors. In a dynamic visual environment that lacks cues identifying the relevance of stimuli for reaching goals, attention requires internal mechanisms to track valuable targets. Reinforcement learning (RL) provides a theoretical framework that is typically applied to describe the mechanisms underlying optimal action control (Lee et al., 2012; Schultz et al., 1997; Sutton and Barto, 1998). Here we propose a RL approach to resolve the credit assignment problem for the deployment of covert attentional selection (Balcarras et al., 2013; Kaping et al., 2011). We compared two types of RL models to explain behavior of two macaques performing a selective attention task during foraging in the space of stimulus features: we show that despite being extensively reinforced on the relevance of stimulus color for reward, monkey behavior does not ...
Neurodynamics of Decision-Making-A Computational Approach
Advances in Cognitive Neurodynamics (V), 2016
Decision-making is a complex process that normally seems to involve several brain structures. In particular, amygdala, orbitofrontal cortex (OFC), and lateral prefrontal cortex (LPFC) seem to be essential in human decision-making, where both emotional and cognitive aspects are taken into consideration. In this paper, we present a stochastic population model representing the neural information processing of decision-making, from perception to behavioral activity. We model the population dynamics of the three neural structures significant in the decision-making process (amygdala, OFC, and LPFC), as well as their interaction. In our model, amygdala and OFC represent the neural correlates of secondary emotion, while the activity of OFC neural populations represents the outcome expectancy of alternatives, and the cognitive aspect of decision-making is controlled by LPFC. The results may have implications for how we make decisions for our individual actions, as well as for societal choices, where we take examples from transport and its impact on climate change.
Essential features of the world are often hidden and must be inferred by constructing internal models based on indirect evidence. Here, to study the mechanisms of inference we established a foraging task that is naturalistic and easily learned, yet can distinguish inference from simpler strategies such as the direct integration of sensory data. We show that both mice and humans learn a strategy consistent with optimal inference of a hidden state. However, humans acquire this strategy more than an order of magnitude faster than mice. Using optogenetics in mice we show that orbitofrontal and anterior cingulate cortex inactivation impact task performance, but only orbitofrontal inactivation reverts mice from an inference-based to a stimulus-bound decision strategy. These results establish a cross-species paradigm for studying the problem of inference-based decision-making and begin to dissect the network of brain regions crucial for its performance.
Model-based decision making in the human brain
2007
also want to thank friends outside of Caltech, who contributed to this work in pretty much the same way. Last of all, I want to thank my parents and sisters for the support and patience they have given during the years. Much of what I am now I owe to them. And to my dearest Allison, for her love and patience, and for knowing how to encourage me when things got tough, and tone me down when I was too excited! v ABSTRACT Many real-life decision making problems incorporate higher-order structure, involving interdependencies between different stimuli, actions, and subsequent rewards. It is not known whether brain regions implicated in decision making, such as ventromedial prefrontal cortex, employ a stored model of the task structure to guide choice (model-based decision making) or merely learn action or state values without assuming higher-order structure, as in standard reinforcement learning. To discriminate between these possibilities we scanned human subjects with fMRI while they performed two different decision making tasks with higher-order structure: probabilistic reversal learning, in which subjects had to infer which of two choices was the more rewarding and then flexibly switch their choice when contingencies changed; and the inspection game, in which subjects had to successfully compete against an intelligent adversary by mentalizing the opponent's state of mind in order to anticipate the opponent's behavior in future. For both tasks we found that neural activity in a key decision making region: ventromedial prefrontal cortex, was more consistent with computational models that exploit higher-order structure, than with simple reinforcement learning. Moreover, in the social interaction game, subjects were found to employ a sophisticated strategy whereby they used knowledge of how their actions would influence the actions of their opponent to guide their choices. Specific computational signals required for the implementation of such a strategy were present in medial prefrontal cortex and superior temporal sulcus, providing insight into the basic computations underlying competitive strategic interactions. These results suggest that brain regions such as ventromedial prefrontal cortex employ an abstract model of task structure to guide behavioral choice, computations that may underlie the human capacity for complex social interactions and abstract strategizing. vi TABLE OF CONTENTS
Role of the frontal cortex in solving the exploration-exploitation trade-off
2010
While many electrophysiological recordings and computational modeling work have investigated the role of the frontal cortex in reinforcement learning (learning by trial-and-error to adapt action values), it is not yet clear how the brain flexibly regulates in a taskappropriate way crucial parameters of learning such as the learning rate and the exploration rate. In a previous work, we proposed a computational model based on the meta-learning theoretical framework where the frontal cortex extracts feedback signals 1) to update action values based on a reward prediction error; 2) to estimate the level of exploration based on the current reward average; 3) to select action based on this exploration rate. This model helped us draw a set of experimental predictions. Here we show a model-based analysis of single-unit recordings in the monkey prefrontal cortex so as to test these predictions. We found neural subpopulations activities consistent with these three functions. We also found global properties of the recorded neural ensemble -such as variations in spatial selectivity -which were predicted by our model. Such an approach, gathering computational modeling and neurophysiology, can help understand complex activities of neural ensembles related to decision making.
Neuroethology of decision-making
Current Opinion in …, 2012
A neuroethological approach to decision-making considers the effect of evolutionary pressures on neural circuits mediating choice. In this view, decision systems are expected to enhance fitness with respect to the local environment, and particularly efficient solutions to specific problems should be conserved, expanded, and repurposed to solve other problems. Here, we discuss basic prerequisites for a variety of decision systems from this viewpoint. We focus on two of the best-studied and most widely represented decision problems. First, we examine patch leaving, a prototype of environmentally based switching between action patterns. Second, we consider social information seeking, a process resembling foraging with search costs. We argue that while the specific neural solutions to these problems sometimes differ across species, both the problems themselves and the algorithms instantiated by biological hardware are repeated widely throughout nature. The behavioral and mathematical study of ubiquitous decision processes like patch leaving and social information seeking thus provides a powerful new approach to uncovering the fundamental design structure of nervous systems.
Building bridges between neural models and complex decision making behaviour
Neural Networks, 2006
Diffusion processes, and their discrete time counterparts, random walk models, have demonstrated an ability to account for a wide range of findings from behavioural decision making for which the purely algebraic and deterministic models often used in economics and psychology cannot account. Recent studies that record neural activations in non-human primates during perceptual decision making tasks have revealed that neural firing rates closely mimic the accumulation of preference theorized by behaviourally-derived diffusion models of decision making.