A framework for studying the neurobiology of value-based decision making (original) (raw)

References

1. Busemeyer, J. R. & Johnson, J. G. in Handbook of Judgment and Decision Making (eds Koehler, D. & Narvey, N.) 133–154 (Blackwell Publishing Co., New York, 2004).
   Book Google Scholar
2. Mas-Colell, A., Whinston, M. & Green, J. Microeconomic Theory (Cambridge Univ. Press, Cambridge, 1995).
   Google Scholar
3. Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction (MIT Press, Cambridge, Massachusetts, 1998).
   Google Scholar
4. Dickison, A. & Balleine, B. W. in Steven's Handbook of Experimental Psychology Vol. 3 Learning, Motivation & Emotion (ed. Gallistel, C.) 497–533 (Wiley & Sons, New York, 2002).
   Google Scholar
5. Dayan, P. in Better Than Conscious? Implications for Performance and Institutional Analysis (eds Engel, C. & Singer, W.) 51–70 (MIT Press, Cambridge, Massachusetts, 2008).
   Google Scholar
6. Balleine, B. W., Daw, N. & O'Doherty, J. in Neuroeconomics: Decision-Making and the Brain (eds Glimcher, P. W., Fehr, E., Camerer, C. & Poldrack, R. A.) 365–385 (Elsevier, New York, 2008).
   Google Scholar
7. Bouton, M. E. Learning and Behavior: A Contemporary Synthesis (Sinauer Associates, Inc., Sunderland, Massachusetts, 2007). This book reviews a large amount of evidence pointing to multiple valuation systems being active in value-based decision making.
   Google Scholar
8. Dayan, P. & Seymour, B. in Neuroeconomics: Decision Making and the Brain (eds Glimcher, P. W., Camerer, C. F., Fehr, E. & Poldrack, R. A.) 175–191 (Elsevier, New York, 2008).
   Google Scholar
9. Dayan, P., Niv, Y., Seymour, B. & Daw, N. D. The misbehavior of value and the discipline of the will. Neural Netw. 19, 1153–1160 (2006). This paper provided several models of how “pathological behaviours” can arise from the competition process between Pavlovian, habitual and goal-directed valuation systems.
   Article PubMed Google Scholar
10. Keay, K. A. & Bandler, R. Parallel circuits mediating distinct emotional coping reactions to different types of stress. Neurosci. Biobehav. Rev. 25, 669–678 (2001).
    Article CAS PubMed Google Scholar
11. Cardinal, R. N., Parkinson, J. A., Hall, J. & Everitt, B. J. Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neurosci. Biobehav. Rev. 26, 321–352 (2002).
    Article PubMed Google Scholar
12. Holland, P. C. & Gallagher, M. Amygdala-frontal interactions and reward expectancy. Curr. Opin. Neurobiol. 14, 148–155 (2004).
    Article CAS PubMed Google Scholar
13. Fendt, M. & Fanselow, M. S. The neuroanatomical and neurochemical basis of conditioned fear. Neurosci. Biobehav. Rev. 23, 743–760 (1999).
    Article CAS PubMed Google Scholar
14. Adams, D. B. Brain mechanisms of aggressive behavior: an updated review. Neurosci. Biobehav. Rev. 30, 304–318 (2006).
    Article PubMed Google Scholar
15. Niv, Y. in Neuroscience (Hebrew University, Jerusalem, 2007).
    Google Scholar
16. Dayan, P. & Abbott, L. R. Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems (MIT Press, Cambridge, Massachusetts, 1999).
    Google Scholar
17. Balleine, B. W. Neural bases of food-seeking: affect, arousal and reward in corticostriatolimbic circuits. Physiol. Behav. 86, 717–730 (2005). This important paper reviews a large amount of evidence pointing to multiple valuation systems being active in value-based decision making.
    Article CAS PubMed Google Scholar
18. Yin, H. H. & Knowlton, B. J. The role of the basal ganglia in habit formation. Nature Rev. Neurosci. 7, 464–476 (2006).
    Article CAS Google Scholar
19. Killcross, S. & Coutureau, E. Coordination of actions and habits in the medial prefrontal cortex of rats. Cereb. Cortex 13, 400–408 (2003).
    Article PubMed Google Scholar
20. Coutureau, E. & Killcross, S. Inactivation of the infralimbic prefrontal cortex reinstates goal-directed responding in overtrained rats. Behav. Brain Res. 146, 167–174 (2003).
    Article PubMed Google Scholar
21. Yin, H. H., Knowlton, B. J. & Balleine, B. W. Blockade of NMDA receptors in the dorsomedial striatum prevents action-outcome learning in instrumental conditioning. Eur. J. Neurosci. 22, 505–512 (2005).
    Article PubMed Google Scholar
22. Wallis, J. D. & Miller, E. K. Neuronal activity in primate dorsolateral and orbital prefrontal cortex during performance of a reward preference task. Eur. J. Neurosci. 18, 2069–2081 (2003).
    Article PubMed Google Scholar
23. Padoa-Schioppa, C. & Assad, J. A. Neurons in the orbitofrontal cortex encode economic value. Nature 441, 223–226 (2006). This paper showed that neurons in the monkey OFC encode the goal value of individual rewarding objects (for example, different liquids) irrespective of the action that needs to be taken to obtain them.
    Article CAS PubMed PubMed Central Google Scholar
24. Wallis, J. D. Orbitofrontal cortex and its contribution to decision-making. Annu. Rev. Neurosci. 30, 31–56 (2007).
    Article CAS PubMed Google Scholar
25. Barraclough, D. J., Conroy, M. L. & Lee, D. Prefrontal cortex and decision making in a mixed-strategy game. Nature Neurosci. 7, 404–410 (2004).
    Article CAS PubMed Google Scholar
26. Schoenbaum, G. & Roesch, M. Orbitofrontal cortex, associative learning, and expectancies. Neuron 47, 633–636 (2005).
    Article CAS PubMed PubMed Central Google Scholar
27. Tom, S. M., Fox, C. R., Trepel, C. & Poldrack, R. A. The neural basis of loss aversion in decision-making under risk. Science 315, 515–518 (2007). This fMRI study showed that the striatal-OFC network encodes a value signal at the time of the goal-directed choice that is consistent with the properties of PT. Furthermore, the study presented evidence that suggests that both the appetitive and the aversive aspects of goal-directed decisions might be encoded in a common valuation network.
    Article CAS PubMed Google Scholar
28. Plassmann, H., O'Doherty, J. & Rangel, A. Orbitofrontal cortex encodes willingness to pay in everyday economic transactions. J. Neurosci. 27, 9984–9988 (2007).
    Article CAS PubMed PubMed Central Google Scholar
29. Hare, T., O'Doherty, J., Camerer, C. F., Schultz, W. & Rangel, A. Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J. Neurosci. (in the press).
30. Paulus, M. P. & Frank, L. R. Ventromedial prefrontal cortex activation is critical for preference judgments. Neuroreport 14, 1311–1315 (2003).
    PubMed Google Scholar
31. Erk, S., Spitzer, M., Wunderlich, A. P., Galley, L. & Walter, H. Cultural objects modulate reward circuitry. Neuroreport 13, 2499–2503 (2002).
    Article PubMed Google Scholar
32. Fellows, L. K. & Farah, M. J. The role of ventromedial prefrontal cortex in decision making: judgment under uncertainty or judgment per se? Cereb. Cortex 17, 2669–2674 (2007).
    Article PubMed Google Scholar
33. Lengyel, M. & Dayan, P. Hippocampal contributions to control: the third way. NIPS [online] (2007).
    Google Scholar
34. Montague, P. R. Why Choose This Book? (Dutton, 2006).
    Google Scholar
35. Fehr, E. & Camerer, C. F. Social neuroeconomics: the neural circuitry of social preferences. Trends Cogn. Sci. 11, 419–427 (2007).
    Article PubMed Google Scholar
36. Lee, D. Game theory and neural basis of social decision making. Nature Neurosci. 11, 404–409 (2008).
    Article CAS PubMed Google Scholar
37. Platt, M. L. & Huettel, S. A. Risky business: the neuroeconomics of decision making under uncertainty. Nature Neurosci. 11, 398–403 (2008).
    Article CAS PubMed Google Scholar
38. Paulus, M. P., Rogalsky, C., Simmons, A., Feinstein, J. S. & Stein, M. B. Increased activation in the right insula during risk-taking decision making is related to harm avoidance and neuroticism. Neuroimage 19, 1439–1448 (2003).
    Article PubMed Google Scholar
39. Leland, D. S. & Paulus, M. P. Increased risk-taking decision-making but not altered response to punishment in stimulant-using young adults. Drug Alcohol Depend. 78, 83–90 (2005).
    Article PubMed Google Scholar
40. Paulus, M. P. et al. Prefrontal, parietal, and temporal cortex networks underlie decision-making in the presence of uncertainty. Neuroimage 13, 91–100 (2001).
    Article CAS PubMed Google Scholar
41. Huettel, S. A., Song, A. W. & McCarthy, G. Decisions under uncertainty: probabilistic context influences activation of prefrontal and parietal cortices. J. Neurosci. 25, 3304–3311 (2005).
    Article CAS PubMed PubMed Central Google Scholar
42. Bossaerts, P. & Hsu, M. in Neuroeconomics: Decision Making and the Brain (eds Glimcher, P. W., Camerer, C. F., Fehr, E. & Poldrack, R. A.) 351–364 (Elsevier, New York, 2008).
    Google Scholar
43. Preuschoff, K. & Bossaerts, P. Adding prediction risk to the theory of reward learning. Ann. NY Acad. Sci. 1104, 135–146 (2007).
    Article PubMed Google Scholar
44. Preuschoff, K., Bossaerts, P. & Quartz, S. R. Neural differentiation of expected reward and risk in human subcortical structures. Neuron 51, 381–390 (2006).
    Article CAS PubMed Google Scholar
45. Tobler, P. N., O'Doherty, J. P., Dolan, R. J. & Schultz, W. Reward value coding distinct from risk attitude-related uncertainty coding in human reward systems. J. Neurophysiol. 97, 1621–1632 (2007).
    Article PubMed Google Scholar
46. Rolls, E. T., McCabe, C. & Redoute, J. Expected value, reward outcome, and temporal difference error representations in a probabilistic decision task. Cereb. Cortex 18, 652–663 (2007).
    Article PubMed Google Scholar
47. Dreher, J. C., Kohn, P. & Berman, K. F. Neural coding of distinct statistical properties of reward information in humans. Cereb. Cortex 16, 561–573 (2006).
    Article PubMed Google Scholar
48. Preuschoff, K., Quartz, S. R. & Bossaerts, P. Human insula activation reflects prediction errors as well as risk. J. Neurosci. 28, 2745–2752 (2008). This fMRI study shows that the human insula encodes risk-prediction errors that could be used to learn the riskiness of different options and that are complementary to reward-prediction errors.
    Article CAS PubMed PubMed Central Google Scholar
49. Tobler, P. N., Fiorillo, C. D. & Schultz, W. Adaptive coding of reward value by dopamine neurons. Science 307, 1642–1645 (2005).
    Article CAS PubMed Google Scholar
50. Platt, M. L. & Glimcher, P. W. Neural correlates of decision variables in parietal cortex. Nature 400, 233–238 (1999).
    Article CAS PubMed Google Scholar
51. Camerer, C. F. & Weber, M. Recent developments in modelling preferences: uncertainty and ambiguity. J. Risk Uncertain. 5, 325–370 (1992).
    Article Google Scholar
52. Hsu, M., Bhatt, M., Adolphs, R., Tranel, D. & Camerer, C. F. Neural systems responding to degrees of uncertainty in human decision-making. Science 310, 1680–1683 (2005).
    Article CAS PubMed Google Scholar
53. Huettel, S. A., Stowe, C. J., Gordon, E. M., Warner, B. T. & Platt, M. L. Neural signatures of economic preferences for risk and ambiguity. Neuron 49, 765–775 (2006).
    Article CAS PubMed Google Scholar
54. Hertwig, R., Barron, G., Weber, E. U. & Erev, I. Decisions from experience and the effect of rare events in risky choice. Psychol. Sci. 15, 534–539 (2004).
    Article PubMed Google Scholar
55. Weller, J. A., Levin, I. P., Shiv, B. & Bechara, A. Neural correlates of adaptive decision making for risky gains and losses. Psychol. Sci. 18, 958–964 (2007).
    Article PubMed Google Scholar
56. De Martino, B., Kumaran, D., Seymour, B. & Dolan, R. J. Frames, biases, and rational decision-making in the human brain. Science 313, 684–687 (2006).
    Article CAS PubMed PubMed Central Google Scholar
57. Frederick, S., Loewenstein, G. & O'Donoghue, T. Time discounting and time preference: a critical review. J. Econ. Lit. 40, 351–401 (2002).
    Article Google Scholar
58. McClure, S. M., Laibson, D. I., Loewenstein, G. & Cohen, J. D. Separate neural systems value immediate and delayed monetary rewards. Science 306, 503–507 (2004). This fMRI study argued that competing goal-directed valuation systems play a part in decisions that involve choosing between immediate small monetary payoffs and larger but delayed payoffs.
    Article CAS PubMed Google Scholar
59. McClure, S. M., Ericson, K. M., Laibson, D. I., Loewenstein, G. & Cohen, J. D. Time discounting for primary rewards. J. Neurosci. 27, 5796–5804 (2007).
    Article CAS PubMed PubMed Central Google Scholar
60. Berns, G. S., Laibson, D. & Loewenstein, G. Intertemporal choice - toward an integrative framework. Trends Cogn. Sci. 11, 482–488 (2007).
    Article PubMed Google Scholar
61. Kable, J. W. & Glimcher, P. W. The neural correlates of subjective value during intertemporal choice. Nature Neurosci. 10, 1625–1633 (2007). This fMRI study argued that a single goal-directed valuation system plays a part in decisions that involve choosing between immediate small monetary payoffs and larger but delayed payoffs.
    Article CAS PubMed Google Scholar
62. Read, D., Frederick, S., Orsel, B. & Rahman, J. Four score and seven years ago from now: the “date/delay” effect in temporal discounting. Manage. Sci. 51, 1326–1335 (1997).
    Article Google Scholar
63. Mischel, W. & Underwood, B. Instrumental ideation in delay of gratification. Child Dev. 45, 1083–1088 (1974).
    Article CAS PubMed Google Scholar
64. Wilson, M. & Daly, M. Do pretty women inspire men to discount the future? Proc. Biol. Sci. 271 (Suppl 4), S177–S179 (2004).
    PubMed PubMed Central Google Scholar
65. Berns, G. S. et al. Neurobiological substrates of dread. Science 312, 754–758 (2006).
    Article CAS PubMed PubMed Central Google Scholar
66. Loewenstein, G. Anticipation and the valuation of delayed consumption. Econ. J. 97, 666–684 (1987).
    Article Google Scholar
67. Stevens, J. R., Hallinan, E. V. & Hauser, M. D. The ecology and evolution of patience in two New World monkeys. Biol. Lett. 1, 223–226 (2005).
    Article PubMed PubMed Central Google Scholar
68. Herrnstein, R. J. Relative and absolute strength of response as a function of frequency of reinforcement. J. Exp. Anal. Behav. 4, 267–272 (1961).
    Article CAS PubMed PubMed Central Google Scholar
69. Mazur, J. E. Estimation of indifference points with an adjusting-delay procedure. J. Exp. Anal. Behav. 49, 37–47 (1988).
    Article CAS PubMed PubMed Central Google Scholar
70. Corrado, G. S., Sugrue, L. P., Seung, H. S. & Newsome, W. T. Linear-nonlinear-poisson models of primate choice dynamics. J. Exp. Anal. Behav. 84, 581–617 (2005).
    Article PubMed PubMed Central Google Scholar
71. Newsome, W. T., Britten, K. H. & Movshon, J. A. Neuronal correlates of a perceptual decision. Nature 341, 52–54 (1989).
    Article CAS PubMed Google Scholar
72. Kim, J. N. & Shadlen, M. N. Neural correlates of a decision in the dorsolateral prefrontal cortex of the macaque. Nature Neurosci. 2, 176–185 (1999).
    Article PubMed Google Scholar
73. Gold, J. I. & Shadlen, M. N. The neural basis of decision making. Annu. Rev. Neurosci. 30, 535–574 (2007).
    Article CAS PubMed Google Scholar
74. Gold, J. I. & Shadlen, M. N. Banburisms and the brain: decoding the relationship between sensory stimuli, decisions, and reward. Neuron 36, 299–308 (2002).
    Article CAS PubMed Google Scholar
75. Gold, J. I. & Shadlen, M. N. Neural computations that underlie decisions about sensory stimuli. Trends Cogn. Sci. 5, 10–16 (2001).
    Article PubMed Google Scholar
76. Heekeren, H. R., Marrett, S. & Ungerleider, L. G. The neural systems that mediate human perceptual decision making. Nature Rev. Neurosci. 9, 467–479 (2008).
    Article CAS Google Scholar
77. Daw, N. D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neurosci. 8, 1704–1711 (2005). This paper proposed a theoretical model of how the brain might assign control to the different goal and habitual systems.
    Article CAS PubMed Google Scholar
78. Frank, M. J., Seeberger, L. C. & O'Reilly, R. C. By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 306, 1940–1943 (2004).
    Article CAS PubMed Google Scholar
79. Frank, M. J. Hold your horses: a dynamic computational role for the subthalamic nucleus in decision making. Neural Netw. 19, 1120–1136 (2006).
    Article PubMed Google Scholar
80. de Araujo, I. E., Rolls, E. T., Kringelbach, M. L., McGlone, F. & Phillips, N. Taste-olfactory convergence, and the representation of the pleasantness of flavour, in the human brain. Eur. J. Neurosci. 18, 2059–2068 (2003).
    Article PubMed Google Scholar
81. de Araujo, I. E., Kringelbach, M. L., Rolls, E. T. & McGlone, F. Human cortical responses to water in the mouth, and the effects of thirst. J. Neurophysiol. 90, 1865–1876 (2003).
    Article PubMed Google Scholar
82. Anderson, A. K. et al. Dissociated neural representations of intensity and valence in human olfaction. Nature Neurosci. 6, 196–202 (2003).
    Article CAS PubMed Google Scholar
83. de Araujo, I. E., Rolls, E. T., Velazco, M. I., Margot, C. & Cayeux, I. Cognitive modulation of olfactory processing. Neuron 46, 671–679 (2005).
    Article CAS PubMed Google Scholar
84. McClure, S. M. et al. Neural correlates of behavioral preference for culturally familiar drinks. Neuron 44, 379–387 (2004).
    Article CAS PubMed Google Scholar
85. Kringelbach, M. L., O'Doherty, J., Rolls, E. T. & Andrews, C. Activation of the human orbitofrontal cortex to a liquid food stimulus is correlated with its subjective pleasantness. Cereb. Cortex 13, 1064–1071 (2003).
    Article CAS PubMed Google Scholar
86. Small, D. M. et al. Dissociation of neural representation of intensity and affective valuation in human gustation. Neuron 39, 701–711 (2003).
    Article CAS PubMed Google Scholar
87. Blood, A. J. & Zatorre, R. J. Intensely pleasurable responses to music correlate with activity in brain regions implicated in reward and emotion. Proc. Natl Acad. Sci. USA 98, 11818–11823 (2001).
    Article CAS PubMed PubMed Central Google Scholar
88. O'Doherty, J. et al. Sensory-specific satiety-related olfactory activation of the human orbitofrontal cortex. Neuroreport 11, 399–403 (2000).
    Article CAS PubMed Google Scholar
89. Small, D. M., Zatorre, R. J., Dagher, A., Evans, A. C. & Jones-Gotman, M. Changes in brain activity related to eating chocolate: from pleasure to aversion. Brain 124, 1720–1733 (2001).
    Article CAS PubMed Google Scholar
90. Breiter, H. C., Aharon, I., Kahneman, D., Dale, A. & Shizgal, P. Functional imaging of neural responses to expectancy and experience of monetary gains and losses. Neuron 30, 619–639 (2001).
    Article CAS PubMed Google Scholar
91. Knutson, B., Fong, G. W., Adams, C. M., Varner, J. L. & Hommer, D. Dissociation of reward anticipation and outcome with event-related fMRI. Neuroreport 12, 3683–3687 (2001).
    Article CAS PubMed Google Scholar
92. Zink, C. F., Pagnoni, G., Martin-Skurski, M. E., Chappelow, J. C. & Berns, G. S. Human striatal responses to monetary reward depend on saliency. Neuron 42, 509–517 (2004).
    Article CAS PubMed Google Scholar
93. Peyron, R. et al. Haemodynamic brain responses to acute pain in humans: sensory and attentional networks. Brain 122, 1765–1780 (1999).
    Article PubMed Google Scholar
94. Davis, K. D., Taylor, S. J., Crawley, A. P., Wood, M. L. & Mikulis, D. J. Functional MRI of pain- and attention-related activations in the human cingulate cortex. J. Neurophysiol. 77, 3370–3380 (1997).
    Article CAS PubMed Google Scholar
95. Seo, H. & Lee, D. Temporal filtering of reward signals in the dorsal anterior cingulate cortex during a mixed-strategy game. J. Neurosci. 27, 8366–8377 (2007).
    Article CAS PubMed PubMed Central Google Scholar
96. Pecina, S., Smith, K. S. & Berridge, K. C. Hedonic hot spots in the brain. Neuroscientist 12, 500–511 (2006).
    Article PubMed Google Scholar
97. Berridge, K. C. & Robinson, T. E. Parsing reward. Trends Neurosci. 26, 507–513 (2003).
    Article CAS PubMed Google Scholar
98. Berridge, K. C. Pleasures of the brain. Brain Cogn. 52, 106–128 (2003).
    Article PubMed Google Scholar
99. Plassmann, H., O'Doherty, J., Shiv, B. & Rangel, A. Marketing actions can modulate neural representations of experienced pleasantness. Proc. Natl Acad. Sci. USA 105, 1050–1054 (2008). This paper showed that the level of “experienced pleasantness” encoded in the medial OFC at the time of consuming a wine is modulated by subjects' beliefs about the price of the wine that they are drinking.
    Article CAS PubMed PubMed Central Google Scholar
100. Montague, P. R., King-Casas, B. & Cohen, J. D. Imaging valuation models in human choice. Annu. Rev. Neurosci. 29, 417–448 (2006).
     Article CAS PubMed Google Scholar
101. Tremblay, L., Hollerman, J. R. & Schultz, W. Modifications of reward expectation-related neuronal activity during learning in primate striatum. J. Neurophysiol. 80, 964–977 (1998).
     Article CAS PubMed Google Scholar
102. Hollerman, J. R. & Schultz, W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neurosci. 1, 304–309 (1998).
     Article CAS PubMed Google Scholar
103. Schultz, W., Dayan, P. & Montague, P. R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997). This seminal paper proposed the connection between the prediction-error component of reinforcement-learning models and the behaviour of dopamine cells.
     Article CAS PubMed Google Scholar
104. Mirenowicz, J. & Schultz, W. Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 72, 1024–1027 (1994).
     Article CAS PubMed Google Scholar
105. Schultz, W. Multiple dopamine functions at different time courses. Annu. Rev. Neurosci. 30, 259–288 (2007).
     Article CAS PubMed Google Scholar
106. Schultz, W. Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology. Curr. Opin. Neurobiol. 14, 139–147 (2004).
     Article CAS PubMed Google Scholar
107. Montague, P. R., Dayan, P. & Sejnowski, T. J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).
     Article CAS PubMed PubMed Central Google Scholar
108. Yacubian, J. et al. Dissociable systems for gain- and loss-related value predictions and errors of prediction in the human brain. J. Neurosci. 26, 9530–9537 (2006).
     Article CAS PubMed PubMed Central Google Scholar
109. Tanaka, S. C. et al. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nature Neurosci. 7, 887–893 (2004).
     Article CAS PubMed Google Scholar
110. Pagnoni, G., Zink, C. F., Montague, P. R. & Berns, G. S. Activity in human ventral striatum locked to errors of reward prediction. Nature Neurosci. 5, 97–98 (2002).
     Article CAS PubMed Google Scholar
111. O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H. & Dolan, R. J. Temporal difference models and reward-related learning in the human brain. Neuron 38, 329–337 (2003).
     Article CAS PubMed Google Scholar
112. Knutson, B., Westdorp, A., Kaiser, E. & Hommer, D. fMRI visualization of brain activity during a monetary incentive delay task. Neuroimage 12, 20–27 (2000).
     Article CAS PubMed Google Scholar
113. Delgado, M. R., Nystrom, L. E., Fissell, C., Noll, D. C. & Fiez, J. A. Tracking the hemodynamic responses to reward and punishment in the striatum. J. Neurophysiol. 84, 3072–3077 (2000).
     Article CAS PubMed Google Scholar
114. Bayer, H. M. & Glimcher, P. W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).
     Article CAS PubMed PubMed Central Google Scholar
115. Bayer, H. M., Lau, B. & Glimcher, P. W. Statistics of midbrain dopamine neuron spike trains in the awake primate. J. Neurophysiol. 98, 1428–1439 (2007).
     Article PubMed Google Scholar
116. Seymour, B., Daw, N., Dayan, P., Singer, T. & Dolan, R. Differential encoding of losses and gains in the human striatum. J. Neurosci. 27, 4826–4831 (2007).
     Article CAS PubMed PubMed Central Google Scholar
117. Daw, N. D., Kakade, S. & Dayan, P. Opponent interactions between serotonin and dopamine. Neural Netw. 15, 603–616 (2002).
     Article PubMed Google Scholar
118. Lohrenz, T., McCabe, K., Camerer, C. F. & Montague, P. R. Neural signature of fictive learning signals in a sequential investment task. Proc. Natl Acad. Sci. USA 104, 9493–9498 (2007).
     Article CAS PubMed PubMed Central Google Scholar
119. Camerer, C. F. & Chong, J. K. Self-tuning experience weighted attraction learning in games. J. Econ. Theory 133, 177–198 (2007).
     Article Google Scholar
120. Olsson, A. & Phelps, E. A. Social learning of fear. Nature Neurosci. 10, 1095–1102 (2007).
     Article CAS PubMed Google Scholar
121. Montague, P. R. et al. Dynamic gain control of dopamine delivery in freely moving animals. J. Neurosci. 24, 1754–1759 (2004).
     Article CAS PubMed PubMed Central Google Scholar
122. Tversky, A. & Kahneman, D. Advances in prospect theory cumulative representation of uncertainty. J. Risk Uncertain. 5, 297–323 (1992).
     Article Google Scholar
123. Kahneman, D. & Tversky, A. Prospect Theory: an analysis of decision under risk. Econometrica 4, 263–291 (1979). This seminal paper proposed the PT model for goal-directed valuation in the presence of risk and provided some supporting evidence. It is one of the most cited papers in economics.
     Article Google Scholar
124. Chen, K., Lakshminarayanan, V. & Santos, L. How basic are behavioral biases? Evidence from capuchin-monkey trading behavior. J. Polit. Econ. 114, 517–537 (2006).
     Article Google Scholar
125. Camerer, C. F. in Choice, Values, and Frames (eds Kahneman, D. & Tversky, A.) (Cambridge Univ. Press, Cambridge, 2000).
     Google Scholar
126. Gilboa, I. & Schmeidler, D. Maxmin expected utility with non-unique prior. J. Math. Econ. 28, 141–153 (1989).
     Article Google Scholar
127. Ghirardato, P., Maccheroni, F. & Marinacci, M. Differentiating ambiguity and ambiguity attitude. J. Econ. Theory 118, 133–173 (2004).
     Article Google Scholar
128. Nestler, E. J. & Charney, D. S. The Neurobiology of Mental Illness (Oxford Univ. Press, Oxford, 2004).
     Google Scholar
129. Kauer, J. A. & Malenka, R. C. Synaptic plasticity and addiction. Nature Rev. Neurosci. 8, 844–858 (2007).
     Article CAS Google Scholar
130. Hyman, S. E., Malenka, R. C. & Nestler, E. J. Neural mechanisms of addiction: the role of reward-related learning and memory. Annu. Rev. Neurosci. 29, 565–598 (2006).
     Article CAS PubMed Google Scholar
131. Redish, A. D. & Johnson, A. A computational model of craving and obsession. Ann. NY Acad. Sci. 1104, 324–339 (2007).
     Article PubMed Google Scholar
132. Redish, A. D. Addiction as a computational process gone awry. Science 306, 1944–1947 (2004). This paper showed how addiction can be conceptualized as a disease of the habit valuation system, using a simple modification of the reinforcement-learning model.
     Article CAS PubMed Google Scholar
133. Paulus, M. P. Decision-making dysfunctions in psychiatry—altered homeostatic processing? Science 318, 602–606 (2007).
     Article CAS PubMed Google Scholar
134. Miller, E. K. & Cohen, J. D. An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 24, 167–202 (2001).
     Article CAS PubMed Google Scholar
135. Hazy, T. E., Frank, M. J. & O'Reilly, R. C. Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system. Philos. Trans. R. Soc. Lond. B Biol. Sci. 362, 1601–1613 (2007).
     Article PubMed PubMed Central Google Scholar
136. Niv, Y., Joel, D. & Dayan, P. A normative perspective on motivation. Trends Cogn. Sci. 10, 375–381 (2006).
     Article PubMed Google Scholar

Download references