Learning the value of information in an uncertain world (original) (raw)

References

1. Ernst, M.O. & Banks, M.S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).
   Article CAS PubMed Google Scholar
2. Kording, K.P. & Wolpert, D.M. Bayesian integration in sensorimotor learning. Nature 427, 244–247 (2004).
   Article PubMed Google Scholar
3. Kahneman, D. & Tversky, A. Choices, Values and Frames (Cambridge University Press, Cambridge, 2000).
   Book Google Scholar
4. Montague, P.R., Dayan, P., Person, C. & Sejnowski, T.J. Bee foraging in uncertain environments using predictive hebbian learning. Nature 377, 725–728 (1995).
   Article CAS PubMed Google Scholar
5. Samejima, K., Ueda, Y., Doya, K. & Kimura, M. Representation of action-specific reward values in the striatum. Science 310, 1337–1340 (2005).
   Article CAS PubMed Google Scholar
6. Daw, N.D., O'Doherty, J.P., Dayan, P., Seymour, B. & Dolan, R.J. Cortical substrates for exploratory decisions in humans. Nature 441, 876–879 (2006).
   Article CAS PubMed PubMed Central Google Scholar
7. Bayer, H.M. & Glimcher, P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).
   Article CAS PubMed PubMed Central Google Scholar
8. Rescorla, R.A. & Wagner, A.R. in Classical Conditioning II: Current Research and Theory (eds. Black, A.H. & Prokasy, W.F.) 64–99 (Appleton-Century Crofts, New York, 1972).
   Google Scholar
9. Sutton, R.S. & Barto, A.G. Reinforcement Learning: an Introduction (MIT Press, Cambridge, Massachusetts, 1998).
   Google Scholar
10. Dayan, P., Kakade, S. & Montague, P.R. Learning and selective attention. Nat. Neurosci. 3 Suppl, 1218–1223 (2000).
    Article CAS PubMed Google Scholar
11. Doya, K. Metalearning and neuromodulation. Neural Netw. 15, 495–506 (2002).
    Article PubMed Google Scholar
12. Pearce, J.M. & Hall, G. A model for Pavlovian learning: variations in the effectiveness of conditioned, but not of unconditioned, stimuli. Psychol. Rev. 87, 532–552 (1980).
    Article CAS PubMed Google Scholar
13. Dickinson, A. & Mackintosh, N.J. Classical conditioning in animals. Annu. Rev. Psychol. 29, 587–612 (1978).
    Article CAS PubMed Google Scholar
14. Cox, R.T. Probability, frequency and reasonable expectaion. Am. J. Phys. 14, 1–13 (1946).
    Article Google Scholar
15. Kakade, S. & Dayan, P. Acquisition and extinction in autoshaping. Psychol. Rev. 109, 533–544 (2002).
    Article PubMed Google Scholar
16. Courville, A.C., Daw, N.D. & Touretzky, D.S. Bayesian theories of conditioning in a changing world. Trends Cogn. Sci. 10, 294–300 (2006).
    Article PubMed Google Scholar
17. Yu, A.J. & Dayan, P. Uncertainty, neuromodulation and attention. Neuron 46, 681–692 (2005).
    Article CAS PubMed Google Scholar
18. Sugrue, L.P., Corrado, G.S. & Newsome, W.T. Matching behavior and the representation of value in the parietal cortex. Science 304, 1782–1787 (2004).
    Article CAS PubMed Google Scholar
19. Kennerley, S.W., Walton, M.E., Behrens, T.E., Buckley, M.J. & Rushworth, M.F. Optimal decision making and the anterior cingulate cortex. Nat. Neurosci. 9, 940–947 (2006).
    Article CAS PubMed Google Scholar
20. Gallistel, C.R., Mark, T.A., King, A.P. & Latham, P.E. The rat approximates an ideal detector of changes in rates of reward: implications for the law of effect. J. Exp. Psychol. Anim. Behav. Process. 27, 354–372 (2001).
    Article CAS PubMed Google Scholar
21. Procyk, E., Tanaka, Y.L. & Joseph, J.P. Anterior cingulate activity during routine and nonroutine sequential behaviors in macaques. Nat. Neurosci. 3, 502–508 (2000).
    Article CAS PubMed Google Scholar
22. Walton, M.E., Devlin, J.T. & Rushworth, M.F. Interactions between decision making and performance monitoring within prefrontal cortex. Nat. Neurosci. 7, 1259–1265 (2004).
    Article CAS PubMed Google Scholar
23. Niki, H. & Watanabe, M. Prefrontal and cingulate unit activity during timing behavior in the monkey. Brain Res. 171, 213–224 (1979).
    Article CAS PubMed Google Scholar
24. Ullsperger, M. & von Cramon, D.Y. Error monitoring using external feedback: specific roles of the habenular complex, the reward system and the cingulate motor area revealed by functional magnetic resonance imaging. J. Neurosci. 23, 4308–4314 (2003).
    Article CAS PubMed PubMed Central Google Scholar
25. Brown, J.W. & Braver, T.S. Learned predictions of error likelihood in the anterior cingulate cortex. Science 307, 1118–1121 (2005).
    Article CAS PubMed Google Scholar
26. Ito, S., Stuphorn, V., Brown, J.W. & Schall, J.D. Performance monitoring by the anterior cingulate cortex during saccade countermanding. Science 302, 120–122 (2003).
    Article CAS PubMed Google Scholar
27. Matsumoto, K., Suzuki, W. & Tanaka, K. Neuronal correlates of goal-based motor selection in the prefrontal cortex. Science 301, 229–232 (2003).
    Article CAS PubMed Google Scholar
28. Smith, S.M. et al. Advances in functional and structural MR image analysis and implementation as FSL. Neuroimage 23 Suppl 1, S208–S219 (2004).
    Article PubMed Google Scholar
29. Koechlin, E., Ody, C. & Kouneiher, F. The architecture of cognitive control in the human prefrontal cortex. Science 302, 1181–1185 (2003).
    Article CAS PubMed Google Scholar
30. Strick, P.L., Dum, R.P. & Picard, N. Motor areas on the medial wall of the hemisphere. Novartis Found Symp. 218, 64–75; discussion 75–80, 104–8 (1998).
    CAS PubMed Google Scholar
31. Van Hoesen, G.W., Morecraft, R.J. & Vogt, B.A. in Neurobiology of Cingulate Cortex and Limbic Thalamus (eds. Vogt, B.A. & Gabriel, M.) (Birkhauser, Boston, 1993).
    Google Scholar
32. McCoy, A.N., Crowley, J.C., Haghighian, G., Dean, H.L. & Platt, M.L. Saccade reward signals in posterior cingulate cortex. Neuron 40, 1031–1040 (2003).
    Article CAS PubMed Google Scholar
33. McCoy, A.N. & Platt, M.L. Risk-sensitive neurons in macaque posterior cingulate cortex. Nat. Neurosci. 8, 1220–1227 (2005).
    Article CAS PubMed Google Scholar
34. Fiorillo, C.D., Tobler, P.N. & Schultz, W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902 (2003).
    Article CAS PubMed Google Scholar
35. Preuschoff, K., Bossaerts, P. & Quartz, S.R. Neural differentiation of expected reward and risk in human subcortical structures. Neuron 51, 381–390 (2006).
    Article CAS PubMed Google Scholar
36. Aston-Jones, G. & Cohen, J.D. An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 28, 403–450 (2005).
    Article CAS PubMed Google Scholar
37. Engle, R.F. Autoregressive conditional Heteroscedasticity with estimates of the variance of UK inflation. Econometrica 50, 987–1008 (1982).
    Article Google Scholar
38. Waelti, P., Dickinson, A. & Schultz, W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412, 43–48 (2001).
    Article CAS PubMed Google Scholar
39. O'Doherty, J. et al. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452–454 (2004).
    Article CAS PubMed Google Scholar
40. Haruno, M. et al. A neural correlate of reward-based behavioral learning in caudate nucleus: a functional magnetic resonance imaging study of a stochastic decision task. J. Neurosci. 24, 1660–1665 (2004).
    Article CAS PubMed PubMed Central Google Scholar
41. Tanaka, S.C. et al. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat. Neurosci. 7, 887–893 (2004).
    Article CAS PubMed Google Scholar
42. Kunishio, K. & Haber, S.N. Primate cingulostriatal projection: limbic striatal versus sensorimotor striatal input. J. Comp. Neurol. 350, 337–356 (1994).
    Article CAS PubMed Google Scholar
43. Amiez, C., Joseph, J.P. & Procyk, E. Reward encoding in the monkey anterior cingulate cortex. Cereb. Cortex 16, 1040–1055 (2006).
    Article CAS PubMed Google Scholar
44. Yoshida, W. & Ishii, S. Resolution of uncertainty in prefrontal cortex. Neuron 50, 781–789 (2006).
    Article CAS PubMed Google Scholar
45. Fitzgerald, K.D. et al. Error-related hyperactivity of the anterior cingulate cortex in obsessive-compulsive disorder. Biol. Psychiatry 57, 287–294 (2005).
    Article PubMed Google Scholar
46. Critchley, H.D., Mathias, C.J. & Dolan, R.J. Neural activity in the human brain relating to uncertainty and arousal during anticipation. Neuron 29, 537–545 (2001).
    Article CAS PubMed Google Scholar
47. Botvinick, M.M., Cohen, J.D. & Carter, C.S. Conflict monitoring and anterior cingulate cortex: an update. Trends Cogn. Sci. 8, 539–546 (2004).
    Article PubMed Google Scholar
48. Rushworth, M.F., Buckley, M.J., Behrens, T.E., Walton, M.E. & Bannerman, D.M. Functional organization of the medial frontal cortex. Curr. Opin. Neurobiol. 17, 220–227 (2007).
    Article CAS PubMed Google Scholar
49. Hampton, A.N., Bossaerts, P. & O'Doherty, J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J. Neurosci. 26, 8360–8367 (2006).
    Article CAS PubMed PubMed Central Google Scholar
50. Preuschoff, K. & Bossaerts, P. Adding prediction risk to the theory of reward learning. Ann. N Y Acad. Sci. 1104, 135–146 (2007).
    Article PubMed Google Scholar

Download references