Multiple reward signals in the brain (original) (raw)

• Fibiger, H. C. & Phillips, A. G. in Handbook of Physiology—The Nervous System Vol. IV (ed. Bloom, F. E.) 647– 675 (Williams and Wilkins, Baltimore, Maryland, 1986 ).
  Google Scholar
• Wise, R. A. & Hoffman, D. C. Localization of drug reward mechanisms by intracranial injections. Synapse 10, 247–263 (1992).
  CAS PubMed Google Scholar
• Robinson, T. E. & Berridge, K. C. The neural basis for drug craving: an incentive-sensitization theory of addiction. Brain Res. Rev. 18, 247–291 (1993).
  CAS PubMed Google Scholar
• Robbins, T. W. & Everitt, B. J. Neurobehavioural mechanisms of reward and motivation. Curr. Opin. Neurobiol. 6, 228–236 (1996)
  CAS PubMed Google Scholar
• Louilot, A., LeMoal, M. & Simon, H. Differential reactivity of dopaminergic neurons in the nucleus accumbens in response to different behavioural situations. An in vivo voltammetric study in free moving rats. Brain Res. 397, 395–400 ( 1986).
  CAS PubMed Google Scholar
• Church, W. H. & Justice, J. B. Jr, Neill, D. B. Detecting behaviourally relevant changes in extracellular dopamine with microdialysis. Brain Res. 41, 397–399 (1987).
  Google Scholar
• Young, A. M. J., Joseph, M. H. & Gray, J. A. Increased dopamine release in vivo in nucleus accumbens and caudate nucleus of the rat during drinking: a microdialysis study. Neuroscience 48, 871– 876 (1992).
  CAS PubMed Google Scholar
• Wilson, C., Nomikos, G. G., Collu, M. & Fibiger, H. C. Dopaminergic correlates of motivated behaviour: importance of drive. J. Neurosci. 15, 5169–5178 (1995).
  CAS PubMed Google Scholar
• Fiorino, D. F., Coury, A. & Phillips, A. G. Dynamic changes in nucleus accumbens dopamine efflux during the Coolidge effect in male rats. J. Neurosci. 17, 4849–4855 (1997).
  CAS PubMed Google Scholar
• Thut, G. et al. Activation of the human brain by monetary reward. NeuroReport 8, 1225–1228 (1997).
  CAS PubMed Google Scholar
• Rogers, R. D. et al. Choosing between small, likely rewards and large, unlikely rewards activates inferior and orbital prefrontal cortex. J. Neurosci. 19, 9029–9038 ( 1999)
  CAS PubMed Google Scholar
• Elliott, R., Friston, K. J. & Dolan, R. J. Dissociable neural responses in human reward systems . J. Neurosci. 20, 6159– 6165 (2000)
  CAS PubMed Google Scholar
• Pavlov, P. I. Conditioned Reflexes (Oxford Univ. Press, London, 1927 ).
  Google Scholar
• Rescorla, R. A. & Wagner, A. R. in Classical Conditioning II: Current Research and Theory (eds Black, A. H. and Prokasy, W. F.) 64–99 (Appleton Century Crofts, New York, 1972).
  Google Scholar
• Mackintosh, N. J. A theory of attention: variations in the associability of stimulus with reinforcement . Psychol. Rev. 82, 276– 298 (1975).
  Google Scholar
• Pearce, J. M. & Hall, G. A model for Pavlovian conditioning: variations in the effectiveness of conditioned but not of unconditioned stimuli . Psychol. Rev. 87, 532– 552 (1980).
  CAS Google Scholar
• Schultz, W. Responses of midbrain dopamine neurons to behavioural trigger stimuli in the monkey. J. Neurophysiol. 56, 1439– 1462 (1986).
  CAS PubMed Google Scholar
• Romo, R. & Schultz, W. Dopamine neurons of the monkey midbrain: contingencies of responses to active touch during self-initiated arm movements . J. Neurophysiol. 63, 592– 606 (1990).
  CAS PubMed Google Scholar
• Schultz, W. & Romo, R. Dopamine neurons of the monkey midbrain: contingencies of responses to stimuli eliciting immediate behavioural reactions . J. Neurophysiol. 63, 607– 624 (1990).
  CAS PubMed Google Scholar
• Ljungberg, T., Apicella, P. & Schultz, W. Responses of monkey dopamine neurons during learning of behavioural reactions. J. Neurophysiol. 67, 145–163 (1992).
  CAS PubMed Google Scholar
• Strecker, R. E. & Jacobs, B. L. Substantia nigra dopaminergic unit activity in behaving cats: effect of arousal on spontaneous discharge and sensory evoked activity. Brain Res. 361 , 339–350 (1985).
  CAS PubMed Google Scholar
• Horvitz, J. C. Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96, 651– 656 (2000).
  CAS PubMed Google Scholar
• Mirenowicz, J. & Schultz, W. Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature 379, 449–451 ( 1996).
  CAS PubMed Google Scholar
• Guarraci, F. A. & Kapp, B. S. An electrophysiological characterization of ventral tegmental area dopaminergic neurons during differential Pavlovian fear conditioning in the awake rabbit. Behav. Brain Res. 99, 169–179 ( 1999).
  CAS PubMed Google Scholar
• Schultz, W. Activity of dopamine neurons in the behaving primate. Semin. Neurosci. 4, 129–138 ( 1992).
  Google Scholar
• Redgrave, P., Prescott, T. J. & Gurney, K. Is the short-latency dopamine response too short to signal reward? Trends Neurosci. 22, 146– 151 (1999).
  CAS PubMed Google Scholar
• Hollerman, J. R. & Schultz, W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neurosci. 1, 304–309 (1998).
  CAS PubMed Google Scholar
• Schultz, W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1–27 (1998 ).
  CAS PubMed Google Scholar
• Schultz, W. & Dickinson, A. Neuronal coding of prediction errors. Annu. Rev. Neurosci. 23, 473– 500 (2000).
  CAS PubMed Google Scholar
• Mirenowicz, J. & Schultz, W. Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 72, 1024–1027 (1994).
  CAS PubMed Google Scholar
• Schultz, W., Dayan, P. & Montague, R. R. A neural substrate of prediction and reward. Science 275, 1593–1599 ( 1997).
  CAS PubMed Google Scholar
• Montague, P. R., Dayan, P. & Sejnowski, T. J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).
  CAS PubMed Google Scholar
• Sutton, R. S. & Barto, A. G. Toward a modern theory of adaptive networks: expectation and prediction. Psychol. Rev. 88, 135–170 (1981). This paper proposed a very effective 'temporal difference' reinforcement learning model that computes a prediction error over time. The teaching signal incorporates primary reinforcers and conditioned stimuli, and resembles in all aspects the response of dopamine neurons to rewards and conditioned, reward-predicting stimuli, although dopamine neurons also report novel stimuli.
  CAS PubMed Google Scholar
• Tesauro, G. TD-Gammon, a self-teaching backgammon program, achieves master-level play . Neural Comp. 6, 215–219 (1994).
  Google Scholar
• Suri, R. E. & Schultz, W. Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Exp. Brain Res. 121, 350–354 (1998).
  CAS PubMed Google Scholar
• Suri, R. & Schultz, W. A neural network with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 91, 871–890 (1999).
  CAS PubMed Google Scholar
• Schultz, W. & Romo, R. Responses of nigrostriatal dopamine neurons to high intensity somatosensory stimulation in the anesthetized monkey . J. Neurophysiol. 57, 201– 217 (1987).
  CAS PubMed Google Scholar
• Abercrombie, E. D., Keefe, K. A., DiFrischia, D. S. & Zigmond, M. J. Differential effect of stress on in vivo dopamine release in striatum, nucleus accumbens, and medial frontal cortex. J. Neurochem. 52, 1655–1658 (1989).
  CAS PubMed Google Scholar
• Hikosaka, O., Sakamoto, M. & Usui, S. Functional properties of monkey caudate neurons. III. Activities related to expectation of target and reward. J. Neurophysiol. 61, 814–832 ( 1989).
  CAS PubMed Google Scholar
• Apicella, P., Ljungberg, T., Scarnati, E. & Schultz, W. Responses to reward in monkey dorsal and ventral striatum. Exp. Brain Res. 85, 491–500 ( 1991).
  CAS PubMed Google Scholar
• Apicella, P., Scarnati, E. & Schultz, W. Tonically discharging neurons of monkey striatum respond to preparatory and rewarding stimuli. Exp. Brain Res. 84, 672–675 (1991).
  CAS PubMed Google Scholar
• Lavoie, A. M. & Mizumori, S. J. Y. Spatial-, movement- and reward-sensitive discharge by medial ventral striatum neurons of rats. Brain Res. 638, 157–168 ( 1994).
  CAS PubMed Google Scholar
• Bowman, E. M., Aigner, T. G. & Richmond, B. J. Neural signals in the monkey ventral striatum related to motivation for juice and cocaine rewards. J. Neurophysiol. 75, 1061–1073 (1996).
  CAS PubMed Google Scholar
• Shidara, M., Aigner, T. G. & Richmond, B. J. Neuronal signals in the monkey ventral striatum related to progress through a predictable series of trials. J. Neurosci. 18, 2613–2625 ( 1998).
  CAS PubMed Google Scholar
• Matsumura, M., Kojima, J., Gardiner, T. W. & Hikosaka, O. Visual and oculomotor functions of monkey subthalamic nucleus. J. Neurophysiol. 67, 1615–1632 (1992).
  CAS PubMed Google Scholar
• Schultz, W. Activity of pars reticulata neurons of monkey substantia nigra in relation to motor, sensory and complex events. J. Neurophysiol. 55, 660–677 (1986).
  CAS PubMed Google Scholar
• Niki, H., Sakai, M. & Kubota, K. Delayed alternation performance and unit activity of the caudate head and medial orbitofrontal gyrus in the monkey. Brain Res. 38, 343–353 ( 1972).
  CAS PubMed Google Scholar
• Niki, H. & Watanabe, M. Prefrontal and cingulate unit activity during timing behaviour in the monkey. Brain Res. 171 , 213–224 (1979).
  CAS PubMed Google Scholar
• Rosenkilde, C. E., Bauer, R. H. & Fuster, J. M. Single cell activity in ventral prefrontal cortex of behaving monkeys. Brain Res. 209, 375 –394 (1981).
  CAS PubMed Google Scholar
• Watanabe, M. The appropriateness of behavioural responses coded in post-trial activity of primate prefrontal units. Neurosci. Lett. 101, 113–117 (1989).
  CAS PubMed Google Scholar
• Tremblay, L. & Schultz, W. Reward-related neuronal activity during go–no go task performance in primate orbitofrontal cortex. J. Neurophysiol. 83, 1864–1876 (2000).
  CAS PubMed Google Scholar
• Niki, H. & Watanabe, M. Cingulate unit activity and delayed response. Brain Res. 110, 381– 386 (1976).
  CAS PubMed Google Scholar
• Nishijo, H., Ono, T. & Nishino, H. Single neuron responses in amygdala of alert monkey during complex sensory stimulation with affective significance. J. Neurosci. 8, 3570–3583 ( 1988).A paper written by one of the few groups investigating neurons in the primate amygdala in relation to reward-related stimuli. They reported a number of different responses to the presentation of natural rewards.
  CAS PubMed Google Scholar
• Nakamura, K., Mikami, A. & Kubota, K. Activity of single neurons in the monkey amygdala during performance of a visual discrimination task. J. Neurophysiol. 67, 1447–1463 (1992).
  CAS PubMed Google Scholar
• Burton, M. J., Rolls, E. T. & Mora, F. Effects of hunger on the responses of neurons in the lateral hypothalamus to the sight and taste of food. Exp. Neurol. 51, 668–677 ( 1976).
  CAS PubMed Google Scholar
• Tremblay, L. & Schultz, W. Relative reward preference in primate orbitofrontal cortex. Nature 398, 704– 708 (1999).
  CAS PubMed Google Scholar
• Pratt, W. E. & Mizumori, S. J. Y. Characteristics of basolateral amygdala neuronal firing on a spatial memory task involving differential reward . Behav. Neurosci. 112, 554– 570 (1998).
  CAS PubMed Google Scholar
• Thorpe, S. J., Rolls, E. T. & Maddison, S. The orbitofrontal cortex: neuronal activity in the behaving monkey. Exp. Brain Res. 49, 93– 115 (1983).
  CAS PubMed Google Scholar
• Rolls, E. T., Murzi, E., Yaxley, S., Thorpe, S. J. & Simpson, S. J. Sensory-specific satiety: food-specific reduction in responsiveness of ventral forebrain neurons after feeding in the monkey . Brain Res. 368, 79–86 (1986).
  CAS PubMed Google Scholar
• Rolls, E. T., Sienkiewicz, Z. J. & Yaxley, S. Hunger modulates the responses to gustatory stimuli of single neurons in the caudolateral orbitofrontal cortex of the macaque monkey. Eur. J. Neurosci. 1, 53– 60 (1989).
  PubMed Google Scholar
• Rolls, E. T., Scott, T. R., Sienkiewicz, Z. J. & Yaxley, S. The responsiveness of neurons in the frontal opercular gustatory cortex of the macaque monkey is independent of hunger. J. Physiol. 397, 1–12 (1988).
  CAS PubMed PubMed Central Google Scholar
• Apicella, P., Legallet, E. & Trouche, E. Responses of tonically discharging neurons in the monkey striatum to primary rewards delivered during different behavioural states . Exp. Brain Res. 116, 456– 466 (1997).
  CAS PubMed Google Scholar
• Apicella, P., Ravel, S., Sardo, P. & Legallet, E. Influence of predictive information on responses of tonically active neurons in the monkey striatum. J. Neurophysiol. 80, 3341– 3344 (1998).
  CAS PubMed Google Scholar
• Apicella, P., Legallet, E. & Trouche, E. Responses of tonically discharging neurons in monkey striatum to visual stimuli presented under passive conditions and during task performance. Neurosci. Lett. 203, 147– 150 (1996).
  CAS PubMed Google Scholar
• Williams, G. V., Rolls, E. T., Leonard, C. M. & Stern, C. Neuronal responses in the ventral striatum of the behaving monkey. Behav. Brain Res. 55, 243–252 (1993).
  CAS PubMed Google Scholar
• Aosaki, T. et al. Responses of tonically active neurons in the primate's striatum undergo systematic changes during behavioural sensorimotor conditioning. J. Neurosci. 14, 3969–3984 (1994).
  CAS PubMed Google Scholar
• Apicella, P., Scarnati, E., Ljungberg, T. & Schultz, W. Neuronal activity in monkey striatum related to the expectation of predictable environmental events. J. Neurophysiol. 68, 945–960 (1992).
  CAS PubMed Google Scholar
• Schultz, W., Apicella, P., Scarnati, E. & Ljungberg, T. Neuronal activity in monkey ventral striatum related to the expectation of reward. J. Neurosci. 12, 4595– 4610 (1992).
  CAS PubMed Google Scholar
• Schoenbaum, G., Chiba, A. A. & Gallagher, M. Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning. Nature Neurosci. 1, 155–159 (1998).
  CAS PubMed Google Scholar
• Hikosaka, K. & Watanabe, M. Delay activity of orbital and lateral prefrontal neurons of the monkey varying with different rewards. Cerebral Cortex 10, 263–271 (2000).
  CAS PubMed Google Scholar
• Hollerman, J. R., Tremblay, L. & Schultz, W. Influence of reward expectation on behavior-related neuronal activity in primate striatum. J. Neurophysiol. 80, 947–963 (1998).
  CAS PubMed Google Scholar
• Tremblay, L., Hollerman, J. R. & Schultz, W. Modifications of reward expectation-related neuronal activity during learning in primate striatum. J. Neurophysiol. 80, 964–977 ( 1998).
  CAS PubMed Google Scholar
• Tremblay, L. & Schultz, W. Modifications of reward expectation-related neuronal activity during learning in primate orbitofrontal cortex. J. Neurophysiol. 83, 1877–1885 (2000).
  CAS PubMed Google Scholar
• Dickinson, A. & Balleine, B. Motivational control of goal-directed action. Animal Learn. Behav. 22, 1– 18 (1994).
  Google Scholar
• Okano, K. & Tanji, J. Neuronal activities in the primate motor fields of the agranular frontal cortex preceding visually triggered and self-paced movement. Exp. Brain Res. 66, 155–166 (1987).
  CAS PubMed Google Scholar
• Romo, R. & Schultz, W. Neuronal activity preceding self-initiated or externally timed arm movements in area 6 of monkey cortex. Exp. Brain Res. 67, 656–662 (1987).
  CAS PubMed Google Scholar
• Kurata, K. & Wise, S. P. Premotor and supplementary motor cortex in rhesus monkeys: neuronal activity during externally- and internally-instructed motor tasks. Exp. Brain Res. 72, 237– 248 (1988).
  CAS PubMed Google Scholar
• Schultz, W. & Romo, R. Neuronal activity in the monkey striatum during the initiation of movements. Exp. Brain Res. 71, 431–436 (1988).
  CAS PubMed Google Scholar
• Watanabe, M. Reward expectancy in primate prefrontal neurons. Nature 382, 629–632 (1996). The first demonstration that expected rewards influence the behaviour-related activity of neurons in a manner that is compatible with a goal-directed account. Neurons in primate dorsolateral prefrontal cortex show different activities depending on the expected reward during a typical prefrontal spatial delayed response task.
  CAS PubMed Google Scholar
• Leon, M. I. & Shadlen, M. N. Effect of expected reward magnitude on the responses of neurons in the dorsolateral prefrontal cortex of the macaque . Neuron 24, 415–425 (1999).
  CAS PubMed Google Scholar
• Kawagoe, R., Takikawa, Y. & Hikosaka, O. Expectation of reward modulates cognitive signals in the basal ganglia. Nature Neurosci. 1, 411 –416 (1998).
  CAS PubMed Google Scholar
• Liu, Z. & Richmond, B. J. Response differences in monkey TE and perirhinal cortex: stimulus association related to reward schedules . J. Neurophysiol. 83, 1677– 1692 (2000).The first demonstration of the prominent reward relationships in neurons of temporal cortex. Neuronal responses to task stimuli in the primate perirhinal cortex were profoundly affected by the distance to reward, whereas neuronal responses in the neighbouring TE area predominantly reflected the visual features of the stimuli.
  CAS PubMed Google Scholar
• Platt, M. L. & Glimcher, P. W. Neural correlates of decision variables in parietal cortex. Nature 400, 233–238 (1999).Neurons in primate parietal association cortex were sensitive to two key variables of game theory and decision making: the quantity and the probability of the outcomes. Based on studies of choice behaviour and decision making in human economics and animal psychology, this is the first application of principles of decision theory in primate neurophysiology.
  CAS PubMed Google Scholar
• Shima, K. & Tanji, J. Role for cingulate motor area cells in voluntary movement selection based on reward. Science 282, 1335–1338 (1998). Neurons in the primate cingulate motor area were active when animals switched to a different movement when continuing to perform the current movement would have produced less reward. This study is interesting from the point of view of movement selection and also for the influence of rewards on behavioural choices.
  CAS PubMed Google Scholar
• Kelley, A. E. Functional specificity of ventral striatal compartments in appetitive behaviours . Ann. NY Acad. Sci. 877, 71– 90 (1999).
  CAS PubMed Google Scholar
• Carelli, R. M., King, V. C., Hampson, R. E. & Deadwyler, S. A. Firing patterns of nucleus accumbens neurons during cocaine self-administration . Brain Res. 626, 14–22 (1993).The first demonstration of drug-related changes in neuronal activity in one of the key structures of drug addiction, the nucleus accumbens. Two important neuronal patterns were found in the rat: activity preceding lever presses that led to acquisition of the drug and activity following drug delivery. These findings have been consistently reproduced by other goups.
  CAS PubMed Google Scholar
• Chang, J. Y., Sawyer, S. F., Lee, R. S. & Woodward, D. J. Electrophysiological and pharmacological evidence for the role of the nucleus accumbens in cocaine self-administration in freely moving rats. J. Neurosci. 14, 1224–1244 (1994).
  CAS PubMed Google Scholar
• Carelli, R. M. & Deadwyler, S. A. A comparison of nucleus accumbens neuronal firing patterns during cocaine self-administration and water reinforcement in rats. J. Neurosci. 14, 7735–7746 (1994).
  CAS PubMed Google Scholar
• Chang, J. Y., Janak, P. H. & Woodward, D. J. Comparison of mesocorticolimbic neuronal responses during cocaine and heroin self-administration in freely moving rats. J. Neurosci. 18, 3098–3115 (1998).
  CAS PubMed Google Scholar
• Carelli, R. M., Ijames, S. G. & Crumling, A. J. Evidence that separate neural circuits in the nucleus accumbens encode cocaine versus 'natural' (water and food) reward. J. Neurosci. 20, 4255–4266 (2000).
  CAS PubMed Google Scholar
• Chang, J. Y., Paris, J. M., Sawyer, S. F., Kirillov, A. B. & Woodward, D. J. Neuronal spike activity in rat nucleus accumbens during cocaine self-administration under different fixed-ratio schedules. J. Neurosci. 74, 483– 497 (1996).
  CAS Google Scholar
• Peoples, L. L., Uzwiak, A. J., Gee, F. & West, M. O. Operant behaviour during sessions of intravenous cocaine infusion is necessary and sufficient for phasic firing of single nucleus accumbens neurons. Brain Res. 757, 280–284 ( 1997).
  CAS PubMed Google Scholar
• West, M. O., Peoples, L. L., Michael, A. J., Chapin, J. K. & Woodward, D. J. Low-dose amphetamine elevates movement-related firing of rat striatal neurons. Brain Res. 745, 331–335 (1997).
  CAS PubMed Google Scholar
• Peoples, L. L. & West, M. O. Phasic firing of single neurons in the rat nucleus accumbens correlated with the timing of intravenous cocaine self-administration. J. Neurosci. 16, 3459–3473 (1996).
  CAS PubMed Google Scholar
• Peoples, L. L., Gee, F., Bibi, R. & West, M. O. Phasic firing time locked to cocaine self-infusion and locomotion: dissociable firing patterns of single nucleus accumbens neurons in the rat. J. Neurosci. 18, 7588–7598 (1998).
  CAS PubMed Google Scholar
• Calabresi, P., Maj, R., Pisani, A., Mercuri, N. B. & Bernardi, G. Long-term synaptic depression in the striatum: physiological and pharmacological characterization. J. Neurosci. 12, 4224–4233 (1992).
  CAS PubMed Google Scholar
• Otmakhova, N. A. & Lisman, J. E. D1/D5 dopamine receptor activation increases the magnitude of early long-term potentiation at CA1 hippocampal synapses. J. Neurosci. 16, 7478–7486 (1996).
  CAS PubMed Google Scholar
• Wickens, J. R., Begg, A. J. & Arbuthnott, G. W. Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex in vitro. Neuroscience 70, 1– 5 (1996).
  CAS PubMed Google Scholar
• Calabresi, P. et al. Abnormal synaptic plasticity in the striatum of mice lacking dopamine D2 receptors. J. Neurosci. 17, 4536–4544 (1997).
  CAS PubMed Google Scholar