Vy Vo - Academia.edu (original) (raw)
Papers by Vy Vo
arXiv (Cornell University), Sep 22, 2022
Graph neural networks (GNNs) have been extensively used for many domains where data are represent... more Graph neural networks (GNNs) have been extensively used for many domains where data are represented as graphs, including social networks, recommender systems, biology, chemistry, etc. Recently, the expressive power of GNNs has drawn much interest. It has been shown that, despite the promising empirical results achieved by GNNs for many applications, there are some limitations in GNNs that hinder their performance for some tasks. For example, since GNNs update node features mainly based on local information, they have limited expressive power in capturing long-range dependencies among nodes in graphs. To address some of the limitations of GNNs, several recent works started to explore augmenting GNNs with memory for improving their expressive power in the relevant tasks. In this paper, we provide a comprehensive review of the existing literature of memory-augmented GNNs. We review these works through the lens of psychology and neuroscience, which has established multiple memory systems and mechanisms in biological brains. We propose a taxonomy of the memory GNN works, as well as a set of criteria for comparing the memory mechanisms. We also provide critical discussions on the limitations of these works. Finally, we discuss the challenges and future directions for this area. Impact Statement-Memory-augmentation of graph neural networks is an emerging research field in the deep graph learning community. These augmentations can enhance GNNs' capabilities for structured representation learning and relational reasoning tasks, such as human-object interaction prediction, question answering, and algorithm reasoning, etc. This paper provides a systematic review of the existing works in this field from the perspective of neuroscience and psychology. As the first review of memory GNNs, we identify the open challenges in this field and shed light on the promising directions for future work. By providing a neuroscience perspective, our work will also help facilitate novel brain-inspired memory model designs to advance graph neural networks for various domains.
Metacognitive skills have been shown to facilitate learning. But do they develop globally or with... more Metacognitive skills have been shown to facilitate learning. But do they develop globally or within content domains? We used an objective measure of metacognition in two different domains to investigate this question in children. 25 subjects (5 to 8 y.o.) made numerosity judgments (which picture has more dots?) and emotion judgments (which picture looks happier?) on a touchscreen. After each judgment, they placed a metacognitive “bet” on their accuracy using a token economy, with immediate feedback (+3/-3 high risk, +1/-1 low risk). We measured metacognition by a phi correlation of risk choice and accuracy, finding that children were significantly metacognitive on both tasks. Higher metacognitive scores on the numerosity task, and not the emotion task, predicted mathematical intelligence. However, metacognitive scores did not predict other measures of ability, such as general IQ. Our study provides evidence that metacognition develops in tandem with domain knowledge, rather than globally.
Psychological Science, 2014
Metacognition, the ability to assess one’s own knowledge, has been targeted as a critical learnin... more Metacognition, the ability to assess one’s own knowledge, has been targeted as a critical learning mechanism in mathematics education. Yet the early childhood origins of metacognition have proven difficult to study. Using a novel nonverbal task and a comprehensive set of metacognitive measures, we provided the strongest evidence to date that young children are metacognitive. We showed that children as young as 5 years made metacognitive “bets” on their numerical discriminations in a wagering task. However, contrary to previous reports from adults, our results showed that children’s metacognition is domain specific: Their metacognition in the numerical domain was unrelated to their metacognition in another domain (emotion discrimination). Moreover, children’s metacognitive ability in only the numerical domain predicted their school-based mathematics knowledge. The data provide novel evidence that metacognition is a fundamental, domain-dependent cognitive ability in children. The find...
Current theories propose that the short-term retention of information in working memory (WM) and ... more Current theories propose that the short-term retention of information in working memory (WM) and the recall of information from long-term memory (LTM) are supported by overlapping neural mechanisms in occipital and parietal cortex. Both are thought to rely on reinstating patterns of sensory activity evoked by the perception of the remembered item. However, the extent of the shared representations between WM and LTM are unclear, and it is unknown how WM and LTM representations may differ across cortical regions. We designed a spatial memory task that allowed us to directly compare the representations of remembered spatial information in WM and LTM. Critically, we carefully matched the precision of behavioral responses in these tasks. We used fMRI and multivariate pattern analyses to examine representations in (1) retinotopic cortex and (2) lateral parietal cortex (LPC) regions previously implicated in LTM. We show that visual memories were represented in a sensory-like code in both t...
Cerebral Cortex, 2021
Current theories propose that the short-term retention of information in working memory (WM) and ... more Current theories propose that the short-term retention of information in working memory (WM) and the recall of information from long-term memory (LTM) are supported by overlapping neural mechanisms in occipital and parietal cortex. However, the extent of the shared representations between WM and LTM is unclear. We designed a spatial memory task that allowed us to directly compare the representations of remembered spatial information in WM and LTM with carefully matched behavioral response precision between tasks. Using multivariate pattern analyses on functional magnetic resonance imaging data, we show that visual memories were represented in a sensory-like code in both memory tasks across retinotopic regions in occipital and parietal cortex. Regions in lateral parietal cortex also encoded remembered locations in both tasks, but in a format that differed from sensory-evoked activity. These results suggest a striking correspondence in the format of representations maintained in WM an...
eneuro, 2019
Navigating through natural environments requires localizing objects along three distinct spatial ... more Navigating through natural environments requires localizing objects along three distinct spatial axes. Information about position along the horizontal and vertical axes is available from an object’s position on the retina, while position along the depth axis must be inferred based on second-order cues such as the disparity between the images cast on the two retinae. Past work has revealed that object position in two-dimensional (2D) retinotopic space is robustly represented in visual cortex and can be robustly predicted using a multivariate encoding model, in which an explicit axis is modeled for each spatial dimension. However, no study to date has used an encoding model to estimate a representation of stimulus position in depth. Here, we recorded BOLD fMRI while human subjects viewed a stereoscopic random-dot sphere at various positions along the depth (z) and the horizontal (x) axes, and the stimuli were presented across a wider range of disparities (out to ∼40 arcmin) compared t...
When a behaviorally relevant stimulus has been previously associated with reward, behavioral resp... more When a behaviorally relevant stimulus has been previously associated with reward, behavioral responses are faster and more accurate compared to equally relevant but less valuable stimuli. Conversely, task irrelevant stimuli that were previously associated with a high reward can capture attention and distract processing away from relevant stimuli (e.g. the chocolate bar in the pantry when you are looking for a nice healthy apple). While increasing the value of task-relevant stimuli systematically up-regulates neural responses in early visual cortex to facilitate information processing, it is not clear if the value of task-irrelevant distractors influences behavior via competition in early visual cortex or via competition at later stages of decision-making and response selection. Here, we measured fMRI in human visual cortex while subjects performed a value-based learning task, and applied a multivariate inverted encoding model to assess the fidelity of distractor representations in e...
Journal of neurophysiology, Jan 28, 2018
Computational models posit that visual attention is guided by activity within spatial maps that i... more Computational models posit that visual attention is guided by activity within spatial maps that index the image-computable salience and the behavioral relevance of objects in the scene. These spatial maps are theorized to be instantiated as activation patterns across a series of retinotopic visual regions in occipital, parietal, and frontal cortex. While previous research has identified sensitivity to either the behavioral relevance or the image-computable salience of different scene elements, the simultaneous influence of these factors on neural 'attentional priority maps' in human cortex is not well understood. We tested the hypothesis that visual salience and behavioral relevance independently impact the activation profile across retinotopically-organized cortical regions by quantifying attentional priority maps measured in human brains using functional MRI while participants attended one of two differentially-salient stimuli. We found that the topography of activation in...
Selective visual attention enables organisms to enhance the representation of behaviorally releva... more Selective visual attention enables organisms to enhance the representation of behaviorally relevant stimuli by altering the encoding properties of single receptive fields (RFs). Yet we know little about how the attentional modulations of single RFs contribute to the encoding of an entire visual scene. Addressing this issue requires (1) measuring a group of RFs that tile a continuous portion of visual space, (2) constructing a population-level measurement of spatial representations based on these RFs, and (3) linking how different types of RF attentional modulations change the population-level representation. To accomplish these aims, we used fMRI to characterize the responses of thousands of voxels in retinotopically organized human cortex. First, we found that the response modulations of voxel RFs (vRFs) depend on the spatial relationship between the RF center and the visual location of the attended target. Second, we used two analyses to assess the spatial encoding quality of a po...
arXiv (Cornell University), Sep 22, 2022
Graph neural networks (GNNs) have been extensively used for many domains where data are represent... more Graph neural networks (GNNs) have been extensively used for many domains where data are represented as graphs, including social networks, recommender systems, biology, chemistry, etc. Recently, the expressive power of GNNs has drawn much interest. It has been shown that, despite the promising empirical results achieved by GNNs for many applications, there are some limitations in GNNs that hinder their performance for some tasks. For example, since GNNs update node features mainly based on local information, they have limited expressive power in capturing long-range dependencies among nodes in graphs. To address some of the limitations of GNNs, several recent works started to explore augmenting GNNs with memory for improving their expressive power in the relevant tasks. In this paper, we provide a comprehensive review of the existing literature of memory-augmented GNNs. We review these works through the lens of psychology and neuroscience, which has established multiple memory systems and mechanisms in biological brains. We propose a taxonomy of the memory GNN works, as well as a set of criteria for comparing the memory mechanisms. We also provide critical discussions on the limitations of these works. Finally, we discuss the challenges and future directions for this area. Impact Statement-Memory-augmentation of graph neural networks is an emerging research field in the deep graph learning community. These augmentations can enhance GNNs' capabilities for structured representation learning and relational reasoning tasks, such as human-object interaction prediction, question answering, and algorithm reasoning, etc. This paper provides a systematic review of the existing works in this field from the perspective of neuroscience and psychology. As the first review of memory GNNs, we identify the open challenges in this field and shed light on the promising directions for future work. By providing a neuroscience perspective, our work will also help facilitate novel brain-inspired memory model designs to advance graph neural networks for various domains.
Metacognitive skills have been shown to facilitate learning. But do they develop globally or with... more Metacognitive skills have been shown to facilitate learning. But do they develop globally or within content domains? We used an objective measure of metacognition in two different domains to investigate this question in children. 25 subjects (5 to 8 y.o.) made numerosity judgments (which picture has more dots?) and emotion judgments (which picture looks happier?) on a touchscreen. After each judgment, they placed a metacognitive “bet” on their accuracy using a token economy, with immediate feedback (+3/-3 high risk, +1/-1 low risk). We measured metacognition by a phi correlation of risk choice and accuracy, finding that children were significantly metacognitive on both tasks. Higher metacognitive scores on the numerosity task, and not the emotion task, predicted mathematical intelligence. However, metacognitive scores did not predict other measures of ability, such as general IQ. Our study provides evidence that metacognition develops in tandem with domain knowledge, rather than globally.
Psychological Science, 2014
Metacognition, the ability to assess one’s own knowledge, has been targeted as a critical learnin... more Metacognition, the ability to assess one’s own knowledge, has been targeted as a critical learning mechanism in mathematics education. Yet the early childhood origins of metacognition have proven difficult to study. Using a novel nonverbal task and a comprehensive set of metacognitive measures, we provided the strongest evidence to date that young children are metacognitive. We showed that children as young as 5 years made metacognitive “bets” on their numerical discriminations in a wagering task. However, contrary to previous reports from adults, our results showed that children’s metacognition is domain specific: Their metacognition in the numerical domain was unrelated to their metacognition in another domain (emotion discrimination). Moreover, children’s metacognitive ability in only the numerical domain predicted their school-based mathematics knowledge. The data provide novel evidence that metacognition is a fundamental, domain-dependent cognitive ability in children. The find...
Current theories propose that the short-term retention of information in working memory (WM) and ... more Current theories propose that the short-term retention of information in working memory (WM) and the recall of information from long-term memory (LTM) are supported by overlapping neural mechanisms in occipital and parietal cortex. Both are thought to rely on reinstating patterns of sensory activity evoked by the perception of the remembered item. However, the extent of the shared representations between WM and LTM are unclear, and it is unknown how WM and LTM representations may differ across cortical regions. We designed a spatial memory task that allowed us to directly compare the representations of remembered spatial information in WM and LTM. Critically, we carefully matched the precision of behavioral responses in these tasks. We used fMRI and multivariate pattern analyses to examine representations in (1) retinotopic cortex and (2) lateral parietal cortex (LPC) regions previously implicated in LTM. We show that visual memories were represented in a sensory-like code in both t...
Cerebral Cortex, 2021
Current theories propose that the short-term retention of information in working memory (WM) and ... more Current theories propose that the short-term retention of information in working memory (WM) and the recall of information from long-term memory (LTM) are supported by overlapping neural mechanisms in occipital and parietal cortex. However, the extent of the shared representations between WM and LTM is unclear. We designed a spatial memory task that allowed us to directly compare the representations of remembered spatial information in WM and LTM with carefully matched behavioral response precision between tasks. Using multivariate pattern analyses on functional magnetic resonance imaging data, we show that visual memories were represented in a sensory-like code in both memory tasks across retinotopic regions in occipital and parietal cortex. Regions in lateral parietal cortex also encoded remembered locations in both tasks, but in a format that differed from sensory-evoked activity. These results suggest a striking correspondence in the format of representations maintained in WM an...
eneuro, 2019
Navigating through natural environments requires localizing objects along three distinct spatial ... more Navigating through natural environments requires localizing objects along three distinct spatial axes. Information about position along the horizontal and vertical axes is available from an object’s position on the retina, while position along the depth axis must be inferred based on second-order cues such as the disparity between the images cast on the two retinae. Past work has revealed that object position in two-dimensional (2D) retinotopic space is robustly represented in visual cortex and can be robustly predicted using a multivariate encoding model, in which an explicit axis is modeled for each spatial dimension. However, no study to date has used an encoding model to estimate a representation of stimulus position in depth. Here, we recorded BOLD fMRI while human subjects viewed a stereoscopic random-dot sphere at various positions along the depth (z) and the horizontal (x) axes, and the stimuli were presented across a wider range of disparities (out to ∼40 arcmin) compared t...
When a behaviorally relevant stimulus has been previously associated with reward, behavioral resp... more When a behaviorally relevant stimulus has been previously associated with reward, behavioral responses are faster and more accurate compared to equally relevant but less valuable stimuli. Conversely, task irrelevant stimuli that were previously associated with a high reward can capture attention and distract processing away from relevant stimuli (e.g. the chocolate bar in the pantry when you are looking for a nice healthy apple). While increasing the value of task-relevant stimuli systematically up-regulates neural responses in early visual cortex to facilitate information processing, it is not clear if the value of task-irrelevant distractors influences behavior via competition in early visual cortex or via competition at later stages of decision-making and response selection. Here, we measured fMRI in human visual cortex while subjects performed a value-based learning task, and applied a multivariate inverted encoding model to assess the fidelity of distractor representations in e...
Journal of neurophysiology, Jan 28, 2018
Computational models posit that visual attention is guided by activity within spatial maps that i... more Computational models posit that visual attention is guided by activity within spatial maps that index the image-computable salience and the behavioral relevance of objects in the scene. These spatial maps are theorized to be instantiated as activation patterns across a series of retinotopic visual regions in occipital, parietal, and frontal cortex. While previous research has identified sensitivity to either the behavioral relevance or the image-computable salience of different scene elements, the simultaneous influence of these factors on neural 'attentional priority maps' in human cortex is not well understood. We tested the hypothesis that visual salience and behavioral relevance independently impact the activation profile across retinotopically-organized cortical regions by quantifying attentional priority maps measured in human brains using functional MRI while participants attended one of two differentially-salient stimuli. We found that the topography of activation in...
Selective visual attention enables organisms to enhance the representation of behaviorally releva... more Selective visual attention enables organisms to enhance the representation of behaviorally relevant stimuli by altering the encoding properties of single receptive fields (RFs). Yet we know little about how the attentional modulations of single RFs contribute to the encoding of an entire visual scene. Addressing this issue requires (1) measuring a group of RFs that tile a continuous portion of visual space, (2) constructing a population-level measurement of spatial representations based on these RFs, and (3) linking how different types of RF attentional modulations change the population-level representation. To accomplish these aims, we used fMRI to characterize the responses of thousands of voxels in retinotopically organized human cortex. First, we found that the response modulations of voxel RFs (vRFs) depend on the spatial relationship between the RF center and the visual location of the attended target. Second, we used two analyses to assess the spatial encoding quality of a po...