Crossmodal Interaction Research Papers - Academia.edu (original) (raw)

Evidence suggests that spatial processing changes across time in naturally cycling women, which is likely due to neuromodulatory effects of steroid hormones. Yet, it is unknown whether crossmodal spatial processes depend on steroid hormones as well. In the present experiment, the crossmodal congruency task was used to assess visuo-tactile interactions in naturally cycling women, women using hormonal contraceptives and men. Participants adopted either a crossed or uncrossed hands posture. It was tested whether a postural effect of hand crossing on multisensory interactions in the crossmodal congruency task is modulated by women's cycle phase. We found that visuotactile interactions changed according to cycle phase. Naturally cycling women showed a significant difference between the menstrual and the luteal phase for crossed, but not for uncrossed hands postures. The two control groups showed no test sessions effects. Regression analysis revealed a positive relation between estradiol levels and the size of crossmodal congruency effects (CCE), indicating that estradiol seems to have a neuromodulatory effect on posture processing.

- by
- •
- Psychology, Crossmodal Interaction, Hormones, Menstrual Cycle

Colors and odors are associated; for instance, people typically match the smell of strawberries to the color pink or red. These associations are forms of crossmodal correspondences. Recently, there has been discussion about the extent to which these correspondences arise for structural reasons (i.e., an inherent mapping between color and odor), statistical reasons (i.e., covariance in experience), and/or semantically-mediated reasons (i.e., stemming from language). The present study probed this question by testing color-odor correspondences in 6 different cultural groups (Dutch, Netherlands-residing-Chinese, German, Malay, Malaysian-Chinese, and US residents), using the same set of 14 odors and asking participants to make congruent and incongruent color choices for each odor. We found consistent patterns in color choices for each odor within each culture, showing that participants were making non-random color-odor matches. We used representational dissimilarity analysis to probe for variations in the patterns of color-odor associations across cultures; we found that US and German participants had the most similar patterns of associations, followed by German and Malay participants. The largest group differences were between Malay and Netherlands-resident Chinese participants and between Dutch and Malaysian-Chinese participants. We conclude that culture plays a role in color-odor crossmodal associations, which likely arise, at least in part, through experience.

The majority of scientific studies on consciousness have focused on vision, exploring the cognitive and neural mechanisms of conscious access to visual stimuli. In parallel, studies on bodily consciousness have revealed that bodily (i.e. tactile, proprioceptive, visceral, vestibular) signals are the basis for the sense of self. However, the role of bodily signals in the formation of visual consciousness is not well understood. Here we investigated how body-related visuo-tactile stimulation modulates conscious access to visual stimuli. We used a robotic platform to apply controlled tactile stimulation to the participants' back while they viewed a dot moving either in synchrony or asynchrony with the touch on their back. Critically, the dot was rendered invisible through continuous flash suppression. Manipulating the visual context by presenting the dot moving on either a body form, or a non-bodily object we show that: (i) conflict induced by synchronous visuo-tactile stimulation in a body context is associated with a delayed conscious access compared to asynchronous visuo-tactile stimulation, (ii) this effect occurs only in the context of a visual body form, and (iii) is not due to detection or response biases. The results indicate that body-related visuo-tactile conflicts impact visual consciousness by facilitating access of non-conflicting visual information to awareness, and that these are sensitive to the visual context in which they are presented, highlighting the interplay between bodily signals and visual experience.

- by Roy Salomon
- •
- Embodiment, Consciousness, Tactile Perception, Crossmodal Interaction

Audio quality is known to cross-modally influence reaction speed, sense of presence, and visual quality. We designed an experiment to test the effect of audio quality on source localization. Stimuli with different MP3 compression rates, as a proxy for audio quality, were generated from drum samples. Participants (n = 18) estimated the position of a snare drum target while compression rate, masker, and target position were systematically manipulated in a full-factorial repeated-measures experiment design. Analysis of variance revealed that location accuracy was better in wide target positions than in narrow, with a medium effect size; and that the effect of target position was moderated by compression rate in different directions for wide and narrow targets. The results suggest that there might be two perceptual effects at play: one, whereby increased audio quality causes a widening of the soundstage, possibly via a SMARC-like mechanism, and two, whereby it enables higher localization accuracy. In the narrow target positions in this experiment, the two effects acted in opposite directions and largely cancelled each other out. In the wide target presentations, their effects were compounded and led to significant correlations between compression rate and localization error.

- by PerMagnus Lindborg
- •
- Auditory Perception, Audio Engineering, Sound, Localization

People often say things like the following: Cabernet Sauvignon tastes differently to an expert wine taster, or Beethoven's Ninth Symphony sounds differently to a seasoned conductor. The examples just described are cases of perceptual learning-cases of long-term changes in perception that result from practice or experience. Philosophers have been discussing such cases for centuries, from the 14 th-century Indian philosopher Vedānta Deśika to Thomas Reid as well as to contemporary philosophers like John McDowell, Ned Block, Susanna Siegel, and Christopher Peacocke. In the book, I use recent evidence from psychology and neuroscience to show that perceptual learning in fact occurs. I also offer a way for philosophers to distinguish between various different types of it. In some cases, perceptual learning involves changes in how one attends; in other cases, it involves a learned ability to differentiate two properties, or to perceive two properties as unified. I show how this taxonomy helps to classify cases in the philosophical literature. I then go on to rethink several domains in the philosophy of perception in terms of perceptual learning, including multisensory perception, color perception, speech perception, and sensory substitution. As a whole, the book offers a theory of the function of perceptual learning. Perceptual learning embeds into our quick perceptual systems what would be a slower task were it to be done in a controlled, cognitive manner. This frees up cognitive resources for other tasks. For instance, a novice wine taster drinking a Cabernet Sauvignon might have to think about its features first and then infer the type of wine, while an expert would be able to identify it immediately. This learned ability to immediately identify the wine enables the expert to think about other things like the vineyard or the vintage of the wine. My account gives us a new way to understand such cases in terms of cognitive resources and cognitive economy.

- by Kevin Connolly
- •
- Cognitive Psychology, Philosophy of Mind, Perception, Learning and Teaching

According to the decomposition thesis, perceptual experiences resolve without remainder into their different modality-specific components. Contrary to this view, I argue that certain cases of multisensory integration give rise to experiences representing features of a novel type. Through the coordinated use of bodily awareness—understood here as encompassing both proprioception and kinaesthesis—and the exteroceptive sensory modalities, one becomes perceptually responsive to spatial features whose instances couldn’t be represented by any of the contributing modalities functioning in isolation. I develop an argument for this conclusion focusing on two cases: 3D shape perception in haptic touch and experiencing an object’s egocentric location in crossmodally accessible, environmental space.

- by Robert Briscoe
- •
- Cognitive Psychology, Cognitive Science, Philosophy, Philosophy of Mind

Musical timbre is often described using terms from non-auditory senses, mainly vision and touch; but it is not clear whether crossmodality in timbre semantics reflects multisensory processing or simply linguistic convention. If multisensory processing is involved in timbre perception, the mechanism governing the interaction remains unknown. To investigate whether timbres commonly perceived as ‘‘bright-dark’’ facilitate or interfere with visual perception (darkness- brightness), we designed two speeded classification experiments. Participants were presented consecutive images of slightly varying (or the same) brightness along with task-irrelevant auditory primes (‘‘bright’’ or ‘‘dark’’ tones) and asked to quickly identify whether the second image was brighter/darker than the first. Incongruent prime-stimulus combinations produced significantly more response errors compared to congruent combinations but choice reaction time was unaffected. Furthermore, responses in a deceptive identical-image condition indicated subtle semantically congruent response bias. Additionally, in Experiment 2 (which also incorporated a spatial texture task), measures of reaction time (RT) and accuracy were used to construct speed-accuracy tradeoff functions (SATFs) in order to critically compare two hypothesized mechanisms for timbre-based crossmodal interactions, sensory response change vs. shift in response criterion. Results of the SATF analysis are largely consistent with the response criterion hypothesis, although without conclusively ruling out sensory change.

- by Zachary Wallmark
- •
- Crossmodal Interaction, Musical Timbre, Timbre Perception

Growing evidence shows that individuals consistently match auditory pitch with visual size. For instance, high-pitched sounds are perceptually associated with smaller visual stimuli, whereas low-pitched sounds with larger ones. The present study explores whether this crossmodal correspondence, reported so far for perceptual processing, also modulates motor planning. To address this issue, we carried out a series of
kinematic experiments to verify whether actions implying size processing are affected by auditory pitch. Experiment 1 showed that grasping movements toward small/large objects were initiated faster in response to high/low pitches, respectively, thus extending previous findings in the literature to more complex motor behavior. Importantly, auditory pitch influenced the relative scaling of the hand preshaping, with high pitches associated with smaller grip aperture compared with low pitches. Notably, no effect of auditory pitch was found in case of pointing movements (no grasp implied, Experiment 2), as well as when auditory pitch was irrelevant to the programming of the grip aperture, that is, in case of grasping an object of uniform size (Experiment 3). Finally, auditory pitch influenced also symbolic manual gestures expressing “small” and “large” concepts (Experiment 4). In sum, our results are novel in revealing the impact of auditory pitch on motor planning when size processing is required, and shed light on the role of auditory information in driving actions.

- by Nicolò Francesco Bernardi and +2
- •
- Motor Control, Crossmodal Interaction, Cross-modality, Pitch Perception

There have recently been various empirical attempts to answer Molyneux’s question, for example, the experiments undertaken by the Held group. These studies, though intricate, have encountered some objections, for instance, from Schwenkler, who proposes two ways of improving the experiments. One is “to re-run [the] experiment with the stimulus objects made to move, and/or the subjects moved or permitted to move with respect to them” (p. 94), which would promote three dimensional or otherwise viewpoint-invariant representations. The other is “to use geometrically simpler shapes, such as the cube and sphere in Molyneux’s original proposal, or planar figures instead of three-dimensional solids” (p. 188). Connolly argues against the first modification but agrees with the second. In this article, I argue that the second modification is also problematic (though still surmountable), and that both Schwenkler and Connolly are too optimistic about the prospect of addressing Molyneux’s question empirically.

- by Tony Cheng
- •
- Perception, Crossmodal Interaction, Touch

Recent developments in neuroscience and psychology have confirmed what many artists have long intuited; that our senses are connected. Research into crossmodal correspondences – the universal tendency of a sensory feature in one modality to be matched with one from another sensory modality – has highlighted strong connections between flavour and sound that have only just begun to be explored by artists working in these sensory realms. This article investigates Oenosthesia, a practice-led art research project that aimed to harness crossmodal correspondences in an artwork that combines a soundscape created from field recordings of the winemaking process with wines consumed as part of the piece. Its success in achieving this was tested through data gathered from participants at presentations of the work in London in September 2016 and in Sydney in March 2017. This article presents the results of this study, which suggest that sound can significantly change perceptions of flavour in an experimental audio-gustatory artwork and highlights the potential for the design of crossmodally congruent sound works that heighten specific flavour characters of a wine.

- by Jo Burzynska
- •
- Crossmodal Interaction, Multisensory Design, Aesthetics of Taste

Public perceptions of nanotechnology are shaped by sound in surprising ways. Our analysis of the audiovisual techniques employed by nanotechnology stakeholders shows that well-chosen sounds can help to win public trust, create value and... more

- by Cormac Deane and +2
- •
- Science Communication, Embodied Cognition, Sound studies, Sound Design

- by Sanne Boesveldt
- •
- Cognitive Psychology, Cognitive Science, Malaysia, China

Auditory and visual signals generated by a single source tend to be temporally correlated, such as the synchronous sounds of footsteps and the limb movements of a walker. Continuous tracking and comparison of the dynamics of auditory-visual streams is thus useful for the perceptual binding of information arising from a common source. Although language-related mechanisms have been implicated in the tracking of speech-related auditory-visual signals (e.g., speech sounds and lip movements), it is not well known what sensory mechanisms generally track ongoing auditory-visual synchrony for nonspeech signals in a complex auditory-visual environment. To begin to address this question, we used music and visual displays that varied in the dynamics of multiple features (e.g., auditory loudness and pitch; visual luminance, color, size, motion, and organization) across multiple time scales. Auditory activity (monitored using auditory steady-state responses, ASSR) was selectively reduced in the left hemisphere when the music and dynamic visual displays were temporally misaligned. Importantly, ASSR was not affected when attentional engagement with the music was reduced, or when visual displays presented dynamics clearly dissimilar to the music. These results appear to suggest that left-lateralized auditory mechanisms are sensitive to auditory-visual temporal alignment, but perhaps only when the dynamics of auditory and visual streams are similar. These mechanisms may contribute to correct auditory-visual binding in a busy sensory environment.

- by Julia Mossbridge
- •
- Engineering, Physics, Chemistry, Music

What does the fact that crossmodal correspondences exist in both chimpanzees and humans teach us? Contrary to recent conclusions that chimpanzees must be synaesthetic, or that certain crossmodal correspondences must be innate, we suggest that the crossmodal correspondence between auditory pitch and visual lightness comes from the internalization of correlations present in the environment.

- by Ophelia DEROY
- •
- Synesthesia, Crossmodal Interaction, Crossmodal Associations

Multisensory processes facilitate perception of currently-presented stimuli and can likewise enhance later object recognition. Memories for objects originally encountered in a multisensory context can be more robust than those for objects encountered in an exclusively visual or auditory context [1], upturning the assumption that memory performance is best when encoding and recognition contexts remain constant [2]. Here, we used event-related potentials (ERPs) to provide the first evidence for direct links between multisensory brain activity at one point in time and subsequent object discrimination abilities. Across two experiments we show that individuals showing a benefit and those

In this chapter I argue for the claim that people born blind taking lysergic acid diethylamide (LSD) have visual illusions even though they fail to report any visual experience. The basis of my claim is the significant neural activation of their visual cortex while on LSD, from retina to late visual processing, and that vision is too strange to be identified as such at first sight. I compare what first sight is like to other kinds of naïve seeing like change blindness, chimerical colors, and synesthesia and more exotic senses like magnetic pole experience, metamorphic octopus proprioception, etc. I conclude the paper by applying this Molyneux intuition-shifting methodology to Molyneux’s question itself, suggesting a training strategy for people born blind in preparation for the novelty of new visual experience.

- by Brian R Glenney
- •
- Perception, Blindness, Crossmodal Interaction, Vision

A robot capable of understanding emotion expressions can increase its own capability of solving problems by using emotion expressions as part of its own decision-making, in a similar way to humans. Evidence shows that the perception of human interaction starts with an innate perception mechanism, where the interaction between different entities is perceived and categorized into two very clear directions: positive or negative. While the person is developing during childhood , the perception evolves and is shaped based on the observation of human interaction, creating the capability to learn different categories of expressions. In the context of human–robot interaction, we propose a model that simulates the innate perception of audio–visual emotion expressions with deep neural networks, that learns new expressions by categorizing them into emotional clusters with a self-organizing layer. The proposed model is evaluated with three different corpora: The Surrey Audio–Visual Expressed Emotion (SAVEE) database, the visual Bi-modal Face and Body benchmark (FABO) database, and the multimodal corpus of the Emotion Recognition in the Wild (EmotiW) challenge. We use these corpora to evaluate the performance of the model to recognize emotional expressions, and compare it to state-of-the-art research.

The sustained periodic modulation of a stimulus induces an entrainment of cortical neurons responding to the stimulus, appearing as a steady-state evoked potential (SS-EP) in the EEG frequency spectrum. Here, we used frequency tagging of SS-EPs to study the crossmodal links in spatial attention between touch and vision. We hypothesized that a visual stimulus approaching the left or right hand orients spatial attention toward the approached hand, and thereby enhances the processing of vibrotactile input originating from that hand. Twenty-five subjects took part in the experiment: 16-s trains of vibrotactile stimuli (4.2 and 7.2 Hz) were applied simultaneously to the left and right hand, concomitantly with a punctate visual stimulus blinking at 9.8 Hz. The visual stimulus was approached toward the left or right hand. The hands were either uncrossed (left and right hands to the left and right of the participant) or crossed (left and right hands to the right and left of the participant). The vibrotactile stimuli elicited two distinct SS-EPs with scalp topographies compatible with activity in the contralateral primary somatosensory cortex. The visual stimulus elicited a third SS-EP with a topography compatible with activity in visual areas. When the visual stimulus was over one of the hands, the amplitude of the vibrotactile SS-EP elicited by stimulation of that hand was enhanced, regardless of whether the hands were uncrossed or crossed. This demonstrates a crossmodal effect of spatial attention between vision and touch, integrating proprioceptive and/or visual information to map the position of the limbs in external space.

- by Elisabeth Colon
- •
- EEG, Crossmodal Interaction, Somatosensory, Steady State Visual Evoked Potentials

The current study aims to examine cross modal transference of valence from the auditory to the visual domain, conducting a simultaneous presentation task with 40 participants. Furthermore, what is of particular interest in this experiment is the attempt to bring in the relationship the aspect of time, as well as the aspect of alexithymia, as a potential factor mediating the relationship. It is assumed that on the early stage of the task emotions have been induced but on the late stage, these emotions have been accumulated to create mood, that is aggregated emotions, the effects of crossmodal transfer will be greater. Results however, were non-significant for all factors, so the null hypotheses were accepted.

- by Natalie Drosoyianni
- •
- Social Psychology, Emotion, Decision Making, Psychology of Music

Restaurants are complex environments where all our senses are engaged. Physical and psychoacoustic factors have been shown to be associated with perceived environmental quality in restaurants. More or less designable sound sources such as background music, voices, and kitchen noises are believed to be important in relation to the overall perception of the soundscape. Previous research publications have suggested typologies and other structured descriptions of sound sources for some environmental contexts, such as urban parks and offices, but there is no detailed account that is relevant to restaurants. While existing classification schemes might be extendable, an empirical approach was taken in the present work. We collected on-site data in 40 restaurants (n = 393), including perceptual ratings, free-form annotations of characteristic sounds and whether they were liked or not, and free-form descriptive words for the environment as a whole. The annotations were subjected to analysis using a cladistic approach and yielded a multi-level taxonomy of perceived sound sources in restaurants. Ten different classification taxa were evaluated by comparing the respondents' Liking of sound sources, by categories defined in the taxonomy, and their Pleasantness rating of the environment as a whole. Correlation analysis revealed that a four-level clade was efficient and outperformed alternatives. Internal validation of the Pleasantness construct was made through separate ratings (n = 7) of on-site free-form descriptions of the environment. External validation was made with ratings from a separate listening experiment (n = 48). The two validations demonstrated that the four-level Sound Sources in Restaurants (SSR) clade had good construct validity and external robustness. Analysis of the data revealed two findings. Voice-related characteristic sounds including a 'people' specifier were more liked than those without such a specifier (d = 0.14 SD), possibly due to an emotional crossmodal association mechanism. Liking of characteristic sounds differed between the first and last annotations that the respondents had made (d = 0.21 SD), which might be due to an initially positive bias being countered by exposure to a task inducing a mode of critical listening. We believe that the SSR taxonomy will be useful for field research and simulation design. The empirical findings might inform theory, specifically research charting the perception of sound sources in multimodal environments.

- by PerMagnus Lindborg
- •
- Environmental Engineering, Mechanical Engineering, Acoustics, Perception

The processing of visual and vestibular information is crucial for perceiving self-motion. Visual cues, such as optic flow, have been shown to induce and alter vestibular percepts, yet the role of vestibular information in shaping visual awareness remains unclear. Here we investigated if vestibular signals influence the access to awareness of invisible visual signals. Using natural vestibular stimulation (passive yaw rotations) on a vestibular self-motion platform, and optic flow masked through continuous flash suppression (CFS) we tested if congruent visual–vestibular information would break interocular suppression more rapidly than incongruent information. We found that when the unseen optic flow was congruent with the vestibular signals perceptual suppression as quantified with the CFS paradigm was broken more rapidly than when it was incongruent. We argue that vestibular signals impact the formation of visual awareness through enhanced access to awareness for congruent multisensory stimulation.

- by Roy Salomon
- •
- Embodiment, Consciousness, Vestibular System, Crossmodal Interaction

Restaurants are complex environments engaging all our senses. More or less designable sound sources, such as background music, voices, and kitchen noises, influence the overall perception of the soundscape. Previous research suggested typologies of sounds in some environmental contexts, such as urban parks and offices, but there is no detailed account that is relevant to restaurants. We collected on-site data in 40 restaurants (n = 393), including perceptual ratings, free-form annotations of characteristic sounds and whether they were liked or not, and free-form descriptive words for the environment as a whole. The annotations were subjected to cladistic analysis, yielding a multi-level taxonomy of perceived sound sources in restaurants (SSR) with good construct validity and external robustness. Further analysis revealed that voice-related characteristic sounds including a 'people' specifier were more liked than those without it (d = 0.14 SD), possibly due to an emotional crossmodal association mechanism. Liking of characteristic sounds differed between the first and last annotations that respondents made (d = 0.21 SD), which might be due to an initially positive bias being countered by exposure to a task inducing a mode of critical listening. Comparing the SSR taxonomy with previous classifications, we believe it will prove useful for field research, simulation design, and sound perception theory.

- by PerMagnus Lindborg
- •
- Environmental Engineering, Acoustics, Perception, Environmental Psychology
- by Onur Güntürkün
- •
- Psychology, Cognitive Science, Visual perception, Magnetic Resonance Imaging

- by Christine Leong
- •
- Cognitive Psychology, Cognitive Science, Malaysia, Culture

views contend that behaviorally-relevant multisensory interactions occur relatively late during stimulus processing and subsequently to influences of (top-down) attentional control. In contrast, work from the last 15 years shows that information from different senses is integrated in the brain also during the initial 100 ms after stimulus onset and within low-level cortices. Critically, many of these early-latency multisensory interactions (hereafter eMSI) directly impact behavior. The prevalence of eMSI substantially advances our understanding of how unified perception and goal-related behavior emerge. However, it also raises important questions about the dependency of the eMSI on top-down, goal-based attentional control mechanisms that bias information processing toward task-relevant objects (hereafter top-down control). To date, this dependency remains controversial, because eMSI can occur independently of top-down control, making it plausible for (some) multisensory processes to directly shape perception and behavior. In other words, the former is not necessary for these early effects to occur and to link them with perception (see ). This issue epitomizes the fundamental question regarding direct links between sensation, perception, and behavior (direct perception), and also extends it in a crucial way to incorporate the multisensory nature of everyday experience. At the same time, the emerging framework must strive to also incorporate the variety of higher-order control mechanisms that likely influence multisensory stimulus responses but which are not based on taskrelevance. This article presents a critical perspective about the importance of topdown control for eMSI: In other words, who is controlling whom?

- by Micah M. Murray
- •
- Cognitive Science, Perception, Selective Attention, Auditory Perception

In this work, we investigate the sense of harmony between sound quality and color sensitivity towards a consumer product. We conducted two sensory tests using vacuum cleaners as a case-study. In the first test, participants evaluated their total impressions of multiple samples having different colors and sounds using given adjective pairs. With the result, we analyzed the effects of sound and color upon the total impression and the interaction between the two modalities. In the second test, the participants evaluated synthetic samples that satisfy a variety of color features (CIE L*c*h) and sound quality metrics (SQM). Based on the result, we found that the sense of harmony between colors and sounds changes depending on the sound sharpness. Using factor analysis, we found that the sound sharpness maximizing the sense of harmony correlated with the potency factor of color perception.

Growing evidence shows that individuals consistently match auditory pitch with visual size. For instance, high-pitched sounds are perceptually associated with smaller visual stimuli, whereas low-pitched sounds with larger ones. The present study explores whether this crossmodal correspondence, reported so far for perceptual processing, also modulates motor planning. To address this issue, we carried out a series of kinematic experiments to verify whether actions implying size processing are affected by auditory pitch. Experiment 1 showed that grasping movements toward small/large objects were initiated faster in response to high/low pitches, respectively, thus extending previous findings in the literature to more complex motor behavior. Importantly, auditory pitch influenced the relative scaling of the hand preshaping, with high pitches associated with smaller grip aperture compared with low pitches. Notably, no effect of auditory pitch was found in case of pointing movements (no grasp implied, Experiment 2), as well as when auditory pitch was irrelevant to the programming of the grip aperture, that is, in case of grasping an object of uniform size (Experiment 3). Finally, auditory pitch influenced also symbolic manual gestures expressing "small" and "large" concepts (Experiment 4). In sum, our results are novel in revealing the impact of auditory pitch on motor planning when size processing is required, and shed light on the role of auditory information in driving actions.

- by Luca Rinaldi
- •
- Psychology, Cognitive Science, Sound, Motor Control

This study analysed high-density event-related potentials (ERPs) within an electrical neuroimaging framework to provide insights regarding the interaction between multisensory processes and stimulus probabilities. Specifically, we identified the spatio-temporal brain mechanisms by which the proportion of temporally congruent and task-irrelevant auditory information influences stimulus processing during a visual duration discrimination task. The spatial position (top/bottom) of the visual stimulus was indicative of how frequently the visual and auditory stimuli would be congruent in their duration (i.e., context of congruence). Stronger influences of irrelevant sound were observed when contexts associated with a high proportion of auditory-visual congruence repeated and also when contexts associated with a low proportion of congruence switched. Context of congruence and context transition resulted in weaker brain responses at 228-257 ms post-stimulus to conditions giving rise to larger behavioural cross-modal interactions. Importantly, a control oddball task revealed that both congruent and incongruent audiovisual stimuli triggered equivalent non-linear multisensory interactions when congruence was not a relevant dimension. Collectively, these results are well explained by statistical learning, which links a particular context (here: a spatial location) with a certain level of top-down attentional control that further modulates cross-modal interactions based on whether a particular context repeated or changed. The current findings shed new light on the importance of context-based control over multisensory processing, whose influences multiplex across finer and broader time scales.

- by Micah Murray
- •
- Cognitive Science, Auditory Perception, Attention, Context
- by Micah Murray
- •
- Psychology, Cognitive Science, Perception, Selective Attention

The "pip-and-pop effect" refers to the facilitation of search for a visual target (a horizontal or vertical bar whose color changes frequently) among multiple visual distractors (tilted bars also changing color unpredictably) by the presentation of a spatially uninformative auditory cue synchronized with the color change of the visual target. In the present study, the visual stimuli in the search display changed brightness instead of color, and the crossmodal congruency between the pitch of the auditory cue and the brightness of the visual target was manipulated. When cue presence and cue congruency were randomly varied between trials (Experiment 1), both congruent cues (low-frequency tones synchronized with dark target states or highfrequency tones synchronized with bright target states) and incongruent cues (the reversed mapping) facilitated visual search performance equally, relative to a no-cue baseline condition. However, when cue congruency was blocked and the participants were informed about the pitch-brightness mapping in the cue-present blocks (Experiment 2), performance was significantly enhanced when the cue and target were crossmodally congruent as compared to when they were incongruent. These results therefore suggest that the crossmodal congruency between auditory pitch and visual brightness can influence performance in the pip-and-pop task by means of top-down facilitation.

- by Charles Spence
- •
- Psychology, Cognitive Science, Psychophysics, Attention

Sweeny, Guzman-Martinez, Ortega, Grabowecky, and Suzuki (2012) demonstrated what we call a reverse McGurk effect: Heard speech sounds modulate the perceived shape of briefly presented visual stimuli. We argue that these findings help to... more

- by Ophelia DEROY
- •
- Speech perception, Crossmodal Interaction, Sound symbolism, Crossmodal Associations

- by Jasper van den Bosch
- •
- Cognitive Psychology, Cognitive Science, Malaysia, Culture

- by Marta Łukowska
- •
- Psychology, Cognitive Science, Embodiment, Consciousness
- by Michael Kubovy
- •
- Psychology, Cognitive Science, Psychophysics, Perception

Voice fMRI Psychophysiological interactions a b s t r a c t Introduction: Crossmodality (i.e., the integration of stimulations coming from different sensory modalities) is a crucial ability in everyday life and has been extensively explored in healthy adults. Still, it has not yet received much attention in psychiatry, and particularly in alcoholdependence. The present study investigates the cerebral correlates of crossmodal integration deficits in alcohol-dependence to assess whether these deficits are due to the mere accumulation of unimodal impairments or rather to specific alterations in crossmodal areas.

- by Philippe Timary
- •
- Clinical Psychology, Cognitive Science, Perception, Drugs And Addiction

- by Julia Mossbridge
- •
- Music, Crossmodal Interaction

Triggered by the Covid-19 pandemic, this paper attempts a problematization of the notion of touch in musical performance. The de facto crisis of musical haptics due to physical and social distancing is here considered in the context of a wider phenomenon, namely the ocularcentric objectification of musical embodiment. This reduction of musical embodiment to its visual dimension has been a long, historical process, accelerated by the political economy of cognitive capitalism, including Covid-19 as a catalyst of
pre-existent tendencies. By revealing a crisis of touch, the ongoing sanitary crisis invites us to further reflect on the meaning of musical haptics, beyond the visual properties of embodied gestures and beyond tactility in the design of tangible user interfaces. In that sense, Covid-19 becomes a modern allegory on the multiple senses of touch, similar to the allegories of the senses in Flemish Renaissance painting.

Although vision is considered the dominant modality, recent studies demonstrate the influence of other modalities on visual perception. For example, in the sound-induced flash illusion, two auditory stimuli cause one visual flash to be... more

- by Ladan Shams
- •
- Psychology, Cognitive Science, Medicine, Crossmodal Interaction

Real-world environments are nearly always multisensory in nature. Processing in such situations confers perceptual advantages, but its automaticity remains poorly understood. Automaticity has been invoked to explain the activation of visual cortices by laterally-presented sounds. This has been observed even when the sounds were task-irrelevant and spatially uninformative about subsequent targets. An auditory-evoked contralateral occipital positivity (ACOP) at ~250ms post-sound onset has been postulated as the event-related potential (ERP) correlate of this cross-modal effect. However, the spatial dimension of the stimuli was nevertheless relevant in all prior studies where the ACOP was observed. By manipulating the implicit predictability of the location of lateralised sounds in a passive auditory paradigm, we tested the automaticity of cross-modal activations of visual cortices. 128-channel ERP data from healthy participants were analysed within an electrical neuroimaging framework. The timing, topography, and localisation resembled previous characterisations of the ACOP. However, the cross-modal activations of visual cortices by sounds were critically dependent on whether the sound location was (un)predictable. Our results are the first direct evidence that this particular cross-modal process is not (fully) automatic; instead, it is context-contingent. More generally, the present findings provide novel insights into the importance of context-related factors in controlling information processing across the senses, and call for a revision of current models of automaticity in cognitive sciences.

- by Micah Murray
- •
- Computer Science, Auditory Perception, Attention, Context

- by Onur Güntürkün
- •
- Psychology, Crossmodal Interaction, Hormones, Menstrual Cycle

- by Michael Dodson and +5
- •
- Cognitive Psychology, Cognitive Science, Malaysia, China
- by Hideyoshi Yanagisawa
- •
- Perception, Crossmodal Interaction, Expectations, Texture
- by Michael Schutz
- •
- Perception, Attention, Crossmodal Interaction