Auditory Influences on Visual Temporal Rate Perception (original) (raw)
Related papers
Attention, perception & psychophysics, 2022
The timing of brief stationary sounds has been shown to alter different aspects of visual motion, such as speed estimation. These effects of auditory timing have been explained by temporal ventriloquism and auditory dominance over visual information in the temporal domain. Although previous studies provide unprecedented evidence for the multisensory nature of speed estimation, how attention is involved in these audiovisual interactions remains unclear. Here, we aimed to understand the effects of spatial attention on these audiovisual interactions in time. We utilized a set of audiovisual stimuli that elicit temporal ventriloquism in visual apparent motion and asked participants to perform a speed comparison task. We manipulated attention either in the visual or auditory domain and systematically changed the number of moving objects in the visual field. When attention was diverted to a stationary object in the visual field via a secondary task, the temporal ventriloquism effects on perceived speed decreased. On the other hand, focusing attention on the auditory stimuli facilitated these effects consistently across different difficulty levels of secondary auditory task. Moreover, the effects of auditory timing on perceived speed did not change with the number of moving objects and existed in all the experimental conditions. Taken together, our findings revealed differential effects of allocating attentional resources in the visual and auditory domains. These behavioral results also demonstrate that reliable temporal ventriloquism effects on visual motion can be induced even in the presence of multiple moving objects in the visual field and under different perceptual load conditions.
Beyond perceptual modality: Auditory effects on visual perception
Acoustical Science and Technology, 2001
Three sets of new findings with regard to modulation of visual perception by auditory stimuli are reviewed. First, we show that visual temporal resolution can be either improved or deteriorated by accompanying sounds, depending on the sequence and delay among the auditory and visual stimuli. Second, a single visual flash can be perceived as multiple flashes when accompanied by multiple sounds. Third, an ambiguous motion display consisting of two objects moving toward each other is perceived as streaming with or without an unsynchronized sound, but as bouncing with a synchronized sound. Based on these findings, we argue, against the traditional belief of visual dominance, that audition can modify vision particularly when it provides strong transient signal(s).
PSYCHOPHYSICAL AND PHYSIOLOGICAL ASPECTS OF AUDITORY TEMPORAL PROCESSING
During the past two decades there has been an increasing interest in how the auditory system extracts information from dynamically varying sounds, sounds whose amplitude and/or frequency vary over time. The importance of such information has been clearly demonstrated in speech perception, sound localization, and in many basic aspects of monaural hearing. This chapter reviews recent experimental and theoretical findings that provide insights into the basic mechanisms involved in temporal processing, specifically those involved in monaural envelope processing. One area of interest is the possibility that the auditory system is "tuned" for envelope periodicity, i.e., that there are filters selectively tuned to the modulation frequency of amplitude modulation (AM). A powerful and provocative theoretical account has recently been proposed based on this notion. New psychophysical data relevant to this issue are discussed and a psychophysically-oriented approach for analyzing physiological data on envelope processing is presented. The second area to be considered is the role of timing information, "phase locking", in monaural hearing. Several contemporary models have proposed that fine-structure timing information in the auditory nerve is fundamental in auditory coding. We present psychophysical data on octave phase effects that can be explained qualitatively by a simple model that does not involve the use of fine-structure information. This suggests that except for relatively slow envelope fluctuations, timing information may play a negligible role in monaural hearing.
Auditory temporal modulation of the visual Ternus effect: the influence of time interval
Experimental Brain Research, 2010
Research on multisensory interactions has shown that the perceived timing of a visual event can be captured by a temporally proximal sound. This effect has been termed 'temporal ventriloquism effect.' Using the Ternus display, we systematically investigated how auditory configurations modulate the visual apparent-motion percepts. The Ternus display involves a multielement stimulus that can induce either of two different percepts of apparent motion: 'element motion' or 'group motion'. We found that two sounds presented in temporal proximity to, or synchronously with, the two visual frames, respectively, can shift the transitional threshold for visual apparent motion . However, such effects were not evident with single-sound configurations (Experiment 2). A further experiment (Experiment 4) provided evidence that time interval information is an important factor for crossmodal interaction of audiovisual Ternus effect. The auditory interval was perceived as longer than the same physical visual interval in the sub-second range. Furthermore, the perceived audiovisual interval could be predicted by optimal integration of the visual and auditory intervals.
Seeing and Perceiving, 2012
Exposure to synchronous but spatially discordant auditory and visual inputs produces adaptive recalibration of the respective localization processes, which manifest themselves in measurable aftereffects. Here we report two experiments that examined the time course of visual recalibration of apparent sound location in order to establish the build-up and dissipation of recalibration. In Experiment 1 participants performed a sound localization task before and during exposure to an auditory-visual discrepancy. In Experiment 2, participants performed a sound localization task before and after 60, 180, or 300 exposures to the discrepancy and aftereffects were measured across a series of post-adaptation sound localization trials. The results show that recalibration is very fast. Substantial aftereffects are obtained after only 18-24 exposures and asymptote appears to be reached between 60 and 180 exposures. The rate of adaptation was independent of the size of the discrepancy. The retention of the aftereffect was strong, as we found no dissipation, not even after as few as 60 exposure trials.
Experimental brain research, 2016
Visual capture and the ventriloquism aftereffect resolve spatial disparities of incongruent auditory visual (AV) objects by shifting auditory spatial perception to align with vision. Here, we demonstrated the distinct temporal characteristics of visual capture and the ventriloquism aftereffect in response to brief AV disparities. In a set of experiments, subjects localized either the auditory component of AV targets (A within AV) or a second sound presented at varying delays (1-20 s) after AV exposure (A2 after AV). AV targets were trains of brief presentations (1 or 20), covering a ±30° azimuthal range, and with ±8° (R or L) disparity. We found that the magnitude of visual capture generally reached its peak within a single AV pair and did not dissipate with time, while the ventriloquism aftereffect accumulated with repetitions of AV pairs and dissipated with time. Additionally, the magnitude of the auditory shift induced by each phenomenon was uncorrelated across listeners and visu...
Audition influences color processing in the sound-induced visual flash illusion
Multisensory interactions can lead to illusory percepts, as exemplified by the sound-induced extra flash illusion (SIFI: Shams, Kamitani, & Shimojo, 2000, 2002). In this illusion, an audiovisual stimulus sequence consisting of two pulsed sounds and a light flash presented within a 100 ms time window generates the visual percept of two flashes. Here, we used colored visual stimuli to investigate whether concurrent auditory stimuli can affect the perceived features of the illusory flash. Zero, one or two pulsed sounds were presented concurrently with either a red or green flash or with two flashes of different colors (red followed by green) in rapid sequence. By querying both the number and color of the participants' visual percepts, we found that the double flash illusion is stimulus specific: i.e., two sounds paired with one red or one green flash generated the percept of two red or two green flashes, respectively. This implies that the illusory second flash is induced at a level of visual processing after perceived color has been encoded. In addition, we found that the presence of two sounds influenced the integration of color information from two successive flashes. In the absence of any sounds, a red and a green flash presented in rapid succession fused to form a single orange percept, but when accompanied by two sounds, this integrated orange percept was perceived to flash twice on a significant proportion of trials. In addition, the number of concurrent auditory stimuli modified the degree to which the successive flashes were integrated to an orange percept vs. maintained as separate red-green percepts. Overall, these findings show that concurrent auditory input can affect both the temporal and featural properties of visual percepts.
Low-Level Visual Processing Speed Modulates Judgment of Audio-Visual Simultaneity
Interdisciplinary Information Sciences, 2015
Temporal consistency between visual and auditory presentations is necessary for integration of visual and auditory information. Subjective simultaneity perception is more important than the synchrony of physical inputs for temporal consistency. Our previous studies have shown that audio-visual integration is difficult even if the visual and auditory inputs are physically synchronous when visual processing is slow. In the present study, we examined the effects of visual processing speed on audio-visual integration using a simultaneity judgment task. Visual processing speed was manipulated by varying the spatial frequency of visual stimuli. High spatial frequency stimuli require a longer processing time because visual responses to high spatial frequencies are slow. The results indicated that the difference between subjective and physical synchrony was larger in high spatial frequency than in low spatial frequency. Thus, the spatial frequency of the visual stimulus affected the judgments of simultaneity for visual and auditory stimuli. The effects of visual processing speed on audio-visual integration are believed to occur at a lower-order stage of sensory processing.
Whereas the visual modality tends to dominate over the auditory modality in bimodal spatial perception, the auditory modality tends to dominate over the visual modality in bimodal temporal perception. Recent results suggest that the visual modality dominates bimodal spatial perception because spatial discriminability is typically greater for the visual than for the auditory modality; accordingly, visual dominance is eliminated or reversed when visual-spatial discriminability is reduced by degrading visual stimuli to be equivalent or inferior to auditory spatial discriminability. Thus, for spatial perception, the modality that provides greater discriminability dominates. Here, we ask whether auditory dominance in duration perception is similarly explained by factors that influence the relative quality of auditory and visual signals. In contrast to the spatial results, the auditory modality dominated over the visual modality in bimodal duration perception even when the auditory signal was clearly weaker, when the auditory signal was ignored (i.e., the visual signal was selectively attended), and when the temporal discriminability was equivalent for the auditory and visual signals. Thus, unlike spatial perception, where the modality carrying more discriminable signals dominates, duration perception seems to be mandatorily linked to auditory processing under most circumstances.