Serial dependence in visual perception (original) (raw)

. Author manuscript; available in PMC: 2014 Nov 1.

Published in final edited form as: Nat Neurosci. 2014 Mar 30;17(5):738–743. doi: 10.1038/nn.3689

Abstract

Visual input often arrives in a noisy and discontinuous stream, owing to head and eye movements, occlusion, lighting changes, and many other factors. Yet the physical world is generally stable—objects and physical characteristics rarely change spontaneously. How then does the human visual system capitalize on continuity in the physical environment over time? Here we show that visual perception is serially dependent, using both prior and present input to inform perception at the present moment. Using an orientation judgment task, we found that even when visual input changes randomly over time, perceived orientation is strongly and systematically biased toward recently seen stimuli. Further, the strength of this bias is modulated by attention and tuned to the spatial and temporal proximity of successive stimuli. These results reveal a serial dependence in perception characterized by a spatiotemporally tuned, orientation-selective operator—which we call a continuity field—that may promote visual stability over time.

Keywords: visual history, aftereffect, tilt aftereffect, visual perception, orientation

A crucial function of vision is detecting important changes in the environment, and sensory adaptation aids in maximizing sensitivity to change. The visual system adapts to properties such as color, orientation, object and scene properties, and many others1,2, thereby optimizing how it responds to changes in these attributes3–5. Adaptation is a simple but powerful mechanism for leveraging past visual input to maximize change sensitivity, but there is a flip side to the coin: the physical world is largely stable and continuous over time. Objects, scenes, and physical properties tend to persist over time, making the recent past a good predictor of the present6. The visual system may therefore delicately balance the need to optimize sensitivity to image changes with the desire to capitalize on the temporal continuity of the physical environment. It may often be advantageous to assume that the present visual environment is similar to the one seen moments ago.

One means of combating noise and stabilizing visual estimates would be to introduce serial dependence to visual perception, to systematically bias perception at the present moment toward input from the recent past. The information necessary for such serial dependence to occur may be retained by the visual system—observers can maintain precise information about basic visual features over long delays7, sometimes even in the face of intervening stimuli8. However, the existence of such a memory trace does not necessarily imply that it alters the perception of future stimuli or that serial dependence occurs in perception. Indeed, maximally independent perception from one moment to the next would carry its own advantages, for example in reducing systematic biases over time. Further, given the known benefits of adaptation and ubiquitous negative aftereffects5, it may be that negative aftereffects dominate over any positive serial dependence in perception. While serially dependent perception would be a simple means of capitalizing on the continuity of the physical environment, whether such an effect actually arises in perception remains to be tested.

Here we tested for serial dependence in visual perception using an orientation judgment task. Subjects viewed a series of randomly oriented gratings presented several seconds apart in time and reported the perceived orientation of each grating using an adjustment response. We found that perceived orientation is strongly and systematically attracted toward orientations seen over the last several seconds. This perceptual serial dependence is modulated by attention and is spatially tuned, occurring more strongly for successive stimuli that appear nearby in space. Several control experiments demonstrate that the perceptual serial dependence we report cannot be explained by any known effect of priming, hysteresis, explicit memory, or expectation. Our results reveal a systematic influence of recent visual input on orientation perception at any given moment: perceived orientation, even of unambiguous stimuli, is attracted toward visual input from the recent past.

Results

Serial dependence in orientation perception

To test for serial dependence in perception, in Experiment 1 we presented subjects with suprathreshold (25% Michelson contrast) Gabor stimuli and asked them to report the orientation of each Gabor by adjusting a response bar (Fig. 1). Stimuli were presented for 500 ms and separated in time by about five seconds (variability in trial duration was introduced by the time it took to make a response). Subjects’ error distributions (reported orientation minus correct orientation; Fig. 2a) revealed that while responses were centered on the correct orientations over the course of the entire experiment, on a trial-by-trial basis the reported orientation was systematically (and precisely) biased in the direction of the orientation seen on the previous trial. For example, when the Gabor on the previous trial was oriented more clockwise than the Gabor on the present trial, subjects perceived the present Gabor as being tilted more clockwise than its true orientation. This attraction followed a derivative-of-Gaussian (DoG) shaped curve (Fig. 2a) with an amplitude of ±8.19° in positive and negative directions for the group (P = 1x10−6; permutation test; see Experiment 1 Methods). Each individual subject also showed a significant attraction (all P < 0.01; permutation tests). On trials where the Gabor orientation was very similar to the orientation presented on the previous trial, the attraction effect was nearly perfect, revealed by the near unit slope around zero on the abscissa. This means that the reported orientation of a Gabor on the current trial could be completely captured by the previously seen orientation. The effect was tuned to the relative orientations of the present and previous stimuli, peaking in amplitude when the difference in orientation between trials was 27.78° in either the clockwise or counter-clockwise direction. Put in context, two gratings that were separated by more than three times the just-noticeable-difference (3 JNDs) could look identical depending on what preceded them (see Supplementary Fig. 7). The amplitude of serial dependence fell off with an increasing number of intervening trials (Fig. 2b), but the attraction was still significant for stimuli seen two and three trials (~15 s) back (two-back amplitude of ±4.89°, P = 0.0004; three-back amplitude of ±2.78°, P = 0.007; permutation tests). A control experiment confirmed that serial dependence in perceived orientation arises for both a counterbalanced stimulus sequence (see Methods) and a fully randomized stimulus sequence (see Supplementary Fig. 2), both of which are free of correlation between the orientation in a given trial and the relative orientations seen in preceding trials (see Supplementary Fig. 3).

Figure 1.

Figure 1

Experiment 1 event sequence. Participants viewed high contrast, suprathreshold Gabor patches presented to either the left or right of fixation (on separate, interleaved runs) and reported the perceived orientation of each Gabor by adjusting the orientation of a response bar.

Figure 2.

Figure 2

Orientation perception is serially dependent. a) Error plot from Experiment 1 for one subject. Positive values on the abscissa indicate that the previous trial was more clockwise than the present trial, and positive errors indicate that the reported orientation was more clockwise than the true stimulus orientation. Gray line is average error; black line shows a DoG curve fit to the data. The peak of the DoG fit gives the amplitude of serial dependence. Each individual subject showed significant serial dependence in orientation perception (all P < 0.01) b) Serial dependence amplitude computed for stimuli presented 1, 2, and 3 trials back from the present trial; significant serial dependence was observed in each case. Error bars are 1 s.d. of the bootstrapped distribution. Data in each bar are based on 4 subjects and 260 data points per subject. c) Experiment 3 results. Thin lines are psychometric curve fits to individual subjects’ 2 AFC data and thick lines are fits to group data. PSE was significantly shifted by the presence of an inducer Gabor at the location of one of the stimuli (mean PSE shift of 3.44°; P = 0.0004 for the group; all subjects P < 0.05; based on three subjects, 448 data points each).

Serial dependence without prior motor responses or recall

It is well established that responses and motor execution can be serially dependent9,10. Experiment 1 controlled for any influence of serial dependence in motor execution by randomizing the initial orientation of the response bar on each trial—motor serial dependence (for example, in how long the subject held down the arrow key during the response) would simply add noise that is uncorrelated to the physical or perceived orientation of the stimulus. In Experiment 2, we further tested whether perceptual serial dependence occurs in the absence of prior motor responses: subjects made no response on 25% of trials; all other aspects of the design were identical to Experiment 1. In trials following those in which no response was made (and hence no carryover effect of a motor response was possible), serial dependence was as strong as in trials that followed a response (amplitude of ±6.76°, s.d. = ±0.91°, P = 0.0002 for trials following a response (104 trials from each of 4 subjects); amplitude of ±8.75°, s.d. = ±0.93°, P = 5x10−6 for trials where no response was made on the previous trial (104 trials from each of 4 subjects); difference: P = 0.12; permutation tests).

Several characteristics of serial dependence distinguish it from the many seemingly similar phenomena previously reported. Unlike prior findings of visual hysteresis using ambiguous11–14 or rivalrous15–19 stimuli, serial dependence occurred for suprathreshold, unambiguous stimuli. A follow-up simulation established that the serial dependence we found is not due to trial order effects or statistical artifacts, which can explain some prior reports of autocorrelation in perception20 (see Supplementary Fig. 8 in the Supplementary Modeling). We also found that the ability to explicitly recall previous stimuli21 is not necessary for serial dependence to occur. Two subjects completed a control experiment with stimuli identical to those in Experiment 1 except that in 25% of trials, after reporting the perceived orientation of the stimulus on the current trial, they were asked whether the orientation of the stimulus seen one or two trials ago was oriented clockwise or counter-clockwise of vertical. Participants were at chance in recalling the orientation from two trials back (53% correct; P = 0.69 for a test of above-chance performance; permutation test based on 104 responses), yet we found significant serial dependence in perceived orientation based on stimuli seen two trials back (amplitude of ±3.12°; s.d. = ±0.90°; P = 0.021; two subjects, 408 data points each). In fact, on those trials in which subjects misremembered the orientation of a previous Gabor patch, the serial dependence was consistent with the previously presented stimulus, not the falsely recalled one.

Serial dependence alters perception

One might still be concerned that the serial effects we found result not from serial dependence of perception per se, but rather from serial dependence in higher-level decision processes independent of perception22. If serial dependence does indeed alter perception, subjects should experience a visual illusion: the perceived orientation of one stimulus should be altered relative to a comparison stimulus visible at the same time. To test for such an illusion, in Experiment 3 subjects saw two Gabors simultaneously on each trial, and reported which of the two was tilted more clockwise (two alternative forced choice; 2AFC; see Supplementary Fig. 4 for a depiction of the stimulus sequence). Prior to this 2AFC judgment, subjects saw another pair of Gabors in the same locations and were cued to report the orientation of one—we tested for serial dependence in the 2AFC responses based on the cued (“inducer”) Gabor (see Methods). We found that the inducer altered the perceived orientation of the Gabor subsequently presented in the same location (Fig. 2c), significantly shifting the point of subjective equality (PSE; offset in orientation required to make the two appear to have identical orientations) for the simultaneously presented Gabors (P = 0.0004 for group PSE shift; all individual subjects P < 0.05; bootstrap tests). This shift in PSE cannot be accounted for by a change in decision criterion or repetition of responses and implies that serial dependence directly alters stimulus appearance.

The results of Experiment 3 also demonstrate a key distinction between serially dependent perception and priming23,24 (and the related notion of object files25). Priming yields an improvement in reaction time and/or discriminability of a repeated stimulus. By contrast, the data from Experiment 3 show that serial dependence can effectively reduce the discriminability of simultaneously presented stimuli by altering their appearance. We compared the slope of a psychometric function fit to all of an observer’s trials (without regard to the orientation of the inducer; 448 trials per subject) with the slopes of psychometric functions fit separately to trials in which the inducer was expected to bias the subject toward a “right” response or toward a “left” response (see Methods for Experiment 3). This comparison tested whether factoring out serial dependence by separating trials according to the expected influence of the inducer yielded an improvement in discrimination over the case of considering all trials together. For the group of three subjects who participated in Experiment 3, discrimination slopes were slightly but significantly steeper when trials were separated by the expected influence of serial dependence (mean slope of 5.69 for separated trials vs. 5.35 for combined trials; P = 0.01; permutation test comparing the measured slope difference to a permuted null distribution; see Experiment 3 Methods). This result is not surprising given that different inducers had different effects on PSE in the 2AFC judgment—fitting a psychometric curve to the entire data set without taking into account these PSE shifts must yield at least a somewhat shallower slope. Importantly, this analysis demonstrates that the variable influence of serial dependence from different preceding stimuli has the potential to reduce the overall discriminability measured within a 2AFC paradigm.

Attention Gates Perceptual Serial Dependence

We next tested the degree to which endogenous attention influences perceptual serial dependence. Experiment 3 hinted that focused attention may enhance serial dependence, but did not directly measure the influence of attention. In Experiment 4, participants viewed eight Gabors organized in a ring around the fixation point and were cued to attend to one of the Gabors prior to stimulus onset (Fig. 3a). The cue indicated with 100% validity which of the Gabors the subject would be required to judge at the end of the trial. Analyzing trials in which the cued location remained constant from the previous to the current trial, we found significant serial dependence, a replication of the effect in the first experiment (P = 1x10−5; Fig. 3b). However, when the cued location changed between trials, perceived orientation was not attracted toward the orientation present at the same retinal location on the previous trial, instead showing a trend toward a negative aftereffect (P = 0.059). Thus, attention plays a significant role in determining the strength of serial dependence. We additionally found that serial dependence was not purely retinotopic: perceived orientation was attracted toward the previously attended orientation, even when it fell in a different retinal location (P = 0.0009). Unlike other aftereffects, notably the tilt aftereffect26, serial dependence in perceived orientation is not determined solely by the retinal location of adaptation27, and it is more strongly modulated by attention28. Attention can carry serial dependence across retinal locations, or, put another way, serial dependence is a property tying together locations that are attended at successive moments.

Figure 3.

Figure 3

Attentional modulation of serial dependence. a) Event sequence for one trial in Experiment 4; subjects reported the orientation of the Gabor at the cued location. b) Serial dependence was strong when the same location was attended on successive trials (red data). There was no serial dependence within a location when the location was unattended on the previous trial (light blue data), but there was significant transfer of serial dependence from one location to another when the two locations were attended on successive trials (dark blue data). Thus, attention is necessary for serial dependence and can carry serial dependence across spatial locations. Error bars are 1 s.d. of the bootstrapped distribution. Significance testing was conducted with permutation tests based on 104 data points from each of three subjects per test.

The spatial tuning of perceptual serial dependence

Given that serial dependence can occur for stimuli appearing in different retinal locations, we tested its tuning across retinal space. In Experiment 5, the location of the Gabor stimulus changed randomly from trial to trial while subjects maintained fixation at the center of the screen; subjects reported the orientation of the Gabor using an adjustment response as in Experiments 1, 2, and 4 (Fig. 4a). Figure 4b shows serial dependence computed within a rolling window over the distance between successive stimuli. The amplitude of serial dependence for one-back trials (blue data) was greatest between stimuli that appeared in nearby locations and fell off with increasing distance. A Gaussian curve fit to the data had a standard deviation of 15.2°—a broad but pronounced spatial tuning. The influence of two-back trials on perceived orientation (Fig. 4b, red data) was likewise spatially tuned, with significantly narrower tuning than the one-back case (standard deviation of 8.6°; P = 2x10−7; bootstrap test for a difference in width between the two curves). We repeated the above binning analysis in two-dimensional space. Each pixel in the resulting visualizations (Fig. 4c) is the amplitude of serial dependence for the collection of trials in which the previous stimulus appeared at that location relative to the stimulus location on the present trial (present trial location indicated by the origin). Serial dependence peaked when successive stimuli fell in the same location and dropped off smoothly with increasing distance for both the one-back and two-back cases. In an additional experiment, we found that serial dependence showed spatial tuning in a spatiotopic (world- or head-centered) coordinate frame as well (see Supplementary Fig. 5).

Figure 4.

Figure 4

The spatial tuning of serial dependence. a) Experimental design for Experiment 5: stimuli and timing were the same as in Experiment 1, but the location of the Gabor was randomized on each trial. b) Serial dependence computed within a rolling window over the spatial distance between the current and previous trials; 1-back data in blue, 2-back data in red. The amplitude of serial dependence fell off with increasing spatial separation between successive trials. Shaded regions show ±1 s.d. of the bootstrapped amplitude. The width of a Gaussian curve fit to the 2-back data was significantly smaller than the width of a Gaussian curve fit to the 1-back data (P = 2x10−7; bootstrap test based on curves fit to the means in 61 bins in each the 1-back and 2-back data). c) Serial dependence computed within a two-dimensional rolling window over the relative positions of the previous and current stimuli, with the current stimulus location plotted at the origin. The results reveal a spatial field within which a prior stimulus attracts the perceived orientation of the present stimulus, which we term a perceptual continuity field (CF).

Using the data from Experiment 5, we also tested whether serial dependence occurs for stimuli seen at the fovea. We collected the trials from Experiment 5 for which the location of the stimulus on the previous trial was less than or equal to three degrees visual angle away from the location of the stimulus on the current trial (the data represented at the origin in Fig. 4c). Within these data, we analyzed trials in which the stimulus was presented at or near the fovea (0° to 2° eccentricity). Using the same analysis as in Experiment 1 (Supplementary Fig. 1), we found significant serial dependence in orientation perception for these foveally presented stimuli (amplitude of ±8.46°, s.d. = ±1.55°, P = 0.026; permutation test based on 268 data points).

Discussion

Here we demonstrate a novel influence of visual history on perception: orientation perception is systematically biased toward stimuli seen up to ten or more seconds in the past. This serial dependence results in a change in appearance, and is not due to hysteresis in motor responses or decision processes (Experiments 2 and 3). Further, the strength of serial dependence is strongly modulated by attention (Experiment 4) and is tuned to the spatial and temporal proximity of successive stimuli (Experiment 5). Thus, serial dependence operates over nearby, successively attended locations. We use the term “continuity field” (CF) to describe the spatial region, or kernel (Fig. 4c), within which orientation perception is attracted toward previous stimuli, thus facilitating continuous orientation perception over time.

How is the serial dependence effect we report related to negative aftereffects that follow adaptation? Empirically, the two are dissociated in several experiments reported here. First, the serial dependence we report is not retinally specific, unlike traditional negative tilt aftereffects29. Further, serially dependent perception is strongly modulated by attention, does not require long-duration adaptation, and has a spatiotopic component. In Experiment 7 (see Supplementary Fig. 6) we found that negative aftereffects can emerge using our stimuli, but only with longer stimulus durations, allowing for adaptation. Finally, Experiment 3 pitted negative aftereffects against serial dependence in the case where the attended location changed between successive trials. Even though two sequential Gabors were presented in the same retinal location, the perceived orientation of the second one depended more strongly on the orientation of a different (attended) Gabor patch located elsewhere. The prior Gabor at the same retinal location did not produce enough of a negative aftereffect to overcome the serial dependence. Serial dependence may be present (though unnoticed) in many published experiments involving brief stimulus exposure, and may actually reduce the measured size of the tilt aftereffect (or other negative aftereffects), at least for briefly presented adapting stimuli.

At a theoretical level, serial dependence and negative aftereffects following adaptation may reflect different, competing goals of the visual system. If we see something for an extended duration, the visual system adapts because lengthy exposure to such a stimulus indicates something has changed about the world (e.g., scene illumination or orientation statistics). Negative aftereffects result from this adaptation1,5, which are a byproduct of a recalibration to the (new) world. On the other hand, encountering a brief stimulus that is no longer visible may be more consistent with the interpretation that we, rather than the world, have changed. In this case, adapting (and the consequent negative aftereffects) would not be optimal. Conversely, if the novel aspects of a brief object are attributed to transient events or internal noise (e.g., transient occlusions, camouflage, blinks, saccades, etc.), then it may be adaptive to assume that any momentary change in the object attributes at this moment may not be inherent to the object itself. Thus, serial dependence emerges. With sufficient exposure, however, this assumption is overridden and the visual system adapts to the (new) stimulus. Negative aftereffects thus emerge only with sufficiently long exposure (Experiment 7; Supplementary Fig. 6). The mechanisms that mediate adaptation (negative aftereffects) and serial dependence (positive aftereffects) need not be shared, and may operate on different time scales and at different levels in the visual system. The resulting negative and positive aftereffects, however, are simultaneously present under many circumstances, as demonstrated in Experiment 3 (Fig. 3b).

Although serially dependent perception is a distinct phenomenon, it could arise from established processes including neural gain changes or tuning shifts. As a simple confirmation of this, we constructed labeled-line models that compute perceived orientation by pooling over a population of orientation-tuned channels (see Labeled-line models and Supplementary Fig. 9 in the Supplementary Modeling). By incorporating either a gain change at the exposed orientation30, or a tuning shift away from the exposed orientation4, an oriented stimulus in the future will produce a population response that is biased toward the current stimulus. Applying this model to the stimulus sequences presented in Experiment 1, we found that the model predictions matched subjects’ perceived orientations well (see Supplementary Fig. 9). Although other similar model frameworks (e.g., a Bayesian estimation framework31) could be modified to produce serial dependence, this modeling confirms that serial dependence is consistent with known neural mechanisms, even though it is phenomenologically surprising.

A class of findings superficially related to ours comes from studies using multistable and ambiguous stimuli. When there are conflicting potential interpretations of a stimulus, prior visual experience can sway the interpretation in one direction or the other11–13. Similarly, in rivalrous displays where perceptual ambiguity comes from viewing different stimuli in the two eyes, the dominance of one interpretation over another can be modulated by recent visual experience15–18 or expectation based on prior learning19. Unlike these cases, we studied perception of suprathreshold, unambiguous stimuli. In our experiments, even though the visual input from a single trial was sufficient to unambiguously perceive the stimulus, we nonetheless found a robust influence of prior visual input. Further, unlike statistical learning effects, serial dependence in perception does not rely on repeated displays, implicit or explicit learning, or long exposure durations32,33. Serial dependence may contribute to the perceptual hysteresis observed with multistable and ambiguous stimuli, but it is a more general mechanism for perceptual stability operating in the absence of conflict.

Chopin and Mamassian34 reported that when observers viewed a series of briefly presented gratings over the course of many minutes, orientations seen as many as two minutes prior to a given trial could influence orientation perception on that trial. On first pass, these results might appear related to the serial dependence in orientation perception we present here. However, Chopin and Mamassian’s findings are not related to ours for several reasons. First, the long-term aftereffects that Chopin and Mamassian report may be the result of a statistical artifact. Maus et al.20 showed that the same pattern of results can arise as a consequence of random fluctuations in the trial sequence coupled with the existence of the well-known negative tilt aftereffect. Our experiments and analysis were designed to avoid such artifacts, and we conducted a simulation (see Negative aftereffect model and Supplementary Fig. 8 in the Supplementary Modeling) to verify that for the sequences of trials that our subjects saw, it is not possible for a negative aftereffect of any strength or duration to produce our pattern of results. Second, when we specifically looked for a long-term positive aftereffect in our data (as reported by Chopin and Mamassian), it was not present: serial dependence dropped to near zero after the three-back trial, essentially opposite their finding. Finally, the serial dependence we demonstrate is fundamentally different from their proposed effect in other ways, including its spatial tuning, orientation tuning, and temporal tuning confined to the past ten to fifteen seconds.

Another line of work has posited that the brain constructs “object files”—temporary object representations that tie together an object’s features and location—in order to help us identify a given object as the same object from moment to moment35,36. The notion of object files exists as a description of tracking ability rather than a mechanism in itself. There are many possible mechanisms – serial dependence being one—that could underlie the ability to track an object’s features in the face of object translation, eye movements, occlusion, and other interference. However, much of the evidence for object files comes from priming studies25,36,37, showing that discrimination of an object’s features is faster and more accurate when the same object was previewed earlier. Our present results differ from these prior findings in that we show that a tracked object looks similar from one moment to the next despite substantial changes in its properties (in this case, orientation)—technically, serial dependence causes observers to be less accurate in their perception of instantaneous object properties, but this illusion, like countless other illusions such as motion-induced mislocalizations38, size illusions39, and contrast illusions40 is adaptive under most circumstances.

In a similar vein to the object file literature, some classes of priming effects have been shown to persist and accumulate over multiple trials. Priming of popout23 is one such example: when viewing pop-out visual search displays (displays in which the target can be rapidly located based on a unique feature), observers are faster to move attention to the target when the pop-out feature of the target is repeated from a previous trial. Similar to our present results, this priming effect persists over many trials. Subsequent work has shown similar priming effects, accumulating over trials, for many variants of the search task, including conjunctive visual search41 (occurring independently for different features42,43), as a result of repeated distractors44, and as a result of repetition of information at a global vs. local scale45. These studies point to the existence of a persistent trace of previously seen stimuli that can facilitate detection and discrimination (improving reaction times and accuracy). Critically, our present results demonstrate a distinct phenomenon: the contents of perception are systematically biased toward previously seen stimuli in a spatially-, temporally-, and attentionally-tuned fashion. As mentioned above, in contrast to priming effects, serial dependence can reduce the discriminability of simultaneously viewed stimuli. While priming can enhance performance by capitalizing on predictable patterns in visual input, serial dependence stabilizes our perceptual experience by integrating information over time.

The constancy of object properties is the norm in the physical world, but not the rule—object features do sometimes change suddenly and unexpectedly. Does serial dependence work against our ability to detect sudden changes? The results of Experiment 1 (Fig. 2a) suggest that it does not—we found no serial dependence when the stimulus orientation changed dramatically between trials. On the other hand, there is ample evidence that humans are susceptible to change blindness46–48: when (unnatural) spontaneous and dramatic changes occur in objects and scenes, they often go unnoticed. Perceptual serial dependence may contribute to change blindness (of orientation information at least) by imposing a stability prior on orientation perception, but crucially, serial dependence is gated by attention (Experiment 4), and thus is not responsible for failures to detect changes due to inattention49,50. Under natural conditions, where object properties do not tend to spontaneously change, serial dependence is useful; when unnatural spontaneous changes are introduced, serial dependence may obscure such changes. The characteristics of serial dependence that we report here are thus well suited to meet the delicate balance between the need for sensitivity to change and the need for sensitivity to physical autocorrelations in the visual environment.

The experiments here demonstrate a surprisingly strong serial dependence in orientation perception, whereby very different stimuli seen in succession can appear similar or even identical. This perceptual serial dependence reveals a novel spatially- and temporally-tuned operator—the continuity field (CF)—that could facilitate perceptual continuity of orientation information over time.

Methods

General Methods

The UC Berkeley and Massachusetts Institute of Technology Institutional Review boards approved all experimental protocols. A total of twelve subjects participated in the experiments; 6 females and 6 males, ranging in age from 20 to 32 years. We required participants to be adults with normal or corrected-to-normal vision. We obtained written informed consent from all subjects prior to their participation.

In all experiments, subjects viewed Gabor stimuli from a chin rest positioned 57 cm from a CRT monitor. The Gabors (windowed sine wave gratings) had a peak contrast of 25% Michelson, a spatial frequency of 0.33 cycles/degree, and a 1.5° s.d. Gaussian contrast envelope. Gabors were presented for 500 ms, after which a noise patch was presented for 1s at the same location. Noise patches were presented to minimize negative aftereffects, and consisted of white noise smoothed with a 0.91° s.d. Gaussian kernel and windowed in a 1.5° s.d. Gaussian contrast envelope. A 0.5° diameter white dot served as a fixation point and subjects were instructed to maintain fixation of the dot for the duration of each experiment while performing the task. In all experiments with the exception of Experiment 3 (see below), on each trial subjects reported the perceived orientation of the Gabor by adjusting the orientation of a response bar (a 0.61° wide white bar windowed in a 1.5° s.d. Gaussian contrast envelope) using the left and right arrow keys. The response bar was initiated in a random orientation on each trial and presented in the same location as the Gabor stimulus.

For the main findings in this study we report P values for individual subjects in addition to reporting group statistics, demonstrating significant within-subject effects. Because the effects are robust even within single subjects, the results do not require large samples, as is established in the psychophysical literature. Statistical tests are two-tailed and Bonferroni-corrected for multiple comparisons.

Experiment 1

Stimuli & Design

Four subjects participated in Experiment 1. Each subject completed ten runs of 104 trials each. Figure 1 shows the event sequence for one trial in Experiment 1: a Gabor stimulus was presented at 6.5° eccentricity (to the left or right of the fixation point in separate, randomly interleaved runs) for 500 ms, followed by a 1 s noise patch. After a 250 ms delay, a response bar appeared at the same location the Gabor was presented in, and subjects adjusted its orientation to match the perceived orientation of the Gabor. After making a response, there was a 2 s delay during which only the fixation point was present prior to the onset of the next trial.

We used two approaches for generating trial sequences in Experiment 1: a fully random sequence of orientations (see Supplementary Fig. 2) and a counterbalanced trial sequence. For counterbalanced runs (data shown in Fig. 2a–b), in the beginning of a run, two baseline orientations were chosen at random from the range of zero to 180 degrees. The trial sequence for a run contained trial pairs, where the baseline orientation was presented in the second trial of the pair and an orientation in the range of −60 to 60° (in increments of 10°) relative to the baseline orientation was presented in the first trial of the pair. All possible pairings of the baseline orientations with orientations in the range of −60 to 60 degrees relative to the baseline orientations were presented within a run. For example, in a run with a baseline orientation of 100°, a 100° oriented Gabor was presented in a trial following a 40° Gabor, in a trial following a 50° Gabor, in a trial following a 60° Gabor, and so on. This counterbalancing was conducted for 1-, 2-, and 3-back trial pairings. We measured serial dependence for the baseline trials only. This approach provided a stringent means of measuring serial dependence because it required that a given orientation could be pulled in both the clockwise and counter-clockwise directions, depending on the preceding stimulus. Thus, while this counterbalancing approach reduced the number of usable trials, it had the advantage of requiring serial dependence to occur within a given orientation. A comparison of serial dependence measured in the counterbalanced design versus a randomized design (see Experiments 3–5) in the same subjects revealed no difference in the amplitude of serial dependence.

Analysis

The general approach to measuring perceptual serial dependence is outlined in Supplementary Figure 1. Within a subject, we first plotted the error on each trial (reported orientation minus presented orientation; positive values indicated errors in the clockwise direction) as a function of the difference in orientation between the Gabor presented on the current and previous trials (previous orientation minus current orientation; positive values indicate that the stimulus on the previous trial was more clockwise than the stimulus on the present trial). Thus, for points where the x and y values had the same sign, the subject’s error on that trial fell in the direction of the orientation on the previous trial. To measure the amplitude of serial dependence, or the degree to which subjects’ errors were pulled toward the orientation of the previous stimulus, we fit the error plot with a 1st derivative of a Gaussian curve (DoG), given by y = xawce−(wx)2 where x is the relative orientation of the previous trial, a is the 2 amplitude of the curve peaks, w is the curve width, and c is the constant 2e-12. The constant c rescales the curve so that the a parameter numerically matches the height of the positive peak of the curve for ease of interpretation: the amplitude of serial dependence (a) is the number of degrees that perceived orientation was pulled in the direction of the previous stimulus for the maximally effective difference in orientation between trials. Error bars were computed by bootstrapping the DoG curve fit 5,000 times, sampling from the data with replacement on each iteration, and taking the standard deviation of the a parameters from the resulting bootstrapped distribution. Significance testing was conducted with a permutation test that similarly refit a DoG curve 100,000 times, shuffling the data labels (relative orientation of the previous trial) on each iteration. This permutation procedure generated a null distribution against which the measured amplitude of serial dependence was compared to obtain a P value. P values were taken as the proportion of amplitude estimates in the bootstrapped null distribution that were equal to or larger in absolute value than the subject’s measured amplitude of serial dependence. In the analysis of Experiment 1 and in subsequent analyses where a permutation test was used to assess significance, the exchangeability requirement for a permutation test is met because under the null hypothesis (here, no systematic relationship between orientation judgment errors and relative previous orientation; a flat line), error distributions would not be expected to differ at different locations along the abscissa, and are hence exchangeable.

Experiment 2

Stimuli & Design

Four subjects participated in Experiment 2. Each subject completed two runs of 208 trials each. The experiment was identical to Experiment 1, except that in 25% of trials (randomly selected but constrained to maintain stimulus sequence counterbalancing as described for Experiment 1), the response bar did not appear and the subject made no response. In these trials, in lieu of the response bar subjects saw the fixation point alone for a period of time determined by the running average of the response period duration for previous trials within the same run. This procedure ensured that trials in which the subject did not make a response had the same average duration as the trials in which the subject made a response.

Analysis

Trials in which the subject made no response were discarded. The remaining trials were separated into two groups: those in which a response was made on the previous trial (“response trials”) and those in which no response was made on the previous trial (“no-response trials”). We separately analyzed the two groups of trials using the same curve fitting method as in Experiment 1 to determine whether the execution of a response on the previous trial influenced the strength of serial dependence.

Experiment 3

Stimuli & Design

Three subjects participated in Experiment 3. Each subject completed four runs of 112 trials each. The event sequence for one trial in Experiment 3 is shown in Supplementary Figure 4. Subjects performed two tasks on each trial; a cue reminded subjects of which task to perform for the upcoming stimuli. At the onset of a trial a cue (a white dot windowed in a 0.7° s.d. Gaussian contrast envelope) appeared at 6.5° eccentricity to the left or right of the fixation point. This cue instructed the subject to judge the orientation of the Gabor that subsequently appeared at the cued location. Following the 350 ms cue and a 350 ms delay period, two Gabors were presented simultaneously to the left and right of fixation at 6.5° eccentricity. The Gabors were presented for 500 ms, followed by the presentation of noise patches for 1 s. The subject then adjusted a response bar, presented at the location of the cued Gabor, to report its orientation. Following the subject’s response there was a 1.5 s delay period (fixation only), followed by a central cue: the fixation point dimmed for 350 ms. This cue instructed the subject to compare the orientations of the subsequent Gabors, judging which was oriented more clockwise (on separate, randomly interleaved runs, subjects judged which Gabor was more counter-clockwise). Following the 350 ms central cue and a 350 ms delay period, two Gabors were presented simultaneously to the left and right of fixation at 6.5° eccentricity. The Gabors were present for 500 ms, followed by the presentation of noise patches for 1 s. Subjects then reported which Gabor was oriented more clockwise (or counter-clockwise) with a button press (two alternative forced choice response (2AFC)). Following the subject’s response, there was a 1.5 s delay period (fixation only) prior to the onset of the next trial.

For the second set of Gabors presented in a trial (2AFC judgment), the difference in orientations between the two Gabors varied from −9 to 9° in increments of 3°; the orientations were constrained to the range of ±14.5° around vertical. For the first set of Gabors presented in a trial (adjustment response), the orientation presented at the cued location was either 20° clockwise or 20° counterclockwise relative to the Gabor that would appear at the same location later in the trial. The Gabor at the uncued location had a random orientation, constrained to the range of −34.5 to 34.5 degrees, the same range that all Gabors in the experiment fell in.

Analysis

We term the cued Gabor in the first set of Gabors that appeared in a trial the “inducer”, as the goal of Experiment 3 was to test whether the orientation of this cued Gabor induced a change in the perceived orientation of the Gabor subsequently presented in the same location in the same trial. We binned trials by the influence that we predicted the inducer would have on the subject’s 2AFC response. Say that on a given trial a 20° oriented inducer was presented in the right visual field, followed by 6° and 0° oriented Gabors in the left and right visual fields, respectively. Because the inducer was oriented more clockwise than the subsequent Gabor that appeared in the same location (the right visual field), we predicted that the inducer would bias the subject to perceive the Gabor in the right visual field as oriented more clockwise than its true orientation. This effect of the inducer would make the subject more likely to report that the Gabor in the right visual field was oriented more clockwise than the Gabor in the left visual field in his/her 2AFC response. Similarly, if an inducer appearing in the left visual field was oriented more counter-clockwise than the subsequent Gabor appearing in the left visual field, it would bias the subject to perceive the Gabor in the left visual field as more counter-clockwise than its true orientation. As in the first example, this effect of the inducer would make the subject more likely to report that the Gabor in the right visual field was oriented more clockwise than the Gabor in the left visual field in his/her 2AFC response. Both of these example trials would be collected together into one bin because the inducer had the same predicted influence on the subject’s 2AFC response. In a second bin, we collected the trials in which we predicted the inducer would make the subject more likely to report that the Gabor in the left visual field was oriented more clockwise than the Gabor in the right visual field in his/her 2AFC response. We then fit psychometric functions to the data in each bin separately (Fig. 2c) using logistic regression of the form where y is the proportion of “right” responses, x is the relative orientation of the left and right Gabors, and a and b scale the slope and intercept of the curve fit, respectively. The point of subjective equality (PSE) for each curve fit was taken as the x value at which y = .5 (subjects were equally likely to respond “left” or “right” in the 2AFC judgment). We tested for a significant difference in the PSE for the data in the two bins using a bootstrapping approach, resampling the data in each bin with replacement 5,000 times, recomputing the PSE estimates on each iteration, and recording the difference in PSE estimates for the data in the two bins (ΔPSE). The bootstrapped distribution of ΔPSE estimates was tested against a null hypothesis of no difference in PSE for the data in the two bins. Note that this approach provided a conservative estimate of the true PSE shift that can be induced by perceptual serial dependence. While the analysis assumed that in the 2AFC judgment only the Gabor at the same location as the inducer Gabor was influenced by it, the results of Experiment 5 show that the influence of serial dependence spreads over a broad region of space (Fig. 4). Thus, in the 2AFC judgment, both Gabors were influenced by the inducer, but the Gabor in the same location as the inducer was influenced more, resulting in the significant PSE shifts that we found. We also verified that the response on the first (adjustment) judgment was not correlated with the response on the second (2AFC) judgment, or with whether the subsequently appearing Gabor would be clockwise or counter-clockwise from the inducer.

To test for a difference in the discrimination slopes found when all trials were analyzed together versus when trials were separated by the expected influence of the inducer, we averaged the slopes of psychometric functions fit separately to trials in which the inducer was expected to bias the subject toward a “left” response and trials in which the inducer was expected to bias the subject toward a “right” response. We subtracted from this average the slope of a psychometric function fit to all trials. We compared the group average of this difference score to a group null distribution, generated by repeating the above procedure 5,000 times and permuting the trial labels on each iteration.

Experiment 4

Stimuli & Design

Three subjects participated in Experiment 4. Each subject completed four runs of 78 trials each. The event sequence for a trial in Experiment 4 is shown in Figure 3a. At the onset of a trial subjects saw a cue (a white dot windowed in a 0.7° s.d. Gaussian contrast envelope) in one of eight possible locations along a 9.0° ring around the fixation point. The cue instructed subjects to attend to the orientation of the Gabor that subsequently appeared at the cued location; all other Gabors were task-irrelevant. The cue was present for 350 ms, followed by a 350 ms delay period (fixation point presented alone). Following the cue and delay period, eights Gabors were presented simultaneously in a 9.0° isoeccentric ring around the fixation point. The Gabors were present for 500 ms, followed by 1 s noise patches presented in the same eight locations. Following a 250 ms delay, a response bar appeared at the cued location and subjects adjusted the bar to report the perceived orientation of the Gabor at that location. There was a 2 s delay period (fixation only) following a subject’s response prior to the onset of the next trial. Given that Experiments 1 & 2 established that serial dependence occurs within a given orientation using a counterbalanced stimulus sequence, in Experiment 4 the orientations of the Gabors presented on each trial were randomly drawn from the range of 0 to180 degrees to increase the number of usable trials.

Analysis

Trials were divided into those in which the cued location was the same as in the previous trial and those in which the cued location differed from the previous trial. We then measured the amplitude of serial dependence as in Experiments 1 & 2 in three separate analyses: i) when the cued location was the same as in the previous trial, we used the orientations of the cued Gabors on the current and previous trials to compute serial dependence; ii) when the cued location differed between successive trials, in one analysis we used the orientations of the cued Gabors on the current and previous trials to compute serial dependence; and iii) when the cued location differed between successive trials, in a second analysis we used the orientation of the cued Gabor on the current trial and the orientation of the (uncued) Gabor presented in the same location on the previous trial (Fig. 3b). This analysis generated three separate estimates of the amplitude of serial dependence, measuring serial dependence within an attended location, across two different attended locations, and within a location that was not previously attended.

Experiment 5

Stimuli & Design

Three subjects participated in Experiment 5. Each subject completed ten runs of 100 trials each. The event timing within a trial was identical to that in Experiment 1 (see Fig. 1), but the Gabor stimuli now appeared at random locations within ±12.5° from fixation in the x and y directions (Fig. 4a). In each trial, the response bar appeared in the same location as the Gabor stimulus. The orientation of the Gabor presented on each trial was chosen randomly from the range of (0, 180] degrees.

Analysis

To measure the spatial tuning of serial dependence in Experiment 5, we binned trials according to the spatial separation between the current and previous stimulus locations. We first binned trials according to the distance between the current and previous stimulus locations (distance between successive stimulus locations was computed as (xcurrent-xprevious)2+(ycurrent-yprevious)2) within a 3° rolling window. For the data within a given window, we computed serial dependence amplitude as in Experiment 1. This analysis yielded a plot depicting the drop-off in the amplitude of serial dependence with increasing distance between the stimuli presented in successive trials (Fig. 4b). We fit Gaussian curves separately to the 1-back and 2-back data and tested for a significant difference in the width (standard deviation) of these curves using a bootstrapping approach as in Experiment 3. We resampled the data from each curve with replacement 100,000 times, recomputing the standard deviation estimates on each iteration and recording the difference in standard deviation estimates for the two curves. The bootstrapped distribution of difference scores was tested against a null hypothesis of no difference in the standard deviation of the two curves. We repeated the analysis using the two-dimensional spatial separation between successive trials (i.e., considering x distance and y distance separately). In this case, the rolling window was a circle with a radius of 3°. This analysis yielded a two-dimensional visualization of the drop-off in amplitude of serial dependence with increasing spatial separation between the stimulus positions on successive trials (Fig. 4c). In Figure 4c, the location of the stimulus on the present trial is represented at the origin, and the position of the stimulus on the previous trial relative to the current trial is represented along the x- and y-axes.

Since the stimuli were positioned randomly on each trial, the number of trials could vary across bins in the above analyses. We used a subsampling approach to equate the effective number of trials across bins. Within each bin, we sampled 200 trials from the total set within the bin and computed serial dependence based on those 200 trials. We repeated this subsampling 5,000 times and used the mean serial dependence amplitude across iterations as the measure of serial dependence within the bin. This approach equated the statistical power across bins.

Negative aftereffect and labeled-line models

The procedures for constructing and testing the negative aftereffect and labeled-line models are described in the Supplementary Modeling. For the negative aftereffect model (see Negative aftereffect model in the Supplementary Modeling), we based the parameters of the negative aftereffect on data from reference #51. For the labeled-line models (see Labeled-line models in the Supplementary Modeling), we based the tuning of orientation-selective channels on data from reference #52.

Supplementary Material

1

10

11

2

3

4

5

6

7

8

9

Acknowledgments

We thank Jennifer Shankey for assistance with data collection, and Gerrit Maus, Santani Teng, and Erica Whitney for comments on the manuscript. This work was supported in part by NIH EY018216 and NSF 1245461 to D.W.

Footnotes

Portions of this work were presented at the annual Vision Sciences Society meeting in 2011 and 2012.

Author Contributions:

J.F. and D.W. designed the experiments. J.F. collected the data and carried out the analyses. J.F. and D.W. wrote the manuscript.

References

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1

10

11

2

3

4

5

6

7

8

9