Multi-step planning of eye movements in visual search (original) (raw)
Related papers
Optimal planning of eye movements
2017
The capability of directing gaze to relevant parts in the environment is crucial for our survival. Computational models based on ideal-observer theory have provided quantitative accounts of human gaze selection in a range of visual search tasks. According to these models, gaze is directed to the position in a visual scene, at which uncertainty about task relevant properties will be reduced maximally with the next look. However, in tasks going beyond a single action, delayed rewards can play a crucial role thereby necessitating planning. Here we investigate whether humans are capable of planning more than the next single eye movement. We found evidence that our subjects’ behavior was better explained by an ideal planner compared to the ideal observer. In particular, the location of the first fixation differed depending on the stimulus and the time available for the search. Overall, our results are the first evidence that our visual system is capable of planning.
Eye movements during visual search: the costs of choosing the optimal path
Vision Research, 2001
Saccadic eye movements are usually assumed to be directed to locations containing important or useful information, but such assumptions fail to take into account that planning saccades to such locations might be too costly in terms of effort or attention required. To investigate costs of saccadic planning, subjects searched for a target letter that was contained in either one of two clusters located on either side of a central fixation target. A target was present on each trial and was more likely (probability =0.8) to appear in one cluster than the other. Probabilities were disclosed by differences in cluster intensities. The distance between each cluster and central fixation varied (60%-300%). The presentation time was limited (500 ms) to ensure that a successful search would require a wisely chosen saccadic plan. The best chance of finding the target would be to direct the first saccade to the high-probability location, but only one of the six subjects tested followed this strategy consistently. The rest (to varying degrees) preferred to aim the first saccade to the closer location, often followed by an attempted search of the remaining location. Two-location searches were unsuccessful; performance at both locations was poor due to insufficient time. Preferences for such ineffective strategies were surprising. They suggest that saccadic plans were influenced by attempts to minimize the cognitive and attentional load attached to planning and to maximize the number of new foveal views that can be acquired in a limited period of time. These strategies, though disastrous in our task, may be crucial in natural scanning, when many cognitive operations are performed at once, and the risk attached to a few errant glances at unimportant places is small.
Where to look next? Eye movements reduce local uncertainty
Journal of Vision, 2007
How do we decide where to look next? During natural, active vision, we move our eyes to gather task-relevant information from the visual scene. Information theory provides an elegant framework for investigating how visual stimulus information combines with prior knowledge and task goals to plan an eye movement. We measured eye movements as observers performed a shape-learning and-matching task, for which the task-relevant information was tightly controlled. Using computational models, we probe the underlying strategies used by observers when planning their next eye movement. One strategy is to move the eyes to locations that maximize the total information gained about the shape, which is equivalent to reducing global uncertainty. Observers' behavior may appear highly similar to this strategy, but a rigorous analysis of sequential fixation placement reveals that observers may instead be using a local rule: fixate only the most informative locations, that is, reduce local uncertainty.
Eye guidance in natural vision: Reinterpreting salience
Journal of Vision, 2011
Models of gaze allocation in complex scenes are derived mainly from studies of static picture viewing. The dominant framework to emerge has been image salience, where properties of the stimulus play a crucial role in guiding the eyes. However, salience-based schemes are poor at accounting for many aspects of picture viewing and can fail dramatically in the context of natural task performance. These failures have led to the development of new models of gaze allocation in scene viewing that address a number of these issues. However, models based on the picture-viewing paradigm are unlikely to generalize to a broader range of experimental contexts, because the stimulus context is limited, and the dynamic, taskdriven nature of vision is not represented. We argue that there is a need to move away from this class of model and find the principles that govern gaze allocation in a broader range of settings. We outline the major limitations of salience-based selection schemes and highlight what we have learned from studies of gaze allocation in natural vision. Clear principles of selection are found across many instances of natural vision and these are not the principles that might be expected from picture-viewing studies. We discuss the emerging theoretical framework for gaze allocation on the basis of reward maximization and uncertainty reduction.
Target Selection for Pursuit and Saccadic Eye Movements in Humans
Eye movements were recorded from three subjects as they initiated tracking of a small circle ("target") moving leftward or rightward, above or below the horizontal meridian, either alone or in the presence of a small square ("distractor") moving leftward or rightward on the other side of the horizontal meridian. At the start of each trial, subjects were provided with either a "form" cue (always centrally positioned and having the circular shape and color of the upcoming moving target) or a "location" cue (a small white square positioned where the upcoming target would appear). The latency of pursuit increased in the presence of an oppositely moving distractor when subjects were provided the form cues but not when they were provided the location cues. The latency of saccades showed similar, but smaller, increases when subjects were given the form cues. On many trials with the form cues, pursuit started in the direction of the distractor and then reversed to follow the target. On these trials, the initial saccade often, but not always, also followed the distractor. These results indicate that the mechanisms of target selection for pursuit and saccades are tightly coordinated but not strictly yoked. The shared effects of the distractor on the latencies of pursuit and saccades probably reflect the common role of visual attention in filtering the inputs that guide these two types of eye movements. The differences in the details of the effects on pursuit and saccades suggest that the neural mechanisms that trigger these two movements can be independently regulated.
Scientific Reports
People must decide where, when, and for how long to allocate gaze to perform different motor behaviours. However, the factors guiding gaze during these ongoing, natural behaviours are poorly understood. Gaze shifts help acquire information, suggesting that people should direct gaze to locations where environmental details most relevant to the task are uncertain. To explore this, human subjects stepped on a series of targets as they walked. We used different levels of target uncertainty, and through instruction, altered the importance of (or subjective value assigned to) foot-placement accuracy. Gaze time on targets increased with greater target uncertainty when precise foot placement was more important, and these longer gaze times associated with reduced foot-placement error. Gaze times as well as the gaze shifts to and from targets relative to stepping differed depending on the target's position in the sequence and uncertainty level. Overall, we show that gaze is allocated to reduce uncertainty about target locations, and this depends on the value of this information gain for successful task performance. Furthermore, we show that the spatial-temporal pattern of gaze to resolve uncertainty changes with the evolution of the motor behaviour, indicating a flexible strategy to plan and control movement. To acquire environmental details necessary for performing a visually guided action, such as locating a landmark, reaching to grasp a glass, avoiding obstacles, and regulating foot placement, appropriate temporal and spatial gaze shifts are required. Consider the situation where you are hiking in the woods; you must identify hazards and obstacles, choose the route you wish to take, and step to desired locations on the ground. Here the decision where, when, and for how long to look has important implications for safety, and thus the coupling between gaze location and foot placement is critical. Although novel stimuli and image salience can capture attention and direct gaze, as supported by computer-based visual tasks and computational models 1 , recent research shows that gaze fixations during more naturalistic behaviours are highly task-relevant 2-9. For instance, when making a sandwich, eye movements are directed to the knife, the jelly jar, the bread, and the plate before each item is manipulated 2. When walking across difficult terrain, people predominantly fixate where they will eventually step 5. However, there is little understanding of how or why task-relevant locations are selected and prioritized, or what determines how much time a location is fixated. Consequently, a central unanswered question emerges: what factors determine how gaze is allocated in visually guided motor behaviours? Several brain regions implicated in the control of eye movements are sensitive to reward probability 10-13. For example, the discharge activity of neurons within the monkey lateral intraparietal area (LIP) varies according to the expected (juice) reward associated with an eye movement to a visual target 13. With walking and other motor actions outside the lab, however, fixating a location does not usually elicit a reward. Rather, gaze shifts help the brain gather relevant details necessary for making a motor decision, such as where to place the foot. Thus, reward alone cannot explain gaze allocation during ongoing, naturalistic behaviours. Interestingly, Foley et al. 14 recently showed that certain LIP neurons change firing rates depending on the expected gain in information needed to perform the second action in a two-step decision task, rather than for the expected reward associated with that subsequent action. This highlights the importance of immediate information gain in shaping action decisions. Since our knowledge of the world is imperfect, our sensory feedback is noisy, and the environment changes as movement unfolds over time, this means that many environmental features relevant to an action are uncertain.
Eye movement trajectories and what they tell us
Neuroscience & Biobehavioral Reviews, 2006
In the last two decades, research has shown that eye movement trajectories can be modified by situational determinants. These modifications can inform us about the mechanisms that control eye movements and they can yield information about the oculomotor, memory and attention system that is not easily obtained via other sources. Eye movement trajectories can deviate either towards or away from elements in the visual field. We review the conditions in which these deviations are found and the mechanisms underlying trajectory deviations. It is argued that deviations towards an element are caused by the unresolved competition in the oculomotor system between elements in a visual scene. Deviations away from an element are mainly observed in situations in which top-down preparation can influence the target selection process, but the exact cause of such deviations remains unclear. q
Looking versus seeing: Strategies alter eye movements during visual search
2010
Visual search can be made more efficient by adopting a passive cognitive strategy (i.e., letting the target "pop" into mind) rather than by trying to actively guide attention. In the present study, we examined how this strategic benefit is linked to eye movements. Results show that participants using a passive strategy wait longer before beginning to move their eyes and make fewer saccades than do active participants. Moreover, the passive advantage stems from more efficient use of the information in a fixation, rather than from a wider attentional window. Individual difference analyses indicate that strategies also change the way eye movements are related to search success, with a rapid saccade rate predicting success among active participants, and fewer and larger amplitude saccades predicting success among passive participants. A change in mindset, therefore, alters how oculomotor behaviors are harnessed in the service of visual search.