List of Contributors
M. Donk
G. Mulder
Faculteit der Psychologie en Pedagogek Vrije Universiteit de Boelelaan 1111 10...
69 downloads
1451 Views
34MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
List of Contributors
M. Donk
G. Mulder
Faculteit der Psychologie en Pedagogek Vrije Universiteit de Boelelaan 1111 1081 HV Amsterdam The Netherlands
Institute of Experimental Psychology University of Groningen Kerklaan 30 9751 NN Haren The Netherlands
M. Eimer
D. Nattkemper
MPI fiir Psychologische Forschung Leopoldstr. 24 80802 Munchen Germany
MPI f6r Psychologische Forschung Leopoldstr. 24 80802 Munchen Germany
J. Everatt
0. Neumann
Department of Psychology University of Surrey Guildford GU2 5XH UK
Abt. fiir Psychologie Universit~it Bielefeld Postfach 10 01 31 33501 Bielefeld Germany
Th. C. Gunter
Institute of Experimental Psychology University of Groningen Kerklaan 30 9751 NN Haren The Netherlands
W. Prinz
MPI fiir Psychologische Forschung Leopoldstr. 24 80802 Munchen Germany
H. Heuer
A. F. Sanders
Institut f6r Arbeitsphysiologie an der Universit~it Dortmund Ardeystr. 67 44139 Dortmund Germany
Faculteit der Psychologie en Pedagogek Vrije Universiteit de Boelelaan 1111 1081 HV Amsterdam The Netherlands
H. S. Koelega E. SchriJger
Psychonomy Department University of Utrecht Heidelberglaan 2 3584 CS Utrecht The Netherlands
MPI f/ir Psychologische Forschung Leopoldstr. 24 80802 Munchen Germany
xi
xii
Contributors
H. G. 0. M. Smid
A. H. C. van der Heijden
Institute of Experimental Psychology University of Groningen Kerklaan 30 9751 NN Haren The Netherlands
Unit of Experimental and Theoretical Psychology Leiden University 2333 AK Leiden The Netherlands
G. ten Hoopen
Unit of Experimental and Theoretical Psychology Leiden University 2333 AK Leiden The Netherlands
M. W. van der Molen
Faculteit der Psychologie Universiteit van Amsterdam Roetersstraat 15 1015 WB Amsterdam The Netherlands A. A. Wijers
G. Underwood
Department of Psychology University of Nottingham Nottingham NG7 2RD UK
Institute of Experimental Psychology University of Groningen Kerklaan 30 9751 NN Haren The Netherlands
Introduction A. F. Sanders* and O. N e u m a n n t
*Free University, Amsterdam, The Netherlands and Biele[eld University, Germany
It is elementary historical knowledge in experimental psychology that the concept of attention had a high status in all classical theorizing, culminating in Titchener's (1908) statement that 'the doctrine of attention is the nerve of the whole psychological system, and that as men judge it, so they shall be judged before the general tribunal of Psychology'. It is equally elementary historical knowledge that the concept was rejected as 'mentalistic' by leading proponents of classical behaviorism and as 'non-existent' by Gestalt psychologists (e.g. Rubin, 1921). Attention research did not vanish during the dominance of these theoretical directions (Lovie, 1983), but attention lost the status of a central theoretical construct. It was brought back to respectability as a consequence of the cognitive shift during the 1950s but, yet, it was never fully restored. Thus, Johnston and Dark (1986) stated that 'at a time of despair and panic we turned to William James (1890) where we found new hope and inspiration'. Although it is intuitively quite clear that 'everyone knows what attention is' (James, 1890), it seems to evade systematic analysis. What is the problem with attention? In fact there seem to be several potential problems and pitfalls. A first problem is methodological and concerns inappropriate use as an ad hoc explanatory concept. It is indeed revealing to see the extent to which attention has been invoked as an 'explanation' of a variety of issues ranging from perceptual phenomena such as figure-ground and binocular rivalry to motor phenomena of programming and preparation. Again, attention has been used as an explanatory concept in the context of consciousness and of action planning, and it has even been used as a synonym for processing capacity (Kahneman, 1973). It should be realized that the uncritical application of the concept has caused, and actually is causing, a justified distrust. For instance, Rubin (1921) correctly objected to its ad hoc application to figure-ground perception, as did K6hler (1925) to M611er's (1923) theory of perceiving Gestalts in terms of attention. In short, attention has been viewed as an explanatory concept for almost any mental phenomenon. This may seem to confirm its central significance, but at the same time the concept becomes void of theoretical meaning. Attention cannot serve the role of a simple deus ex machina, but represents a field of inquiry requiring its own processing models. Trivial as this may sound, the issue remains a pitfall against which modern studies are certainly not safeguarded. Handbook of Perception and Action, Volume 3 ISBN 0-12-516163-8
Copyright 9 1996 Academic Press Ltd All rights of reproduction in any form reserved
2
A.F. Sanders and O. Neumann
This leads to a second problem: it has been extremely difficult to define proper research questions about attention without confounding it with issues of either perceptual or motor or central processing as such. If, following James, attention is viewed as a state of selection 'of one out of what seem several simultaneously possible objects or trains of thought' (p. 403), research on attention should deal with the dynamics and limits of selective activity, but not with the ongoing processes themselves. This seemingly trivial point has still been one of the major weaknesses from which the field has suffered, perhaps because attention cannot be studied in isolation but always accompanies some other process. Thus, most topics listed under attention in Woodworth's (1938) classical text would not qualify today as typical for attentional control. In the context of his theoretical scheme, Titchener was fully aware of the danger of confounding when he defined attention as sensory clearness. This had the implication that attention serves only the role of modulating sensory processes. Unfortunately, his introspective experimental technique led into a deadlock and was definitively abandoned after Titchener's death in 1929. It was undoubtedly the merit of Donald Broadbent (1958) that attention regained respectability in the 1950s since he was the first to formulate a workable set of questions, a testable theory and a set of fruitful experimental paradigms. The filter theory encompassed a variety of issues which, also intuitively, belong to the attentional domain: selective listening, automatic detection, the orientation reaction, distraction, vigilance, and doing two things at once. Broadbent clearly saw that a theory of attention should deal with selective as well as with intensive attention. Again, the filter could be either strategically set toward certain signal sources, or automatically attracted by certain signals, thus addressing what James had called active and passive attention. Broadbent also realized the role of motivation in the decision of what to attend (witness the experiments on the effects on environmental stresses and human performance). In monotonous conditions the filter becomes 'satiated' in selecting the same type of information over and over again. Finally, the filter emphasized the control function of attention, since it decided which stimuli would have priority and which not. Filter theory could be shown to be wrong on many occasions but it should be realized that it was the first time an attentional theory was sufficiently predictive that it could be shown to be wrong! Some main problems were probably that it was tuned too much to sensory intake, that it was too much all-or-none, and that it was too exclusively concerned with focused attention. Further, it used Shannon and Weaver's (1949) concept of a limited-capacity channel without adopting the quantitative methods of information theory. Broadbent was the first to acknowledge some of these shortcomings in his 1971 summary by introducing 'pigeonholing' as a counterpart of filtering and by allowing for 'attenuation' instead of all-or-none filtering, both principles elegantly framed in the context of the signal detection model. Apart from the theoretical issues it was probably of importance that the original results on filtering suffered from limitations of the specific experimental paradigms. Thus the interpretation of results on dichotic listening in terms of constraints in shifting attention proved doubtful, while the results on shadowing were limited to the presentation of two auditory messages. The picture was further complicated by pronounced practice effects, apparently leading to automatic processing. The lack of generalization from results obtained in simple experimental paradigms obviously brings back the danger of confounding attentional control with the properties of the domain of processes under investigation.
Introduction
3
These last points suggest a third main problem for studying attention, namely the enormous variety of mental phenomena with which attentional control may be concerned. It has been amply demonstrated that this variety precludes a common heading of 'processing capacity' (Kahneman, 1973), which proposes that subjects may carry out simultaneous tasks as long as capacity limits are not exceeded. As has been repeatedly pointed out, the consequence of assuming multiple processing resources (Navon and Gopher, 1979) implies giving up a unitary concept of attentional control. Perhaps, therefore, attention can be only used in a descriptive sense to indicate a complex field of study (Neumann, 1992), characterized by functionally different systems, such as orienting, sustained action and multiple task performance (Posner, 1978). If this is the case the future is to define specific paradigms, each allowing for the separation of processing as such and attentional control within the constraints of its domain, without any claim about a common mechanism across domains. Whether there is an ultimate unitary attentional coordination remains to be seen, but for the moment the emphasis is on diversity rather than on unity. It should be added that the presently popular connectionist language provides ample opportunities for modeling attention in terms of selective enhancement or inhibition of connections, and of treating problems of attentional control and systems separation, arising in cases of mutually interfering pathways. Accepting diversity is precisely what has happened during the last decades. The developments concern proliferation of various attentional domains such as orienting in space, object localization in visual search, central processing bottlenecks, attention for action, attentional control in dual-task performance, energetical models of effort, etc. Together with this proliferation a score of new experimental paradigms and measurement techniques have been developed and investigated, among which are cost-benefit analysis for separating controlled versus automatic processing (Posner, 1978), consistent mapping and the development of automatic processing (Shiffrin and Schneider, 1977), the analysis of brain potentials (N/i/it/inen, 1982) and recent neurophysiological developments on regional cerebral blood flow (Posner and Petersen, 1990). In particular, the neurophysiological correlates of performance may allow a more precise distinction between processing proper and its attentional control, thus enhancing the theoretical status of the field. Many of these themes are examined by the contributors of this volume. Eight of the 10 chapters can be subdivided into three groups dealing with (1) information intake, (2) central control functions, and (3) the intensity aspect. With respect to information intake a division is made between attention accompanying auditory and visual information processing. With respect to visual processing, separate chapters are concerned with visual search and with stimulus identification and localization, largely reflecting the distinction between divided and focused attention. The central control function of attention is reflected in three chapters, one on dual-task performance, one on involuntary attention and one on automatic versus controlled information processing. There are also two chapters with an emphasis on the intensity aspect of attention, one relating to the energetics of reaction processes and one to sustained attention. The chapter on brain potentials reflects the prospects of electrophysiological techniques, while the final theoretical chapter overviews topics from the nine other chapters in a search for communalities and divergences. It is inevitable that the different chapters reflect equally many biases of individual authors concerning their area of discussion. Some chapters consist of a critical
4
A.F. Sanders and 0. Neumann
review, others highlight specific developments; some are more theoretical and others more experimental; and some are more optimistic and others more pessimistic about the prospects of their area. Perhaps most conspicuously, they are rooted in widely different research t r a d i t i o n s - from h u m a n performance to psycholinguistics, from m e m o r y research to the psychology of music, from the control of eye m o v e m e n t s to the estimation of time and the effects of stress. The diversity of the approaches that can be found in this v o l u m e thus reflects the ubiquity of the selective and control m e c h a n i s m that make up 'attention'.
REFERENCES Broadbent, D. E. (1958). Perception and Communication. London: Pergamon. Broadbent, D. E. (1971). Decision and Stress. New York: Academic Press. James, W. (1890). The Principles of Psychology. New York: Holt. Johnston, W. A. and Dark, D. J. (1986). Selective attention. Annual Review of Psychology, 37, 43-75. Kahneman, D. (1973). Attention and Effort. Englewood Cliffs, NJ: Prentice Hall. K6hler, W. (1925). Gestalttheorie und Komplextheorie. Psychologische Forschung, 6, 458-416. Lovie, A. D. (1983). Attention and behaviorism- fact and fiction. British Journal of Psychology, 74, 301-310. MiiUer, G. E. (1923). Komplextheorie und Gestalttheorie: Ein Beitrag zur Wahrnehmungspsychologie. G6ttingen. N~i~it~inen, R. (1982). Processing negativity: An evoked potential reflection of selective attention. Psychological Bulletin, 92, 605-640. Navon, D. and Gopher, D. (1979). On the economy of human processing systems. Psychological Review, 86, 214-255. Neumann, O. (1992). Theorien der Aufmerksamkeit: von Metaphern zu Mechanismen. Psychologische Rundschau, 43, 83-101. Posner, M. I. (1978). Chronometric Explorations of Mind. Hillsdale, NJ: Erlbaum. Posner, M. I. and Petersen, S. E. (1990). The attention system of the human brain. Annual Review of Neuroscience, 13, 25-42. Posner, M. I. and Snyder, C. R. R. (1975). Facilitation and inhibition in the processing of signals. In P. M. A. Rabbitt and S. Dornic (Eds), Attention and Performance V. London: Academic Press. Rubin, E. (1921). Die Nichtexistenz der Aufmerksamkeit. Bericht iiber den 9. Kongress fiir experimentelle Psychologie (Jena), pp. 211-212. Shannon, C. and Weaver, W. (1949). The Mathematical Theory of Communication. Urbana, IL: The University of Illinois Press. Shiffrin, R. M. and Schneider, W. (1977). Controlled and automatic information processing: Perceptual learning, automatic attending and a general theory. Psychological Review, 84, 127-189. Titchener, E. B. (1908). Lectures on the Elementary Psychology of Emotion and Attention. New York: MacMillan. Woodworth, R. S. (1938). Experimental Psychology. New York: Holt.
Chapter 1 V i s u a l Attention A. H. C. van der Heijden Unit of Experimental and Theoretical Psychology, Leiden University, The Netherlands
'Modern' attention r e s e a r c h - t h e research that contemporary attention research builds u p o n - started at the end of the 1950s (see Lovie, 1983, for a description of this beginning). The auditory attention branch had its beginning in the UK. It started with the introduction of the multi-channel tape recorder and used 'listening to speech' as its main task (see Moray, 1969, for an overview of this early research). The visual attention branch had its origin in the USA and Canada. It started with the rediscovery of an afterimage and had 'reading of letters' as its main task (see Neisser, 1967, for an overview). The outcomes and accomplishments of older, mainly German, research projects were nearly completely neglected (Neumann, in press). Apparently, the hope was that a fresh start, a new language (the information transmission and information processing language) and an improved methodology ('objective' measurements of behavior instead of 'subjective' introspective reports) could bring the progression that psychology looked for. This chapter deals with modern visual attention research. It deals with it in an idiosyncratic way, with regard to the topics chosen as well as the way of presenting these topics. During the past 20 years, there has been an explosion of important empirical and theoretical work in visual attention. But attention still is an elusive concept and many interpretations are presently in vogue. So, selections are necessary and in selections of necessity a personal bias is involved. My bias consists of the view that attention is a selection mechanism dealing with regions in visual space (Van der Heijden, 1992). As a result, in this chapter important lines of research and ingenious contributions to theory will not receive the attention many readers might have wished. Moreover, a way of dealing with the selected topics has to be chosen. In this chapter I opted for a hybrid approach: partly historical, partly systematic and partly personal. In the historical part I try to do some justice to the tremendous formative influence of the initial experimentation and theorizing in modern visual attention research. In the systematic part I try to outline the important achievements of further ingenious experimentation and theorizing. In the personal part I present my view on visual information processing and the role of selective attention therein. Handbook of Perception and Action, Volume 3 ISBN 0-12-516163-8
Copyright 9 1996 Academic Press Ltd All rights of reproduction in any form reserved
6
A . H . C . Van der Heijden
My wish is that these selections will induce sufficient frustration and inspiration that the coming 30 years of research will bring the same tremendous advances as the past 30 years have done.
1
THE START
The afterimage that visual attention research started with is a phenomenon that was already well documented in the German literature (see also Sperling, 1960, one of the few publications that recognized earlier contributions). Von Helmholtz (1871, 1894), for instance, described experiments in which an electric spark was used to illuminate a page with printed letters: 'The electric discharge illuminated the printed page for an indivisible instant during which its image became visible and remained for a very short while as a positive afterimage. Thus, the duration of the perceptibility of the picture was limited to the duration of the afterimage' (Warren and Warren, 1968, translators), and that duration was much longer than the duration of the spark ('an indivisible instant'). Also Wundt (1899), for instance, convincingly argued that it is blatantly false to assume that, in tachistoscopic experiments with short stimulus durations, the duration that a stimulus is seen exactly equals the physical stimulus duration. The duration of the visual sensation evoked by a short stimulus presentation appreciably exceeds the objective exposure duration. There is something like a positive afterimage (see Sperling, 1960, pp. 2223, for further information and details). At the end of the 1950s and the beginning of the 1960s this positive afterimage gave rise to a series of investigations in Canada and the USA. This research was not really seen as visual attention research but as, what can be called, visual memory research. Except Sperling's (1960) 'The information available in brief visual presentations', all landmark papers had 'memory', in one form or another, in their title: 'Paced memorizing in a continuous task' (Mackworth, 1959); 'Short-term memory in vision' (Averbach and Coriell, 1961); 'Short-term storage of information in vision' (Averbach and Sperling, 1961); 'The visual image and the memory trace' (Mackworth, 1962); 'The relation between the visual image and post-perceptual immediate memory' (Mackworth, 1963); 'A model for visual memory tasks' (Sperling, 1963); 'Successive approximations to a model for short-term memory' (Sperling, 1967). Nevertheless, it was from this 'memory-in-vision' research that modern visual attention research gradually emerged. The basic observation these investigators all started with is easily reproduced in a simple experiment, a whole-report experiment. A subject is briefly shown a visual stimulus containing many items, e.g. letters or digits, and is instructed to name as many of these items as possible. It then appears that there is a limit to the number of items that the subject can name. The subject can report only about six or seven items. In fact, this limit is a classic result (Cattell, 1885; Erdman and Dodge, 1898; Glanville and Dallenbach, 1920; all cited in Woodworth and Schlosberg, 1954). However, the very fact that, around 1960, this simple phenomenon was known under a great variety of n a m e s - 'span of perception', 'span of apprehension', 'span of attention', 'span of immediate memory' (Averbach and Sperling, 1961)- already indicates that it was far from clear what exactly determined this limit in performance. Mackworth, Sperling, and Averbach and Coriell were all concerned with elucidating aspects of this problem.
Visual attention
I
A1 I I I
7
I
A2
I
"-
I I
Figure 1.1. Two-store model for visual information processing. Store A: visual representation; A1, visual representation during stimulus presentation; A2, visual representation after stimulus presentation (positive afterimage). Store B: memory for identified items. The first arrow indicates a transition or translation stage between A and B, the second arrow the production of the response.
For a clear explanation of what aspects they were concerned with, it is necessary to make explicit the general framework with which these investigators started (Figure 1.1). One conviction was that, to explain the data obtained in whole-report tasks, at least two stores or memories had to be postulated. First, upon stimulus presentation, visual information enters the visual system and is stored during (A1), and also for a short while after (A2; the positive afterimage), stimulus presentation as a visual representation (store A). However, because responding can be delayed without severe consequences or simply because there is a limit to performance, a second store, containing representations of recognized items, must also be involved (store B). Of course, a transition or translation stage between A and B has to be postulated (the first arrow in Figure 1.1). Finally, the information appears in a (spoken or written) overt response (the second arrow). We are now in a position to have a closer look at the experiments, experimental results and theoretical views of Mackworth, Sperling, and Averbach and Coriell.
1.1
Mackworth: Whole-Report Task
In line with the general scheme just presented, Mackworth (1959) assumed that in the execution of the whole-report task two 'traces' are involved: a 'direct perceptual trace' (store A) and a more durable 'second trace' (store B). She presumed that '... the first trace may be a direct representation of the visual situation, while the second trace tends to be a verbal one. Thus the first direct trace must be translated by the brain into a verbal one to be stored in a more durable form' (Mackworth, 1959, p. 211). After a suggestion by Glanville and Dallenbach (1920), Mackworth (1962) assumed that, with very short stimulus presentations, this translation consists of a reading of the items from the afterimage, i.e. from A2 in our scheme. To further investigate this process, but also to assess the duration of the afterimage and the influence of possible limitations of store B on overt performance, Mackworth (1962, 1963) used a whole-report task and varied the exposure duration. Mackworth (1962) used stimulus cards with digits as items. The exposure durations used ranged from 62 ms to 16 s. Subjects had to write down as many digits as possible and in the correct position (i.e. identification and localization was required). Figure 1.2 presents the, for present purposes, most interesting part of the results. For exposure durations between 62 ms and I s a linear relationship between number of elements named and stimulus exposure time was found. The equation
8
A . H . C . Van der Heijden
7
B
4 I,..,
i,., O
3
e~
2
o~.,.
! i I
1
1"-6'~
1 1
~"
I 1
~
I 3
-d"
I 1
1
1 .~.
Seconds per message duration Figure 1.2. The effect of exposure duration upon the number of digits recalled.
for this relation is approximately: Digits correct = 4 + 3 x exposure time (in seconds). For exposure times longer than I s this linear relationship no longer holds. The increase in number of digits named is then much smaller. Mackworth (1962) regarded this outcome as a complete confirmation of her a priori theoretical views. The linear relationship is taken as evidence for a serial reading (or recognition) process. From the increase in the number of elements reported when the exposure time was increased from 62 ms to I s, it is concluded that the elements are read from the stimulus representation, i.e. when the stimulus is present (A1), at a rate of about three items per second (333 ms per digit). The additional four items that, according to the equation, are reported with a 0 ms exposure duration must, of course, have been read after stimulus presentation from the afterimage (i.e. from A2). Assuming a constant reading rate of three items per second, it follows that this image was readable during 4 x 333 ms, approximately 1350 ms (the intersection with the x-axis). Apparently, with exposure durations longer than I s, capacity limitations of the second store are reached. Therefore the linear relationship breaks down. Mackworth (1963) replicated and extended these results. Besides digits, now also letters, colors and shapes were used as items. Subjects had to name as many elements as possible in the correct order. For exposure times from 100 ms to I s, for
Visual attention
9
all four types of material, a linear relation between exposure time and number of elements reported was found. Of course, this outcome invited the same explanation as the digit results reported by Mackworth (1962): the slope of the linear function directly reflects the reading (or naming) rate and the intercept is an estimate of the number of items read from the afterimage. But now there was even more supportive evidence for this interpretation. Slope and intercept were largest for the function for digits, somewhat smaller for the function for letters, again somewhat smaller for the function for colors and smallest for the function for shapes. This detailed outcome pattern perfectly corresponded with the reading rates for these different types of material observed in an independent part of the experiment (3.6 items per second for digits, 3.1 for letters, 2.2 for colors and 1.9 for shapes). Multiplication of intercept and reading time produced an estimated usable afterimage duration of about 1.5 s for all types of material. Taken all together, according to Mackworth, items are literally read, and it is the exposure time plus afterimage duration and the reading rate that limit the amount reported in whole-report tasks. Only when exposure times are greater than 1 s do 'verbal-memory' limitations influence performance. For all exposure durations the number of items read from an afterimage with a duration of about 1.5 s considerably contributes to performance, i.e. there is an afterimage with a spectacularly long duration.
1.2 Sperling: Partial-Report Task Like Mackworth before him, Sperling (1960) started from the assumption that in the performance of whole-report tasks two 'stores' are involved: one subserving what the subject sees (store A) and one subserving what the subject knows (store B). But, contrary to Mackworth, Sperling (1960) was convinced that, to find out what is really going on in whole-report tasks, it is better to use exposure durations that do not allow useful eye movements, i.e. exposure durations shorter than 150ms. Moreover, Sperling was convinced that in such a situation store B, immediate memory in his terminology, already imposes a limit on what can be reported. From the literature (he refers to Miller, 1956) and from two experiments (Sperling, 1960, experiments 1 and 2), he knew that the capacity of this immediate m e m o r y - the span of immediate m e m o r y - is about 4.5 items. Sperling's main interest, however, was not in the capacity of this immediate memory. What he wanted to know was: How much can be seen in a single brief exposure? His main interest was in the capacity of store A. But, just because store B imposes a low limit on overt performance, a whole-report procedure cannot be used as a measuring stick to determine this capacity. If a subject's introspective reports about 'what is seen' are regarded as inadequate and an objective measurement is required, a trick has to be i n v e n t e d - a trick that ensures the limit of store B plays no role in the performance of the task. The technique for circumventing the limit of store B is the partial-report technique (Sperling, 1960). With this technique subjects are not required to name as many elements as they can as in a whole-report task; they have to report only a sample from the total number of items presented. Obviously, the maximum size of this sample is chosen in such a way that it is smaller than the span of immediate memory.
10
A. H. C. Van der Heijden
In his first partial-report experiment, Sperling implemented this technique in the following way. Stimuli containing rows of letters, e.g. three rows of three letters, were used. A stimulus was exposed for 50 ms. Immediately after the stimulus was turned off, one out of three different tones was presented. The pitch of the tone indicated which row had to be reported: high t o n e - u p p e r row, middle t o n e middle row, low t o n e - bottom row. Of course, the tone as a coded instruction can be given at any moment before, during and after stimulus presentation. In a second partial-report experiment, Sperling systematically varied the moment of presentation of the tone relative to the display. With the indicator just before stimulus presentation and at stimulus offset, the partial-report method resulted in nearly perfect performance. With increasing indicator delays, performance decreased to about 60% correct. By using the rule that the proportion of the sample equals the proportion seen of the whole stimulus, Sperling was able to calculate the number of items seen, the value he was interested in. Figure 1.3 depicts the results obtained. Sperling (1960, p. 20) concluded that with the indicator at stimulus offset '... between two and three times more information is available for partial-reports than for the whole reports.' The decreasing performance observed with increasing indicator delays indicated to him that 'This discrepancy between the two kinds of report is short-lived. Information in excess of that indicated by the whole-report was available to the Ss for only a fraction of a second following the exposure. At the end of this time, the accuracy of partial-reports is no longer very different from that of whole reports' (Sperling, 1960, p. 20). In summary, Sperling showed that many more items are seen than can be reported in a whole-report task. Moreover, the results obtained with the delayed indicator, together with the subject's phenomenological reports, the effects of visual masking and the known facts about the persistence of sensation, reinforced his a priori conviction that the items are still seen for a short interval of time after
9
100
!
75
50
..d
_ ,i 25
!
! / .10 0
! .15
I .30
! .50
l 1.0
0
Delay of instruction tone (see) Figure 1.3. Decay of available information with a stimulus containing 3 x 3 letters. The duration of the light flash is shown on the same time scale at the lower left. The bar at the right indicates the whole-report performance for this material.
Visual attention
11
stimulus exposure, or that the 'information is initially stored as a visual image and that the Ss can effectively utilize this information in their partial reports' (Sperling, 1960, p. 21).
1.3
Averbach and Corielh Bar-Probe Task
Averbach and Coriell (1961) knew that a brief visual stimulation gives rise to a much longer-lasting 'short-term storage in the visual system', called 'positive afterimage', 'retinal persistence', 'persistence of vision', etc., by writers on perception (A2). They also seemed to be convinced that A2 limits the span of perception (Averbach and Coriell, 1961, pp. 310-311). In general, they wanted to learn about the functional properties of this storage, 'its decay, readout and erasure'. Specifically, they wanted to know whether it is the capacity or the storage duration of store A that limits performance in whole-report tasks. Like Sperling (1960), they knew that an eye-fixation is the basic temporal building block of the visual system and that it is of fundamental importance to study what is happening during one such eye-fixation. They therefore used short exposure durations that mimic a single eye-fixation. Averbach and Coriell's partial-report bar-probe task, designed to investigate the properties of this visual storage, is a visual variant of Sperling's partial-report task. Visual displays consisting of two horizontal strings of eight random letters are briefly (50 ms) exposed. A visual spatial cue - a black bar above or below one of the 16 letter positions- is used to indicate a single letter. The subjects have to report the letter indicated and to guess if unsure. As in Sperling's task this visual indicator can be presented at various moments in time relative to the letter display. Averbach and Coriell used this possibility for investigating store A. As far as the temporal properties of store A are concerned, the results obtained essentially replicated Sperling's results. When the indicator preceded the display or was shown simultaneously with it, accuracy was rather high, about 75%. Accuracy gradually decreased when the cue was delayed. At a delay of about 200ms accuracy reached an asymptotic level of about 35%. From the finding that the decay curve did not fall to zero, Averbach and Coriell concluded that performance not only taps store A, but also contains a more permanent memory, i.e. store B, component. Ultimately they arrived at an estimated store A duration of about 250 ms. The spatial properties of store A were really surprising. Figure 1.4 presents the percentages correct per indicated position. The distribution of correct responses over positions is strongly W-shaped. Averbach and Coriell remark that individual letters in all 16 positions were clearly legible. So 'The explanation seems to lie.., in the fact that letters in some positions, although perfectly legible by themselves, are not legible in the context of the array' (p. 315). 'Resolution of the s t o r a g e - o r ease of reading-out- is disturbed when too much data is put in' (p. 316), and that in a very peculiar and intriguing way. Averbach and Coriell (p. 326) summarized their, for our purposes relevant, findings by stating that 'The visual process involves a buffer storage whose read-in is very fast and read-out relatively slow... The storage time is of the order of one-quarter second. The storage capacity is more difficult to assess...'
12
A. H. C. Van der Heijden
100
80
0 o
K 40
20 top row bottom row 1
2
3
4
5
6
7
8
target position Figure 1.4. Accuracy of report in Averbach and Coriell's (1961) partial-report bar-probe study as a function of target position.
1.4
Reading, Selection and Attention
The foregoing does not really do justice to the, often highly ingenious, initial experimental and theoretical work. Nevertheless, the information given suffices to indicate what was regarded as of real importance and where the bulk of modern research in visual attention started from (see Moray, 1969, and van der Heijden, 1992, for a description of some other important research traditions). As the titles of the initial papers, mentioned earlier, already suggested, and as is also readily apparent from the brief descriptions just given, the afterimage-the availability of usable visual information after stimulus exposure- and the possibility of assessing some of its properties with objective measurement techniques was regarded as the most important and exciting finding. And that excitement was infectious. Numerous investigators started to investigate this afterimage, or 'shortterm store', 'visual image', 'persisting sensory trace', 'visual information storage', or, as Neisser (1967) called it, 'icon'. A diversity of methods for investigating its content and duration was developed. A kind of subdiscipline within experimental psychology arose (Haber and Standing, 1969; Haber and Hershenson, 1974), flowered (Coltheart, 1980; Long, 1980), was questioned (Haber, 1983) and seems to have largely disappeared by now. Most of this research is not of any importance for the topic of our concern: attention in vision. We return to a few exceptions later in this chapter.
Visual attention
13
With regard to the information processing operation that is going on in their different types of tasks the investigators were unanimous. The subjects read the items. The observer reads a visual image (Sperling, 1960, p. 27), the process of reading the indicated letter takes time (Averbach and Coriell, 1961, p. 314), and the linear function shows the amount that can be read from the stimulus itself and from the visual image (Mackworth, 1962, p. 58). That there is reading in a reading task is self-evident, so this 'analysis' is not of great interest. In an instruction 'that was well understood' by his subjects, Sperling (1960, p. 11) gives some indications how reading was conceptualized. 'You will see letters illuminated by a flash that quickly fades o u t . . . You will hear a tone during the flash or while it is fading which will indicate which letters you are to attempt to read. Do not read the card until you hear the tone, [etc.]'. So, the idea is that the subject is in complete control of his or her reading, even in a single eye-fixation and when reading is the task. With regard to the speed of this reading process, however, the investigators had different opinions. As we have already seen, Mackworth was convinced that the reading of a single item takes a considerable amount of time: between 300 and 500 ms, depending on the kind of item. By using a chain of assumptions, Averbach and Coriell (1961) were able to estimate the time needed for 'the process of detecting a marker and reading a letter'. Their estimate was that for maximum performance between 200 and 270 ms are needed. For Sperling, however, these common-sense reading rates were not that self-evident. According to Sperling (1960, p. 24), reading goes much faster. He referred to data reported by Baxt (1871) indicating that the time required to read a letter is about 10 ms. We return to this extremely fast reading further on in this chapter. Sperling (1960) and Averbach and Coriell (1961) were well aware that in their respective tasks a 'choice' or a 'selection' (i.e. the act of choosing) is also involved. (Mackworth, 1963, p. 75, mentioned 'a selective process' but had no further use for it in her theoretical elaborations.) Sperling (1960, p. 23) stated 'If more information is available to him than he can remember, the S must "choose" a part of it to remember. In doing so, he has chosen the part to forget...Ss exercised only locational choices... Locational choices are probably not the only effective choices that the S can m a k e . . . Usually, what he does, or attempts to do, is determined by the instructions.' Averbach and Coriell (1961, p. 316) concluded that the visual system stores information and that 'This storage can be tapped selectively on a signal given by the experimenter'. However, not the 'selectivity', but the 'storage' is what was regarded as of importance. Sperling (1960) and Averbach and Coriell (1961) also assumed that in their tasks attention was involved, and that attention had something to do with the 'reading' and with the 'selection' (to the best of my knowledge, Mackworth did not use the word attention, but I am not a wordprocessor). Sperling (1960, p. 24) explained that the unobservable reaction to a high signal tone is 'looking up', where 'looking up' has to be described in terms of a shift of 'attention', where 'attention' has to be conceived as a preparation for, or sensitization to, the correct row. It is, however, far from clear what attention exactly does. Attending is not reading: 'Once his attention is directed to the appropriate row, the S still has to read the letters' (p. 24). Or possibly it is: 'The choice of what part of the stimulus to attend to or of which letters to read is the choice of what fraction of the stimulus information to utilize' (p. 25). To the best of my knowledge, Averbach and Coriell (1961, p. 324) used the
14
A. H. C. Van der Heijden
word attention only once, when they stated '... there is a selective process, which occurs only after the marker appears, when the subject has been cued to direct his attention to the single desired letter.' And, of course, that remark does not really elucidate the functional role of attention in their partial-report bar-probe task. It is not really surprising, and also not of real importance, that in the studies discussed up to now the notion of 'attention' remains rather vague and that its relation with 'reading' and 'selection' is not spelled out in detail. As described, the studies were not really concerned with these topics, but with the duration and capacity of memories, and especially of a memory in vision. That subjects can read, attend and select was taken for granted and gratefully used as a tool to investigate other issues. What is of importance, however, is that in these tasks something like reading, attending and selecting is involved in a nontrivial way. With regard to the bar-probe task Eriksen and Collins (1969, p. 254) remark: 'The perseverating image, or as Neisser (1967) termed it, icon, has generated considerable interest, but an equally if not more interesting characteristic of this experimental technique is the means by which attention can be selectively directed in a matter of milliseconds to the relevant stimulus item.' And, indeed, only a change in emphasis or perspective was needed to change the tasks from visual memory tasks into attention or selection tasks (see also Eriksen, 1990).
2
THE THEORY
The results described were incorporated in, and became supporting evidence for, two very influential general theories concerned with information processing and selection of information: the theories of Neisser (1967) and Broadbent (1971). These two theories had much in common. Moreover, they shared one fundamental, a priori assumption that purportedly made clear why was observed what was observed. The assumption is that the human information processing system has a central capacity limitation (see Kahneman, 1973, for further support). The theories can be regarded as 'one theory' because of their similarity and as 'the theory' because of their influence. On the basis of auditory research, Broadbent (1958, p. 297) concluded 'A nervous system acts to some extent as a single communication channel, so that it is meaningful to regard it as having a limited capacity.' This conclusion reappeared at the start of Broadbent's (1971) study, which also deals with visual information processing. 'Large though the brain is, any conceivable mechanism which could cope simultaneously with all possible states of the eye, the ear, and our other receptors, would probably be even larger' (Broadbent, 1971, p. 9). Neisser (1967, p. 87) agreed that 'To deal with the whole visual input at once, and make discriminations based on any combination of features in the field, would require too large a brain, or too much "previous experience" to be plausible.' 'If we allow several figures to appear at once, the number of possible input configurations is so very large that a wholly parallel mechanism, giving a different output for each of them, is inconceivable' (Neisser, 1967, p. 94). Brains are obviously too small. They have a limited capacity.
Visual attention
15
Of course, the problem then is how to make an appropriate use of this limited brain capacity. Broadbent's (1958, p. 297) second conclusion gave the answer: 'A selective operation is performed upon the input to this channel...'. Broadbent (1971, p. 9) repeated this conclusion when he states 'The workings of the nervous system then are likely to incorporate a good many devices aimed at economizing on the mechanism necessary...Broadbent (1958)...held that the limited capacity portion of the nervous system was preceded and protected by a selective device or filter, which would pass only some of the incoming information.' And again, Neisser (1967, p. 94) agreed: 'To cope with this difficulty [of limited capacity], even a mechanical recognition system must have some way to select portions of the incoming information for detailed analysis.' So, selection is the answer. Appropriate selection guarantees appropriate use of the limited capacity brain. But selection requires something, e.g. a display of possibilities, from which to select. Broadbent (1958, p. 298) had already introduced this provision. He stated that 'Incoming information may be held in a temporary store at a stage previous to the limited capacity channel: it will then pass through the channel when the class of events to which it belongs is next selected.' So, information is selected from a 'buffer or a temporary store' (Broadbent, 1971, p. 10). Also Neisser (1967, p. 94) was clearly aware that for selection a range of alternatives is required. He stated that selection '... implies the existence of two levels of analysis... '. One level of analysis provides the information to select from, the second level deals with the information selected in detail (see also Neisser, 1967, p. 90). And, of course, for the first level the icon comes in very handy. In the foregoing I stressed the correspondences between Neisser's (1967) and Broadbent's (1958, 1971) views, but, of course, there are also differences. For instance, the difference in terms used to characterize the information processing sequence just described is striking. Broadbent talks in terms of 'channels' containing representations of physical features, 'filter settings' for selecting this information and 'identification or categorization' for the operation of the limited capacity portion. He virtually refuses to use the word attention (the word does not appear in the subject index of his 1971 book). Neisser talks about 'preattentive analysers' which form segregated 'objects', about 'attention' as the allotment of analyzing mechanisms to a limited region of the segregated field, and about an 'act of focal attention' in which objects are identified. There are also substantial differences in opinion, for instance, about what 'identification' exactly is: a result of a passive analysis (Broadbent, 1958, 1971) or the outcome of an active construction (Neisser, 1967). For present purposes, however, all this is not of very much importance (except for the point that more languages were made available for expressing basically the same theoretical point of view; see also Kahneman, 1973). Of importance is how the detailed data, provided by Mackworth, Sperling, and Averbach and Coriell, are dealt with.
2.1
Selection for Processing
From the foregoing it will be clear that the evidence showing the existence of a positive afterimage was gratefully accepted. It appeared, under the terms 'buffer store' (Broadbent, 1971) and 'icon' (Neisser, 1967), as an essential component in the
16
A. H. C. Van der Heijden
visual information processing theories. However, because our main interest is not in positive afterimages but in visual attention, we can immediately turn to reading, selection and attention. With regard to the processing, i.e. the identification or categorization, of the information in the tasks discussed, both Neisser (1967) and Broadbent (1971) are extremely clear. Of course, the items are simply read. For both theorists the 'wholereport t a s k ' - t h e task Mackworth was mainly concerned w i t h - p r o v i d e d the essential information. Even before he has given his theoretical analysis, Neisser (1967, p. 42) already knows that 'The fact that the span of apprehension averages only four or five... probably results from the high rate of encoding. In a tachistoscopic experiment the subject must read the fading icon as rapidly as possible... '. Later on this view is elaborated: 'The attentive synthesis of any particular letter or figure takes an appreciable time, of the order of 1 0 0 m s . . . I f a whole row of letters is to be identified, they must be synthesized one at a time... To "identify" generally means to name, and hence to synthesize not only a visual object but a linguistic-auditory one... Hence the span of apprehension is limited to what can be synthesized, and then verbally s t o r e d . . . ' (Neisser, 1967, p. 103). With regard to Mackworth's data (see also Figure 1.2) Broadbent (1971, pp. 173174) remarks: 'We therefore have the evidence we wanted, that the selective process is limited in its rate of working.., we have a buffer store lasting a fixed length of time, and a subsequent serial process which can extract one item after another from the buffer until the limit of time is exceeded. The speed of the serial process is less when, for example, naming colours rather than digits; and it seems therefore to have to do with the allocation of responses to stimuli. In our present terms, it is the elicitation of a category state rather than the mere transmission of evidence about the occurrence of a physical event.' According to Broadbent, the speed of the serial process is the speed specified by Mackworth. Of course, this serial identification also requires a serial selection of the information to be identified. This serial selection has not only to make clear how in Mackworth's tasks the responses appear in the correct order, but also how in Sperling's task responding is restricted to one row and in Averbach and Coriell's task to a single letter. These problems are not dealt with in very much detail; for instance neither Broadbent nor Neisser really explains Averbach and Coriell's results. In general, for Broadbent (1971) the relevant selective process is 'filtering' (or 'stimulus set') guided by instruction, with filtering conceived as a hierarchical process in which stimuli are examined for the presence or absence of some key features (e.g. being red, or on the upper row). Only if these features are present are all other features of the stimulus processed (e.g. the red one is an 'A', or on the upper row is 'X, G, T'). For Neisser (1967) the relevant processes are the processes that determine the priority of encoding. Two processes capable of doing this are distinguished: perceptual set, based on (reading) habits or induced by instruction (pp. 39 and 103), but also the preattentive processes or the outcome of preattentive processes e.g. motion is specified as an effective cue (p. 92). As already indicated, Broadbent (1971) had little use for the word attention. The word is used in the section heading 'Visual experiments in selective attention' but does not show up in the subject index. Following Mackworth's lead, Broadbent managed without attention. Neisser, on the contrary, knows what attention is and
Visual attention
17
is not: '...attention is not a mysterious concentration of psychic energy; it is simply an allotment of analyzing mechanisms to a limited region of the field. To pay attention to a figure is to make certain analyses of, or certain constructions in, the corresponding part of the icon' (Neisser, 1967, pp. 88-89). In fact, for the tasks we are concerned with, attending is (simply) reading after selection (and that is not too different from Sperling's and Averbach and Coriell's views). The foregoing does not really do justice to these two major modern information processing and attention theories. Not mentioned, for instance, has been that both theories recognize other forms of selection than we described. In Neisser's theory a second form of selection is hidden in 'the synthesizing of an object'. In his view 'Paying attention is not just analyzing carefully; rather, it is a constructive act... What we build has only the dimensions we have given it' (Neisser, 1967, p. 96), or '... the detailed properties and features we ordinarily see in an attended figure are, in a sense, "optional"' (Neisser, 1967, p. 94). Broadbent (1971; see also Treisman, 1964, 1969) is very explicit about a second form of selection: response set. This form of selection is induced by an instruction such as 'name the letters, not the digits', and operates, in one way or another, via semantic class (see Broadbent, 1971, pp. 177-180 for details). These alternatives are certainly real and of importance in visual information processing (Bundesen, 1990; Phaf, Van der Heijden and Hudson, 1990; Van der Heijden, 1992). However, because they are not central in this stream of theorizing, for this moment, we can safely set them aside. For present purposes, then, our exposition suffices, because it is now clear that two topics became central in modern theorizing: (1) One concerned with a limited capacity information processing system that identifies items one after another (identification). (2) Another concerned with (selection) mechanisms that regulate that this system is working on only a subset of the visual information at a time, i.e. performs 'selection for processing' (selection).
3
FURTHER DEVELOPMENTS
As stated, Broadbent's (1958, 1971) and Neisser's (1967) theories were very influential. Indeed, they were so influential that it is not difficult to characterize the bulk of theorizing since then and up to now. The nearly generally accepted view became, and still is, that the information processing system has indeed a limited central capacity for processing information and that a selective mechanism, early in a linear stream of information processing, has to protect this system against 'overload' by feeding it with appropriate bits and pieces of information to work upon. In this 'limited-capacity early-selection' view, selection precedes recognition and only what is selected can be recognized. Models expressing this view, with slight elaborations, modifications, variations, extensions and other niceties, were proposed by Kahneman (1973), Coltheart (1972, 1975), Rumelhart (1970), Eriksen and Rohrbaugh (1970), Francolini and Egeth (1980), Johnston and Dark (1985, 1986), Kahneman and Treisman (1984) and many others. Nowadays, a large part of experimentation and theorizing is dominated by Treisman's 'feature integration theory of attention', which is simply a direct descendant and further elaboration of Neisser's (1967) view (see Treisman, 1988, for an overview).
18
A. H. C. Van der Heijden
However, there were also dissidents. These theorists rejected the kernel assumption of limited central capacity. Either their a priori theoretical views or some experimental evidence forced them to defend one or another variant of the theoretical position that the central information processing system is not hindered by capacity limitations or that the capacity issue is simply irrelevant for theory. The first to propose such a view were Deutsch and Deutsch (1963). Related proposals came for example from Norman (1968), Morton (1969), Keele (1973), Shiffrin and Schneider (1977), Duncan (1980), Coltheart (1984), Pashler (1987), Posner (1978), Mewhort and Campbell (1981) and Van der Heijden (1981). In line with the 'unlimited capacity' assumption, most of these dissidents also suggested that selection of information takes place at a stage that contains identified information, i.e. that selection occurs relatively 'late' in a linear stream of information processing. Selection follows pattern recognition and can be based on recognized information. So, there was an alternative for the 'early-selection limited-capacity' view: the 'unlimited-capacity late-selection' view. Because of all kinds of major and minor similarities and differences between and within these two groups of views, and because there is also a substantial number of hybrids, it is impossible to present an intelligible overview. Fortunately, that is not really necessary. We are now in a position to have a more systematic look at some important 'further' developments. Obviously, two lines of further development are of critical importance: developments in the domain of information processing and developments in the domain of information selection.
3.1 Parallel Processing 3.1.1 Retinal Acuity and Eye Movements The first further development is not really a further development. It concerns the delayed recognition within visual information processing psychology of an already long-established fact- the fact that the peripheral eye, or, in general, the peripheral visual system, has severe and peculiar capacity limitations that show up in performance (e.g. Hall and Von Kries, 1879; Poffenberger, 1912; both mentioned in Eriksen and Schultz, 1977). As stated, Broadbent (1971) imported his kernel assumption-limited central capacity for categorization or identification in vision- from auditory information processing, and, possibly, in audition this assumption reflects something real (Neumann, Van der Heijden and Allport, 1986). Neisser (1967) imported his basic assumption- limited central capacity for figural synthesis- from artificial intelligence ('... even a competent automaton would require processes of figure-formation and attention...' (p. 94)), and, of course, we now know that for artifacts this central limitation is certainly real. The important point, however, is whether for visual information processing this assumption also hits something real, and there are strong reasons to doubt that it does so. Until now, virtually all major theories of attention in vision have completely neglected the fact that the visual system has severe peripheral limitations. For instance, no implications whatsoever are attached to the astonishing information, provided by neurophysiology, that there are only two million (peripheral) retinal
Visual attention
19
ganglion cells connecting eye and cortex but that the number of (central) neurons in the cortex concerned with visual information processing is thousands of times larger (Barlow, 1981, 1985). The reason for this neglect seems obvious and even understandable. We 'see' a detailed and panoramic world. (But it is far from clear how the reasoning then proceeds and leads to limited central capacity. We 'see' so much, so much too much information is presented to the eyes, so there must be a limited capacity for really 'seeing'?) The amount of information 'seen in the world' or 'presented to the eyes' is of no relevance, however. Of relevance is only the information the eye presents to the brain. And eyes are very 'limited' in this respect. For instance, the human daylight visual system is, on average, very limited in its capacity for resolving detail. Moreover, the eye is markedly inhomogeneous with respect to this capacity. The highest visual acuity is found in a very small, central region of the retina: the fovea. Most retinal ganglion cells are devoted to the transmission of information from this region to the cortex. From the fovea towards the periphery, acuity falls off progressively and dramatically (Anstis, 1974; Haber and Hershenson, 1974). Appreciably fewer retinal ganglion cells are involved in transporting information from these parafoveal and peripheral regions. In short, the eye sends detailed information to the brain about only a very limited region of the visual world. Just because the eyes send only a limited amount of information to the brain, it seems a priori highly unlikely that the brain cannot deal with the information it receives from the eyes, i.e. has capacity limitations. It is much more likely that the brain is tailor-made for dealing with the information received from the eyes. Indeed, it is well known that most of the visual brain's computing power is devoted to information received from the fovea (see Cowey, 1981, and Levi, Klein and Aitsebaomo, 1985, for the evidence). So, instead of assuming that the brain has severe limitations in the identification or categorization of information, and therefore needs a central protective filter, it seems much better to assume that the eyes, as peripheral filters, prevent the brain from reaching its limits by passing on only a limited amount of information about the world. The eyes as interfaces between world and brain seem perfectly suited for solving Broadbent's capacity problem. Normally, the eyes point only in one direction at a time. Together with their limited capacity, this means that at any moment in time the eyes can sample only a small part of the visual world. Only through a series of time-consuming eye movements and fixations can a larger region of the visual world be effectively dealt with. (Indeed, the perception of the detailed, panoramic world is a process extended in time; a limited-capacity process with the eye as the bottleneck.) So, instead of assuming a central shifting around of processing mechanisms, it seems much better to assume that the eye movements are the mechanisms that allocate central analyzing mechanisms to a limited region of the visual field. By means of eye movements the peripheral visual system seems perfectly capable of solving Neisser's analyzer allocation problem. At least and in general, before central limited capacity is invoked as an explanatory construct, peripheral limitations and peripheral contributions have to be adequately acknowledged (but words such as 'retina', 'acuity', 'retinal acuity', etc., virtually never appear in the 'subject indexes' of major treatments of attention in vision). Of course, aspects of the operation of this peripheral limited-capacity system will show up in experiments and can be mistakenly taken as evidence for central capacity limitations. For instance, in experiments in which eye movements are
20
A. H. C. Van der Heijden
possible and the eyes can provide the central system with a series of new detail, indicators for the operation of the central system will be hidden behind a massive, serial contribution of the peripheral sensory system. This was (almost) certainly the case in Mackworth's experiments in which exposure times were used that allowed more than a single eye-fixation (exposure times greater than 150-200 ms; Figure 1.2). Therefore Mackworth's, and consequently Broadbent's and Neisser's, serial reading at an extremely low rate has to be seriously doubted. To learn something about the properties, e.g. the capacity, of the central information processing system, short exposure times, mimicking what happens in one single eye-fixation, have to be used. But even when exposure times are used that do not allow useful refixations the same interpretation error can arise. For instance, the differential retinal acuity means that, with linear arrays as used in the research of Mackworth, Sperling, and Averbach and Coriell, the assumption that all information will be available simultaneously to some central processor is seriously in error (see Eriksen and Schultz, 1977, for relevant evidence). The differential central arrival times for a n d / o r the differential central quality of information projected on different regions of the retina can induce the spurious impression of a limited central capacity a.nd serial processing (an example follows later). If one wants to eliminate this artifact, circular arrays centered around the fixation point have to be used, as Eriksen did in his line of research (see Eriksen and Steffy, 1964, for the introduction of the circular arrays; see Eriksen, 1990, for an overview of this research).
3.1.2
Perceptual Independence
and Lateral Interference
Within a single eye-fixation there i s - besides retinal a c u i t y - a n o t h e r peripheral (and possibly also central) factor that severely limits performance in multiple-item tasks such as whole-report, partial-report and bar-probe tasks. Averbach and Coriell (1961, p. 316) hinted at this factor when they wrote: '...letters in some positions, although perfectly legible by themselves, are not legible in the context of the array.' In their view this factor (let us call it 'lateral interference') is responsible for the peculiar W-shaped serial position curve for correct reports (see Figure 1.4). Later research produced abundant evidence for the importance of lateral interference, i.e. the adverse effect on identification of a letter caused by a (mask) letter in close spatial and temporal proximity. Eriksen and associates clearly saw that it is futile to ask questions about the capacity of the central information processing system when it is not known whether the peripheral sensory apparatus is capable of transmitting sufficient information to work upon. First it has to be ascertained whether there are enough 'independent sensory channels' to make the information available to the central system. In their research the concept of 'perceptual independence' is central. With perceptual independence, 'At any moment in time the varying sensitivities, or alternatively, error factors, in the visual perceptual system are uncorrelated for elements associated with foveal areas separated by some minimal distance, and further, there is a lack of interaction between forms, simultaneously presented on these separate foveal areas' (Collins and Eriksen, 1967, p. 369). From their research it appears that there is perceptual independence when the items are separated by more than 1 degree (Collins and Eriksen, 1967; Eriksen, 1966; Eriksen and Lappin, 1967), while
Visual attention
21
with a separation of less than 1 degree the retinal sensitivity or error factor is correlated (Collins and Eriksen, 1967; Eriksen, Munsinger and Greenspon, 1966). Subsequent research provided more evidence for the importance of this factor and substantially more detailed information about its ways of operation. It appeared that lateral interference, as Eriksen's research had already shown, is indeed stronger with close item-item spacings than with wide spacings (Bouma, 1970; Wolford and Chambers, 1983). It is more detrimental in peripheral regions of the retina than in the fovea (Bouma, 1970; Wolford and Hollingsworth, 1974). It is not only observed with short but also with extended exposure durations (Estes, Allmeyer and Reder, 1976; Taylor and Brown, 1972). The effect is asymmetric: a mask-item placed on the peripheral side of a target-item is more detrimental for performance than one placed on the foveal side (Bouma, 1970, 1973; Chastain, 1983; Chastain and Lawson, 1979). Finally, lateral interference is probably to some extent feature specific; the accuracy-reducing interaction appears larger for identical or similar letters than for dissimilar letters (La Heij and Van der Heijden, 1983; Shapiro and Krueger, 1983). Estes (1978, p. 188) argued that exactly this factor, lateral interference, in combination with retinal acuity can wholly account for the W-shaped serial position curve found with partial-report bar-probe tasks using linear arrays. Acuity is highest in the fovea and declines progressively towards the periphery. This explains the high level of performance observed with items in the middle of the array. Lateral interference is greatest in the interior of the array where each item has two neighbors and least at the ends of the array where the items have only one neighbor at the foveal side. This explains the high level of performance for the two end items. The two explanations combined can account for the shape of the position curve (see Van der Heijden, 1987, and Hagenzieker, Van der Heijden and Hagenaar, 1990, for numerical examples that illustrate the power of Estes' explanation). Nevertheless, as we will see later, this retinal acuity-lateral interference story cannot form the complete explanation of the W-shaped serial position curve. In partial-report bar-probe tasks, more than identification alone is involved and other factors contribute to the position curve. However, recent experiments and analyses have shown that at least a substantial W-shaped component of this curve has indeed to be attributed to the combination of retinal acuity and lateral interference (Hagenaar, 1990; Hagenzieker et al., 1990; Van der Heijden, 1992). Like retinal acuity, lateral interference as a peripheral effect can seriously hinder us in assessing the capacity of the central information processing system. A peripheral 'limitation' precedes the central system and imposes its characteristics on the total stream of information processing. The peripheral limitation is then easily mistaken as a central limitation and thereby as spurious evidence in favor of limited central capacity theories. If lateral interference is (also) a central effect (Cowey, 1979, 1981), then it is a sort of central 'capacity limitation' not intended by Broadbent's, Neisser's or others' limited capacity theories. Lateral interference has then to be regarded as a property of Broadbent's limited capacity channel and no filter can protect this channel against its own properties. Or, lateral interference is then an inherent property of Neisser's analyzing mechanisms and will accompany them to all limited regions where focal atttention goes. In short, if lateral interference is also a central effect, then it is better not to regard it as a central limitation as intended by limited capacity theories. It is then much more likely that it has a positive function in visual
22
A. H. C. Van der Heijden
information processing (see Phaf et al., 1990, and Van der Heijden, 1990, for some suggestions).
3.1.3 Successive Masking and Parallel Processing To gain more insight into the real capacity and mode of operation of the central information processor, Sperling (Averbach and Sperling, 1961; Sperling, 1963, 1967) used a whole-report task. After Baxt (1871), he tried to get rid of the visual image, and therefore be able to control the duration of the visual sensation, by using a masking stimulus. The stimulus containing the information for whole-report was exposed from 5 up to 60 ms (Sperling, 1963) and was, at termination, immediately followed by a patterned mask which consisted of parts of letters spread randomly over the field. Sperling assumed that this visual noise field would erase the visual image, because 'The noise field would effectively mask the stimulus even if both fields were on simultaneously; therefore, it is assumed to stop any possible persistence of the stimulus' (Averbach and Sperling, 1961, p. 202). The results of these experiments were spectacular. With each 10 ms increase in stimulus exposure time, an additional letter could be named, up to a total of about four letters. Sperling (1963) took this finding as evidence for a very fast serial 'scanning' process (the word 'reading' could obviously no longer be used). However, after a further detailed analysis, Sperling (1967, p. 290) concluded that it must be a parallel scanning process. 'The observation that all locations begin to be reported at better than chance levels even at the briefest exposures, may be interpreted as evidence of an essentially parallel process for letter recognition. This process gives the illusion of being serial because the different locations mature at different rates...' In his model Sperling (1967) assumed that the scanned elements are stored in a 'recognition buffer memory' as a 'program of motor instructions'. 'The important idea.., is that the program of motor instructions for a rehearsal can be set up in a very short time (e.g. 50 ms for 3 letters) compared to the time necessary to execute it (e.g. 500 ms for 3 letters)' (Sperling, 1967, p. 291). Indeed, the important idea here is that 'reading' can be decomposed into a fast parallel recognition component and a slow, subsequent, selection component. This important idea was not accepted, however. Broadbent (1971, p. 17) simply left 'these very rapid processes' aside because '... the area is changing rapidly, is somewhat outside our main interests, and at the time of writing is rather confused...'. But Neisser (1967, p. 33) came up with an important argument against Sperling's experiments: 'In my opinion, this and other "erasure" experiments demonstrate only the effect of exposure time on legibility... The subject reads letters not only from the stimulus when it is on, but from the icon afterwards, despite the presence of the mask' (see also Kahneman, 1968, p. 417). And, indeed, because in Sperling's experiments the stimulus duration covaried with the interval between onset of stimulus and onset of mask (the SOA), it cannot be excluded that the quality of the visual image improved with increased exposure durations and therefore was better able to resist the interfering effects of the masking stimulus. Phrased in a different way, Sperling (1963, 1967) subscribed to an 'interruption theory' of backward masking. He assumed that the number of items recognized, X, equals X=F~(t)
(1)
Visual attention
23
where t is the exposure duration of the stimulus (or the moment of presentation of the mask) and F~ stands for the recognition operation performed before interruption or erasure. There is, however, abundant evidence for an alternative theory of backward masking: the 'integration theory' (Eriksen, 1966; Eriksen and Collins, 1967; Kahneman, 1968, pp. 416-418; Kinsbourne and Warrington, 1962; Schultz and Eriksen, 1977). This theory holds that successive visual events are summed or integrated into a composite, something like a montage. This target-mask montage still allows items to be recognized. According to this view, what Sperling really measured equals X = F~(t) + F2(t).
(2)
Here, Fl(t) again represents the number of elements recognized before mask presentation and F2(t) represents the number recognized from the composite, i.e. the number recognized despite the presence of the mask. And because it is likely that F2(t) increases with increasing t (the quality of the stimulus contribution to the composite increases with increased exposure durations), it follows that Sperling's recognition rates, estimated with equation (1), overestimated the real recognition rates. Sperling mistakenly took F1 + F2 for F~. However, t h i s - i n principle v a l i d - a r g u m e n t can be defeated with its own weapons. Imagine a situation in which both target duration and mask duration are held constant and only the dark interval (the ISI) between target offset and mask onset is increased in length. In such a situation F~(t) in equation (2) can still be interpreted as the number of items recognized before presentation of the mask and F2(t) as the number read from the composite. In this situation, however, there is no reason at all to assume that F2(t) increases with t (i.e. with target duration + ISI). Because the visual image of the target fades during the ISI, it becomes increasingly susceptible to masking and the (undecayed) representation of the mask will therefore increasingly dominate in the composite. So, a decreasing contribution of F2(t) with increasing t is to be expected. In such a situation, Sperling's recognition rates estimated with equation (1) will underestimate the real recognition rates. Van der Heijden (1971) reported an experiment using the conditions specified. The results were essentially the same as those reported by Sperling (1963, 1967). So, it seems that there is not very much reason to doubt Sperling's (very fast) parallel 'scanning' process (see also Coltheart, 1972, for essentially the same conclusion). As stated, Sperling (1963) initially assumed serial processing of items. Later, Sperling (1967) wrote that the assumption of parallel processing was more in accordance with the data. A similar transition is to be found in the theoretical work of Estes and associates in relation with their 'detection' method. Initially their theorizing was mainly concerned with serial scanning (Estes and Taylor, 1964, 1966). Later on, Wolford, Wessel and Estes (1968, p. 444) concluded '... the type of model favored by our results involves parallel rather than serial processing.' With regard to these serial-to-parallel transitions it is important to realize that the evidence for serial processing was obtained from experiments in which the number of elements presented simultaneously was varied. So it is not unlikely that most of this early evidence for serial processing has to be ascribed to the decreasing average retinal acuity and the increasing lateral interference with increasing number of items presented.
24
A. H. C. Van der Heijden
The (unlimited central capacity) parallel processing versus (limited central capacity) serial processing issue has proven to be one of the most unruly issues in visual information processing research. Not only is there the problem to converge upon experimental paradigms that eliminate spurious peripheral influences, resulting from, for example, saccadic eye movements, covarying retinal acuity and differential lateral interference (see Bouma, 1970, for the virtual omnipresence of the latter factor), there are also severe problems of interpretation because parallel and serial mechanisms can mimic each other's outcomes (Snodgrass and Townsend, 1980; Townsend, 1971, 1972, 1974, 1990). Last, but not least, there is the system whose properties are searched for, which most likely works with an intricate combination of parallel and serial processes (Allport, 1989; Van der Heijden, 1992). Nevertheless, despite all this, nowadays there is abundant evidence for various forms of parallel processing. Space does not allow me to deal with this evidence in detail, so I restrict myself to a listing of what I regard as the important paradigms: (1) visual search tasks (see Duncan and Humphreys, 1989; Egeth, Folk and Mullin, 1989, for recent overviews; see also Chapter 2); (2) varieties of the Stroop task (see Van der Heijden, 1992, for an overview); (3) the negative priming task (Driver and Tipper, 1989; see Allport, 1989, for a brief overview). Taken all together, the best bet seems to be that the brain has no problems in dealing with the information provided by the eyes, or that the central system has no capacity limitations in this respect (see also Van der Heijden, 1987, 1992; for related conclusions, mainly reached on the basis of theoretical considerations, see also Allport, 1987, 1989; Neumann, 1987, 1990; Van der Heijden, 1990).
3.2 3.2.1
Early Selection Physical Cues and Unlimited Capacity
Of fundamental importance for the study of (selective) attention in vision is the research concerned with the problem of which stimulus features give a good partial-report performance in Sperling's (1960) partial-report task, or, more generally, the research concerned with which stimulus characteristics afford efficient selection. Sperling showed that when the tone directly indicated a position, efficient selection was observed and that result was often replicated. From subsequent research it appeared that efficient selection was also possible on the basis of color (Clark, 1969; Dick, 1972; Von Wright, 1968, 1972; see also Bundesen, Pedersen and Larsen, 1984; Bundesen, Shibuya and Larsen, 1985), brightness (Von Wright, 1968; see also Bundesen et al., 1984), shape (Turvey and Kravetz, 1970; Von Wright, 1968; see also Bundesen et al., 1984) and size (Von Wright, 1970). So, 'simple physical features' afford efficient selection. Sperling (1960, experiment 6) also investigated what happened when, instead of the members of one of the rows, the tone indicated the members of the derived category 'letters' or 'digits'. He found that, even when the tone preceded the letter-digit array, performance was not better than when half of the items were selected at random. Von Wright (1968, 1970) replicated this negative result. The
Visual attention
25
same outcome was obtained when the cue specified the members of a category characterized by a nonvisual property such as 'vowel' versus 'consonant' (Von Wright, 1970) or 'letters ending with the v o w e l / E / ' versus 'letters ending with the v o w e l / I / ' (Coltheart, Lea and Thompson, 1974). Subsequent research showed that under some circumstances efficient selection by a derived property, such as letter-digit, is possible (see Bundesen, 1987; Bundesen et al., 1984, 1985; Duncan, 1983; Merikle, 1980; Shibuya and Bundesen, 1988). However, because in these experiments not so much attention, but something like 'expectation', might have been the prime selective factor, we will not discuss these results further (see Bundesen, 1990; Van der Heijden, 1992). These results considerably helped in shaping and strengthening the conviction, already put forward by Mackworth, Sperling, and Averbach and Coriell, '... that the original display must produce some effect which is unselective, and which at first contains all the information from all parts of the visual field' and that then '... a selective process which picks out some parts of this information...' comes in (Broadbent, 1971, p. 171). Moreover, the outcome of the partial-report experiments seemed to provide the insight into the nature of the information initially registered in the 'icon' (Neisser, 1967) or the "buffer' (Broadbent, 1971). The empirical generalization was that 'raw', 'elementary', 'precategorical' or 'physical' features such as position, color, brightness, size and shape were represented (and therefore afforded efficient selection) and 'processed', 'derived', 'categorical', or 'semantic' properties such as letter, digit, vowel or consonant were not (and therefore could not serve as the basis for selection) (Coltheart et al., 1974). The observation that there is no efficient selection on the basis of derived properties was taken as evidence that identity information did not become available automatically. Identity information for the stimuli was not only not in the initial store, it was also not somewhere else in the information processing system. So, this pattern of results also strengthened the various variants of the view '...that selection takes place in order to protect a mechanism of limited capacity... (serial processing system, or categorizing mechanism)' (Broadbent, 1971, p. 178). However, neither the point of view that only raw physical features are represented nor the point of view that identity information is not represented are easily substantiated. The basic reason is that the reasoning, starting with data about selection and ending with conclusions about representation and capacity, is not valid. This is easily shown by means of experimental results. Von Wright (1970) showed that when the cue specified 'upright letters' versus 'letters rotated through 180 degrees' no efficient selection was found. Bundesen et al. (1984) found that selection of 'angular letters' and 'curved letters' was inferior to selection by category membership. However, nobody doubts that these subtle visual properties were represented somewhere in the visual system. So, from 'does not afford (efficient) selection' it does not follow 'was not represented in the visual system'. Mere presence is no warrant for efficient selection. Also something like 'conspicuity' (see Broadbent, 1971, p. 180), or 'discriminability' (Duncan, 1980; Nissen, 1985) is a factor of importance (see also Allport, 1989, p. 637), and the distinction between vowels and consonants, letters and digits, upright letters and rotated letters, or angular and curved letters is not very conspicuous. We return to this issue later in the chapter. That appreciably more, and very sophisticated, visual information processing than that resulting in representations of 'raw' and 'simple' physical features
26
A. H. C. Van der Heijden
precedes selection was already recognized by Neisser (1967) and Kahneman (1973). As stated, in Neisser's view, 'preattentive processes', working in parallel over the entire visual field, first structure that field and '... produce the objects which later mechanisms are to flesh out and interpret.' (p. 89). These, what must be, highly complex computational processes are responsible for the Gestalt factors of perceptual organization and perceptual grouping. There is indeed abundant evidence that these factors profoundly affect visual selection (Allport, 1989; Duncan, 1984; Fryklund, 1975; Kahneman and Henik, 1981; Merikle, 1980; Prinzmetal, 1981). Moreover, in visual search tasks not only simple physical features such as line orientation or color can be detected in parallel. There are also three-dimensional emergent features that stand out in visual search (Enns, 1990). As stated, the point of view that identity information does not become available automatically is not easily substantiated on the basis of the selection data. In general, '... behavioral evidence about the relative efficiency of "selection"... is simply irrelevant to questions about the level of processing accorded to the "unselected information"' (Allport, 1987, p. 409; see also Van der Heijden, 1981; 1992). Limited central capacity requires efficient selection, but efficient selection does not require limited central capacity. With the selection data as described it is perfectly possible that identity information becomes available automatically and in parallel but that it cannot serve as a starting point for efficient selection (Coltheart, 1984; Duncan, 1981; Pashler, 1984; Van der Heijden, 1992). In summary, the importance of these experiments is neither in the information they provide about the kind of information stored in 'iconic memory' nor in the insight they provide into the kind of information processing that is going on. The real importance is in the information they provide about the kinds of selection that can be done. It appears that especially conspicuous visual characteristics afford efficient selection. Only in this sense can it be said that the selective operation addresses a stage of precategorical representation or that selection is 'early' selection. However, it is better to say that visual selection is really selection in vision (Van der Heijden, 1992). There is no reason to reject the assumption that the central system suffers no capacity limitations. 3.2.2
Location Errors a n d Early Selection
Also of fundamental importance for the study of selective attention in vision is the research concerned with the problem of what type of errors are made in Averbach and Coriell's (1961) partial-report bar-probe task. Initially it was simply assumed that all errors are errors of identification. Later it was realized that in this task localization errors could also occur (Eriksen and Rohrbaugh, 1970). Therefore two types of errors were distinguished. An error response was counted as an intrusion error when the subject named a letter not present in the array and as an location error when the subject named a letter from the array but from a wrong position (Townsend, 1973). Intrusion errors were regarded as indicators of identification problems and location errors as indicators of problems of localization. The amazing outcome of the research using this analysis is that, with linear arrays and with the letters close together, overwhelmingly more location errors are found than are to be expected on the basis of the traditional identification-problem view (Butler, 1980; Chow, 1986; Hagenaar, 1990; Hagenzieker et al., 1990; Matsuda,
Visual attention
27
1988; Mewhort and Campbell, 1978; Mewhort, Marchetti and Campbell, 1982; Mewhort et al., 1981, 1984; Townsend, 1973). Most of the location errors consist of correct responses to an item adjacent to the item indicated by the probe, i.e. 'near' location errors (Campbell and Mewhort, 1980; Hagenaar, 1990; Hagenzieker et al., 1990; Mewhort and Campbell, 1978; Mewhort et al., 1981). Location errors are distributed in the form of an M over the serial positions and nearly completely complement the W-shape for correct reports, presented in Figure 1.4 (Campbell and Mewhort, 1980; Hagenaar, 1990; Hagenzieker et al., 1990; Mewhort and Campbell, 1978; Mewhort et al., 1981). The decrease in accuracy as a result of delaying the cue and of applying a backward mask is largely reflected by an increase in location errors (Butler, 1980; Butler, Mewhort and Tramer, 1987; Campbell and Mewhort, 1980; Hagenaar, 1990; Mewhort and Campbell, 1978; Mewhort et al., 1981, 1984). Intrusion errors show only minor variations with the different experimental conditions and manipulations. This pattern of results appears completely opposite to that predicted by the reading-of-a-fading-image analogy. The data strongly suggest that it is not so much problems of identification, but problems of localization that impose the limitations in bar-probe performance. To account for the results, Mewhort and associates proposed a 'dual-buffer' model. A representation of the stimulus is first registered in a 'feature buffer'. An 'identifier', working in parallel on all the information in this buffer, identifies the letters and stores its results, together with location information, in a 'character buffer'. An 'attentional mechanism' operates in this store, and selects and passes a code, suitable for output (see especially Mewhort and Campbell, 1981, for a detailed description). Coltheart (1980, 1984) presented a generalized version of this model. In his version of the character buffer, a system composed of 'lexical entries', there is not only identity and location information. The lexical entries are 'tagged' with information about the complete physical manifestation of the item that triggered the entry. It is of fundamental importance to see how in these models 'reading' is conceptualized. Contrary to the traditional views, but in line with the proposal we put forward earlier, it is assumed that all information present in a single eyefixation is processed, i.e. identified, automatically and in parallel. However, also contrary to the received view, but now also contrary to the view we put forward in the previous section, it is assumed that selection takes place at a postcategorical level. Episodic information at this level- the tags in Coltheart's view or an abstract representation of relative spatial position in Mewhort's v i e w - is used for distinguishing relevant information from irrelevant information, and the location errors are explained in terms of saliency of position information and loss of position information in the character buffer. So, in these views selection is not really early selection in vision, but is late selection in a memory containing 'abstract' information. While in these models the existence of an initial, high-capacity, fast-decaying analog representation, a sensory buffer or 'icon', is still recognized, it plays no role in the selection of information. However, from the observation that in bar-probe tasks most errors are location errors, and from the conclusion, based on this observation, that problems of localization rather than problems of identification impose the major limitation, it in no way follows that selection takes place in a store containing identified information and abstract position information. Localization problems can also be perceptual problems. Indeed, with short stimulus-probe intervals it is mainly visual factors,
28
A. H. C. Van der Heijden
such as small inter-item spacings (Eriksen and Rohrbaugh, 1970; Mewhort et al., 1982) and backward masking (see above), that induce the large number of location errors. More, in general, it is not easily substantiated that the initial, high-capacity, analog representation, i.e. vision, plays no role in bar-probe performance. In a series of bar-probe studies, with relatively large inter-item spacings and therefore only few location errors, Van der Heijden and associates obtained clear evidence that with increasing probe delays a progressively decaying visual represention was addressed (Van der Heijden, 1984, 1986; Van der Heijden et al., 1987; Keele and Chase, 1967). With a modified bar-probe task, Pashler (1984) showed that a quality manipulation, which could affect the visual representation but not the derived representation, strongly influenced performance (see Mewhort, Johns and Coble, 1991, for a number of replications). Recent analyses strongly suggest that bar-probe performance is limited by two factors: identification and localization (Hagenaar, 1990; Hagenzieker et al., 1990; Mewhort et al., 1988; Van der Heijden, 1992). Identification is likely to be automatic parallel identification as suggested by Sperling, Coltheart, and Mewhort and associates, with retinal acuity and lateral interference determining the quality of this process. With short stimulus-probe intervals the localization problems are likely to be localization problems in vision, not problems in a higher-order memory. In general, it is not only true that '...behavioral evidence about the relative efficiency of "selection" .. . is simply irrelevant to questions about the level of processing accorded to the "unselected information"'. Evidence suggesting highly efficient (unlimited capacity parallel) processing is also simply irrelevant to the question at what level the selective mechanism intervenes. Unlimited-capacity parallel processing affords late selection, but unlimited-capacity parallel processing certainly does not command late selection (Van der Heijden, 1981, 1987, 1990, 1992). 3.2.3
S p a t i a l L o c a t i o n a n d t h e P o s i t i o n of Objects
As stated, most errors observed in partial-report bar-probe tasks are location errors and most of these location errors are near-location errors. Instead of naming the target indicated by the bar marker, the subject names one of its immediate neighbors. The abundance of near-location errors is not only observed with post-cues but also with pre-cues (Hagenzieker et al., 1990). Of course, this pattern of results strongly indicates that subjects are dealing with a subregion in visual space, and, moreover, that there are limits to the precision with which this can be done. At present it is far from clear what is at the basis of this imprecision. It is possible that, with the short exposure durations used, the spatial resolving power of the visual system is not sufficient for perfect visual localization of letters or bar marker or both. As will become apparent at the end of this chapter, this topic deserves much more investigation. In the present context, however, the precise explanation is not of real importance. Of real importance is that an abundance of near-location errors is not only found when a bar directly indicates the position of the target; the same result is obtained when a property of the target, other than its position, signifies its relevance. Snyder (1972) asked his subjects to name a letter specified by color (e.g. name the red one among the black ones), by fragmentation or by inversion. Incorrect responses tended to come from positions adjacent to the target. Fryklund (1975) asked his
Visual attention
29
subjects to name the five red letters among the 20 black ones. He reports that '...intrusions come from positions immediately adjacent to target positions' (p. 380). Butler et al. (1987) showed their subjects a row of eight letters with a red (black) target embedded in seven black (red) letters and asked them to name the target. These authors stress that the results obtained do not deviate from those obtained with a position cue, so there must have been many near-location errors. In theory, in these experiments position is irrelevant. It is color, inversion or fragmentation, and not position, that distinguishes the target from irrelevant elements, and theories can be invented in which position plays no special role in the selection of information (Broadbent, 1958, 1971; Bundesen, 1990; Duncan, 1984; Phaf et al., 1990). In the reality of empirical results, however, the spatial imprecision indicates that space intermediates in the selection of the target. Color, inversion or fragmentation appears to indicate firstly a position in space, just as a bar marker, and the target appears to be selected via its position in visual space. In a recent study, Tsal and Lavie (1988) provided further evidence for this special role of position in visual selection tasks. In their first experiment a circular array with nine letters was briefly shown. Three letters were red, three green and three brown. The subjects were instructed to report first a letter of a given color (e.g. report first a red letter) and then, without restrictions, as many of the rest as possible. After adjusting the outcome for chance, it appeared that, among the letters reported in addition to the first letter, there were significantly more 'near-location reports', i.e. reports of letters from the two positions adjacent to the letter reported first, than letters with the relevant color or neutral letters. In two subsequent experiments in which first one (experiment 2) or two (experiment 3) curved letters, presented together with angular letters, had to be reported, essentially the same result was obtained. The authors conclude that '.... the direction of attention to a relevant spatial location seems to be a general and mandatory process that takes place irrespective of the dimension according to which the stimulus was initially selected...' (p. 19). They also wonder why spatial position functions as an intermediate because '... this proposed sequence of processing does not seem to be a particularly parsimonious operation of the attentional system' (p. 20). We return to these attentional issues in the last section. This space-as-a-not-avoidable-intermediate is compatible with the abundant evidence, produced in the last two decades, in favor of the view that attention can be conceived as a spotlight that can be directed at circumscribed regions in visual space. Space does not allow me to go into the details of this topic. Suffice it to say that benefits as a result of foreknowledge of the position at which a target is going to appear have been demonstrated with location cues (e.g. a short duration dot on the position of the impending target) and symbolic cues (e.g. an arrow in a neutral position pointing to the right and indicating that the target will appear at the right), in single-item tasks and in multiple-item tasks, with detection, localization, search and recognition as the task to be performed, and with latency and accuracy as the dependent variable (Egly and Homa, 1984; Eriksen, 1990; Eriksen and Hoffman, 1972, 1974; Eriksen and Rohrbaugh, 1970; Eriksen and St James, 1986; Jonides, 1980, 1981, 1983; Posner, 1980; Posner and Cohen, 1984; Posner, Nissen and Ogden, 1978; Van der Heijden and Eerland, 1973; Van der Heijden, Schreuder and Wolters, 1985; Van der Heijden et al., 1987; see Van der Heijden, 1992, for an overview). Indeed, as Duncan (1981, p. 92) states '... it must be important that advance knowledge of position affords such an excellent (perhaps the best) selection cue.'
30
A. H. C. Van der Heijden
This space as an intermediate is also compatible with the evidence that suggests that, at least in a number of circumstances, selection in vision is object selection (Allport, 1987, 1989; Duncan, 1984; Kahneman, 1973; Kahneman and Chajczyk, 1983; Kahneman and Henik, 1981; Kahneman and Treisman, 1984; Neisser, 1967; Treisman, Kahneman and Burkell, 1983). Some thought reveals that '... a physical object may be thought of as a region of the sensory environment that can be separately acted upon in some w a y . . . ' (Allport, 1987, p. 412; my italics). Clearly, in vision an object can be approached and selected via the exact visual location it occupies. So, object selection in vision is simply one specific case of location selection in vision. Given that position is the ultimate intermediate in selection in vision, the problem of the efficiency of other selection criteria, such as color, shape, inversion, letter, vowel, etc., reduces to the problem of the efficiency of these criteria in specifying a position in visual space. As Duncan (1981) argues for all selection criteria, given the theory as it stands, this is purely an empirical matter. Fortunately, empirical research strongly suggests that criteria closely related to position, such as movement of a target (Nakayama and Silverman, 1986; Neisser, 1967), the abrupt onset of a target (Jonides and Yantis, 1988; Yantis and Jonides, 1984) and (stereoscopic) depth (Enns, 1990; Nakayama and Silverman, 1986) are uniquely efficient. And, when in a multiple-item array all items are equivalent with regard to position or attributes related to position, the most likely complete empiric rule seems to be the one that also determines efficiency of visual search: efficiency of search increases continuously with (1) decreasing similarity between target and nontargets and (2) increasing similarity between the nontargets (Duncan, 1989; Duncan and Humphreys, 1989). For the tasks discussed in this chapter only the first part of the rule is of relevance (for certain types of search tasks also the second part is of importance). Using this part of the rule it follows that, for example, the localization of a red item among black ones or a bright item among dim ones is very efficient. The target is the element in the visual field that is most unlike its neighbors, i.e. target-nontarget similarity is appreciably lower than nontarget-nontarget similarity. It also follows that, for example, the localization of a letter among digits or a vowel among consonants will be highly inefficient. There is no reason to assume that the target is most unlike its neighbors, or that target-nontarget similarity is lower than nontarget-nontarget similarity. So, it appears very likely that an attribute's efficiency in specifying a position in visual space corresponds with its efficiency in serving as a selection criterion in vision. The reader is referred to Duncan (1989) and Duncan and Humphreys (1989) for several intriguing suggestions about how this localization is performed.
4
A MODEL
From the further developments reported in the preceding sections, three outcomes are essential because they strongly point at, and easily lead to, a parsimonious model for information processing, selection and attention for the aspects of task performance with which we were concerned (see Van der Heijden, 1992, for a more disciplined derivation and a more complete account). To see that, we have to keep
Visual attention
31
in mind that: (1) '...behavioral evidence about the relative efficiency of "selection"... is simply irrelevant to questions about the level of processing accorded to the "unselected information"' (Allport, 1987, p. 409). While limited central capacity requires early selection, early selection does not require limited central capacity. (2) '... evidence suggesting highly efficient (unlimited-capacity parallel) processing is simply irrelevant to the question at what level the selective mechanism intervenes' (Van der Heijden, 1981). While unlimited central capacity affords late selection, unlimited central capacity does not command late selection. In other words, we have to realize that the limited capacity versus unlimited capacity issue and the early selection versus late selection issue are largely independent issues (see also Allport, 1987, 1989; Kahneman and Treisman, 1984, p. 55; Van der Heijden, 1981, 1987, 1992). The three essential outcomes mentioned earlier are: (1) Within the limits set by retinal acuity and lateral masking there is, within a single eye-fixation, an unlimited central capacity for processing, i.e. identification or categorization, of visual information. (2) Selection of information is early selection, or selection is selection in vision (obviously, this selection is not selection for 'processing' as suggested by orthodox early selection theories). (3) This early selection, whether controlled by verbal instructions, position cues, criterion attributes or whatever else one can have (e.g. by 'the subject'), is ultimately accomplished via a region in visual space. That these three outcomes easily lead to a parsimonious model is most readily seen when we first briefly turn to some suggestions from contemporary neurophysiology that are of help for the interpretation of 'unlimited capacity for processing within a single eye-fixation'. Here we can only briefly indicate these suggestions (for more detailed, general, accounts see Allport, 1989; De Yoe and Van Essen, 1988; Livingstone and Hubel, 1988; Maunsell, 1987; Mishkin, Ungerleider and Macko, 1983; Neumann, 1990; Zeki and Shipp, 1988). The evidence indicates that there are basically two channels in the visual information processing system: a magno channel and a parvo channel. These channels diverge and are specialized in coding different stimulus properties. The magno channel, which leads to the parieto-occipital cortex, carries information about position (and possibly global form). The parvo system (which further subdivides) has the temporo-occipital region as its destination, and handles color and detailed form (but not position). In other words, the parvo system is thought to be concerned primarily with 'what', i.e. with identity processing, and the magno system primarily with 'where', i.e. with location processing. Obviously, given such an architecture, the translation of the unlimited-capacity processing assumption has to be approximately as depicted in Figure 1.5. Visual information enters the central processing system in an 'input map' (the striate cortex?). Here all information is represented, albeit not yet explicit. The information is sent on in an identity channel (the parvo channel) and a location channel (the magno channel). Ultimately the identities are calculated in an 'identity domain' (the
32
A. H. C. Van der Heijden
ID
LO
IN
STIMULI Figure 1.5. Unlimited-capacity identity and location processing model. Information enters the input map (IN) and is sent on to an identity domain (ID) and a location map (LO). [Adapted from Van der Heijden, 1992._]
inferior temporal cortex?) and the positions, or, in general, the spatial relations, are calculated in a 'location map' (the posterior parietal cortex?). The interesting, and in my view theoretically important, point is now that only a single, simple addition suffices to complete this scheme in such a way that it is also compatible with the remaining two essential outcomes mentioned at the beginning of this section: selection is early selection and selection is ultimately accomplished via a region in visual space. Figure 1.6 depicts this completed schematic representation of unlimited-capacity parallel processing. Added are 'feedback lines' from the location map back to the input map. If we assume that selection takes place through enhanced activation via subsets of these feedback or 're-entry' lines then selection is early selection (the lines re-enter the input map) and selection is accomplished via regions in space (the lines originate in the location map). In this scheme unlimited capacity and early selection are harmoniously combined because 'In a massively parallel, activation-coded system, selective enhancement provides a . . . mechanism of selective cueing that need not entail the exclusion from further processing of the noncued information, beyond the level of encoding at which selective enhancement first occurs' (Allport, 1987, p. 410). As in my earlier work (Van der Heijden, 1992), I propose to call the differential or selective operation of the system of feedback lines 'selective attention'. Selective attention consists of the enhanced activation of subsets of these feedback lines.
33
Visual attention
11)
LO
STIMULI Figure 1.6. Completed unlimited-capacity processing model. Position information is fed back from the location map to the input map. [Adapted[tom Van der Heijden, 1992.] By combining early selection and unlimited capacity, this conceptualization is fundamentally at variance with all views that postulate there is early selection in order to protect a central system with a limited information processing capacity against confusion and overload. However, an alternative, and much more positive, function for early selection is easily found. Early selection has an important function in the direction and execution of a selected action (Van der Heijden, 1990, 1992). By means of early selection, attention solves the problem of which object to act upon by determining from what region in visual space the set of parameters is taken that is allowed to specify a selected action in detail (Allport, 1987; Neumann, 1987). In fact, by replacing 'attention directed at position' by 'position directed as attention' (Van der Heijden, 1992), the model is completely compatible with, and an implementation of, the action-based views of attention as proposed by Allport (1987, 1989) and Neumann (1987, 1990). As a theory, this conceptualization is also diametrically opposed to the 'feature integration' theory proposed by Treisman, after the suggestions by Neisser (1967; see Crick, 1984; Treisman, 1977, 1982; Treisman and Gelade, 1980; Treisman and Schmidt, 1982; for an overview see Treisman, 1988). In feature integration theory the starting point is that the modularity of the visual system introduces a problem. (How do we see integrated objects or a unified world?) That problem has to be solved by attention. (Focal attention integrates the information in the different modules.) In the view presented here, (selective) attention, as an animistic deus ex
34
A. H. C. Van der Heijden
machina, is the real problem. That problem is adequately taken care of by the
modularity of the visual system and appropriate (map-map) connectivity. With the scheme presented in Figure 1.6 as a starting point and with attention as defined, topics with regard to the mode of operation of visual atttention, e.g. the topic of divided and focused attention, reduce to the topic of the shape of the distribution of (enhanced) activation in the location map. Topics with regard to what exactly is selected, e.g. the topic of whether attention selects an 'object' or an approximate region in visual space, reduce to the problem of the exact region of (enhanced) activation in the location map. Topics with regard to the control of selective attention, e.g. the topic of involuntary and voluntary attention, reduce to the topic of the mechanisms that are capable of modulating the pattern of (enhanced) activation in the location map. In my view, it is these areas with which contemporary research in visual attention and visual search is basically concerned. However, an appropriate interpretation of contemporary research in these terms needs divided attention. In the past, issues of visual information processing and selection were generally discussed in terms of unidimensional or linear information processing models (processing ~ more processing ~ still more processing; Neisser, 1976). In the present scheme, identification and localization are separated and assigned to two distinct, parallel information processing channels that, clearly, have their own properties and characteristics. Therefore the scheme also provides the possibility to reformulate and revisit interesting and long-standing research problems in visual information processing and attention. How fast is recognition and how slow is subsequent selection? (see Sperling, 1967, for the relevant research). What exactly is the effect of a pattern mask? (see Mewhort et al., 1981, 1984, for good starting assumptions). How to conceptualize lateral interference? (see Butler and Currie, 1986, for useful ideas). Where do 'Gestalt properties' enter the selection process? (see Kahneman and Henik, 1981; Livingstone and Hubel, 1988, p. 748, for good starting assumptions). What is the exact difference between search tasks and recognition tasks? (see Duncan and Humphreys, 1989; Kahneman and Treisman, 1984, for good starting points). How does attention 'move' in the visual field? (see Eriksen and Murphy, 1987; Yantis, 1988, for good starting points). And why do we perceive a stable visual world despite (saccadic) eye movements? (see Bridgeman, Van der Heijden and Velichkovsky, 1990, for a start). However, space does not allow me to go into these intriguing issues in any detail and I leave them as exercises for the reader. Duncan (1981, p. 93) ended his paper on 'Directing attention in the visual field' with the question: 'Why is it that advance knowledge of position affords such an excellent cue for attentional selection?' I end this chapter with my answer to that question: 'Because "advance knowledge of position" is the attention doing the selection.'
REFERENCES Allport, D. A. (1987). Selection for action: Some behavioral and neurophysiological considerations of attention and action. In H. Heuer and A. F. Sanders (Eds), Perspectives on Perception and Action. Hillsdale, NJ: Erlbaum.
Visual attention
35
Allport, D. A. (1989). Visual attention. In M. I. Posner (Ed.), Foundations of Cognitive Science. Cambridge, MA: MIT Press. Anstis, S. M. (1974). A chart demonstrating variations in acuity with retinal position. Vision Research, 14, 589-592. Averbach, E. and Coriell, A. S. (1961). Short-term memory in vision. Bell System Technical Journal, 40, 309-328. Averbach, E. and Sperling, G. (1961). Short-term storage of information in vision. In C. Cherry (Ed.), Information Theory. London: Butterworth. Barlow, H. B. (1981). Critical limiting factors in the design of the eye and visual cortex. The Ferrier Lecture. Proceedings of the Royal Society (London) B, 212, 1-34. Barlow, H. B. (1985). The twelfth Bartlett memorial lecture: The role of single neurons in the psychology of perception. Quarterly Journal of Experimental Psychology, 37A, 121-145. Baxt, N. (1871). Ueber die Zeit welche n6tig ist, damit ein Gesichtseindruck zum Bewusstsein kommt und fiber die Gr6sse der bewussten Wahrnehmung bei einen Gesichtseindruck von gegebener Dauer. Pflugers Archiv fiir die Gesamte Physiologie des Menschen und der Tiere, 4, 325-336. Bouma, H. (1970). Interaction effects in parafoveal letter recognition. Nature, 226,, 177-178. Bouma, H. (1973). Visual interference in the parafoveal recognition of initial and final letters of words. Vision Research, 13, 767-782. Bridgeman, B., Van der Heijden, A. H. C. and Velichkovsky, B. M. (1990). Visual stability and saccadic eye movements. Report of the research group "'Mind and Brain", ZiF, Bielefeld, No. 58. Broadbent, D. E. (1958). Perception and Communication. London: Pergamon Press. Broadbent, D. E. (1971). Decision and Stress. London: Academic Press. Bundesen, C. (1987). Visual attention: Race models for selection from multielement displays. Psychological Research, 49, 113-121. Bundesen, C. (1990). A theory of visual attention. Psychological Review, 97, 523-547. Bundesen, C., Pedersen, L. F. and Larsen, A. (1984). Measuring efficiency of selection from briefly exposed visual displays: A model for partial report. Journal of Experimental Psychology: Human Perception and Performance, 10, 329-339. Bundesen, C., Shibuya, H. and Larsen, A. (1985). Visual selection from multielement displays: A model for partial report. In M. I. Posner and O. S. M. Marin (Eds), Attention and Performance XI. Hillsdale, NJ: Erlbaum. Butler, B. E. (1980). Selective attention and stimulus localization in visual perception. Canadian Journal of Psychology, 34, 119-133. Butler, B. E. and Currie, A. (1986). On the nature of perceptual limits in vision: A new look at lateral masking. Psychological Research, 48, 201-210. Butler, B. E., Mewhort, D. J. K. and Tramer, S. C. (1987). Location errors in tachistoscopic recognition: Guesses, probe errors, or spatial confusions? Canadian Journal of Psychology, 41, 339-350. Campbell, A. J. and Mewhort, D. J. K. (1980). On familiarity effects in visual information processing. Canadian Journal of Psychology, 34, 134-154. Cattell, J. McK. (1885). Ueber die Zeit der Erkennung und Benennung von Schriftzeichen, Bildern und Farben. Philosophische Studien, 2, 635-650. Chastain, G. (1983). Parafoveal identification asymmetry is a lateral masking effect. Journal of General Psychology, 109, 77-81. Chastain, G. and Lawson, L. (1979). Identification asymmetry of parafoveal stimulus pairs. Perception and Psychophysics, 26, 363-368. Chow, S. L. (1986). Iconic memory, location information, and partial report. Journal of Experimental Psychology: Human Perception and Performance, 12, 455-465. Clark, S. E. (1969). Retrieval of color information from preperceptual memory. Journal of Experimental Psychology, 82, 263-266. Collins, J. F. and Eriksen, C. W. (1967). The perception of multiple simultaneously presented forms as a function of foveal spacing. Perception and Psychophysics, 2, 369-373.
36
A. H. C. Van der Heijden
Coltheart, M. (1972). Visual information processing. In P. C. Dodwell (Ed.), New Horizons in Psychology 2. Harmondsworth: Penguin. Coltheart, M. (1975). Iconic memory: A reply to Professor Holding. Memory and Cognition, 3, 42 -48. Coltheart, M. (1980). Iconic memory and visible persistence. Perception and Psychophysics, 27, 183-228. Coltheart, M. (1984). Sensory m e m o r y - A tutorial review. In H. Bouma and D. G. Bouwhuis (Eds), Attention and Performance X: Control of Language Processes. London: L.E.A. Coltheart, M., Lea, C. D. and Thompson, K. (1974). In defense of iconic memory. Quarterly Journal of Experimental Psychology, 26, 633-641. Cowey, A. (1979). Cortical maps and visual perception: The Grindley memorial lecture. Quarterly Journal of Experimental Psychology, 31, 1-17. Cowey, A. (1981). Why are there so many visual areas? In F. O. Schmitt, F. G. Worden, G. Adelman and S. G. Dennis (Eds), The Organisation of the Cerebral Cortex. Cambridge, MA: MIT Press. Crick, F. H. C. (1984). Function of the thalamic reticular complex: The searchlight hypothesis. Proceedings of the National Academy of Sciences USA, 81, 4586-4590. Deutsch, J. A. and Deutsch, D. (1963). Attention: Some theoretical considerations. Psychological Review, 70, 80-90. DeYoe, E. A. and Van Essen, D. C. (1988). Concurrent processing streams in monkey visual cortex. TINS, 11, 219-226. Dick, A. O. (1972). Parallel and serial processing in tachistoscopic recognition: Two mechanisms. Journal of Experimental Psychology, 96, 60-66. Driver, J. and Tipper, S. P. (1989). On the nonselectivity of 'selective' seeing: Contrasts between interference and priming in selective attention. Journal of Experimental Psychology: Human Perception and Performance, 15, 304-314. Duncan, J. (1980). The demonstration of capacity limitation. Cognitive Psychology, 12, 75-96. Duncan, J. (1981). Directing attention in the visual field. Perception and Psychophysics, 30, 90-93. Duncan, J. (1983). Perceptual selection based on alphanumeric class: Evidence from partial reports. Perception and Psychophysics, 33, 533-547. Duncan, J. (1984). Selective attention and the organization of visual information. Journal of Experimental Psychology: General, 113, 501-517. Duncan, J. (1989). Boundary conditions on parallel processing in human vision. Perception, 18, 457-469. Duncan, J. and Humphreys, G. W. (1989). Visual search and stimulus similarity. Psychological Review, 96, 433-458. Egeth, H. E., Folk, C. L. and Mullin, P. A. (1989). Spatial parallelism in the processing of lines, letters and lexicality. In B. E. Shepp and S. Ballesteros (Eds), Object Perception: Structure and Process. Hillsdale, NJ: Erlbaum. Egly, R. and Homa, D. (1984). Sensitization of the visual field. Journal of Experimental Psychology: Human Perception and Performance, 10, 778-793. Enns, J. T. (1990). Three-dimensional features that pop out in visual search. In D. Brogan (Ed.), Visual Search. London: Taylor and Francis. Erdman, B. and Dodge, R. (1898). Psychologische Untersuchungen iiber das Lesen. Halle: M. Niemeyer. Eriksen, C. W. (1966). Independence of successive inputs and uncorrelated error in visual form perception. Journal of Experimental Psychology, 72, 26-35. Eriksen, C. W. (1990). Attentional search of the visual field. In D. Brogan (Ed.), Visual Search. London: Taylor and Francis. Eriksen, C. W. and Collins, J. F. (1967). Some temporal characteristics of visual pattern perception. Journal of Experimental Psychology, 74, 476-484. Eriksen, C. W. and Collins, J. F. (1969). Visual perceptual rate under two conditions of search. Journal of Experimental Psychology, 80, 489-492.
Visual attention
37
Eriksen, C. W. and Hoffman, J. E. (1972). Temporal and spatial characteristics of selective encoding from visual displays. Perception and Psychophysics, 12, 201-204. Eriksen, C. W. and Hoffman, J. E. (1974). Selective attention: Noise suppression or signal enhancement? Bulletin of the Psychonomic Society, 4, 587-589. Eriksen, C. W. and Lappin, J. S. (1967). Independence in the perception of simultaneously presented forms at brief durations. Journal of Experimental Psychology, 73, 468-472. Eriksen, C. W., Munsinger, H. L. and Greenspon, T. S. (1966). Identification versus samedifferent judgment: An interpretation in terms of uncorrelated perceptual error. Journal of Experimental Psychology, 72, 20-25. Eriksen, C. W. and Murphy, T. D. (1987). Movement of attentional focus across the visual field: A critical look at the evidence. Perception and Psychophysics, 42, 299-305. Eriksen, C. W. and Rohrbaugh, J. W. (1970). Some factors determining efficiency of selective attention. American Journal of Psychology, 83, 330-343. Eriksen, C. W. and Schultz, D. W. (1977). Retinal locus and acuity in visual information processing. Bulletin of the Psychonomic Society, 9, 81-84. Eriksen, C. W. and St James, J. D. (1986). Visual attention within and around the field of focal attention: A zoom lens model. Perception and Psychophysics, 40, 225-240. Eriksen, C. W. and Steffy, R. A. (1964). Short-term memory and retroactive interference in visual perception. Journal of Experimental Psychology, 68, 423-434. Estes, W. K. (1978). Perceptual processing in letter recognition and reading. In E. C. Carterette and M. P. Friedman (Eds), Handbook of Perception, Vol IX. New York: Academic Press. Estes, W. K., Allmeyer, D. H. and Reder, S. M. (1976). Serial position functions for letter identification at brief and extended exposure durations. Perception and Psychophysics, 19, 1-15. Estes, W. K. and Taylor, H. A. (1964). A detection method and probabilistic models for assessing information processing from brief visual displays. Proceedings of the National Academy of Sciences USA, 52, 46-54. Estes, W. K. and Taylor, H. A. (1966). Visual detection in relation to display size and redundancy of critical elements. Perception and Psychophysics, 1, 9-16. Francolini, C. N. and Egeth, H. E. (1980). On the non-automaticity of automatic activation: Evidence of selective seeing. Perception and Psychophysics, 27, 331-342. Fryklund, I. (1975). Effects of cued-set spatial arrangement and target-background similarity in the partial-report paradigm. Perception and Psychophysics, 17, 375-386. Glanville, A. D. and Dallenbach, K. M. (1920). The range of attention. American Journal of Psychology, 41, 387-393. Haber, R. N. (1983). The impending demise of the icon: A critique of the concept of iconic storage in visual information processing. Behavioral and Brain Sciences, 6, 154. Haber, R. N~ and Hershenson, M. (1974). The Psychology of Visual Perception. London: Holt, Rinehart and Winston. Haber, R. N. and Standing, L. G. (1969). Direct measures of short-term visual storage: Quarterly Journal of Experimental Psychology, 21, 43-54. Hagenaar, R. (1990). Visual selection in letter naming. PhD Thesis, University of Leiden, The Netherlands. Hagenzieker, M. P., Van der Heijden, A. H. C. and Hagenaar, R. (1990). Time courses in visual information processing: Some empirical evidence for inhibition. Psychological Research, 52, 13-21. Hall, G. S. and Von Kries, J. (1879). Ober die Abh/ingigkeit der Reaktionszeit vom Ort des Reizes. Archives of Anatomy and Physiology: Leipzig Supplement, 1-10. Johnston, W. A. and Dark, V. J. (1985). Dissociable domains of selective processing. In M. I. Posner and O. S. M. Martin (Eds), Attention and Performance, XI: Mechanisms of Attention. Hillsdale, NJ: Erlbaum.
38
A. H. C. Van der Heijden
Johnston, W. A. and Dark, V. J. (1986). Selective attention. Annual Review of Psychology, 37, 43-75. Jonides, J. (1980). Towards a model of the mind's eye's movement. Canadian Journal of Psychology, 34, 103-112. Jonides, J. (1981). Voluntary versus automatic control over the mind's eye's movement. In J. B. Long and A. D. Baddeley (Eds), Attention and Performance IX. Hillsdale, NJ: Erlbaum. Jonides, J. (1983). Further toward a model of the mind's eye's movements. Bulletin of the Psychonomic Society, 21, 247-250. Jonides, J. and Yantis, S. (1988). Uniqueness of abrupt visual onset in capturing attention. Perception and Psychophysics, 43, 346-354. Kahneman, D. (1968). Method, findings and theory in studies of visual masking. Psychological Bulletin, 70, 404-425. Kahneman, D. (1973). Attention and Effort. Englewood Cliffs, NJ: Prentice-Hall. Kahneman, D. and Chajczyk, D. (1983). Tests of the automaticity of reading: Dilution of Stroop effects by color-irrelevant stimuli. Journal of Experimental Psychology: Human Perception and Performance, 9, 497-509. Kahneman, D. and Henik, A. (1981). Perceptual organization and attention. In M. Kubovy and J. R. Pomerantz (Eds), Perceptual Organization. Hillsdale, NJ: Erlbaum. Kahneman, D. and Treisman, A. (1984). Changing views of attention and automaticity. In R. Parasuraman and P. R. Davies (Eds), Varieties of Attention. New York: Academic Press. Keele, S. W. (1973). Attention and Human Performance. Pacific Palisades, CA: Goodyear. Keele, S. W. and Chase, W. G. (1967). Short-term visual storage. Perception and Psychophysics, 2, 383-386. Kinsbourne, M. and Warrington, E. K. (1962). The effect of an aftercoming random pattern on the perception of brief visual stimuli. Quarterly Journal of Experimental Psychology, 14, 223-234. La Heij, W. and Van der Heijden, A. H. C. (1983). Feature specific interference in letter identification. Acta Psychologica, 53, 37-60. Levi, D. M., Klein, S. A. and Aitsebaomo, A. P. (1985). Vernier acuity, crowding and cortical magnification. Vision Research, 25, 963-977. Livingstone, M. and Hubel, D. (1988). Segregation of form, color, movement, and depth: Anatomy, physiology, and perception. Science, 240, 740-749. Long, G. M. (1980). Iconic memory: A review and critique of the study of short-term visual storage. Psychological Bulletin, 88, 785-820. Lovie, A. D. (1983). Attention and behaviourism-fact and fiction. British Journal of Psychology, 74, 301-310. Mackworth, J. F. (1959). Paced memorizing in a continuous task. Journal of Experimental Psychology, 58, 206-211. Mackworth, J. F. (1962). The visual image and the memory trace. Canadian Journal of Psychology, 16, 55-59. Mackworth, J. F. (1963). The relation between the visual image and post-perceptual immediate memory. Journal of Verbal Learning and Verbal Behavior, 2, 75-85. Matsuda, M. (1988). Identification and localization in tachistoscopic partial report. Japanese Psychological Research, 30, 33-41. Maunsell, J. H. R. (1987). Physiological evidence for two visual subsystems. In L. M. Vaina (Ed.), Matters of Intelligence: Conceptual Structures in Cognitive Neuroscience. Dordrecht: Reidel. Merikle, P. M. (1980). Selection from visual persistence by perceptual groups and category membership. Journal of Experimental Psychology: General, 109, 279-295. Mewhort, D. J. K., Butler, B. E., Feldman-Stewart, D. and Tramer, S. (1988). 'Iconic memory', location information, and the bar-probe task: A reply to Chow (1986). Journal of Experimental Psychology: Human Perception and Performance, 14, 729-737.
Visual attention
39
Mewhort, D. J. K. and Campbell, A. J. (1978). Processing spatial information and the selective-masking effect. Perception and Psychophysics, 24, 93-101. Mewhort, D. J. K. and Campbell, A. J. (1981). Toward a model of skilled reading: An analysis of performance in tachistoscopic tasks. Reading Research: Advances in Theory and Practice, 3, 39-118. Mewhort, D. J. K., Campbell, A. J., Marchetti, F. M. and Campbell, J. I. D. (1981). Identification, localisation, and 'iconic memory': An evaluation of the bar-probe task. Memory and Cognition, 9, 50-67. Mewhort, D. J. K., Johns, E. E. and Coble, S. (1991). Early and late selection in partial report: Evidence from degraded displays. Perception and Psychophysics, 50, 258-266. Mewhort, D. J. K., Marchetti, F. M. and Campbell, A. J. (1982). Blank characters in tachistoscopic recognition: Space has both a symbolic and a sensory role. Canadian Journal of Psychology, 36, 559-575. Mewhort, D. J. K., Marchetti, F. M., Gurnsey, R. and Campbell, A. J. (1984). Information persistence: A dual-buffer model for initial visual processing. In H. Bouma and D. G. Bouwhuis (Eds), Attention and Performance X: Control of Language Processes. London: Erlbaum. Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81-97. Mishkin, M., Ungerleider, L. G. and Macko, K. A. (1983). Object vision and spatial vision: Two cortical pathways? Trends in Neurosciences, 6, 414-417. Moray, N. (1969). Attention: Selective Processes in Vision and Hearing. London: Hutchinson Educational. Morton, J. (1969). Interaction of information in word recognition. Psychological Review, 76, 165-178. Nakayama, K. and Silverman, G. H. (1986). Serial and parallel processing of visual feature conjunctions. Nature, 320, 264-265. Neisser, U. (1967). Cognitive Psychology. New York: Appleton-Century-Crofts. Neisser, U. (1976). Cognition and Reality. San Francisco, CA: Freeman. Neumann, O. (1987). Beyond capacity: A functional view of attention. In H. Heuer and A. F. Sanders (Eds), Perspectives on Perception and Action. Hillsdale, NJ: Erlbaum. Neumann, O. (1990). Visual attention and action. In O. Neumann and W. Prinz (Eds), Relationships Between Perception and Action. Berlin: Springer. Neumann, O. (1996). Konzepte der Aufmerksamkeit. G6ttingen: Hogrefe, in press. Neumann, O., Van der Heijden, A. H. C. and Allport, D. A. (1986). Visual selective attention: Introductory remarks. Psychological Research, 48, 185-188. Nissen, M. J. (1985). Accessing features and objects: Is location special? In M. I. Posner and O. S. M. Martin (Eds), Attention and Performance XI. Hillsdale, NJ: Erlbaum. Norman, D. A. (1968). Towards a theory of memory and attention. Psychological Review, 75, 522-536. Pashler, H. (1984). Evidence against late selection: Stimulus quality effects in previewed displays. Journal of Experimental Psychology: Human Perception and Performance, 10, 429448. Pashler, H. (1987). Detecting conjunctions of color and form: Reassessing the serial search hypothesis. Perception and Psychophysics, 41, 191-201. Phaf, R. H., Van der Heijden, A. H. C. and Hudson, P. T. W. (1990). SLAM: A connectionist model for attention in visual selection tasks. Cognitive Psychology, 22, 273-341. Potfenberger, A. T. (1912). Reaction time to retinal stimulation with special reference to time lost in conduction through nerve centers. Archives of Psychology, 3, 1-73. Posner, M. I. (1978). Chronometric Exploration of the Mind. Hillsdale, NJ: Erlbaum. Posner, M. I. (1980). Orienting of attention. The VIIth Sir Frederic Bartlett Lecture. Quarterly Journal of Experimental Psychology, 32, 3-25.
40
A. H. C. Van der Heijden
Posner, M. I. and Cohen, Y. (1984). Components of visual orienting. In H. Bouma and D. G. Bouwhuis (Eds), Attention and Performance X. Hillsdale, NJ: Erlbaum. Posner, M. I., Nissen, M. J. and Ogden, W. C. (1978). Attended and unattended processing modes: The role of set for spatial location. In H. L. Pick and I. J. Saltzman (Eds), Modes of Perceiving and Processing Information. HiUsdale, NJ: Erlbaum. Prinzmetal, W. (1981). Principles of feature integration in visual perception. Perception and Psychophysics, 30, 330-340. Rumelhart, D. E. (1970). A multicomponent theory of the perception of briefly exposed visual displays. Journal of Mathematical Psychology, 7, 191-218. Schultz, D. W. and Eriksen, C. W. (1977). Do noise masks terminate target processing? Memory and Cognition, 5, 90-96. Shapiro, R. G. and Krueger, L. E. (1983). Effect of similarity of surround on target-letter processing. Journal of Experimental Psychology: Human Perception and Performance, 9, 547559. Shibuya, H. and Bundesen, C. (1988). Visual selection from multielement displays: Measuring and modeling effects of exposure duration. Journal of Experimental Psychology: Human Perception and Performance, 14, 591-600. Shiffrin, R. M. and Schneider, W. (1977). Controlled and automatic human information processing: II. Perceptual learning, automatic attending, and a general theory. Psychological Review, 84, 127-190. Snodgrass, J. G. and Townsend, J. T. (1980). Comparing parallel and serial models: Theory and implementation. Journal of Experimental Psychology: Human Perception and Performance, 6, 330-354. Snyder, C. R. R. (1972). Selection, inspection and naming in visual search. Journal of Experimental Psychology, 92, 428-431. Sperling, G. (1960). The information available in brief visual presentations. Psychological Monograph, 74 (whole no. 498). Sperling, G. (1963). A model for visual memory tasks. Human Factors, 5, 19-31. Sperling, G. (1967). Successive approximations to a model for short-term memory. Acta Psychologica, 27, 285-292. Taylor, S. G. and Brown, D. R. (1972). Lateral visual masking: Supraretinal effects when viewing linear arrays with unlimited viewing time. Perception and Psychophysics, 12, 97-99. Townsend, J. T. (1971). A note on the identifiability of parallel and serial processes. Perception and Psychophysics, 10, 161-163. Townsend, J. T. (1972). Some results on the identifiability of parallel and serial processes. British Journal of Mathematical and Statistical Psychology, 25, 168-199. Townsend, J. T. (1974). Issues and models concerning the processing of a finite number of inputs. In B. H. Kantowitz (Ed.), Human Information Processing: Tutorials in Performance and Cognition. Hillsdale, NJ: Erlbaum. Townsend, J. T. (1990). Serial vs. parallel processing: Sometimes they look like Tweedledum and Tweedledee but they can (and should) be distinguished. Psychological Science, 1, 46-54. Townsend, V. M. (1973). Loss of spatial and identity information following a tachistoscopic exposure. Journal of Experimental Psychology, 98, 113-118. Treisman, A. M. (1964). Verbal cues, language, and meaning in selective attention. American Journal of Psychology, 77, 206-219. Treisman, A. M. (1969). Strategies and models of selective attention. Psychological Review, 76, 282-299. Treisman, A. M. (1977). Focussed attention in the perception and retrieval of multidimensional stimuli. Perception and Psychophysics, 22, 1-11. Treisman, A. M. (1982). Perceptual grouping and attention in visual search for features and for objects. Journal of Experimental Psychology: Human Perception and Performance, 8, 194-214. Treisman, A. M. (1988). Features and objects: The fourteenth Bartlett memorial lecture. Quarterly Journal of Experimental Psychology, 40A, 201-237.
Visual attention
41
Treisman, A. M. and Gelade, G. (1980). A feature integration theory of attention. Cognitive Psychology, 12, 97-136. Treisman, A. M., Kahneman, D. and Burkell, J. (1983). Perceptual objects and the cost of filtering. Perception and Psychophysics, 33, 527-532. Treisman, A. M. and Schmidt, H. (1982). Illusory conjunctions in the perception of objects. Cognitive Psychology, 14, 107-141. Tsal, Y. and Lavie, N. (1988). Attending to color and shape: The special role of location in selective visual processing. Perception and Psychophysics, 44, 15-21. Turvey, M. T. and Kravetz, S. (1970). Retrieval from iconic memory with shape as the selection criterion. Perception and Psychophysics, 8, 171 - 172. Van der Heijden, A. H. C. (1971). The processing of tachistoscopic displays as a function of effective stimulus duration. Acta Psychologica, 35, 233-242. Van der Heijden, A. H. C. (1981). Short-Term Visual Information Forgetting. London: Routledge and Kegan Paul. Van der Heijden, A. H. C. (1984). Postcategorical filtering in a bar-probe task. Memory and Cognition, 12, 446-457. Van der Heijden, A. H. C. (1986). On selection in vision. Psychological Research, 48, 211-219. Van der Heijden, A. H. C. (1987). Central selection in vision. In H. Heuer and A. F. Sanders (Eds), Perspectives on Perception and Action. Hillsdale, NJ" Erlbaum. Van der Heijden, A. H. C. (1990). Visual information processing and selection. In O. Neumann and W. Prinz (Eds), Relationships Between Perception and Action. Berling: Springer. Van der Heijden, A. H. C. (1992). Selective Attention in Vision. London: Routledge. Van der Heijden, A. H. C. and Eerland, E. (1973). The effects of cueing in a visual signal detection task. Quarterly Journal of Experimental Psychology, 25, 496-503. Van der Heijden, A. H. C., Schreuder, R., De Loor, M. and Hagenzieker, M. (1987). Early and late selection: Visual letter confusions in a bar-probe task. Acta Psychologica, 65, 75-89. Van der Heijden, A. H. C., Schreuder, R. and Wolters, G. (1985). Enhancing single-item recognition accuracy by cueing spatial locations in vision. Quarterly Journal of Experimental Psychology, 37a, 427-434. Van der Heijden, A. H. C., Wolters, G., Groep, J. C. and Hagenaar, R. (1987). Single-letter recognition accuracy benefits from advance cueing of location. Perception and Psychophysics, 42, 503-509. Von Helmholtz, H. (1871). Ueber die Zeit welche n6tig ist, damit ein Gesichtseindruck zum Bewusstsein kommt. Berliner Monatsberichte, 8 Juni, 333-337. Von Helmholtz, H. (1894). Handbuch der physiologischen Optik. Hamburg: L. Vos. Von Wright, J. M. (1968). Selection in visual immediate memory. Quarterly Journal of Experimental Psychology, 20, 62-68. Von Wright, J. M. (1970). On selection in visual immediate memory. Acta Psychologica, 33, 280-292. Von Wright, J. M. (1972). On the problem of selection in iconic memory. Scandinavian Journal of Psychology, 13, 159-171. Warren, R. M. and Warren, R. P. (1968). Helmholtz on Perception: Its Physiology and Development. New York: John Wiley. Wolford, G. and Chambers, L. (1983). Lateral masking as a function of spacing. Perception and Psychophysics, 33, 129-138. Wolford, G. and Hollingsworth, S. (1974). Lateral masking in visual information processing. Perception and Psychophysics, 16, 315-320. Wolford, G. L., Wessel, D. L. and Estes, W. K. (1968). Further evidence concerning scanning and sampling assumptions of visual detection models. Perception and Psychophysics, 3, 439-444. Woodworth, R. S. and Schlosberg, H. (1954). Experimental Psychology. London: Methuen. Wundt, W. (1899). Zur Kritik tachistoscopischer Versuche. Philosophische Studien, 15, 287-317.
42
A. H. C. Van der Heijden
Yantis, S. (1988). On analog movements of visual attention. Perception and Psychophysics, 43, 203-206. Yantis, S. and Jonides, J. (1984). Abrupt visual onsets and selective attention: Evidence from visual search. Journal of Experimental Psychology: Human Perception and Performance, 10, 601-621. Zeki, S. and Shipp, S. (1988). The functional logic of cortical connections. Nature, 355, 311-317.
Chapter 2 Visual Search A. F. Sanders and M. Donk Department of Psychology, Vrije Universiteit, Amsterdam, The Netherlands
In his review of the literature Monk (1984) correctly noted that the term visual search is generally used as a loose label for a variety of experimental paradigms which share the broad feature that there is a need for spatial uncertainty reduction. Spatial uncertainty refers to the unknown position of one or more targets, either predefined and known in advance of a trial or unknown but still sufficiently deviant from the other elements of the visual scene to enable detection. All search conditions have in common that they suffer from parameter overspecification and hence require selective attention for action (Neumann, 1987). Apart from this communality, studies on visual search differ in many respects. One reason for this might be that the field has its roots in applied rather than in basic research, with the consequence that several classic experiments aimed at simulating a real-life task instead of addressing a theoretical issue. Theoretical questions evolved from the applied work, but they were often connected to the original experimental settings, which has the consequence that a general theory of visual search is lacking. Instead, there are various smaller, more or less isolated, frameworks, subsumed under the general search umbrella. Two aspects of visual search are often distinguished, one relating to structural constraints, and the other to functional strategic factors. They can certainly not be strictly separated, but they still represent two traditions. The symposium on Search and the Human Observer (Clare and Sinclair, 1979) and Rabbitt's (1984) chapter in Varieties of Attention (Parasuraman and Davies, 1984) are prototypical. Characteristic is their emphasis on either one of two broad questions. Structural constraints are usually concerned with the question of what kind of information is extracted from fixations during visual search. It is not surprising that this tradition relies heavily on studies of single eye-fixations. Yet, structural constraints can be described on at least three levels: the sensory level of a single fixation, the anatomical level of eye and head movements and, finally, the information processing level. The other question is what determines the position of the next fixation; is the locus of control in search externally (bottom-up) or internally (top-down) guided? This second tradition is dominated by studies on free continuous search with the aim of describing strategies. The division is of course not absolute: a combination of constraints and strategies is thought to determine the final search pattern. Handbook of Perception and Action, Volume 3
Copyright 9 1996 Academic Press Ltd All rights of reproduction in any form reserved
ISBN 0-12-516163-8
43
44
A. F. Sanders and M. Donk
The study of visual search during a single fixation derives from the wellestablished evidence that there is no useful vision during saccadic eye movements (see Matin, 1974, for a review), which implies that search processes must occur during fixations. Studying a single fixation, then, has the advantage of allowing rigid experimental control of the stimulus conditions, and in particular of the limiting effects of peripheral vision. The advantage is that the experimenter decides what the subject is going to see, without worrying too much about scanning and strategy. Visual search during a single fixation constitutes a major paradigm in the study of visual attention. It is at the basis of many theoretical issues such as early versus late selection (Duncan, 1980; Van der Heijden, 1987), automatic versus controlled processing (Schneider, Dumais and Shiffrin, 1984), feature versus conjunction search (Treisman and Gelade, 1980; Treisman and Sato, 1990; Wolfe, Cave and Franzel, 1989) and spotlight versus zoomlens (Eriksen, 1990; Posner, Snyder and Davidson, 1980). A more detailed discussion of these issues is found in other chapters of this volume, since they are more concerned with mechanisms of attention than with search per se. Here, mechanisms of visual attention will be only touched upon as far as the issues are relevant to free continuous visual search. The first section is concerned with perceptual limits during a single fixation as expressed in visibility and conspicuity (Engel, 1976). These sensory constraints are generally described on a psychophysical level. Visual search may extend to a large field and include eye and head movements (Sanders, 1970). Different search modes are involved when the display exceeds a certain limit (Sanders, 1963, 1970; Sanders and Houtmans, 1984). The second section deals with anatomical constraints when the size of the display requires eye and head movements (Sanders, 1963, 1970). The third section is concerned with information processing constraints. Here, the major interest is in attentional processing limits (Neisser, 1963, 1967; Prinz, 1987). The reported studies mainly deal with prescribed scanning, such as scanning lines as opposed to free scanning in which any direction or order of fixation is permitted. In prescribed scanning, there is little space for strategy, since search is fully determined by the stimulus configuration. It should be noted in passing that the analysis of reading is beyond the scope of this chapter, since it is not really a search problem. The fourth section is concerned with relations between eye movements and conspicuity (Engel, 1976), followed by a discussion on the relations between eye movements and attention. Apparently, both sections are related in that eye movements elicited by conspicuity may very well be accompanied or even preceded by attention shifts. The main argument to separate both sections is that the first is explicitly concerned with research on eye movements and conspicuity, while in the second section the emphasis is on the interrelation between eye movements and attention. The paper continues with a section entitled 'Patterns of eye movements in visual search', which outlines two alternative mathematical models describing visual search behavior in homogeneous noisy displays. The next section deals with cognitive models of visual search followed by a discussion about visual sampling strategies. The final section is concerned with effects of stresses and abnormal conditions on search. Obviously, the last three sections are more related to functional strategic factors in visual search.
Visual search
1
SENSORY
45
CONSTRAINTS
It is an almost trivial statement that the most effective search is no search at all (Corbin et al., 1958). Obviously, this refers to conditions where a target is sufficiently conspicuous to be immediately detected. Engel (1971, 1977) defined visual conspicuity as the degree of perceptual prominence of a visible object in its surroundings regardless of knowledge about the location of the object. According to Engel, an object's conspicuity is determined by simple features such as brightness, color, size, shape and motion. The visual lobe, or conspicuity area, represents the peripheral area around the central fixation point within which a target can be detected with a certain probability within a single glance. Basically, the conspicuity area is assumed to represent the area within which a target 'pops out' without the need for a serial attentional scan. The size of the visual lobe depends on variables such as display density and target-background similarity. The basic reason for measuring the visual lobe is that the number of fixations required to find a target is supposed to be a direct function of the size of the visual lobe for that target. Usually, the visual lobe has been measured with tachistoscopical techniques. The method is rather simple in that a target is briefly presented at different eccentricities from the central point of fixation. For each distance, the subject indicates whether, and eventually where, a target has been seen. In this way, a series of contours around the fixation point can be constructed representing probabilities of target detection (Bellamy, 1984; Bellamy and Courtney, 1981; Engel, 1976; Hughes and Cole, 1986; Kraiss and Knaeeper, 1982). In view of the large number of readings required to map the lobe fully (Chalkin, Corbin and Volkmann, 1962), generally only one or two axes have been measured. Thus, some studies measured four meridians around the fixation point (Johnston, 1965) while others only used one or two (Engel, 1976; Erickson, 1964). The main advantage of the tachistoscopical technique is that it enables visual lobe measurement without the influence of eye movements, thus providing an objective measure of peripheral sensitivity. Although the visual lobe has been widely used in models of visual search, there is no agreed definition of the lobe. In fact, the definition depends on the method of measurement, which varies substantially among the different studies. A major difference between the various studies concerns the critical detection contour. Conspicuity is usually expressed as a distance associated with a certain detection probability. Thus, the 50% detection probability contour (hard-shelled lobe) is usually taken as the edge of the conspicuity area (Chalkin et al., 1962; Engel, 1976). It is assumed that a target occurring inside this area will be directly found. Sometimes another detection probability contour is taken (Johnston, 1965) and, more often, the area with a small probability of detection is included in the lobe (soft-shelled lobe). Another measure of conspicuity starts from the probability of detection at a certain fixed distance from the central point of fixation. For example, Boynton, Elsworth and Palmer (1958) measured acuity at 3 ~ from the fixation point while Erickson (1964) measured at 6 ~ In a field study, Hughes and Cole (1986) simply measured the frequency of target detection as an indication of the conspicuity. A second main difference between the various studies is related to the way in which the lobe area is measured. In order to determine the area of the visual lobe,
46
A. F. Sanders and M. Donk
two different dependent variables have been used. Bellamy and Courtney (1981) established an estimate of the lobe based on location errors committed by subjects when attempting to report the target position. In this procedure, subjects indicated the position of the target after each brief presentation of the display. After transformation of the location error scores, a linear relationship could be established between error scores and target eccentricity (see also Courtney, 1984; Courtney and Chan, 1985, 1986). Another method relies on the proportion of detected targets as a function of the eccentricity of the fixation point (Engel, 1976). This method merely requires detection and no location of the target. The subject has to indicate after each display presentation whether or not he or she had seen a target. A third issue concerns the presentation time of the stimulus display. There is recent evidence that presentation time affects the size of the lobe (Sanders and Br/ick, 1991). However, different studies have used different presentation times of the stimulus display. In some studies the display was presented for an extremely brief period of time (Engel, 1976) whereas in other studies the display was presented until the subject responded (Hughes and Cole, 1986). In most studies, however, the display was presented for about 250 ms (Bellamy, 1984; Courtney and Chan, 1986). This duration is supposed to be short enough to exclude saccadic eye movements during presentation and, at the same time, long enough to be comparable to fixation durations in free search. A final issue concerns the divergence in instructions and, consequently, the response criterion of the subjects. Courtney and Chan (1985) instructed their subjects to indicate a target position after each presentation, even if the target had not been seen (see also Bellamy and Courtney, 1981; Courtney, 1984; Courtney and Chan, 1986). On the other hand, Hughes and Cole (1986) required their subjects to report only clearly detected targets. Similarly, Engel's subjects indicated with a push button whether or not they had detected the target (Engel, 1976). Obviously, either instruction might result in a different response bias of the subject and, therefore, in a different visual lobe estimate. When comparing the results of the various studies, there are large differences in outcome with respect to either size, shape or variability of the visual lobe. For example, Courtney and Chan (1985) reported irregular lobe boundaries, while Bellamy and Courtney (1981) found regular elliptical lobe shapes. Some investigators reported no difference between horizontal and vertical axes while other studies showed a longer horizontal than vertical length (Bellamy and Courtney, 1981; Courtney and Chan, 1986; Chalkin et al., 1962). Furthermore, various visual lobe sizes have been reported, varying from rather small to relatively large (Bellamy and Courtney, 1981; Courtney and Chan, 1986; Engel, 1976; Hughes and Cole, 1986). Unfortunately, there are too many unsystematic methodological differences between the various studies to elucidate the divergences. In addition, there is a total lack of theoretical background with respect to the target-background configurations which have been used and their effect on detection. Yet, in its own specific target-background configuration, each individual visual lobe estimate might be a valuable predictor of search performance. Indeed, it has been repeatedly demonstrated that lobe area is inversely proportional to search time and proportional to the probability of detecting the target in a single eye-fixation during search (Bellamy and Courtney, 1981; Bloomfield and Howarth, 1969; Engel, 1976). However, it is surprising that most correlations between visual lobe parameters and search times are not as significant as might have been expected (Courtney and
Visual search
47
Chan, 1986; Erickson, 1964; Johnston, 1965). The relatively low correlations might be caused by two factors. First, the use of tachistoscopical measurements in assessing the lobe area might be inappropriate. Klein and Farrell (1989) have shown that a short exposure duration is not a particularly good way to control eye-fixation because subjects adopt special strategies to deal with a rapidly decaying icon. This might obviously lead to a different performance level in comparison with situations in which eye movements are allowed. Various alternative lobe measurement techniques have been proposed in which the visual lobe is estimated from free search behavior. For example, Mackworth (1976) presented his subjects displays consisting of horizontal lines separated by a variable vertical distance. Subjects were asked to search each strip consisting of several rows of circles with the aim of locating a square among the circles. They were further instructed to scan the lines from left to right without stopping or looking back. Mackworth (1976) theorized that the smallest vertical distance between the horizontal lines that caused the pattern of fixation to deviate significantly from a horizontal scan was equivalent to lobe width. Indeed, several studies (Mackworth, 1981) suggested effects of distance between the lines as well as of display density. However, the main drawback of Mackworth's studies is that a subject's scanning pattern might also be affected by speed-accuracy trade-off. Subjects were free to choose their preferred fixation duration and saccade length. These variables surely affect the size of the visual lobe and have, therefore, to be taken into account in order to properly estimate the lobe size (Sanders and Briick, 1991). Prinz and his colleagues (Nattkemper and Prinz, 1984, 1990; Prinz and Kehrer, 1982) developed a similar method in which subjects were instructed to scan a list of letters row by row, from top to bottom (as in reading), and to search for a target as quickly as possible. A trial ended when the target was detected or when the end of the list was reached. Eye movements were measured, to determine the vertical detection distance, that is the vertical distance from the actual fixation point to the target location at the moment of detection. On the basis of this vertical detection distance, the integration area, i.e. the field around the fixation point from which information is processed, could be estimated. Prinz and his colleagues used this method to study the effects of visual density (Nattkemper and Prinz, 1990), the difference between the upper and lower hemiretina (Prinz, 1984) and the effect of redundancy (Nattkemper and Prinz, 1984). A quite different method was used by Barbur, Forsyth and Wooding (1993). Basically, they presented their subjects with multiple stimuli equidistant to the central fixation point, i.e. stimuli were arranged in a circle. Subjects were required to make one single saccade to a predefined target in the circle. The visual lobe then was equal to the radius of the circle at which 50% of all searches required only one saccade. Widdel (1983) suggested still another technique for measuring the visual lobe. His basic idea was that a successful fixation implies that, during that fixation, the target item is detected in the periphery. The next saccade is determined by this peripheral stimulus so that the next fixation will attain the target. When the target is fixated at fixation n, fixation n - 1 is the successful fixation because the target was detected peripherally during that fixation. In a series of search trials, Widdel (1983) measured the distances of the fixations n - 1 to the targets and computed
48
A. F. Sanders and M. Donk
their distribution, thus developing a method that certainly deserves attention. Although his method is rather complicated, its main advantage is the absence of tachistoscopic presentation techniques. Furthermore, he excluded the influence of individual scanning strategies by only using the last previous fixation distance to the target. Although promising, the dynamic techniques are usually more complicated than static lobe determinations. In addition, it has not yet been tested whether dynamic lobe measurements are really better correlated with search time than are static lobe measurements. A second major factor that might be responsible for the relatively low correlations between lobe measures and search time is the fact that the visual lobe is not the only determinant of visual search behavior. The strategy adopted in search may also have a considerable effect on search time (see the section on 'Patterns of eye movements in visual search'). In addition, attending to a certain location, the novelty of targets or the meaning of targets may exert influence on search time. Directed attention to a location generally decreases the peripheral sensitivity threshold at that location resulting in a higher detection probability. Engel (1976) recognized this and distinguished, therefore, between the conspicuity and the visibility area. The visibility area represents the peripheral area around the central fixation point within which a target can be detected with foreknowledge about its location. The distinction between the visibility area and the conspicuity area is useful in that it reflects the difference between attending and not attending a location (Posner and Snyder, 1975). However, the extent to which, and the conditions under which, the visibility area is a good predictor for search time is still unclear. Yet, it seems plausible that the combination of the conspicuity area and the visibility area might be a better predictor than either one in isolation.
2
ANATOMICAL
CONSTRAINTS
Spatial uncertainty and the need for search increases with the size of the display for the simple reason that there are more demands on the visual periphery. One question is to what extent the same search principles apply to displays of a different size. It may be useful to distinguish here between the macrostructure and the microstructure of the visual field. The microstructure is characterized by local detail which is only visible in the fovea and parafovea whereas the macrostructure concerns large-sized stimuli with contours that are still visible at a wider visual angle. The microstructure was addressed in the previous section on the visual lobe. The macrostructure will be treated in the present section. Sanders (1963, 1970, 1993; Sanders and Houtmans, 1984) argues that different search modes are involved if a display exceeds the limits set by the need for eye and head movements. In a standard experiment, two widely separated signals are presented at about eye level and at equal distances from the meridian. At the start of a trial the left signal is fixated; this signal should be processed at presentation, followed by a saccade to the right signal whereupon a trial ends with a same-different response. The basic phenomenon, which has been confirmed in a score of experiments, is that the time elapsing from the completion of the saccade (the start of fixating the right signal) until the completion of the same-different response is fairly constant as long
Visual search
49
as the total display angle is within the limits of the 'eye field', but increases with some 100 ms as soon as head movements are required to supplement the saccade, i.e. when the display is in the 'head field'. In the case of the stimuli that were commonly used in the original studies (columns of either four or five dots) the head field started at a binocular display angle of about 80-90 ~ and at about 60 ~ in the case of a more complex stimulus configuration. The original studies also showed another increase in processing time at a display angle at which a saccade was first needed, i.e. at about 25 ~. It was assumed that at smaller angles both signals could be identified by mere peripheral viewing. Together, these results suggested three processing modes: as long as no saccade is needed to arrive at the same-different response (the 'stationary field'), the stimuli are encoded as an integrated whole. In the eye field, the stimuli are independently encoded, but, while encoding the left signal, a hypothesis arises about the right signal which then speeds up the response time from the moment that the right signal is fixated. In the head field, finally, the left and right signal are processed as fully independent events. In other words, the results suggest that, whatever position is fixated, the visual scene can use a constant background schema in the eye field; search can consist of scanning relevant parts of the same schema. In contrast, changes of the background are required when the display angle is within the head field; establishing a new background adds to searching for the target. For the theory of search it would follow that, as long as the display is within the eye field, hypotheses about peripherally located signals are o b t a i n e d - a s far as allowed by peripheral vision- directing the gaze and enhancing the efficiency of subsequent identification. Thus, the eye field might be the limiting case of the lobe. The problem is of course that the experiments, as outlined above, are not proper search studies, so that the validity of the extrapolation remains dubious. However, Sanders (1963, 1970) also carried out a search experiment in which six columns of dots were presented in a horizontal row. All columns but one contained four dots, while one column had five dots. The task was to identify the position of the five-dot column and press a corresponding response key, after which a new display was presented without delay. Apart from a repetition effect in the case of successive five-dot columns at the same position, the results showed a perfectly linear relationship between response time and the visual angle between the target positions at successive presentations as long as the total display was within the eye field. The slope of this function could be largely ascribed to the time taken by a single saccade between the target position at successive trials. This suggests that a reliable hypothesis was obtained about the location of the five-dot column. When the total display exceeded the eye field, response times increased by 100 ms if the angle between the target positions at successive trials exceeded 35 ~. This suggests that, in the head field, the display is inspected in two successive percepts, one covering the area in the immediate vicinity of the target position of the previous trial, and a second one covering the remaining dot columns. The results suggested that the distinction between the processing modes in the eye and head field have some generality. Later studies have confirmed the generality of the distinction between the eye and the head field, but have also shown that the difference between the stationary field and the eye field depends on the specific stimulus configuration. Thus, Sanders and Houtmans (1985a) found evidence for the stationary field only if the stimuli were dot columns and not if they were digits. They concluded that two
50
A. F. Sanders and M. Donk
adjacent stimuli can only be encoded as a single 'whole' if the configuration actually permits recoding. This is intuitively appealing when the signals consist of two columns containing four or five dots. Perhaps this allows recoding in terms of a 'missing dot'. The results on digits as stimuli do not suggest recoding since, in that case, the processing times in the stationary field and the eye field hardly differed. Sanders and Houtmans proposed to maintain the distinction between the stationary field and the eye field with the restriction that a stationary field is more efficient only if recoding occurs. In fact it might serve as a check that such a condition has been realized. Further studies have confirmed the view that, in the eye field, the shorter response latency following fixation of the right signal is due to a hypothesis about the right signal, which arises during the fixation of the left signal. The main evidence came from a study in which the presentation of the right signal was somewhat delayed so that it either still occurred during fixation of the left signal, during the saccade, or even after fixation of the position of the right signal. The shorter response latency in the eye field was found only in the first case, confirming that fixation of the left signal is a prerequisite. In the head field the moment of presentation did not affect response latency, which should be expected if the signals are processed as independent percepts. Together with additional data from Sanders and Reitsma (1982b), these results also excluded an alternative explanation that ascribes the shorter response latency following fixation of the right signal to the possibility that, in the eye field, subjects are capable of starting processing of the right signal immediately on completion of a saccade, whereas in the head field, this is not possible. Sanders and Reitsma found that, in comparison with a normal two-choice reaction time, the latency measured from the fixation of the right signal to the response was shorter in the eye field but about equally long in the head field. Research on the nature of the hypothesis about the right signal, obtained during fixation of the left signal, has so far failed to deliver a satisfactory picture. Houtmans and Sanders (1984) varied the quality of the right signal and found that, in the case of a degraded right signal, the shorter response latency in the eye field was reduced but not eliminated. They also found that when intact and degraded right signals were mixed in a block of trials, and when intact signals were relatively infrequent, the response latency was about equally reduced, irrespective of the quality of the right signal. These results go beyond the trivial conclusion that an intact right signal delivers more peripheral information than a degraded one. First, an intact right signal was advantageous only when it was expected to occursuggesting the involvement of attentional control- and, second, there was still a reduction in latency in the case of a degraded right signal, in which condition subjects were incapable of identifying the right signal during fixation of the left one. Another issue of interest is the end-product of a single fixation. In the context of the functional visual field this can be studied by analyzing the properties of the fixation time of the left signal. Several interesting results have emerged: first, there is no evidence that acquisition of peripheral information about the right signal delays the fixation duration of the left signal. It is not uncommon to find a somewhat longer fixation duration in the head field, but that can also be ascribed to more complex motor programming of a combined eye-head movement. Second, Sanders and Houtmans (1985b) found that, when the left signal is degraded, the fixation duration of the left signal increases by about the same amount as in a traditional choice reaction study. Thus, the eyes appear not to leave the left signal
Visual search
51
before all perceptual problems have been solved. This is not related to intake of information, since presentation duration of the left signal did not appear to affect its fixation duration. Sanders and Rath (1991) found that, when under heavy time pressure, subjects may actually refrain from perceptual processing during the fixation of the left signal. However, this results in prolonged processing during fixation of the right signal. Together these results also suggest that a saccade blocks not only sensory processes, but also central perceptual processing of information, obtained during the previous fixation. Boer and Van de Weijgert (1988) found that, when the left signal had to be classified (target/no target), classification time was not reflected in the left fixation time, so that finishing a fixation and starting a saccade may be bounded by perceptual processing and not by further central processing. Boer and Van de Weijgert's results also suggest that target classification may indeed occur during the saccade. More recently, these results were further confirmed by Van Duren and Sanders (1992) in an extensive series of experiments which showed that target classification as well as response selection can occur during a saccade. A final issue concerns the output of a fixation. This was addressed in a study by Hansen and Sanders (1988) who varied the signal quality of both the left and right signals and found that correspondence of signal quality facilitated the required same/different response. Thus, if both signals were intact, processing time during fixation of the right signal was faster than when the left signal was degraded. However, if both signals were degraded there was also faster processing of the right signal in comparison with the condition where the left signal was intact and the right signal degraded. This result suggests that the stored code of the left signal still contains the features, both relevant and irrelevant, of the last fixated signal. Encoding may have the effect of inventarizing features and making them individually accessible, but not of getting rid of irrelevant features. Alternatively, it could be that the observed result was due to a closer correspondence of processing demands of homogeneous stimuli, i.e. ~tact/intact or degraded/degraded, compared with heterogeneous stimuli, i.e. intact/degraded or degraded/intact (Los, 1994).
3
INFORMATION
PROCESSING
CONSTRAINTS
An account of information processing constraints on visual search may conveniently start with the work of Neisser (1963, 1967) who introduced a continuous visual search task in which structured letter matrices were presented. The instruction was to search either for the presence of a target, which occurred only once in the whole matrix, or for the absence of a target, which always occurred once in each row with the exception of the row that should be detected. Subjects scanned the rows of letters in a prescribed order so as to rule out influences of strategy. Search time, plotted as a function of the location of the target in the matrix, showed a perfectly linear function with a search rate of between 3 and 10 items per second, depending on the similarity between the features of the target and the background, i.e. the more similar the features, the slower the rate, and on whether subjects searched for presence or absence of the target, i.e. search for presence was much faster. 1
52
A. F. Sanders and M. Donk
In line with early selection theory, Neisser suggested that subjects do not identify each individual signal but start with a fast and parallel preattentive search, operating on general physical features rather than on local letters. A letter is identified by w a y of focal attention when all feature tests favor the target, according to a 'shouting demon' principle (Selfridge and Neisser, 1959). Neisser's two-process theory, i.e. preattentive search followed by focal attentive identification, assumes that subjects maintain an active template of the target. Non-targets play a role only in that, as target and non-targets are more similar, there are more critical features to be tested, so as to differentiate between targets and non-targets, which has the effect that search slows down. When searching for absence rather than for presence of a target, each row requires a time-consuming full identification of the target. The finding that search for absence is much slower poses obvious problems for a simple late selection theory which states that individual letters are always fully identified. Such a theory would require the additional hypothesis that deciding about the presence of a signal is particularly attention-demanding and time-consuming. It would then follow that a signal is easier rejected than confirmed, which is clearly at odds with various bodies of research, among which same-different judgements (Farrell, 1985) and target classification are prime examples. Neisser's two-process theory can easily accommodate various well-established findings on effective search for targets with a distinctive physical feature such as color, size and form. Thus, in a free search task, Williams (1966) measured saccades and fixations on large displays, containing two-digit numbers on differently colored, shaped and sized fields. The target was always a two-digit number with a prescribed feature. Numbers with a prescribed color were more frequently fixated, but, in contrast, shape hardly served as a distinctive feature in this study. In the same vein, Green and Anderson (1956) found that search rate depended on the number of non-targets with the same color as the target (see also Noble and Sanders, 1981), suggesting simple early selection with color as selection criterion. They also found that varying the number,of irrelevant colors of non-targets hardly affected search speed. As another illustration, Willows and McKinnon (1973) found that when alternating lines had a different color subjects could limit themselves to reading the lines with the same color, without any distraction from the lines with the other color. Neisser's (1963) own studies had shown that search for a straight letter amid curved non-targets, or vice versa, increased search rate. In a situation where subjects had the task to count the total number of targets on the display, Gould and Dill (1969) found that targets were fixated longer than non-targets, which was taken as evidence that non-targets are less deeply analysed. The longer fixation of targets might as well reflect the extra time taken by counting, but Gould and Dill also found that similar non-targets were fixated longer than nonsimilar ones, suggesting at least a contribution of differential depth of analysis. Again, Corcoran and Jackson (1979) had subjects search for a ' ~ ' among straight or curved letters as non-targets and found that changes within either set of non-targets, or
1Search rates in the absence of saccades (Sperling et al., 1971) are considerably faster, which can be only partially explained by the time taken by the saccades. It has been suggested that, in addition, search in the Neisser type of task slows down as a consequence of, first, lateral interference between letters (Bouma, 1978) and, second, the time taken by programming a saccade during each fixation. This last suggestion entails that search and motor programming cannot occur in parallel, which is more consistent with discrete than with continuous models of processing information (Miller, 1988; Sanders, 1990).
Visual search
53
even a transition from straight to curved ones, did not affect search rate. Apparently subjects could either use the ' 9 or the ' / ' as criterion for their search. This is consistent with the further finding of a pronounced negative effect on search rate when non-targets consist of a combination of straight and curved shapes. The finding that shape has more pronounced effects in some studies than in others might well be related to line search or free search: the less powerful shape feature might be less effective in free search, in which it must determine the direction of the saccades, than in line search, in which it plays a role only in distinguishing incoming stimuli.
3.1
Criticisms of the Two-Process Theory
Despite the impressive set of supporting data, Neisser's two-process model has not been undebated. As in other paradigms on selective attention, the discussion centers around the nature of the criteria that can be used for separating targets and non-targets. In turn, this issue is connected to the question about the extent to which stimuli are fully encoded during search, which is at the basis of the late-selection view of attention. The main reason for this discussion in the area of visual search is that variables affecting non-targets appear more relevant contributors to search rate than originally envisaged by the early selection dual-process theory. For example, in contrast to Green and Anderson's (1956) results, Cahill and Carter (1976) found that heterogeneity of irrelevant colors did affect search for a specific color. Yet, this contradiction might still be explained by dual-process theory in terms of a limited set of feature maps. In that case more heterogeneity increases the probability that the critical feature map is also activated by nontargets. In line with this reasoning, Moraglia et al. (1989) distinguished between non-target heterogeneity and similarity, and found that search time was affected only by similarity and not by heterogeneity per se. However, a simple early selection model faces some problems, which concentrate around findings that composite features may be used as a global search criterion, the physical nature of which is rather hard to imagine, and that, in fact, such composite features can be learned. The observation that, without identification, a target letter can be detected among non-target digits, and vice versa (Brand, 1971), is a case in point. Searching for a digit among letters is relatively easy in that the digit also appears to pop out (Jonides and Gleitman, 1972), although this effect has been found to disappear when searching for various alternative digits (Francolini and Egeth, 1979). The observation of the 'oh-zero' effect, i.e. that, depending on instruction, the ' 9 may pop out as a letter among digits or a digit among letters, has been taken as evidence that the distinctive features may not invariably reflect physical features but can be semantic as well, which is in clear conflict with the dual-process notion. However, the oh-zero effect is not easily reproduced (Duncan, 1983) which renders the issue unsettled. There is additional evidence that, under certain conditions at least, non-targets are fully analyzed, despite the presence of a distinctive physical characteristic. For example, when searching for an 'A' amidst a, E, c and C as background, subjects appear to be distracted by the 'a', suggesting an effect on the name level, despite the presence of shape as a distinctive feature (Hendersen and Chard, 1971).
54
A. F. Sanders and M. Donk
Other evidence stems from the well-known work of Shiffrin and Schneider (1977), Schneider and Shiffrin (1977) and Schneider et al. (1984) suggesting that 'automatic' detection of target letters among non-target letters is possible when subjects receive massive amounts of practice in which targets and non-targets are consistent in that they never change roles. Again, Rabbitt (1964, 1967) found that changing the non-target elements, which all consisted of straight-line letters, slowed down search for a straight-line letter from the target set. This suggests that subjects learn certain cues that are no longer valid when they have to look for the same straight-line targets among other straight-line non-targets. The effect proved to be stronger when searching for one out of eight target letters than when searching for one out of two alternative letters, which suggests a more refined set of distinctive features in the former case. The further observation that it matters whether subjects have to detect 'targets' or to respond differentially to different targets (Rabbitt, 1981) supports the idea that one deals with complex features, defining a learned category, which is almost certainly beyond physical properties. This is of course the critical deviation from early selection: effects of practice in learning how to categorize physical target and background features can be accounted for by early selection, but the features should always be on a global physical level. Conditions in which full identification of all items is observed are obviously in line with late selection theory. Yet, as noted before, full identification is not the usual finding in visual search, but is largely limited to studies on target detection during single fixations. In fact, there might be no proper physical criterion in those conditions, with the consequence that response set, i.e. number of alternative responses, rather than stimulus set, i.e. number of alternative stimuli, tends to dominate (Broadbent, 1971). Response set phenomena might still be of some relevance when search goes on in a systematic sequence of fixations, such as in line scanning, but largely disappear in free search where positional information is more relevant for deciding where to fixate next. The question remains whether phenomena of response set can be reconciled with a global parallel scan as implied by the two-process theory. According to this theory a global always precedes a local analysis, so that an item cannot be identified unless the global level has been successfully passed. As an alternative, Prinz (1987) has suggested that selection may occur simultaneously at a sensory (early-global-nonattentive) level and at a semantic (late-local-attentive) level. This is consistent with recent views, which no longer view attentional selection in terms of rejection or attenuation so as to overcome a limited capacity bottleneck but as creating a functional difference between relevant and irrelevant information (Allport, 1987; Humphreys and M/iller, 1993; Neumann, 1987; Van der Heijden, 1987, 1992; see also Chapter 1). Prinz (1987) has suggested three experimental results as evidence for simultaneous selection in search. The first concerned the size of the lobe in a Neisser type of line search, which turned out to be larger for round target letters among straight non-target letters than for straight target letters among straight non-target letters (Prinz, 1983). Yet, a dual-process theory might explain this result simply in terms of lateral inhibition, which has a stronger damaging effect on complex than on simple features. The second result concerned the effect of secondary task load: a semantically demanding second task slowed down search but increased the lobe for simple features. This was explained in terms of selective interference of the second task with semantic processing, leading to slower search. Since processing
Visual search
55
the simple features was not affected, they might profit from slower search rate and, hence, have a greater lobe (Prinz and Nattkemper, 1986). Yet, this result could equally well be interpreted in terms of more interference as the conditions ask for more detailed processing, reflecting response set. The simultaneous sensory and semantic processing interpretation of Prinz and Nattkemper would require an additional experiment in which a second task differentially affected sensory selection, with the prediction that, in that case, semantic selection would be undisturbed. Finally, Prinz, Meinecke and Hielscher (1987) studied the category effect on the basis of semantic features. They found that, when degrading letters in a Neisser task, the category effect remained, while the difference between categories disappeared with respect to their effect on the lobe. Yet, a dual-process notion might explain this result by a sequential full analysis in the case of the degraded letters. Simultaneous selection at the featural and semantic level has also been proposed by Ellis and Chase (1971), but their study was concerned with memory search of a single stimulus. Target identification on the basis of a feature was found to be faster only in the case of a larger memory set but not when a small memory set was used. Along the lines of Posner and Boies (1971), they suggested a parallel search for physical and name features, a horse race deciding which process would be first completed. Yet, as cogently argued by Broadbent (1982) and by Kahneman and Treisman (1984), a single stimulus fully excludes a stimulus set, thus evoking a competitive analysis between the various stimulus dimensions, featural and semantic alike. A similar problem might have occurred in studies such as those of Cavanagh and Chase (1971), who found full processing when two stimuli (a target and a non-target) were briefly presented in close vicinity. All these situations seem quite different from real overload conditions where a target must be found in the presence of many competing visual signals during free search. A global-to-local search may largely apply to the latter case.
3.2 Target Versus Background Control Prinz combined the simultaneous selection hypothesis with an additional principle that is also found in Chase (1986). The starting point of this view is the widespread a s s u m p t i o n - among others in the literature on the orientation response (Sokolov, 1963)- that people have an internal model of the perceptual environment. Perception, then, is viewed as an exchange process between stimulus information and the internal model, serving the primary function of deciding whether reality fits the model rather than deciding what is actually there. When searching for a target, subjects have an internal model of the background and check this background for deviations. Targets are detected by default, in that they deviate from non-targets. This view differs from the traditional view which holds that people keep an active representation of the target, aiming at a fit of that representation with a particular element in the environment. Instead, search becomes a matter of analyzing the context, so as to detect deviations, rather than of checking target representations. Context control can occur both on the semantic and on the featural level and is not incompatible, therefore, with either the notion of simultaneous selection, as advocated by Prinz, or even of successive preattentive and attentive selection as implied by Treisman and Neisser.
56
A. F. Sanders and M. Donk
Various general results on search are consistent with either a notion of target or context control. Thus, targets are usually incidental, which qualifies them as deviations from normal. This explains that searching for the presence of a target in a Neisser task is faster than searching for its absence. Again, the more a target differs from the context, the faster a deviation is noted. As specific support for the context theory, Prinz (1987) has summarized various experimental results. First, there is the phenomenon of pseudo-targets, referring to the observation that subjects spend relatively more time on new non-targets, i.e. non-targets that did not yet occur in the list (Prinz, Tweer and Feige, 1974). The possibility that new non-targets share more features with the target than with the existing non-targets was well controlled in this study, which makes the result hard to explain in terms of target control. A second result concerns the effect of non-target sequential redundancy. Autocorrelation of the non-targets speeds up search, which suggests that relations between non-targets are analyzed (Prinz, 1979). Nattkemper and Prinz (1984) found that, in a Neisser task, lists with predictable non-targets were searched slower in early practice, allowing for the build-up of an internal model, which then leads to a more efficient search once the model had been established. Again, the finding that, in the case of multiple targets, a target is sometimes detected prior to its identification follows easily from the view that a deviation from the context is detected. Here, the traditional view of target control faces the problem mentioned earlier of describing the specific distinctive features defining the target set. Clusters of non-targets, allowing rapid rejection, may be gradually learned. Yet, the evidence suggests that not all effects of non-targets, such as the pseudo-target effect, depend on practice. Prinz suggests that information about non-targets is retained in a short-term visual store, ready for immediate use at the feature level for evaluating the next trial; this predicts a considerable increase in search speed when the same non-targets are repeated, and this has been found. Autocorrelations between non-targets, on the other hand, would be processed on the semantic level, and therefore depend more on practice. Together these results are consistent with those of Rabbitt (1981), who found that when subjects search for specific targets, they adopt a fixed optimal scan path when the background is familiar, while the scan is unsystematic in the case of an unfamiliar display. Context encoding is certainly a viable alternative to target representation, in particular because it is also relevant to the interpretation of results on searching more structured environments. Yet, it may not be the only principle in search. For instance, Rabbitt, Cumming and Vyas (1977) found that search is fast when the same target is presented on successive displays, and even faster when appearing in the same spatial position. The variance in background is irrelevant in that case; it seems that subjects start the search with a rapid identity check, which is based on target representation. The identity check extends to irrelevant aspects of the target (Jordan and Rabbitt, 1977), suggesting that a complete signal, including irrelevant dimensions, is stored for comparison. Again, Rabbitt (1984) observed that negative transfer, induced by changing background items, ceases to exist after very prolonged practice. Presumably, a unique complex set of features has been acquired which promotes checking the target rather than the background. So far the discussion has been restricted to 'homogeneous' backgrounds, such as when scanning rows of letters, inspecting briefly presented displays with at most
Visual search
57
four items, or in free search of homogeneous noisy fields containing a slightly deviant target. Background effects are much more pronounced in the case of a coherent meaningful or a structured meaningless background. The classic studies of Biederman (1972) are a case in point: coherent or jumbled scenes were presented for a brief period of time, preceded or followed by an arrow pointing to an object on the picture. In all cases it was easier to identify the object when subjects had seen the coherent picture. Biederman, Glass and Stacey (1973) also found that detection of the absence of an object was also much easier on a coherent picture. A coherent background corresponds to internal schemata, i.e. abstract cognitive structures with both invariant and variable properties, which may be summarized by the label 'knowledge of the world'. When scanning a picture, a schema is initiated and parameter values are assigned to the various variables. This explains why 'the gaze selects informative details within pictures' (Mackworth and Morandi, 1967). The background plays also a dominant role in the case of a structured Gestalt, as abundantly shown in the Gestalt literature (Metzger, 1954). A classic example concerns the Gottschaldt hidden figures, where a target is very hard to detect when embedded in a well-structured background. More recent demonstrations stem, among others, from studies by Banks and Prinzmetal (1976) who showed that targets outside a cluster of background elements are more rapidly detected. Farmer and Taylor (1980) showed that backgrounds of different hues and brightness were more rapidly scanned when grouped than when scattered. The finding that similar effects occur with letter search (Marken and Patterson, 1979), i.e. AAAPPPQQQ faster than, say, APQAPQ, is in line with Prinz's results on background redundancy. The effect of the background organization was also confirmed in a study by Moraglia (1989) who had subjects search for a horizontal line segment through displays containing varying numbers of elements, differing from the target and from each other in orientation. The detection latency was considerably longer when the line segments were randomly positioned than when they were arranged in concentric circles. Background organization directs an observer's search along regular paths, from which search may benefit (if the path leads to the target) or suffer (if the path leads away from the target). One application is the sweepline on radar displays. When subjects follow the sweep, they automatically cover the whole display so as to increase the probability of target detection (Teichner and Mocharnuk, 1979). Effects of background organization show that search uses redundancy in the perceptual environment and is guided by known environmental structures. The first element fits traditional early selection theory, which has always maintained that perceptual structure reduces overload, and hence reduces the need for selective attention (Broadbent, 1958). The second is harder for theories that conceive selective attention as an indirect process of comparison of features from a target and its neighbors (Treisman and Gelade, 1980). The point is that it reflects neither a successive nor a simultaneous featural and semantic guidance of search, but, instead, a top-down local guidance in search for deviations from a meaningful schema. This seems more in line with a context-related approach, which holds that the conspicuity of a target depends on whether it fits or disrupts a total stimulus pattern (Duncan and Humphreys, 1989; Humphreys, Quinlan and Riddoch, 1989; Humphreys, Riddoch and Quinlan, 1985). The evidence in favor of this theory
58
A. F. Sanders and M. Donk
stems from studies in which targets with the same feature sets are used, with the difference that the target either fits or disrupts the background (Smets and Stappers, 1990).
3.3
Discussion
The foregoing discussion has made abundantly clear that there is little uniformity with respect to emphases, paradigms or methods used in the investigation of the structural constraints of visual search behavior. Sensory constraints are mainly described in terms of psychophysical limits while neglecting theoretical notions on either anatomical constraints or information processing. Anatomical constraints are probably important when inspecting large displays. However, neither conspicuity nor attentional limits were controlled for, which renders it doubtful whether the results on the functional visual field can be easily extrapolated to other stimulus/background configurations. Information processing constraints, finally, appear in the literature as a wide range of theoretical considerations about the relation between attentional notions on the one hand and display characteristics on the other hand. The main criticism probably concerns the lack of a uniform taxonomy. Paradigms stem from either an early selection view (Broadbent, 1958), a late selection tradition (Deutsch and Deutsch, 1963), or even from a functional view in which attention generally serves the function of creating a difference between relevant and irrelevant information. In addition, most experiments about information processing constraints fail to control for either target conspicuity, as psychophysically defined by Engel (1976), or display angle, which might be important in view of the findings of Sanders (1963, 1970). Thus, research on visual search might profit considerably from a more integrated approach in which all but one type of constraint is kept constant. Effects on search time might then be validly ascribed to the varying constraint.
4
EYE M O V E M E N T S
AND CONSPICUITY
The effects of peripheral vision on spontaneous eye movements have been systematically investigated by Findlay (1980), who developed a coherent psychophysics of saccade generation by using a paradigm developed by Levy-Schoen (1974). Two visual stimuli were simultaneously presented at opposite sides of a fixation point and subjects made a saccade to either one or the other stimulus. By varying the properties of the visual stimuli, he aimed at determining which details of a stimulus attracted saccades. The main conclusion was that the location on the retina (proximity to the fovea) and the amount of transient stimulation were important variables. Spatial detail seemed to be ineffective, at least for the generation of spontaneous eye movements. Although spontaneous eye movements might not be guided by some salient spatial detail, eye movements during visual search might be. In a task where
Visual search
59
subjects searched a heterogeneous visual field, Williams (1966) found that color information could be used effectively to guide the eye to a target, while size and shape information were not effective. Williams (1967) suggested that, when certain characteristics of a target are specified, the searcher attends to objects having those characteristics and attempts to fixate those objects. Objects, then are better fixated on the basis of color than on the basis of either form or size. When the target is specified by all three dimensions, subjects generally select on the basis of only one dimension, e.g. color. There are, however, indications that shape information might also be used to guide the eyes. Gould (1967) showed that, in an array of nine well-spaced patterns, subjects looked more often and longer at non-target patterns as pattern similarity with the target increased. Gould and Dill (1969) replicated this result in that they found that the probability of fixating a pattern increased as the shape of the pattern was more similar to the target shape. Gould (1967) suggested that peripherally discriminable patterns do not require foveal fixation. Functionally, peripheral vision would not fundamentally differ from foveal vision. Only the quality of peripherally presented information is worse than foveal information. Both Gould (1967) and Williams (1967) emphasized the capability of the observer to decide on the basis of extrafoveal information about whether a stimulus is a target or not and, thus, whether or not it requires fixation. According to Engel (1976), conspicuity is the main determinant of an eye movement. He emphasized that parafoveal irregularities automatically attract the eye. He presented his subjects with a homogeneous display of many randomly located background disks among which were two deviant disks. One of the target disks was smaller and one was larger than the background disks. Subjects searched for either the small or the large one. During search for the target there appeared to be a tendency to fixate the irrelevant deviant disk, which appeared to be related to the conspicuity of the irrelevant disk. Thus, when deviating stimuli are present in the periphery, an eye movement will be initiated towards them. The decision as to whether a stimulus is a target is generally taken on the basis of foveal information while extrafoveal information serves to detect irregularities in the total stimulus pattern. In general, eye movement behavior seems to depend strongly on the search display. In homogeneous fields, peripheral vision may serve to select a point for the next fixation which shows at least some irregularity in comparison with the background (Engel, 1977). Peripheral vision in heterogeneous search fields may merely serve to select a point that shares common features with the target (Gould, 1967; Williams, 1967). However, when targets cannot be discriminated from distractors on the basis of parafoveal vision, the adopted search strategy might be the main determinant of search behavior.
5
EYE M O V E M E N T S
AND
ATTENTION
The previous section was explicitly concerned with the question to what extent eye movements are evoked by stimulus characteristics. The main question in the present section is whether eye movements depend on attentional shifts. Logically, the relationship between eye movements and spatial attention might have several forms (Shepherd, Findlay and Hockey, 1986).
60
A. F. Sanders and M. Donk
At one extreme, the processes involved in generating an eye movement as well as attention to a location in space might be completely identical. However, this is unlikely because many experiments have shown that attention can be moved to different parts of the visual field in the absence of overt eye movements (Eriksen and Eriksen, 1974; Eriksen and Hoffman, 1972, 1973; Posner, 1980; Posner, Nissen and Ogden, 1978). At the other extreme, the processes involved in eye movements and attention might be fully independent of each other. This hypothesis is also unprobable. For example, Crovitz and Davies (1962) found that preparation to make an eye movement towards a stimulus facilitates subsequent perception of that stimulus. Again, Nissen, Posner and Snyder (1978) found that, relative to other locations, reaction time to the onset of a stimulus at the saccade target position was faster prior to the start of the eye movement. This suggests that attention shifts and saccades may share a common movement mechanism. A less extreme possibility would be that, at some stage of their execution, the processes involved in generating eye movements and attention share a common element; execution of either process might then be facilitated or inhibited by the other depending on their respective spatial goals. Several studies appear to provide evidence for this position. Broadly, a distinction can be made between studies investigating the effect of eye movements on attending a location and studies emphasizing the influence of attending a location on eye movements. Effects of an eye movement on attention allocation are usually inferred from probe reaction times or from proportions of correctly reported probes, presented on different locations at various time intervals following a cue indicating the direction of the required saccade. The main emphasis is on the question of what happens to the allocation of attention when a saccade is carried out? Thus, Nissen et al. (1978) performed an experiment in which they summoned saccades by peripheral cues. Reaction times to subsequent probe stimuli, occurring at the peripheral cue position or at the central fixation position, indicated that preparing a saccade necessarily involved allocation of attention to the target position in advance of the actual eye movement. At least under peripheral cueing conditions, there seemed to be a strong link between preparing a saccade and allocation of attention to the target position. Remington (1980) has suggested that attention allocation and eye movements are elicited by the same system but are separately controlled. In some of his conditions, he also used a peripheral cue to indicate the target position for an eye movement. Subjects were required to report the presence or the absence of a probe stimulus which occurred on 50% of trials at one out of four positions and at various time intervals following the cue. In another experiment, he used central cues to indicate the direction of the saccade. He concluded that a peripheral stimulus summoned both attention and an eye movement since detection of the target position steadily improved until it was overruled by saccadic suppression. When saccades were directed by a central cue, there was no evidence for prior allocation of attention to the target location of the eye movement. These results indicate a tighter connection between the peripheral cue and attention than between attention and the saccade. Shepherd et al. (1986), on the other hand, found evidence for a closer relation between attention and eye movements. In their experiments, spatial attention was manipulated by varying the probability that a peripheral probe stimulus would appear in different positions, while saccades were directed by a central arrow, enabling separation of the effects of attention and eye movements. When the
Visual search
61
saccade was directed away from the most likely position of the probe, the effect of moving the eyes proved to be stronger than the effect of spatial cueing until well after the saccade had finished. This suggests that making a voluntary saccade necessarily involves allocation of attention to the target position, which competes for resources with the processes that underlie the allocation of attention to the other position. In line with these findings are the results of Henderson (1993) who found that peripheral preview of a letter string facilitated the response to an identical target string when a saccade had to be directed toward the position of that letter string. No facilitation was found when subjects had to remain fixated on the central fixation point. In summary, the evidence concerning attention shifts in relation to saccadic eye movements is not unequivocal. Although the results of Shepherd et al. (1986) and Henderson (1993) suggest that attention shifts necessarily accompany eye movements, Remington's (1980) results suggest otherwise. In Henderson's experiment subjects had to respond to letter strings whereas Remington and Shepherd et al. used probe reaction time tasks. This renders a direct comparison difficult. Also, the experiments of Remington and Shepherd et al. are quite different from each other: a major difference concerns the presentation time of the probe. While Remington (1980) presented probes in the form of a very brief (3 ms) near-threshold luminance increase, Shepherd et al. (1986) presented probes until the subject had responded. The probe in Remington's experiment was a fairly bright box subtending a visual angle of 1.37 ~ whereas in the experiment of Shepherd et al. (1986) the probe was a dark square of about 0.3 ~. Finally, Remington (1980) determined the proportion of hits while Shepherd et al. (1986) measured probe reaction times. To what extent these differences are responsible for the differences in outcomes is unclear. Any general conclusion as to whether eye movements elicit attention shifts seems to be premature. The reverse question, i.e. whether attention shifts elicit eye movements, has also stimulated some research. For instance, Shepherd et al. (1986) investigated the effect of probe stimuli on saccade latencies: probe stimuli appearing before the saccade shortened saccade latencies if they appeared at the saccade's target and lengthened saccade latencies if they appeared on the opposite side of the saccade's target. Thus, attention seems to play an important role in the generation of voluntary eye movements. Saslow (1967) reported a study in which he had a temporal separation, i.e. temporal gap, between the offset of a central fixation point and the onset of a peripheral target light that elicited a saccade. Temporal separation decreased the saccadic reaction time to about 150 ms compared with 250 ms when the central fixation point remained visible. Fischer and Breitmeyer (1987) presented subjects with three different conditions. In one condition, the gap paradigm was used, i.e. the fixation point switched off 200 ms prior to the peripheral target. In another condition, the central fixation point remained on and subjects were instructed to direct attention to it. The third condition was similar to the second one except for the instruction, in which subjects were told to ignore the central fixation point. The results indicated that both the separation of the central fixation light and the peripheral cue and ignoring the central fixation light reduced the latencies of saccades. It should be added, however, that the effect of temporal separation was much more pronounced. In another experiment (Mayfrank, Mobashery, Kimmig and Fischer, 1986), subjects were instructed to direct their gaze to the middle of a
62
A. F. Sanders and M. Donk
screen without a fixation point. Attention had to be directed to a peripheral light. The subsequent target appeared either 4 ~ to the right or to the left of the center of the screen. In the separation condition, in which the attender peripheral light disappeared 200 ms prior to the presentation of the target for the saccade, saccade latencies were generally short. In other conditions, in which the peripheral light source remained on, saccade latencies were longer, even in the condition in which the peripheral light was located at the same position as the target for the saccade. These results were taken as evidence for the hypothesis that a disengagement of attention is necessary before the saccade can be initiated. This disengagement of attention is assumed to take some extra time. The results of Shepherd et al. (1986) suggest that eye movements are facilitated by prior allocation of attention to the target for the saccade while Fischer and Breitmeyer (1987) suggest that starting an eye movement requires a general disengagement from attention. However, the experiments vary with respect to the manipulation of attention. While Shepherd et al. (1986) manipulated attention by means of expectation, Fischer and Breitmeyer (1987) simply told their subjects where to direct their attention. More importantly, a temporal separation might function as a warning signal for the saccadic response. A decrease in saccadic latency might then not be related to disengagement of attention but merely to the constant period preceding the presentation of the target for the saccade. In general, there is some evidence that the prior allocation of attention to the saccade target shortens the saccade latency (Shepherd et al., 1986). Whether attention necessarily moves with the eyes is not clear. Additional studies are required.
6
PATTERNS
O F EYE M O V E M E N T S
IN VISUAL
SEARCH
Eye movements during visual search might be conceived of as including a stochastic process in which the observer samples portions of the display at each fixation (Monk, 1984). Sampling has been modeled as either a random (Krendel and Wodinsky, 1960) or a systematic process (Williams, 1966). Random search implies that the probability of finding a target remains constant over successive fixations and, furthermore, does not depend on events during earlier fixations. Thus, a random search model assumes that the observer is memoryless, so that the same item might be repeatedly fixated. Krendel and Wodinsky (1960) have suggested that a random search strategy can be inferred from the cumulative distribution of search times. A random search strategy expects an exponential cumulative distribution of search times. In contrast, systematic search (Williams, 1966) implies that the same item is never inspected again, so that the observer is supposed to have a perfect memory. Consequently, search times cannot exceed a certain maximum, i.e. the required time for one complete inspection of the display. Engel (1976) has shown that if, as assumed by the systematic search model, search is carried out without repeated fixations, the cumulative distribution function is linear. Mean search time with a systematic search strategy would take about half the time obtained with random search (Howard and Bloomfield, 1968).
Visual search
63
It has been commonly found that search times are exponentially distributed with the result that the random search model has generally been accepted as the most appropriate predictor of search performance (Bloomfield, 1972; Engel, 1976; Howard and Bloomfield, 1969; Krendel and Wodinsky, 1960). Megaw and Richardson (1979) also found that the observed cumulative distributions of search times are best fitted by an exponential distribution. Yet, after a more thorough analysis, a better fit was obtained with a mixed model. Morawski, Drury and Karwan (1980) suggested that some combination of both models should be taken into account. They claim that real performance distributions are usually in between the two functions as described by the random and the systematic model. A major problem with the interpretation of cumulative distributions of search times concerns the fact that sometimes observers look at a target and, yet, totally fail to see it (Mackworth, Kaplan and Metlay, 1964). This looking without seeing might result in an exponential distribution of search times even when a display is systematically scanned. Memory might be perfect while perception fails. In the same vein, Williams (1966) demonstrated that exponential distributions are obtained when subjects employ a systematic strategy with a detection probability of 0.6 instead of 1.0. According to Williams, there starts another systematic search when the whole display area has been searched without successful target detection. These objections suggest that the question of whether search is random or systematic cannot be simply inferred from cumulative search time distributions. A better approach may be the actual recording of eye movements and, even then, firm conclusions seem prohibited as long as one does not know which lobe size subjects actually have in free search.
7
COGNITIVE
MODELS
A more cognitive approach towards the notion of scanning strategies stems from Noton and Stark (1971) and Stark and Ellis (1981) (see also Stark, Yamashita, Tharp and Ngo, 1993). According to their scan path theory, eye movements are controlled by cognitive models that are already present in the brain. These models are perceptual hypotheses to be tested against the complex world of sensory experiences. In the checking phase of what appears to be a hierarchical pattern recognition process, scan paths or repetitious sequences of saccades are generated. Actually, most research on scan paths has been done with pictures. The few studies on homogeneous fields have only led to a few systematic observations. Thus, people tend to avoid the edges of a display and, more generally, more fixations occur in the center than in the periphery (Enoch, 1959). The edge effect can be easily overcome through biasing either by instruction or by display structure (Baker, 1958; Yarbus, 1967). It should be added that scan paths are usually hard to analyze and show considerable individual variation. Using pictures, Mackworth and Morandi (1967) found that regions, rated as highly informative were more frequently fixated. As discussed previously, a meaningful display structure, providing a schema for search, constitutes a prime example of cognitive control. The effect of familiarity of the display structure on the
64
A. F. Sanders and M. Donk
scan pattern (Rabbitt, 1981) is another demonstration of the operation of a cognitive model: confronted with a scene, an observer can recognize within a few milliseconds whether or not that scene has been met before. In another experiment, Friedman (1979; see also Antes and Penland, 1981) informed subjects verbally about the theme of a scene that would be presented. Following presentation, unexpected objects were fixated longer than expected objects. This result suggests that the abstract verbal information is a sufficient basis for generating a concrete set of objects on the basis of which a picture is scanned. The extra processing of unexpected objects was interpreted as evidence for bottom-up processing of elements that did not fit the schema. Loftus and Mackworth (1978) also found that unexpected objects tended to be fixated before expected ones. Expectations are also important in the determination of saccade length. Jacobs (1986, 1987) had subjects search for a target (C, c, k, x') inserted in lines of x's. Jacobs analyzed those lines where no target had occurred and found that saccade size depended on the size of the expected spatial visibility limits, i.e. the visual lobe or visual span in Jacob's terminology, for a particular target. Jacobs (1987) suggested that there is a decisional process at each fixation concerning the presence of a target, ending with a saccade to that target or a shift that is sufficiently outside the visual span to enable an efficient next fixation.
8
OPTIMAL
SCANNING
AND
MONITORING
The most elaborate theoretical analysis of scanning strategies stems undoubtedly from research on scanning discrete displays for relevant signals. The task is usually to detect either a drift of a signal from a predefined optimum or the presence of a signal in some danger zone. Common examples of real-life environments are the cockpit, where several displays inform the pilot about the current state of the aircraft, and the control room, where a human observer has the task of detecting significant deviations from the optimal positions on the display. Some of the earlier applied work in this area has been engaged with search experiments in which sequences of fixations, relative fixation frequencies and durations were measured during a real or a simulated flight. This research (Fitts, Jones and Milton, 1950) resulted in a score of recommendations about improving the arrangement of instruments, so as to enable more efficient control. These include the frequency-ofuse principle, the sequence-of-use principle and the importance principle (Sanders and McCormick, 1987). The major theoretical contributions, the formulation of which started during the 1950s and 1960s, aimed at describing the operator's strategy in deciding where to look at each given moment. The emphasis was on mathematical modeling, usually derived from prevailing concepts. Several alternatives have been proposed but, unfortunately, empirical evidence is scant (Carbonell, 1966; Kvalseth, 1978; Senders, 1983; Sheridan, 1970; Stein and Wewerinke, 1983). The models have in common that they start from some a priori principle of optimal scanning, which subsequently leads to a prediction about what human operators do or should do, provided that they are rational processors of information along the lines of the model's prescrip-
Visual search
65
tion. They also share the assumption that visual scanning strategies depend almost completely on variables related to the dynamics of the displays. Practice has the effect that sampling can be controlled by an internal representation of the process dynamics, which, in principle, should correspond closely to the actual values. The first model (Senders, 1955) proposed that the operator attempts a continuous reconstruction of the various displays. In accord with information theory, this requires that each display should be periodically sampled with a frequency of at least twice its bandwidth. To attain perfect reconstruction, a perfect internal model is required. According to Senders (1983), this might be established only after considerable practice. As a test of this model, Senders carried out experiments on scanning four or six instruments. In the four-instrument condition, the observed and predicted values were in reasonable agreement, but in the six-instrument condition the data deviated from the predicted values: low-bandwidth instruments, i.e. relatively slowly changing signals, were sampled more frequently and highbandwidth instruments, i.e. relatively rapidly changing signals, were sampled less frequently than predicted by the model. The assumption of periodic sampling proved to be untenable. As a post-hoc explanation, Senders suggested that the introduction of two extra signals could have led to short-term forgetting of the signal readings with the consequence that especially low-bandwidth signals must have been inspected more frequently in order to maintain a representation of the actual situation. This oversampling of the low-bandwidth signals could have created an overload situation which resulted in a reduction of time to be allocated to the higher-bandwidth signals. As a main alternative to the information-based reconstruction model, decision-based conditional sampling models were developed during the 1960s. These models suggest that subjects only sample when the risk of missing a critical signal exceeds a certain limit. This means that, in addition to considering bandwidth, observers sample more frequently as the previous reading of a display is closer to the critical zone. In the case of a 'safe' reading, the next reading can be postponed until the previous reading is fully dated, i.e. when the autocorrelation function approaches zero. A main advantage of conditional sampling models is their greater intuitive plausibility: in the reconstruction notion, subjects play simply the passive role of mechanical transmitters of information. In the conditional models, subjects maintain a decision criterion with respect to their strategy. This means that they behave on the basis of both motivational and cognitive factors. It is obvious that in this way both models beautifully reflect the dominant traditions of information and decision theory as formulated in the 1950s and 1960s. Senders (1983) has described several variants of the conditional sampling model, which are all concerned with setting a strategic risk criterion. In one case, subjects only sample when the probability that a signal has entered the critical zone is maximal. This means that the aim is not primarily to detect the moment of entering, but rather whether a signal has actually entered the danger zone. In another version the accepted risk is defined by a threshold, while in still another version the accepted risk varies as a function of the distance of the signal from the danger zone. The variants obviously affect the predicted frequencies of observing, which are smaller when subjects accept the possibility of detecting an error post factum. Yet, the predicted frequencies of observation are always more than in the reconstruction
66
A. F. Sanders and M. Donk
model. This means that the observed undersampling is even more pronounced, and at odds, therefore, with the predictions of the model. Conditional sampling is not the only decision-theoretical prescription for scanning discrete displays. For example, Carbonell (1966) proposed a queueing model in which signal sources queue for inspection with the most urgent cases as first in line. Since displays are assumed to be only serially monitored, there is always a probability of missing a signal. This requires the observer to make a rational decision where to sample next, so as to minimize risk. The optimal strategy is to sample at each moment the display with the highest expected costs of a failure. The model assumes a quick calculation of the costs, connected to each display, before deciding where to fixate next. Sheridan (1970) has also proposed a model with the cost-benefit matrix of observations and actions as main determinant. As in the case in most other models, Sheridan assumes that the controller calls upon internally represented distributions of the processes. In an experimental study, Sheridan and Rouse (1971) found smaller fixation intervals than predicted, and suggested two alternative post-hoc explanations, one cognitive and one motivational. First, subjects could have a deviant (suboptimal or incomplete) internal representation of the processes, which might be repaired by additional practice. Second, subjects might suffer from risk aversion, and not act in accordance with the presumed strategy of risk minimization. There are additional normative formulations along either decision and information theoretical lines (Gai and Curry, 1976; Kvalseth, 1978; Stein and Wewerinke, 1983). They differ from each other, and from the earlier discussed models, with respect to the emphasis on varying aspects of the task. However, all models have in common that they rely largely on external factors, or their internal representation, as the main principle governing search. Even with respect to decision criteria, the models start from the assumption that subjects adhere to externally prescribed costs and benefits. The determining factors include: (a) the rate at which a process generates external uncertainty; (b) the permissible error, or the accuracy with which the current value of a function should be read; (c) the limits which a process should not exceed; (d) the probability that a source shows a critical value, while the observer is attending another source; and, finally, (e) the pay-off matrix associated with missed critical events and the costs of making observations. The main problem for the normative models is that empirical evidence is not only scant, but also in conflict with the predictions of the models. This became clear with respect to Senders' models, but it also applies to Sheridan's m o d e l - general oversampling- and to Kvalseth (1979). The major reasons underlying these experimental failures may well relate to the assumptions of perfect memory, perfectly veridical and independent mental representations of the processes, perfect calculation of costs and benefits, and independent control of the displays. In an analogy to research on decision making (Hogarth, 1990), one might still defend the position that, in principle, the models provide a correct description of how humans search, but that the human elements are imperfect- show forgetting (Moray, Richard and Low, 1980), have an imperfect representation, or fail to make the correct calculations (Sheridan and Rouse, 1971). The much more drastic alternative is that human search strategies depend on principles that basically differ from statistical prescriptions. For instance, van Delft (1987) has suggested that, rather than considering representations of the individual processes, humans might simply search displays on the basis of some fixed sequences, perhaps modulated by relations between
Visual search
67
bandwidths. In support of this view, van Delft found that the frequency of fixating an individual display depended on whether the other displays were relatively slow or fast. The rate of search might depend on the highest bandwidth, so that a medium-bandwidth signal would be inspected more often when it is the slowest than when it is the fastest of the set. Recently, Donk (1994a, b) and Donk and Hagemeister (1994) carried out a series of studies aimed at testing some of the basic assumptions of conditional sampling. In one study (Donk and Hagemeister, 1994), subjects had a self-paced card sorting task in combination with monitoring a display for the occurrence of a critical reading. The rationale of this study was to test whether deviations from optimal inspection rate might be due to failures of memory, in the case of a slow signal, or of time estimation, in the case of a fast signal. This was done by varying selective memory interference through the similarity between the cards and the display, and by varying information load in card sorting, which is known to affect time estimates. The usual phenomenon of oversampling slow signals and undersampling fast signals was found. However, the variations in memory interference and information load did not affect the sampling rate, which casts some doubt on explanations in terms of failures of memory and time estimates. In the same study, it was found that for slow signals sampling was conditional on the distance from the signal to the critical zone at the previous reading, but this variable did not affect sampling fast signals. This was confirmed in another study (Donk, 1994b) where subjects monitored one display in combination with a tracking task. The results suggest that there is no homogeneous search strategy. Instead, sampling might be so frequent for the high-bandwidth signals that subjects no longer bother about the position of the last reading. In contrast, that position might be relevant for slow signals in view of longer intervals between readings. The results demonstrated that signal dynamics play at least a role in shaping a subject's inspection strategy. On the other hand, when subjects monitored four or six displays at the same time, Donk (1994a) also found that, irrespective of bandwidth, horizontal saccades were more frequent than diagonal ones. A contribution of a perceptual variable, relating to the arrangement of the displays, may not surprise, consistent as it is with reading habits. Yet, it is outside the realm of the normative models, which start from rationally controlled internal representations of the outside world rather than from immediate control by outside stimuli or from internal rules, that are independent of the environment.
9
S E A R C H A N D STRESS
It is a classic hypothesis that search strategies may change when people suffer from environmental or organic stresses such as prolonged performance, noise, drugs, sleep loss, or overload in dual-task performance. In particular some stresses would lead to funneling and others to widening of attention. 2 This hypothesis was explicitly stated by Drew (1940), while funneling was also mentioned by Bartlett (1953) as a promising criterion for fatigue. On closer inspection, however, the literature is far from consistent, which is at least partly due again to wide variations 2Mackworth's (1965) reference to tunnel vision as a consequence of visual noise is not included in this discussion, since these results can be readily explained by lateral interference.
68
A. F. Sanders and M. Donk
in search conditions. For example, the edge effect (see the section on 'Cognitive models') did not appear to change as a function of time-on-task, which seems to argue against funneling caused by fatigue (Baker, 1958; Colquhoun and Edwards, 1970; Schoonaard, Gould and Miller, 1973). On the other hand, Broadbent (1950) found in his 'twenty dials' test a tendency towards a greater neglect of peripheral signals as time progressed, while Sanders (1963) also reported that subjects inspected the periphery more frequently in the beginning than toward the end of a long work spell. In both cases the display constituted a head field, so that these instances of funneling might be limited to cases where the task asks for frequent head movements. Sanders and Reitsma (1982a, b) also found more pronounced effects of sleep loss on processing extremely peripheral signals in a study on visual orienting and in one on the functional visual field. In the last case, processing time increased as a function of time-on-task when the display constituted a head field but not when it constituted an eye field. However, these last results should be interpreted with care, since Sanders and Reitsma also found a stronger effect of sleep loss when the signals were successively presented without a need to shift the head. Hence, in this experiment, the effects of sleep loss may be on integrating successive percepts rather than on the size of the visual field. Effects of funneling have also been studied in dual-task studies, where subjects performed a combination of a central task, e.g. pursuit tracking, target classification, and a peripheral signal detection task. For instance, Michon and Kirk (1962) found that detection of peripheral signals depended on the load of the central task, and that eye fixations in the periphery were significantly less frequent towards the end of a work spell. In a classic study by Bursill (1958) the percentage of missed peripheral signals increased as a function of time-on-task, and in particular under hot and humid conditions. Similarly, Bahrick, Fitts and Rankin (1952) found that incentives improved central tracking performance at the cost of peripheral detection. In further studies, Hockey (1970a, b) found that loud noise, supposedly producing high arousal, had the effect of narrowing attention to the central tracking task, while sleep loss, producing low arousal, had the effect of leveling attentional selectivity. This is of course the opposite of the early suggestion that funneling would occur when one is fatigued. Again, Hockey (1970c) found evidence that his earlier results should not be interpreted as evidence for visual funneling but rather for a stronger or weaker bias towards one task. Finally, it should be noted that the Hockey results are not easily replicable (Forster and Grierson, 1978; Loeb and Jones, 1978) and have found little follow-up in recent years. Indeed, Hockey and Hamilton's (1983) comment that replicability might be hard in view of the strategic nature of the changes, representing a delicate balance of task priorities, does not inspire a more detailed inquiry into the matter. In all these studies eye movements were allowed. This was not the case in the work of Abernethy and Leibowitz (1971) and of Leibowitz and Appelle (1969) who found raised peripheral luminance thresholds when subjects performed a central task. In the same way, Ikeda and Takeuchi (1975) reported a decrease in the accuracy of localizing thresholds as foveal load increased. However, Holmes, Cohen, Haith and Morrison (1977) warned against an interpretation of such results in terms of funneling. They could equally well be a matter of general interference in that allocating attention to the periphery is less possible, as the central task needs more attentional resources and has priority. Funneling would require a more pronounced effect of dual-task load on more peripheral signals, while general
Visual search
69
interference would predict little dependence on visual angle, which is what the authors found. In contrast, Williams (1989) found a stronger effect on more peripheral stimuli with a combination of a central memory search task and identification of a parafoveal (maximally 5 ~ target during a brief presentation. However, the fact that he used only parafoveal stimuli casts some doubt on the interpretation, since funneling is commonly understood with respect to the further periphery. In a recent study, Van de Weijgert (1991, 1993) investigated the effect of foveal load on identifying peripheral stimuli with wider visual angles. She found an almost equal effect of foveal load on peripheral identification at different angles, suggesting suppression of allocating attention rather than funneling.
10
DISCUSSION
Visual search behavior has been described at both a structural and a functional level. While the structural approach is primarily concerned with 'what is processed during visual search', the functional approach is mainly interested in 'the determinants of visual search behavior and the resulting visual search pattern'. Either approach has its prospects and limits. A structural approach might be characterized as atomistic: sensory, anatomic and information processing limits are described without too much emphasis on their mutual interplay. This means that various search limits have been investigated in relative isolation. On the other hand, either theory, within its own confined area, enables accurate predictions and provides a rather elaborate account of factors influencing visual search. A functional approach is more holistic. Either theory within this tradition is concerned with the search process as a whole. In contrast to the structural approach, functional theories might not be considered as mutually complementary. Generally, search behavior has been described on a cognitive and a mathematical level. The major limit of a cognitive approach (see section on 'Cognitive models') consists of a lack of methods to test its premises appropriately. Neither schema nor scan path theory predicts eye movement behavior. It merely assumes a large variety of scan patterns, each of which reflects the operation of some cognitive model of the outside world. A cognitive model concerning scanning strategies certainly has a large external validity, but it lacks predictive power. A normative mathematical approach (see section on 'Optimal scanning and monitoring'), on the other hand, precisely predicts search or sampling behavior of the human observer. The main problem, however, is that empiric evidence deviates from the predictions of the models. Normative mathematical models neither explain nor describe human search behavior; they merely prescribe behavior. Functional approaches toward visual search should start from the accumulating knowledge about structural limits in perception and information processing. A large amount of knowledge is already available about structural constraints in search. A major task for future research consists of developing strategic functional principles, which follow from the structural constraints. Admittedly, this sounds like adhering to a largely bottom-up approach. However, it remains to be seen to what extent strategic principles are top-down determined or, in other words, are related to structural constraints.
70
A. F. Sanders and M. Donk
REFERENCES Abernethy, C. N. and Leibowitz, H. W. (1971). The effect of feedback on luminance thresholds for peripherally presented stimuli. Perception and Psychophysics, 10, 172-174. Allport, A. (1987). Selection for action: Some behavioral and neurophysiological considerations of attention and action. In H. Heuer and A. F. Sanders (Eds), Perspectives on Perception and Action. Hillsdale, NJ: Erlbaum. Antes, J. R. and Penland, J. G. (1981). Picture context effects on eye movements. In D. F. Fisher, R. A. Monty and J. W. Senders (Eds), Eye Movements: Cognition and Visual Perception. Hillsdale, NJ: Erlbaum. Bahrick, H. P., Fitts, P. M. and Rankin, R. E. (1952). Effect of incentives upon reactions to peripheral stimuli. Journal of Experimental Psychology, 44, 400-416. Baker, C. H. (1958). Attention to visual displays during a vigilance task: Biasing attention. British Journal of Psychology, 49, 279-288. Banks, W. P. and Prinzmetal, W. (1976). Configurational effects in visual information processing. Perception and Psychophysics, 19, 361-367. Barbur, J. L., Forsyth, P. M. and Woodings, D. S. (1993). Eye movements and search performance. In D. Brogan, A. Gale and K. Carr (Eds), Visual Search II (pp. 253-264). London: Taylor and Francis. Bartlett, F. C. (1953). Psychological criteria of fatigue. In W. F. Floyd and A. T. Welford (Eds), Symposium on Fatigue. London: Klewis. Bellamy, L. J. (1984). The application of visual lobe measurement to visual perception. In A.G. Gale and F. Johnson (Eds), Theoretical and Applied Aspects of Eye Movement Research. Amsterdam: Elsevier. Bellamy, L. J. and Courtney, A. J. (1981). Development of a search task for the measurement of visual acuity. Ergonomics, 24, 497-509. Biederman, I. (1972). Perceiving real-world scenes. Science, 77, 77-80. Biederman, I., Glass, A. L. and Stacey, E. W. (1973). Searching for objects in real-world scenes. Journal of Experimental Psychology, 97, 22-27. Bloomfield, J. R. (1972). Visual search in complex fields: Size differences between target discs and surrounding disks. Human Factors, 14, 139-148. Bloomfield, J. R. and Howarth, C. I. (1969). Testing visual search theory. In H. W. Leibowitz (Ed.), Image Evaluation. Proceedings of NATO Advisory Group on Human Factors. Boer, L. C. and van de Weijgert, E. C. M. (1988). Eye movements and stages of processing. Acta Psychologica, 67, 3-18. Bouma, H. (1978). Visual search in reading: Eye movements and the functional visual field. In J. Requin (Ed.), Attention and Performance, vol. 7. Hillsdale, NJ : Erlbaum. Boynton, R. M., Elsworth, C. and Palmer, R. (1958). Laboratory studies pertaining to visual air reconnaissance. Part 3. Technical Report 55-304, AML-WADC, Wright-Patterson, Ohio. Brand, J. (1971). Classification without identification in visual search. Quarterly Journal of Experimental Psychology, 23, 178-186. Broadbent, D. E. (1950). The twenty-dials test under quiet conditions. APU Report. Broadbent, D. E. (1958). Perception and Communication. London: Pergamon. Broadbent, D. E. (1971). Decision and Stress. London: Academic Press. Broadbent, D. E. (1982). Task combination and selective intake of information. Acta Psychologica, 50, 253-290. Bursill, A. E. (1958). The restriction of peripheral vision during exposure to hot and humid conditions. Quarterly Journal of Experimental Psychology, 10, 113-130. Cahill, M. C. and Carter, R. C. (1976). Color code size for searching displays of different density. Human Factors, 18, 273-280. Carbonell, J. R. (1966). A queuing model of visual sampling: Experimental validation. IEEE Transactions on Man Machine Systems, MMS-9, 82-87.
Visual search
71
Cavanagh, J. P. and Chase, W. G. (1971). The equivalence of target and non-target processing in visual search. Perception and Psychophysics, 9, 493-495. Chalkin, J. D., Corbin, H. H. and Volkmann, J. (1962). Mapping a field of short-time visual search. Science, 138, 1327-1328. Chase, W. G. (1986). Visual information processing. In K. R. Boff, L. Kaufman and J. P. Thomas (Eds), Handbook of Perception and Human Performance, Vol. 2, chapter 28. New York: Wiley. Clare, J. N. and Sinclair, M. A. (1979). Search and the Human Observer. London: Taylor and Francis. Colquhoun, P. and Edwards, J. A. (1970). Practice effects on a visual vigilance task with and without search. Human Factors, 12, 537-546. Corbin, H., Carter, J., Reese, E. P. and Volkmann, J. (1958). Experiments on Visual Search 1956-1957. Psychological Research Unit, Mount Holyoke College. Corcoran, D. W. and Jackson, A. (1979). Flexibility in the choice of distinctive features in visual search with random cue blocked designs. Perception, 6, 629-633. Courtney, A. J. (1984). A search task to assess visual lobe size. Human Factors, 23, 289-298. Courtney, A. J. and Chan, H. S. (1985). Simple measures of visual lobe size and search performance. Ergonomics, 28, 1319-1332. Courtney, A. J. and Chan, H. S. (1986). Visual lobe dimensions and search performance for targets on a competing homogeneous background. Perception and Psychophysics, 40, 39-44. Crovitz, H. F. and Davies, W. (1962). Tendencies to eye movement and perceptual accuracy. Journal of Experimental Psychology, 63, 495-498. Deutsch, J. A. and Deutsch, D. (1963). Attention: Some theoretical considerations. Psychological Review, 70, 80-90. Donk, M. (1994a). Human monitoring behavior in a multiple instrument setting: Independent sampling, sequential sampling or arrangement-dependent sampling. Acta Psychologica, 86, 31-55. Donk, M. (1994b). The effect of secondary task load on visual sampling behavior. Ergonomics, 37, 1089-1096. Donk, M. and Hagemeister, C. (1994). Visual instrument monitoring as affected by simultaneous self-paced card sorting. IEEE Transactions on Systems, Man and Cybernetics, 24, 926-931. Drew, G. C. (1940). Mental Fatigue. Flying Personnel Research Committee, no. 227. Duncan, J. (1980). The locus of interference in the perception of simultaneous stimuli. Psychological Review, 87(3), 272-300. Duncan, J. (1983). Category effects in visual search: A failure to replicate the oh-zero phenomenon. Perception and Psychophysics, 34, 221-232. Duncan, J. and Humphreys, G. W. (1989). Visual search and stimulus similarity. Psychological Review, 98(3), 433-458. Ellis, S. H. and Chase, W. G. (1971). Parallel processing in item recognition. Perception and Psychophysics, 10, 379-384. Engel, F. L. (1971). Visual conspicuity, directed attention, and retinal locus. Vision Research, 11, 563-576. Engel, F. L. (1976). Visual conspicuity as an external determinant of eye movements and selective attention. Thesis. TH Eindhoven. Engel, F. L. (1977). Visual conspicuity, visual search and fixation tendencies of the eye. Vision Research, 17, 95-108. Enoch, J. M. (1959). Effect of the size of a complex display upon visual search. Journal of the Optical Society of America, 49, 280-286. Erickson, R. A. (1964). Relation between visual search time and peripheral visual acuity. Human Factors, 6, 165-178. Eriksen, B. A. and Eriksen, C. W. (1974). Effects of noise letters on the identification of a target letter in a non-search task. Perception and Psychophysics, 16, 143-149.
72
A. F. Sanders and M. Donk
Eriksen, C. W. (1990). Attentional search of the visual field. In D. Brogan (Ed.), Visual Search (pp. 3-19). London: Taylor and Francis. Eriksen, C. W. and Hoffman, J. E. (1972). Some characteristics of selective attention in visual perception determined by vocal reaction time. Perception and Psychophysics, 11, 169-171. Eriksen, C. W. and Hoffman, J. E. (1973). The extent of processing of noise elements during encoding of visual displays. Perception and Psychophysics, 14, 155-160. Farmer, E. W. and Taylor, R. M. (1980). Visual search through color displays: Effects of target-background similarity and background uniformity. Perception and Psychophysics, 27, 267-272. Farrell, B. (1985). 'Same-different' judgments: A review of current controversies in perceptual comparisons. Psychological Bulletin, 98, 419-456. Findlay, J. M. (1980). The visual stimulus for saccadic eye movements in human observers. Perception, 9, 7-21. Fischer, B. and Breitmeyer, B. (1987). Mechanisms of visual attention revealed by saccadic eye movements. Neuropsychologia, 25, 73-83. Fitts, P. M., Jones, R. E. and Milton, J. L. (1950). Eye movements of aircraft pilots during instrument landing approaches. Aeronautical Engineering Review, 9, 1-5. Forster, P. M. and Grierson, A. T. (1978). Noise and attentional selectivity: A reproducible phenomenon? British Journal of Psychology, 69, 482-498. Francolini, C. M. and Egeth, H. A. (1979). Perceptual selectivity is task-dependent: The pop-out effect poops out. Perception and Psychophysics, 25, 99-110. Friedman, A. (1979). Framing pictures: The role of knowledge in automatized encoding and memory for gist. Journal of Experimental Psychology: General, 108, 316-355. Gai, E. G. and Curry, R. E. (1976). A model of the human observer in failure detection tasks. IEEE Transactions on Systems, Man, and Cybernetics, SMC-6, 85-91. Gould, J. D. (1967). Pattern recognition and eye movement parameters. Perception and Psychophysics, 6, 311-320. Gould, J. D. and Dill, A. (1969). Eye movement patterns and pattern recognition. Perception and Psychophysics, 6, 311-320. Green, B. F. and Anderson, L. K. (1956). Color coding in a visual search task. Journal of Experimental Psychology, 51, 19-24. Hansen, W. and Sanders, A. F. (1988). On the output of encoding during stimulus fixation. Acta Psychologica, 69, 95-107. Hendersen, L. and Chard, J. (1971). Semantic effects in visual word detection with visual similarity controlled. Perception and Psychophysics, 23, 290-298. Henderson, J. M. (1993). Visual attention and saccadic eye movements. In G. d'Ydewalle and J. Van Rensbergen (Eds), Perception and Cognition (pp. 37-50). Amsterdam: Elsevier Science. Hockey, G. R. J. (1970a). Effect of loud noise on attentional selectivity. Quarterly Journal of Experimental Psychology, 22, 28-36. Hockey, G. R. J. (1970b). Signal probability and spatial location as possible bases for increased selectivity in noise. Quarterly Journal of Experimental Psychology, 22, 37-42. Hockey, G. R. J. (1970c). Changes in attention allocation in a multicomponent task under loss of sleep. British Journal of Psychology, 61, 473-480. Hockey, G. R. J. and Hamilton, P. (1983). The cognitive patterning of stress-states. In G. R. J. Hockey (Ed.), Stress and Fatigue in Human Performance. New York: Wiley. Hogarth, R. M. (1990). Judgment and Choice. New York: Wiley. Holmes, D. L., Cohen, K. M., Heith, M. M. and Morrison, F. J. (1977). Peripheral visual processing. Perception and Psychophysics, 22, 571-577. Houtmans, M. J. M. and Sanders, A. F. (1984). Perception of signals presented in the periphery of the visual field. Acta Psychologica, 55, 143-155. Howard, C. I. and Bloomfield, J. R. (1968). Towards a theory of visual search. In AGARD Conference Proceedings (no. 41), Brussels.
Visual search
73
Howard, C. I. and Bloomfield, J. R. (1969). A rational equation for predicting search times in simple inspection tasks. Psychonomic Science, 17, 225-226. Hughes, P. K. and Cole, B. L. (1986). Can the conspicuity of objects be predicted from laboratory experiments? Ergonomics, 29, 1097-1111. Humphreys, G. W. and Miiller, H. J. (1993). Search via Recursive Rejection (SERR): A connectionist model of visual search. Cognitive Psychology, 25, 43-110. Humphreys, G. W., Quinlan, P. T. and Riddoch, M. J. (1989). Grouping processes visual search: Effects with single- and combined-feature targets. Journal of Experimental Psychology: General, 118, 258-279. Humphreys, G. W., Riddoch, M. J. and Quinlan, P. T. (1985). Interactive processes in perceptual organisation: Evidence from visual agnosia. In M. I. Posner and O. S. M. Marin (Eds), Attention and Performance 11. Hillsdale, NJ: Erlbaum. Ikeda, M. and Takeuchi, T. (1975). Influence of foveal load on the functional visual field. Perception and Psychophysics, 18, 255-260. Jacobs, A. M. (1986). Eye movement control in visual search: How direct is visual span control? Perception and Psychophysics, 39, 47-58. Jacobs, A. M. (1987). Towards a model of eye movement control in visual search. In J. K. O'Regan and A. Levy-Schoen (Eds), Eye Movements: From Physiology to Cognition. Amsterdam: North Holland. Johnston, D. M. (1965). Search performance as a functon of peripheral acuity. Human Factors, 7, 528-535. Jonides, J. and Gleitman, H. (1972). A conceptual category effect in visual search: O as a letter or a digit. Perception and Psychophysics, 12, 457-460. Jordan, T. C. and Rabbitt, P. M. (1977). Response times to stimuli of increasing complexity as a function of ageing. British Journal of Psychology, 68, 289-301. Kahneman, D. and Treisman, A. M. (1984). Changing views of attention and automaticity. In R. Parasuraman and R. Davies (Eds), Varieties of Attention. New York: Academic Press. Klein, R. and Farrell, M. (1989). Search performance without eye movements. Perception and Psychophysics, 46, 476-482. Kraiss, K. F. and Knaeeper, A. (1982). Using visual lobe area measurements to predict visual search performance. Human Factors, 24, 673-682. Krendel, E. S. and Wodinsky, J. (1960). Visual search in an unstructured visual field. Journal of the Optical Society of America, 50, 562-568. Kvalseth, T. (1978). Human and bayesian information processing during probabilistic inference tasks. IEEE Transactions on Systems, Man and Cybernetics, SMC-8, 224-229. Kvalseth, T. (1979). A decision theoretic model of the sampling behavior of the human process monitor. Ergonomics, 21, 671-686. Leibowitz, H. W. and Appelle, S. (1969). The effect of a central task on luminance thresholds for peripherally presented stimuli. Human Factors, 11, 387-392. Levy-Schoen, A. (1974). Le champ d'activit6 du regard: Donn6es experimentales. L'Ann& Psychologique, 74, 43-66. Loeb, M. and Jones, P. D. (1978). Noise exposure, monitoring and tracking performance as a function of signal bias and tak priorit. Ergonomics, 21, 265-272. Loftus, G. R. and Mackworth, N. H. (1978). Cognitive determinants of fixation location during picture viewing. Journal of Experimental Psychology: Human Perception and Performance, 4, 565-572. Los, S. A. (1994). Procedural differences in processing intact and degraded stimuli. Memory and Cognition, 22, 145-156. Mackworth, N. H. (1965). Visual noise causes tunnel vision. Psychonomic Science, 3, 67-68. Mackworth, N. H. (1976). Stimulus density limits in the useful field of view. In R. A. Monty and J. W. Senders (Eds), Eye Movements and Psychological Processes. Hillsdale, NJ: Erlbaum. Mackworth, N. H. (1981). Stimulus density limits the useful field of view. In D. F. Fisher, R. A. Monty and J. W. Senders (Eds), Cognition and Visual Perception. Hillsdale, NJ: Erlbaum.
74
A. F. Sanders and M. Donk
Mackworth, N. H., Kaplan, I. T. and Metlay, W. (1964). Eye movements during vigilance. Perceptual and Motor Skills, 20, 549-554. Mackworth, N. H. and Morandi, A. J. (1967). The gaze selects informative details within pictures. Perception and Psychophysics, 2, 547-552. Marken, R. and Patterson, J. (1979). Effects of sequence and variety of irrelevant items in visual search. Perceptual and Motor Skills, 49, 315-318. Matin, E. (1974). Saccadic suppression: A review and an analysis. Psychological Bulletin, 81, 899-917. Mayfrank, L., Mobashery, M., Kimmig, H. and Fischer, B. (1986). The role of fixation and visual attention on the occurrence of express saccades in man. European Journal of Psychiatry and Neurological Science, 235, 269-275. Megaw, E. D. and Richardson, J. (1979). Target uncertainty and visual scanning strategies. Human Factors, 21, 303-315. Metzger, W. (1954). Gesetze des Sehens. Darmstadt: Huber. Michon, J. A. and Kirk, N. S. (1962). Eye movements in radar watchkeeping. IZF-report 17. Miller, J. O. (1988). Discrete and continuous models of human information processing: Theoretical distinctions and empirical results. Acta Psychologica, 67, 191-257. Monk, T. H. (1984). Search. In J. S. Warm (Ed.), Sustained Attention in Human Performance. New York: Wiley. Moraglia, G. (1989). Display organisation and the detection of horizontal line segments. Perception and Psychophysics, 45, 265-272. Moraglia, G., Maloney, K. P., Fekete, E. M. and Albasi, K. (1989). Visual search along the color dimension. Canadian Journal of Psychology, 43, 1-12. Morawski, T. B., Drury, C. G. and Karwan, M. H. (1980). Predicting search performance for multiple targets. Human Factors, 22, 707-718. Moray, N., Richards, M. and Low, J. (1980). The Behaviour of the Fighter Controllers, Technical Report. London: Ministry of Defence. Nattkemper, D. and Prinz, W. (1984). Costs and benefits of redundancy in visual search. In A. G. Johnson and F. Johnson (Eds), Theoretical and Applied Aspects of Eye Movement Research. Amsterdam: North Holland. Nattkemper, D. and Prinz, W. (1990). Local and global amplitude and fixation duration in continuous visual search. In R. Groner, G. d'Ydewalle and R. Parham (Eds), From Eye to Mind: Information Acquisition in Perception, Search and Reading. Amsterdam: NorthHolland. Neisser, U. (1963). Decision time without reaction time: Experiments on visual scanning. American Journal of Psychology, 76, 376-395. Neisser, U. (1967). Cognitive Psychology. New York: Appleton-Century-Crofts. Neumann, O. (1987). Beyond capacity: A functional view of attention. In H. Heuer and A. F. Sanders (Eds), Perspectives on Perception and Action. Hillsdale, NJ: Erlbaum. Nissen, M. J., Posner, M. I. and Snyder, C. R. R. (1978). Relationships between attention shifts and saccadic eye movements. Paper to the Psychonomic Society. Noble, M. E. and Sanders, A. F. (1981). Searching for traffic signals while engaged in compensatory tracking. Human Factors, 22, 89-102. Noton, D. and Stark, L. (1971). Eye movements and visual perception. Scientific American, 224, 34-43. Parasuraman, R. and Davies, D. R. (1984). Varieties of Attention. Orlando, FL: Academic Press. Posner, M. I. (1980). Orienting of attention. Quarterly Journal of Experimental Psychology, 32, 3-25. Posner, M. I. and Boies, S. J. (1971). Components of attention. Psychological Review, 78,391-408. Posner, M. I., Nissen, M. J. and Ogden, W. C. (1978). Attended and unattended processing modes: The role of set for spatial location. In H. L. Pick and J. J. Salzman (Eds), Modes of Perceiving and Processing Information. Hillsdale, NJ: Erlbaum.
Visual search
75
Posner, M. I. and Snyder, C. R. R. (1975). Facilitation and inhibition in the processing of signals. In P. M. A. Rabbitt and S. Dornic (Eds), Attention and Performance 5. New York: Academic Press. Posner, M. I., Snyder, R. R. and Davidson, D. J. (1980). Attention and the detection of signals. Journal of Experimental Psychology: General, 109, 160-174. Prinz, W. (1979). Integration of information in visual search. Quarterly Journal of Experimental Psychology, 31, 287-304. Prinz, W. (1983). Asymmetrical control areas in continuous visual search. In R. Groner, C. Menz, D. F. Fisher and R. A. Monty (Eds), Eye Movements and Psychological Functions. International Views. Hillsdale, NJ: Erlbaum. Prinz, W. (1984). Attention and sensitivity in visual search. Psychological Research, 45, 355-366. Prinz, W. (1987). Continuous selection. Psychological Research, 48, 231-238. Prinz, W. and Kehrer, L. (1982). Recording detection distances in continuous visual search. In R. Groner and P. Fraisse (Eds), Cognition and Eye Movements. Amsterdam: North Holland. Prinz, W., Meinecke, C. and Hielscher, M. (1987). Effects of stimulus degradation on category search. Acta Psychologica, 64, 187-206. Prinz, W. and Nattkemper, D. (1986). Effects of secondary task on search performance. Psychological Research, 48, 47-52. Prinz, W., Tweer, R. and Feige, R. (1974). Context control of search behaviour: Evidence from a hurdling technique. Acta Psychologica, 38, 72-80. Rabbitt, P. M. A. (1964). Ignoring irrelevant information. British Journal of Psychology, 55, 403-414. Rabbitt, P. M. A. (1967). Learning to ignore irrelevant information. American Journal of Psychology, 80, 1-13. Rabbitt, P. M. A. (1981). Visual selective attention. In C. R. Puff (Ed.), Handbook of Research Methods in Human Memory and Cognition. New York: Academic Press. Rabbitt, P. M. A. (1984). The control of attention in visual search. In R. Parasuraman and D. R. Davies (Eds), Varieties of Attention. Orlando, FL: Academic Press. Rabbitt, P. M. A., Cumming, G. and Vyas, S. M. (1977). Modulation of selective attention by sequential effects in visual search tasks. Quarterly Journal of Experimental Psychology, 31, 305-317. Remington, R. W. (1980). Attention and saccadic eye movements. Journal of Experimental Psychology: Human Perception and Performance, 6, 726-744. Sanders, A. F. (1963). The Selective Process in the Functional Visual Field. Assen: van Gorcum. Sanders, A. F. (1970). Some aspects of the selective process in the functional visual field. Ergonomics, 13, 101-117. Sanders, A. F. (1990). Issues and trends in the debate on discrete vs. continuous processing of information. Acta Psychologica, 74, 123-167. Sanders, A. F. (1993). Processing information in the functional visual field. In G. d'Ydewalle and J. van Rensbergen (Eds), Perception and Cognition (pp. 3-22). Amsterdam: Elsevier Science. Sanders, A. F. and Br6ck, R. (1991). The effect of presentation time on the size of the visual lobe. Bulletin of the Psychonomic Society, 29, 206-208. Sanders, A. F. and Houtmans, M. J. M. (1984). The functional visual field revisited. In A. J. van Doorn, W. A. van der Grind and J. J. Koenderink (Eds), Limits in Perception. Utrecht: VNU Science. Sanders, A. F. and Houtmans, M. J. M. (1985a). Perceptual processing modes in the functional visual field. Acta Psychologica, 58, 251-262. Sanders, A. F. and Houtmans, M. J. M. (1985b). There is no central stimulus encoding during saccadic eye shifts: A case against general parallel processing models. Acta Psychologica, 60, 323-338.
76
A. F. Sanders and M. Donk
Sanders, A. F. and Rath, A. (1991). Perceptual processing and speed-accuracy trade-off. Acta Psychologica, 77, 275-291. Sanders, A. F. and Reitsma, W. D. (1982a). Lack of sleep and covert orienting of attention. Acta Psychologica, 52, 137-145. Sanders, A. F. and Reitsma, W. D. (1982b). The effect of sleep-loss on processing information in the functional visual field. Acta Psychologica, 51, 149-162. Sanders, M. S. and McCormick, E. J. (1987). Human Factors in Engineering and Design. New York: McGraw Hill. Saslow, M. G. (1967). Effects of components of displacement-step stimuli upon latency for saccadic eye movements. Journal of the Optical Society of America, 57, 1024-1029. Schneider, W., Dumais, S. T. and Shiffrin, R. M. (1984). Automatic and controlled processing and attention. In R. Parasuraman and D. R. Davies (Eds), Varieties of Attention. Orlando, FL: Academic Press. Schneider, W. and Shiffrin, R. M. (1977). Controlled and automatic human information processing. I: Detection search and attention. Psychological Review, 84, 1-66. Schoonaard, J. W., Gould, J. D. and Miller, L. A. (1973). Studies of visual inspection. Ergonomics, 16, 365-379. Selfridge, O. G. and Neisser, U. (1959). Pattern recognition by machine. Scientific American, 203, 60-68. Senders, J. W. (1955). Man's capacity to use information from complex displays. In H. Quastler (Ed.), Information Theory in Psychology. Glencoe: Free Press. Senders, J. W. (1983). Visual Sampling Processes. Hillsdale, NJ: Erlbaum. Shepherd, M., Findlay, J. M. and Hockey, R. J. (1986). The relationship between eye movements and spatial attention. Quarterly Journal of Experimental Psychology, 38, 475-491. Sheridan, T. B. (1970). On how often the supervisor should sample. IEEE Transactions on Systems, Science and Cybernetics, SSC-6, 140-145. Sheridan, T. B. and Rouse, W. B. (1971). Supervisory sampling and control: Sources of suboptimality in a prediction task. NASA Annual Conference on Manual Control. Shiffrin, R. M. and Schneider, W. (1977). Controlled and automatic human information processing. II: Perceptual learning, automatic attending and a general theory. Psychological Review, 84, 127-190. Smets, G. J. F. and Stappers, P. J. (1990). Do invariants of features determine the conspicuity of forms? In D. Brogan (Ed.), Visual Search. London: Taylor and Francis. Sokolov, E. N. (1963). Perception and the Conditioned Reflex. New York: Pergamon. Sperling, G., Budiansky, Spivak, J. G. and Johnson, M. C. (1971). Extremely rapid visual search: The maximum rate of scanning letters for the presence of a numeral. Science, 174, 307-311. Stark, L. and Ellis, S. R. (1981). Scan path revisited: Cognitive models direct active looking. In D. F. Fisher, R. A. Monty and J. W. Senders (Eds), Eye Movements: Cognition and Visual Perception. Hillsdale, NJ: Erlbaum. Stark, L., Yamashita, I., Tharp, G. and Ngo, H. X. (1993). Keynote lecture: Search patterns and search paths in human visual search. In D. Brogan, A. Gale and K. Carr (Eds), Visual Search II (pp. 37-58). London: Taylor and Francis. Stein, W. and Wewerinke, P. W. (1983). Human display monitoring and failure detection: Control theoretic models and experiments. Automatica, 19, 711-718. Teichner, W. H. and Mocharnuk, J. B. (1979). Visual search for complex targets. Human Factors, 21, 259-275. Treisman, A. M. and Gelade, G. (1980). A feature integration theory of attention. Cognitive Psychology, 12, 97-136. Treisman, A. M. and Sato, S. (1990). Conjunction search revisited. Journal of Experimental Psychology: Human Perception and Performance, 16(3), 459-478. Van Delft, J. H. (1987). The development of a response sequence: A new description of human sampling behavior with multiple independent sources of information. Proceedings of the Human Factors Society, 31st Annual Meeting, pp. 151-155.
Visual search
77
Van der Heijden, A. H. C. (1987). Central selection in vision. In H. Heuer and A. F. Sanders (Eds), Perspectives on Perception and Action (pp. 421-446). Hillsdale, NJ: Erlbaum. Van der Heijden, A. H. C. (1992). Selective Attention in Vision. London: Routledge. Van de Weijgert, E. C. M. (1991). Foveal load and peripheral task performance: Tunnel vision or general interference. The Second International Conference on Visual Search, Durham, UK, Sept. 1990. Van de Weijgert, E. C. M. (1993). Foveal load and peripheral task performance: Tunnel vision or general interference. In D. Brogan, A. Gale and K. Carr (Eds), Visual search II (pp. 341-348). London: Taylor and Francis. Van Duren, L. L. and Sanders, A. F. (1992). The output code of a visual fixation. Bulletin of the Psychonomic Society, 30{4}, 305-308. Widdel, H. (1983). A method for measuring the visual lobe area. In R. Groner, C. Menz, D. F. Fisher and R. A. Monty (Eds), Eye Movements and Psychological Functions: International Views. Hillsdale, NJ: Erlbaum. Williams, L. G. (1966). The effect of target specification on objects fixated during visual search. Perception and Psychophysics, 1, 315-318. Williams, L. G. (1967). The effect of target specification on objects fixated during visual search. Acta Psychologica, 27, 355-360. Williams, L. G. (1989). Foveal load affects the functional field of view. Human Performance, 2, 1-28.
Willows, D. M. and McKinnon, G. E. (1973). Selective reading attention to the 'unattended' lines. Canadian Journal of Psychology, 27, 292-304. Wolfe, J. M., Cave, K. R. and Franzel, S. L. (1989). Guided search: An alternative to the feature integration model for visual attention. Journal of Experimental Psychology: Human Perception and Performance, 15(3), 419-433. Yarbus, A. L. (1967). Eye Movements and Vision. New York: Plenum Press.
Chapter 3 Auditory Attention G. ten Hoopen Leiden University, The Netherlands
In this chapter I shall present three topics, one old topic of auditory attention and two new ones. The old o n e - auditory selective attention, often treated under the heading 'cocktail party p r o b l e m ' - w i l l be surveyed globally. There are many texts available that deal extensively with the matter. Some relevant titles will be mentioned in this section and in the epilogue section. However, since I could not find overviews in which the more recent studies on auditory selection of Johnston and coworkers were included, I will treat their work in more detail. Furthermore will I discuss doubts that could be cast upon some well-known studies that expressed much optimism about the fate of unattended messages. Moreover I will attempt to advocate that there is an inherent confounding of attentional selection features in the dichotic paradigm. Although it is mostly reported that location per se is a good selector, it is probably only so in combination with other features such as frequency or timbre. In the section on one of the new topics, 'Streaming, attention and auditory illusions', this latter suspicion will be discussed again. I decided to include the topic of streaming because it is often neglected in surveys of auditory attention. This section is therefore more specific and technical than are most sections in handbooks. Nevertheless one can get only a partial glimpse there of this fascinating and huge field of recent research, which is concerned with how the listener organizes the auditory environment. In the epilogue section of this chapter I shall mention the relevant references, since neither of them carries the word attention in its title. Since I surmised that the classical student of attention might not be well acquainted with the streaming literature, I tried to sketch the commonalities between the old cocktail party studies and the more recent streaming studies as far as possible. A far more dangerous endeavor was to write the middle section, embraced by the 'cocktail parties' and the 'streams'. Relatively few studies in the literature were devoted to the question of whether temporal allocation of attention to sound signals exists or not. Hence this topic is often missing in surveys on auditory attention. Still, 'set' is a legitimate candidate that should also be covered by the term attention. Moray (1969) already mentioned set in his list of six categories of attention: mental concentration, vigilance, selective attention, search, activation and set. The section on set ('attention in single auditory streams') is also a little bit more technical than could be expected from surveys. Handbook of Perception and Action, Volume 3
Copyright 9 1996 Academic Press Ltd All rights of reproduction in any form reserved
ISBN 0-12-516163-8
79
80
G. ten Hoopen
Thus, this chapter presents three aspects of attention: (1) attention as selection (the cocktail party studies); (2) attention as set (the temporal allocation studies); and (3) attention as organizing the auditory environment (the streaming and auditory illusion studies).
AUDITORY ATTENTION: PARTY SOLUTION
THE COCKTAIL
An important ability of the human listener is to follow one speaker in the midst of others. Although this faculty is often discussed under the heading 'cocktail party problem', it may be better to speak of 'cocktail party solution', since it is the machine rather than the person that is having a hard time disentangling concurrent speakers. Research on auditory attention started in the 1950s to get a better understanding of how humans, faced with a mixture of auditory (and visual) inputs, manage to pay attention to one message (focused attention) or to keep track of more messages at the same time (divided attention). The choice of the terms focused and divided attention seems a little unlucky, since it might erroneously be suggested that the listener can indeed process the auditory environment either way. The terms 'focused' and 'divided' only describe the instructions of the experimenter. Cherry's (1953) experiments are the classical example of studying the cocktail party solution. In his first set of experiments, two readings by the same speaker (messages A and B) were recorded on audiotape in a superimposed way, i.e. in 'mono'. The task for the listener was to separate message A from B and dictate A as well as possible to the experimenter who recorded the response. The task of verbally repeating a message word by word or phrase by phrase is called 'shadowing'. Since nearly all separation cues that are normally present at a real drinking party (binaural cues, timbre cues, loudness cues) were stripped off, the listener could only with great difficulty manage to shadow message A. Syntactic features such as syllable and word order, and suprasegmental features such as stress pattern, which still differed between the messages, enabled the listener to achieve separation, but only after listening to the record over and over again. The shadowing task became a lot easier when A and B were channeled to different ears (Cherry's second set of tests), even though the speaker of the messages was the same. This suggests that the 'ear' is a good separator. However, it should be emphasized that, although many studies used headphones, the same results are obtained when A and B are played back over different loudspeakers. In the latter case both ears receive both messages. Hence, perceived location, rather than the ear, is the selection principle, a fact already mentioned by Broadbent (1958). In the dichotic paradigm, which is of course more convenient in many noisy university laboratories, the lateralization of the messages is 180 ~ Spieth, Curtis and Webster (1954) demonstrated that selection of a message in the localization paradigm was already possible when the angular separation between the speakers was 10-20 ~. The same authors showed that listeners could also select between messages when one message was routed via a low-pass filter and the other one via a high-pass filter. Thus, frequency bands can also serve as a basis for selection. A very trivial cause of the pervading dichotic paradigm should not be overlooked: two-channel
Auditory attention
81
tape recorders could and still can be afforded financially much easier by university researchers than bandpass filters. Auditory environment, also in air traffic control towers, however, is not dimensioned only by location but also by frequency and time. In the remainder of this chapter I will advocate that these latter dimensions could be better selectors than location. A methodological problem in using concurrent speech messages in the dichotic shadowing task is that there is no adequate control over the 'dichotic quality' of the stimulus. According to Tartter (1988): 'The dichotic listening task involves the simultaneous or near-simultaneous presentation of two different stimuli, one to each ear.' (p. 283). It should be obvious that this criterion is often violated when using spoken messages. Relatively long silences in one message might occur at moments the other one comprises signals and vice versa, allowing the listener to make attention shifts. Better control over the dichotic situation is achieved by the split-span paradigm, in which different words are more or less synchronized in time, and routed to different ears. Broadbent (1954), for instance, presented three digits to one ear and three digits to the other ear in a dichotic manner (split-span paradigm) and found that listeners preferred to order the digits by ear rather than by time when recalling them. On the basis of these results and those of the shadowing studies, Broadbent proposed his now classical 'filter model'. Messages that are presented at the same time enter a sensory buffer in parallel. One of the messages is routed through a filter on the basis of physical features, the other ones residing in the sensory buffer. A limited-capacity mechanism (the so-called r-system), beyond the filter, further processes the message. See Broadbent (1958) for the original description and textbooks such as those of Massaro (1975), Moray (1969) and Norman (1976). As to many initial models, many amendments were also made to Broadbent's filter model. At least two experiments convincingly demonstrated that 'channels' in Broadbent's buffer could be organized not only along physical stimulus characteristics, but also according to meaning. Treisman (1960) implemented a clever variation on the dichotic shadowing task. Her subjects had to shadow the right ear message, while ignoring the left ear message. At unexpected moments, however, the messages were reversed between ears, whereupon subjects continued shadowing the 'primary' message, only switching back to the right ear after a couple of seconds. This result is evidence that selection was based not solely on location but also on content. Also devastating to the notion of genuine physical separation of messages was the well-known experiment of two undergraduate students, Gray and Wedderburn (1960). Applying exactly the same split-span paradigm as Broadbent did, they presented three digits alternating between ears and three words forming a sentence alternating in antiphase. One ear received, for instance, the sequence ' w h o ' - ' 6 ' 'there', and the other ear received ' 4 ' - ' g o e s ' - ' 1 ' in synchrony. The order of report in the recall task was not by ear, as predicted by physical separation, but by meaning: 'who goes there' and '4-6-1'. In line with Broadbent's model, the shadowing studies reported mostly that no meaning was processed of the message to be ignored. However, not being aware of meaning does not necessarily imply that a stimulus is not processed for meaning. Three kinds of studies seem to shed light on that problem: recall, conditioning and reaction time studies.
82
G. ten Hoopen
Norman (1969) tested the fate of the secondary message in the following way. During the shadowing task subjects were unexpectedly interrupted by the experimenter for an immediate recall test. It turned out that also the last few words of the secondary message were recognized. This can be taken as evidence that the secondary message also accessed the e-system, but that it was not stored in long-term memory. An alternate interpretation is that the filter might have switched quickly to the contents of the other physical channel in the buffer. In that view, Broadbent's model can cope with such results. I will, therefore, discuss other paradigms that attempted to trace the fate of the secondary message. Corteen and Wood (1972) shock-conditioned their subjects to names of cities before having them participate in a dichotic shadowing experiment. On the channel to be ignored, the conditioned city names as well as other city names appeared periodically. The authors reported that, although the subjects claimed not to be aware of city names, they showed a significant galvanic skin response (GSR), not only to the shock-conditioned cities but even to non-conditioned city names. Corteen and Wood stated that their results conform to the position held by Deutsch and Deutsch (1963). Their model also posits that non-target messages do achieve full perceptual analysis. Corteen and Dunn (1974) replicated these findings and concluded that there can be some semantic processing without awareness. Similar results have been reported by yon Wright, Anderson and Stenman (1975). They found an appreciable GSR to conditioned words in the secondary message and a smaller but noticeable GSR to homonyms and synonyms. Although these conditioning studies seem to offer good convergent evidence refuting early selection notions, they should be taken with some caution: Wardlaw and Kroll (1976), in a meticulous replication of the Corteen studies, failed to find differential GSRs. It is a salient fact that, although these authors communicated with Corteen, they were unable to obtain his original audiotapes for a replication in their own laboratory. Lewis (1970) performed two experiments in which subjects were presented with two synchronized lists of words, one to each ear, and were required to shadow one list. (The mean synchronization error between word pairs was 25 ms, therefore the messages obeyed the criterion 'dichotic' rather well.) Verbal reaction time (RT), that is, the time elapsing from the onset of the stimulus word to the onset of the pronunciation of it, was registered. Some of the words in the other ear were semantically related to their counterparts in the list to be shadowed. The remarkable result was that semantically related words in the ear to be ignored nevertheless affected the verbal RT to their counterparts in the shadowed list. Reaction time to words, which were accompanied by synonyms in the other ear, was increased from 699 ms (base rate of unrelated words) to 726 ms, whereas RT decreased to 643 ms when antonyms arrived at the ear to be rejected. Lewis' conclusion was that: 'unattended messages are processed at a semantic level.' (Lewis, 1970, p. 228). Unfortunately Lewis did not report an analysis of the interaction between semantic relationship and ear to be shadowed, although half of his subjects received the primary message in the left ear, and the other half in the right ear. Kimura (1961), who applied Broadbent's split-span procedure, reported a right ear advantage (REA) for verbal material. Since then, hundreds of studies in the field of neuropsychology have used the dichotic listening procedure as a non-invasive test for establishing language lateralization, and the REA appears to be a robust effect (Bryden, 1988). It is an intriguing question whether the semantic interfering effects
Auditory attention
83
on the RT in Lewis' study differed depending on the ear to be ignored. One could hypothesize that, when the semantic counterparts (to be ignored) were presented to the right ear, they could affect shadowing RT much more than when presented to the left ear. Such an analysis would have refined the model of auditory selective attention. Lewis posited that his results supported the Deutsch and Deutsch (1963) model, rather than Treisman and Geffen's (1967) perceptual decrement model. It might not be surprising that, at the battlefield of models, Treisman, Squire and Green (1974) replicated Lewis' study. The authors reported that the Lewis effect dissipated in the course of the dichotic list, i.e. the semantic effect of a non-target word on its dichotic cohort was confined to the first few positions of the list. According to Treisman et al. the listener needs some time to build up full attention to the primary message. It is worthwhile to square this notion with a process operating in auditory stream formation. If tones are rapidly alternated between frequencies far apart, listeners hear two sound sequences (called 'streams'), instead of one. However, Bregman (1978a) demonstrated that the listener needs a few cycles of alternation to interpret the sequence as two streams. I shall return to the phenomenon of streaming extensively in the third section of this chapter. Johnston and Heinz (1979) found that the Lewis effect could be diminished if target and non-target words were made more discriminable. The aim of their study was to pit 'late selection' theory (Deutsch and Deutsch, 1963) against 'multi-loci' theory (Broadbent, 1970; Norman, 1969). Late selection theory posits that all input will automatically receive full sensory and semantic analysis to the extent that some late selection device can screen unimportant messages. It is a so-called 'late bottleneck' theory. In multiple loci theory, on the other hand, the bottle is malleable: the neck can be moved. According to Johnston and Heinz (1979), it is the very essence of attention that selective operations can be carried out anywhere along the continuum from early to late in perceptual processing. So the processing of targets and non-targets can also vary from superficial to deep. If this supposition is true, then it should be possible to manipulate the Lewis effect. If conditions are created in which the non-targets receive deeper processing, the Lewis effect should be magnified, but this effect should be attenuated when the non-targets are processed shallowly. In the latter case no semantic interference with the words to be shadowed (targets) should occur. Johnston and Heinz devised experiments in which sensory discriminability between the primary and secondary messages varied. In the 'low discriminable' condition the dichotic target and non-target words were spoken by the same male voice whereas in the 'high discriminable' condition these words were in a male and a female voice. The results showed that varying the sensory discriminability affected the Lewis effect in the hypothesized direction. According to the authors the 'late selection' theory can not predict such phenomena. Johnston and Wilson (1980) offered more evidence for 'multi-loci' theory. As targets in dichotic pairs ambiguous words were presented and the task of the subject was to identify them as belonging to certain categories (e.g. articles of clothing). The meaning of the non-target cohort could be appropriate in the sense that it could, if processed, disambiguate the target such that the latter belonged to the designated category (e.g. target: sock, non-target: smelly). The non-target could also be inappropriate and, if processed, disambiguate the target so that it did not fit the category (e.g. target: sock, non-target: punches). The non-target could also be
84
G. ten Hoopen
neutral. The second factor, which was covaried with semantic relationship, was the listening condition. In the focused listening condition the targets were presented to one ear only, while in the divided condition the target could arrive at either ear. The results showed a significant interaction between semantic relationship and listening condition. In the focused condition target identification was not affected by the semantic features of the non-target. In the divided condition, however, targets accompanied by non-targets that could appropriately disambiguate them were identified much better than targets accompanied by inappropiate non-targets. Consequently, the authors claimed that the depth of processing of the non-target is not fixed (neither shallow as in the early selection stance, nor deep as in the late selection stance), but variable instead. It should be mentioned that the Johnston stance (see also Johnston and Heinz, 1978) is quite similar to the compromise between early and late selection notions already proposed by Norman in 1968. In Norman's model, competing messages can be discarded anywhere in the processing sequence. Both the physical input and the so-called 'pertinence' of information (based on expectation and linguistic rules) determine what will be screened and what will be selected for further processing. As far as I know, no more important theoretical contributions to selective attention modeling were reported after 1980 by students of auditory attention. However, theorizing about selective attention was intensified in the field of visual perception. I refer to Neumann, van der Heijden and Allport (1986) for a creditable explanation of this theory shift by juxtaposing auditory and visual selection clearly. Theorizing about attention was moved to the field of motor behavior too. The latter shift might be due to the fact that many researchers located the bottleneck (limitedcapacity system) far away toward the response site (see Chapter 4). If selective attention is indeed response competition, then the field of motor behavior seems an appropriate lodging for theorists of attention too.
2
ATTENTION
IN A SINGLE AUDITORY
STREAM
In this section I shall discuss studies that share the supposition that the listener, guided by recurrent patterns in the ongoing sound stream (e.g. rhythm, a pitch pattern, an accent pattern), is set to foster certain moments in the flow of time. The idea is that more or less attention can be invested at temporal positions as a function of the pattern structure. A related notion is 'temporal priming': sounds occupying certain positions in a temporal pattern might be processed in favor of other sounds as a function of the priming pattern. The studies stem from various fields of research: speech perception, music perception and time perception.
2.1 Speech A good example of a study that explicitly investigated the temporal allocation of attention in a single speech stream is that of Shields, McHugh and Martin (1974). The study was based on Martin's (1972) model for rhythmic structure in speech and other behavior. According to this model, speech elements are hierarchically organized, thus providing an underlying temporal structure within which elements are
Auditory attention
85
related instead of being simply concatenated. The rhythmic rules derived from the hierarchy impose an accent structure (a prosodic structure) on the speech sequence. According to Shields et al., a smart perceptual system would make use of the prosodic pattern of the sequence. That is, if a rhythmic structure has been detected, predictions can be made about accents as they should unfold in time. Such a perceptual system could benefit from attentional focusing on elements in temporal positions still to come. This presupposes that listeners, after having locked on to the rhythm, are able anticipatorily to focus their attention on accented syllables. Shields et al. tested their hypothesis by means of the classic phoneme monitoring task (this task consists of monitoring for some predesignated phoneme in a message and reacting to its occurrence, often manually; cf. Foss, 1969; Hakes and Foss, 1970). The prediction was that the monitoring reaction time (RT) to phonemes in accented syllables would be shorter than the RT to phonemes in unaccented syllables. Subjects were confronted with two kinds of material: (1) sentences containing the target phoneme ( / b / ) either in accented or unaccented nonsense syllables; and (2) strings of nonsense words, one of which comprised the accented or unaccented syllable, which had been spliced from the sentence material. By means of this splicing procedure, acoustic parameters were held constant across materials. The prediction was that the RTs to t h e / b / t a r g e t would be shorter when they appeared in accented syllables than when they were presented in unaccented ones, and that this effect would be larger in the sentences than in the nonsense words context. (The nonsense words were scrambled in order to prevent a natural binary rhythm pattern.) Table 3.1 displays the results which supported the hypothesis: the RT improvement for accented target positions was significant in the (rhythmic) sentence context (63 ms), whereas it was not significant (16 ms) in the nonrhythmic context. An interesting variable also included in this experiment was the position of the syllable that contained the target phoneme. This could be at the beginning, middle or end of the sentence or string. If the rhythm hypothesis is correct, then the allocation of temporal attention might improve as more instances of the beat structure (unaccented-accented) could be gathered. This leads to the prediction that RT to the accented target in the sentence will diminish as a function of the target's temporal position. This pattern could, indeed, be observed when comparing the initial position with the middle position. However, the pattern reversed
Table 3.1. Monitoring reaction times (ms) to the phoneme/b/in unaccented and accented nonsense syllables embedded in a sentence or in a nonsense word string (data from Shields et al., 1974)
Context Accent
Sentence
Nonsense word string
Unaccented Accented Difference
612 549 63*
687 671 16 (N.S.)
*Significantdifference. N.S., not significant.
86
G. ten Hoopen
when comparing the middle with the end position: RTs to accented targets were 10 ms longer in the end position than in the middle position, whereas there was a decrease of 78 ms for unaccented targets. Perhaps the classic 'ageing foreperiod effect' (Drazin, 1961) 1 interfered, but it is not clear why it should have interacted with accented and unaccented targets. Unfortunately, the authors mentioned positional RTs only for their sentence condition and not for their nonsense word string control condition. Therefore, this experiment offers only limited support for the interesting idea that rhythmic attention might build up as a function of time. Pitt and Samuel (1990) coined the term 'attentional bounce hypothesis' (ABH) for the notion that the listener's attention can lock in to the rhythmicity of the speaker. The authors criticized some aspects of the Shields et al. study discussed above, one criticism concerning the fact that the target phonemes were embedded in nonwords